Alignment techniques for percutaneous access

Abstract

This invention provides a technique for aligning a medical device for percutaneous entry into a human anatomical structure. The technique may include: determining the orientation of the medical device, determining a target location within the human anatomical structure, and determining a plane including the target location. Furthermore, the technique may include: determining the projected position of the medical device on the plane, and generating interface data indicating the distance between the projected position and the target location on the plane.

Description

[0001] Related applications

[0002] This application claims priority to U.S. Provisional Application No. 62 / 956,019, filed on December 31, 2019, entitled “ALIGNMENT TECHNIQUES FOR PERCUTANEOUS ACCESS,” the disclosure of which is incorporated herein by reference in its entirety. Background Technology Technical Field

[0004] This disclosure pertains to the field of medical procedures.

[0005] Description of related technologies

[0006] Various medical procedures involve the use of one or more devices configured to penetrate human anatomy to reach a treatment site. Some procedures may involve inserting one or more devices through the patient's skin and other anatomical structures to reach a treatment site and remove an object (such as a kidney stone) from the patient's body. Summary of the Invention

[0007] In some embodiments, this disclosure relates to a method comprising: receiving sensor data indicating the position of at least a portion of a medical device; determining, at least partially based on the sensor data, an orientation of the medical device; determining a target location within an organ of a patient; and determining a plane including the target location. The method further comprises: determining, at least partially based on the orientation of the medical device, a projected position of the medical device on the plane; generating interface data representing one or more interface elements indicating a first distance between the projected position on the plane and the target location; and displaying, at least partially based on the interface data, the one or more interface elements.

[0008] In some embodiments, determining the plane includes determining the plane such that the direction of travel of the medical device is perpendicular to the plane. The method may further include updating the orientation of the plane as the orientation of the medical device changes, such that the direction of travel of the medical device remains perpendicular to the plane.

[0009] In some embodiments, determining the plane includes: determining a line between the distal end of the medical device and the target location, and determining the plane such that the line is perpendicular to the plane. The method may also include maintaining the orientation of the plane as the orientation of the medical device changes.

[0010] In some implementations, determining the target location is based at least in part on additional sensor data from the endoscopic mirror. Furthermore, determining the plane includes determining the plane such that the direction of travel of the endoscopic mirror is perpendicular to the plane.

[0011] In some embodiments, displaying the one or more interface elements includes displaying a first interface element at a second distance from the center of a second interface element among the one or more interface elements. This second distance may be at least partially based on the first distance and the insertion distance between the tip of the medical device and the target location on the plane.

[0012] In some embodiments, the method further includes: displaying a progress indicator indicating the proximity of the tip of the medical device to the target position; setting a progress change parameter of the progress indicator to a first value, the progress change parameter indicating the amount of progress change of the progress indicator relative to a unit of movement of the medical device; determining that the tip of the medical device has moved closer to the target position; and setting the progress change parameter to a second value based at least in part on the determination that the tip of the medical device has moved closer to the target position. The second value may be associated with a larger amount of progress change of the progress indicator relative to that unit of movement of the medical device than the first value.

[0013] In some embodiments, this disclosure relates to a medical system comprising: a communication interface configured to receive sensor data from a medical device configured to percutaneously enter a human anatomical structure; and control circuitry communicatively coupled to the communication interface. The control circuitry is configured to: determine, at least in part, based on the sensor data, an orientation of the medical device; determine a target location within the human anatomical structure; determine a plane including the target location; determine, at least in part, based on the orientation of the medical device, a projected position of the medical device on the plane; and generate interface data representing one or more interface elements indicating the distance between the projected position on the plane and the target location.

[0014] In some embodiments, the medical system further includes an endoscope configured to access the target location via the lumen of the human anatomy. The endoscope may include sensors configured to provide additional sensor data to the communication interface. The control circuitry may be configured to determine the target location based at least in part on the additional sensor data.

[0015] In some embodiments, the control circuit is configured to determine the plane such that the direction of travel of the medical device is perpendicular to the plane. The control circuit may also be configured to update the orientation of the plane as the orientation of the medical device changes, such that the direction of travel of the medical device remains perpendicular to the plane.

[0016] In some embodiments, the control circuit is configured to determine the plane by defining a line between the distal end of the medical device and the target location, and by defining the plane such that the line is perpendicular to the plane. The control circuit may be further configured to maintain the orientation of the plane as the orientation of the medical device changes.

[0017] In some embodiments, the medical system includes a robotic system coupled to a device configured to generate a signal, and the sensor data is based at least in part on the signal. The control circuitry may be further configured to: determine the coordinate system of the medical device based at least in part on the coordinate system of the robotic system; and cause one or more interface elements to move within the interface in a direction associated with the direction of movement of the medical device relative to its coordinate system.

[0018] In some embodiments, the control circuit is further configured to: set a position change parameter to a first value; determine that the medical device has moved closer to the target position; and set the position change parameter to a second value. The position change parameter may indicate the amount of positional change of the one or more interface elements within the interface relative to the movement of the medical device. The second value may be associated with a larger amount of positional change of the one or more interface elements within the interface relative to the movement of the medical device than the first value.

[0019] In some embodiments, the control circuitry is further configured to: cause a progress indicator to be displayed, the progress indicator indicating the proximity of the medical device to the target position; set a progress change parameter of the progress indicator to a first value; determine that the medical device has moved closer to the target position; and set the progress change parameter to a second value. The progress change parameter may indicate the amount of progress change of the progress indicator relative to a unit of movement of the medical device. The second value may be associated with a larger amount of progress change of the progress indicator relative to that unit of movement of the medical device than the first value.

[0020] In some embodiments, this disclosure relates to one or more non-transitory computer-readable media storing computer-executable instructions that, when executed by control circuitry, cause the control circuitry to perform operations including: determining the orientation of a medical device configured for percutaneous insertion into a human anatomical structure; determining a target location within the human anatomical structure; determining a plane including the target location; determining a projected position of the medical device on the plane, at least in part based on the orientation of the medical device; and generating interface data representing one or more interface elements indicating a first distance between the projected position on the plane and the target location.

[0021] In some embodiments, the direction of travel of the medical device is perpendicular to the plane. The operation may also include updating the orientation of the plane as the orientation of the medical device changes, such that the direction of travel of the medical device remains perpendicular to the plane.

[0022] In some embodiments, the plane is perpendicular to the line between the distal end of the medical device and the target location. The operation may also include maintaining the orientation of the plane as the orientation of the medical device changes.

[0023] In some embodiments, the operation further includes displaying the first interface element of the one or more interface elements at a second distance from the center of the second interface element of the one or more interface elements. The second distance may be at least partially based on the first distance. The second distance may further be at least partially based on the insertion distance between the tip of the medical device and the target location on the plane.

[0024] In some embodiments, the operation further includes: causing a progress indicator to be displayed, the progress indicator indicating the proximity of the medical device to the target location; updating the progress indicator by a second amount based at least in part on the medical device moving closer to the target location by a first amount for the first time; and updating the progress indicator by a third amount greater than the first amount based at least in part on the medical device moving closer to the target location by a second amount based on the medical device moving closer to the target location by the first amount for the second time.

[0025] In some implementations, the operation further includes: generating progress region data indicating multiple regions of the environment of the medical device; determining, at least in part, the region in which the medical device is located from the multiple regions based on the first sensor data and the progress region data; determining, at least in part, the state of the medical device based on the region; and causing an indication to be displayed on the state.

[0026] In some embodiments, this disclosure relates to a method comprising: receiving first sensor data from a needle configured for percutaneous insertion into a patient by control circuitry of a medical system; determining, by the control circuitry, an orientation of the needle based at least in part on the first sensor data; receiving, by the control circuitry, second sensor data from an endoscope disposed at least in part within an anatomical cavity of the patient; determining, by the control circuitry, a target location within an organ of the patient based at least in part on the second sensor data; determining, by the control circuitry, a plane including the target location; determining, by the control circuitry, a projected position of the needle on the plane based at least in part on the orientation of the needle; and generating interface data representing one or more interface elements indicating a distance between the projected position on the plane and the target location.

[0027] In some implementations, the method further includes: displaying the one or more interface elements in the interface; identifying a scaling ratio between the size represented on the plane and the size represented on the interface; determining that the needle has moved closer to the target position; and updating the scaling ratio at least in part based on the determination that the needle has moved closer to the target position.

[0028] In some embodiments, the method further includes: setting a position change parameter to a first value; determining that the medical device has moved closer to the target location; and setting the position change parameter to a second value, at least in part based on the determination that the medical device has moved closer to the target location. The position change parameter may indicate the amount of positional change of the one or more interface elements within the interface relative to a unit of position or orientation change of the needle. The second value may be associated with a larger amount of positional change of the one or more interface elements within the interface relative to that unit of position or orientation change of the needle.

[0029] In some embodiments, the medical system includes a robotic arm coupled to an electromagnetic field generator, and the first sensor data is based at least in part on an electromagnetic field from the electromagnetic field generator. The method may further include: determining a world coordinate system of a robotic system associated with the robotic arm; representing the needle's orientation in the world coordinate system; representing the plane in the world coordinate system; determining a target coordinate system for the plane represented in the world coordinate system based at least in part on the needle's orientation in the world coordinate system; determining the needle's coordinate system based at least in part on the target coordinate system; and moving the one or more interface elements within the interface in a direction related to the direction of movement of the needle relative to its coordinate system.

[0030] To summarize this disclosure, certain aspects, advantages, and features have been described. It should be understood that not all of these advantages need to be achieved according to any particular implementation. Therefore, the disclosed implementation may be carried out in a manner that achieves or optimizes one or more advantages taught herein, without necessarily achieving the other advantages taught or suggested herein. Attached Figure Description

[0031] For illustrative purposes, various embodiments are depicted in the accompanying drawings and should not in any way be construed as limiting the scope of this disclosure. Furthermore, various features of the different disclosed embodiments can be combined to form additional embodiments, which are part of this disclosure. Throughout the drawings, reference numerals may be repeatedly used to indicate correspondences between reference elements.

[0032] Figure 1 An implementation of a medical system configured to implement the techniques discussed herein is shown, according to one or more embodiments.

[0033] Figure 2 An exemplary interface for providing information about the location and / or orientation of a medical device, according to one or more embodiments, is shown.

[0034] Figure 3 The illustration shows a device arranged to assist in inserting a trocar into a patient, according to one or more embodiments. Figure 1 A top view of the medical system.

[0035] Figure 4 A navigation endoscope arranged inside a patient's body according to one or more embodiments is shown. Figure 1 A top view of the medical system.

[0036] Figure 5 The diagram illustrates an arrangement, according to one or more embodiments, to assist in the insertion of a needle into a patient. Figure 1 A top view of the medical system.

[0037] Figures 6-1 to 6-11 An exemplary interface is shown, according to one or more embodiments, for providing information about the alignment and / or progress of a medical device during a procedure.

[0038] Figure 7 An exemplary flowchart is shown, according to one or more embodiments, of a process for determining the alignment of a medical device relative to a target trajectory and presenting information about that alignment.

[0039] Figure 8 An exemplary flowchart is shown, according to one or more embodiments, of a process for presenting information about the orientation of a medical device.

[0040] Figure 9 An exemplary flowchart is shown, according to one or more embodiments, of a process for presenting information about the proximity of a medical device to a target location.

[0041] Figure 10 An exemplary flowchart is shown, according to one or more embodiments, of a process for setting and / or updating position change parameters associated with an instrument alignment element.

[0042] Figure 11 The following are illustrated according to one or more implementation schemes. Figure 1 Exemplary details of the robotic system.

[0043] Figure 12 The following are illustrated according to one or more implementation schemes. Figure 1 Exemplary details of the control system.

[0044] Figures 13-1 to 13-3 Exemplary techniques for mapping the orientation of a medical device onto a representation / plane that remains fixed as the orientation of the medical device changes, according to one or more embodiments, are illustrated.

[0045] Figures 14-1 to 14-3 Exemplary techniques for mapping the orientation of a medical device onto a representation / plane that is updated as the orientation of the medical device changes, according to one or more embodiments, are illustrated.

[0046] Figures 15-1 to 15-3 Another example is shown of mapping the orientation of a medical device onto a representation / plane that is updated as the orientation of the medical device changes, according to one or more embodiments.

[0047] Figures 16-1 to 16-3 Another example is shown of mapping the orientation of a medical device onto a representation / plane that is updated as the orientation of the medical device changes, according to one or more embodiments.

[0048] Figure 17 Exemplary techniques for establishing one or more coordinate systems to correlate the movement of a medical device with the movement of interface elements within the interface, according to one or more embodiments, are illustrated.

[0049] Figure 18 illustrates an exemplary adaptive target determination technique for an interface according to one or more implementations.

[0050] Figure 19-1 and Figure 19-2 An exemplary technique for scaling a target plane based on the insertion distance of a medical device, according to one or more embodiments, is shown.

[0051] Figure 19-3An exemplary graph showing the scaling of the target plane relative to the distance from the medical device to the target location according to one or more embodiments is shown.

[0052] Figure 20 An exemplary anatomical visualization section, which can be provided via an interface to help a user navigate a medical device according to one or more embodiments, is shown.

[0053] Figure 21 Exemplary areas that can be implemented / generated according to one or more implementation schemes to determine the state of a medical device are shown.

[0054] Figure 22 An exemplary graph showing the percentage progress relative to the distance from the medical device to the target location, according to one or more embodiments, is shown.

[0055] Figure 23 An exemplary flowchart is shown, according to one or more embodiments, of a process for generating data indicating the distance between a projected position of a medical device on a plane and a target position on the plane.

[0056] Figure 24 An exemplary flowchart is shown, according to one or more embodiments, of a process for updating the progress amount of a movement unit indication for a medical device.

[0057] Figure 25 An exemplary flowchart is shown, according to one or more embodiments, of a process for updating position change parameters that indicate the amount of position change of an interface element relative to the orientation / movement change unit of a medical device.

[0058] Figure 26 This is an exemplary flowchart of a process for updating the scaling ratio of a plane and an interface according to one or more implementations.

[0059] Figure 27 An exemplary flowchart is shown, according to one or more embodiments, of a process for determining the coordinate system of a medical device and / or associating the movement of interface elements with the movement of the medical device.

[0060] Figure 28 An exemplary flowchart is shown, according to one or more embodiments, of a process for determining the status of a medical device based on the region where the medical device is located. Detailed Implementation

[0061] The headings provided herein are for convenience only and do not necessarily affect the scope or meaning of the disclosure. While certain preferred embodiments and examples are disclosed below, this subject matter extends beyond the specific disclosed embodiments to other alternative embodiments and / or uses, modifications, and equivalents thereof. Therefore, the scope of the claims that may appear herein is not limited to any particular embodiment described below. For example, in any method or process disclosed herein, the behavior or operation of the method or process may be performed in any suitable order and is not necessarily limited to any particular disclosed sequence. Various operations may be described sequentially as multiple discrete operations in a manner that helps to understand certain embodiments; however, the order of description should not be construed as implying that these operations are sequentially related. Additionally, the structures, systems, and / or apparatuses described herein may be implemented as integrated components or separate components. For comparison of the various embodiments, certain aspects and advantages of these embodiments are described. Not all of these aspects or advantages must be achieved by any particular embodiment. Thus, for example, various embodiments may be performed in a manner that achieves or optimizes one or more advantages taught herein without necessarily achieving other aspects or advantages taught or suggested herein.

[0062] Regarding preferred embodiments, certain standard positional anatomical terms may be used herein to refer to the anatomy of animals (i.e., humans). Although certain spatial relative terms, such as “external,” “internal,” “upper,” “lower,” “below,” “above,” “vertical,” “horizontal,” “top,” “bottom,” and similar terms, are used herein to describe the spatial relationship of one device / element or anatomical structure to another device / element or anatomical structure, it should be understood that these terms are used herein for ease of description to describe the positional relationship between elements / structures, as illustrated in the figures. It should be understood that spatial relative terms are intended to encompass different orientations of elements / structures in use or operation, other than those depicted in the figures. For example, an element / structure described as “above” another element / structure may indicate a position relative to the patient being examined or an alternating orientation of the element / structure below or beside such other elements / structures, and vice versa.

[0063] Overview

[0064] This disclosure relates to systems, devices, and methods for assisting physicians or other users in orienting medical devices for percutaneous access to a location within human anatomy. While certain aspects of this disclosure (such as kidney stone removal / treatment procedures) are described in detail herein within the context of kidney, urinary, and / or renal protocols, it should be understood that such context is provided for convenience and clarity, and that the concepts disclosed herein apply to any suitable medical protocol. However, as mentioned, a description of renal / urinary anatomy and associated medical issues and protocols is presented below to aid in the description of the concepts disclosed herein.

[0065] Kidney stones (also known as urolithiasis) are a relatively common medical condition involving the formation of solid masses in the urinary tract, called "kidney stones" (also known as renal calculi, renal lithiasis, or nephrolithiasis) or "urinary stones." Urinary stones can form and / or be found in the kidneys, ureters, and bladder (called "bladder stones"). These types of urinary stones are formed from concentrated minerals and can cause severe abdominal pain once the stone is large enough to obstruct urine flow through the ureter or urethra. Urinary stones can be formed from calcium, magnesium, ammonia, uric acid, cysteine, and / or other compounds.

[0066] To remove urinary stones from the bladder and ureters, a surgeon inserts a ureteroscope into the urinary tract through the urethra. Typically, the ureteroscope includes an endoscope at its distal end, which is configured to allow visualization of the urinary tract. The ureteroscope may also include a stone retrieval mechanism to capture or break up the stones. During a ureteroscopy procedure, one physician / technician controls the position of the ureteroscope, while another physician / technician controls the stone retrieval mechanism.

[0067] To remove relatively large stones (i.e., "kidney stones") from the kidneys, physicians can use percutaneous nephrolithotomy ("PCNL"), which involves inserting a nephroscope through the skin to break up and / or remove the stones. Fluoroscopy can be used to locate the kidney stones, providing a target for nephroscope insertion. However, fluoroscopy increases the cost of kidney stone removal surgery due to the cost of fluoroscopy itself and the cost of the technician operating the fluoroscope. Fluoroscopy also exposes patients to radiation for extended periods. Even with fluoroscopy, accurately accessing the kidney stone through the percutaneous incision can be difficult and inaccurate. Furthermore, some kidney stone removal surgeries require a two- or three-day hospital stay. In summary, certain kidney stone removal surgeries are relatively expensive and problematic for patients.

[0068] In some specific implementations, this disclosure relates to techniques and systems for assisting in the alignment of a medical device for percutaneous entry into a target location within a human anatomy. For example, to perform a medical procedure, a physician or other user may use a medical device to enter a target location within a patient to remove a kidney stone located within the kidney. The target location may represent the desired location of the medical device within the patient's anatomy, such as the desired nipple or other location within the kidney. The techniques and systems discussed herein can provide information about the orientation / position of the medical device to assist the physician or other user in aligning the medical device appropriately and / or inserting it into the patient to reach the target location. For example, the techniques and systems may provide a visual representation indicating the current orientation of the medical device relative to a target trajectory / or posture, a visual representation indicating the proximity of the medical device to the target location, and / or other information about the medical device and / or the procedure. The target trajectory may represent the desired path used to enter the target location from an entry point on the patient (such as a location on the patient's skin). By providing such information, the physician or other user can accurately manipulate / manipulate the medical device to reach the target location and perform the medical procedure in a manner that minimizes damage to the patient's anatomy.

[0069] In many embodiments, these techniques and systems are discussed in the context of percutaneous procedures, which may include any procedure that obtains access to a target location by creating punctures and / or incisions in the skin, mucous membranes, and / or other body layers. However, it should be understood that these techniques and systems can be implemented in the context of any medical procedure, including, for example, minimally invasive procedures (e.g., laparoscopy), non-invasive procedures (e.g., endoscopy), therapeutic procedures, diagnostic procedures, percutaneous procedures, non-percutaneous procedures, or other types of procedures. Endoscopic procedures may include bronchoscopy, ureteroscopy, gastroscopy, nephroscopy, nephrectomy, etc. In some embodiments, in the context of a laparoscopic procedure or another procedure, these techniques and systems may be used to align a first medical device with a second medical device / anatomical location, such as guide port placement (e.g., aligning a first cannula with a second cannula / anatomical location). Furthermore, in some implementations, within the context of diagnostic procedures, these technologies and systems can be used to align an ultrasound probe equipped with electromagnetic sensors with anatomical targets or to guide the user to a set of target orientations to reconstruct anatomical structures, such as three-dimensional (3D) renal anatomy. Additionally, in some implementations, within the context of endoscopic procedures, these technologies and systems can be used to guide the placement of a bronchoscope while performing a biopsy at a marked location, such as a tumor site.

[0070] healthcare system

[0071] Figure 1An exemplary medical system 100 for performing various medical procedures according to aspects of this disclosure is illustrated. Medical system 100 includes a robotic system 110 configured to engage and / or control a medical device 120 to perform procedures on a patient 130. Medical system 100 also includes a control system 140 configured to interact with the robotic system 110, providing information about the procedures and / or performing various other operations. For example, control system 140 may include a display 142 to present information to assist physician 160. Medical system 100 may include a workbench 150 configured to hold the patient 130. System 100 may also include an electromagnetic (EM) field generator 180, which may be held by one or more robotic arms 112 of robotic system 110 or may be a stand-alone device. In this example, medical system 100 may also include an imaging device 190, which may be integrated into a C-arm and / or configured to provide imaging during procedures such as fluoroscopic procedures. Despite Figure 1 As shown in the figure, but in some embodiments the imaging device 190 is omitted.

[0072] In some implementations, medical system 100 can be used to perform percutaneous procedures. For example, if a patient 130's kidney stone is too large to be removed through the urinary tract, physician 160 can perform a procedure to remove the kidney stone through a percutaneous entry point on the patient 130. For illustration, physician 160 can interact with control system 140 to control robotic system 110 to advance and navigate a medical device 120 (e.g., a speculum) from the urethra, through the bladder, up the ureter, and into the kidney where the stone is located. Control system 140 can provide information about medical device 120 via display 142 to assist physician 160 in navigating medical device 120, such as real-time images captured with it.

[0073] Once the site of the kidney stone is reached (e.g., within the renal calyx), medical device 120 can be used to designate / mark the target location (e.g., the desired point of entry into the kidney) for percutaneous entry of medical device 170 into the kidney. To minimize damage to the kidney and / or surrounding anatomy, physician 160 may designate a specific papilla as the target location for entry of medical device 170 into the kidney. However, other target locations may be designated or identified. To assist physician in inserting medical device 170 into patient 130 through a specific papilla, control system 140 provides device alignment interface 144, which may include a visualization for indicating the alignment of the orientation of medical device 170 relative to a target trajectory (e.g., the desired entry path), a visualization for indicating the progress of insertion of medical device 170 toward the target location, and / or other information. Once medical device 170 reaches the target location, physician 160 may use medical device 170 and / or another medical device, such as through a percutaneous entry point, to remove the kidney stone from patient 130.

[0074] Although the above-described percutaneous procedures and / or other procedures have been discussed in the context of the use of medical device 120, in some specific implementations, percutaneous procedures can be performed without the assistance of medical device 120. Furthermore, medical system 100 can be used to perform a variety of other procedures.

[0075] Furthermore, although many embodiments describe physician 160 using medical device 170, medical device 170 can also be used by components of medical system 100. For example, medical device 170 can be held / manipulated by robotic system 110 (e.g., one or more robotic arms 112), and the techniques discussed herein can be implemented to control robotic system 110 to insert medical device 170 in an appropriate orientation to reach a target location.

[0076] exist Figure 1In the example, medical device 120 is implemented as a endoscopic endoscope, while medical device 170 is implemented as a needle. Therefore, for ease of discussion, medical device 120 is referred to as "endoscopic endoscope 120" or "lumen-based medical device 120," and medical device 170 is referred to as "needle 170" or "percutaneous medical device 170." However, medical device 120 and medical device 170 can each be implemented as suitable types of medical devices, including, for example, endoscopic endoscopes (sometimes called "endoscopes"), needles, catheters, guidewires, lithotripters, basket retrieval devices, forceps, vacuum devices, scalpels, imaging probes, clamps, scissors, graspers, needle holders, microscalpels, staplers, flathead staplers, aspiration / irrigation tools, clamp applicators, etc. In some embodiments, the medical device is a manipulable device, while in other embodiments, the medical device is a non-manipulable device. In some embodiments, surgical instruments refer to devices configured to puncture or insert through human anatomical structures, such as needles, scalpels, guidewires, etc. However, surgical tools can also refer to other types of medical devices.

[0077] In some embodiments, the medical device (such as endoscope 120 and / or needle 170) includes a sensor configured to generate sensor data that can be transmitted to another device. In examples, the sensor data may indicate the position / orientation of the medical device and / or be used to determine the position / orientation of the medical device. For example, the sensor may include an electromagnetic (EM) sensor having a coil of conductive material. Here, an EM field generator (such as EM field generator 180) provides an EM field detected by the EM sensor on the medical device. The magnetic field may induce a small current in the coil of the EM sensor, which may be analyzed to determine the distance and / or angle / orientation between the EM sensor and the EM field generator. Furthermore, the medical device may include other types of sensors configured to generate sensor data, such as cameras, distance sensors, radar devices, shape-sensing fibers, accelerometers, gyroscopes, satellite-based positioning sensors (e.g., Global Positioning System (GPS)), radio frequency transceivers, etc. In some embodiments, the sensor is positioned on the distal end of the medical device, while in other embodiments, the sensor is positioned at another location on the medical device. In some implementations, sensors on the medical device may provide sensor data to the control system 140, and the control system 140 may perform one or more positioning techniques to determine / track the position and / or orientation of the medical device.

[0078] The term "scope" or "endoscope" is used herein in its broad and general sense and can refer to any type of elongated medical device having image generation, observation, and / or capture capabilities and configured to be introduced into any type of organ, cavity, lumen, chamber, and / or space of the body. For example, a scope or endoscope as referred to herein can mean a ureteroscope (e.g., for access to the urinary tract), a laparoscope, a nephroscope (e.g., for access to the kidney), a bronchoscope (e.g., for access to the airway, such as the bronchi), a colonoscope (e.g., for access to the colon), an arthroscope (e.g., for access to a joint), a cystoscope (e.g., for access to the bladder), a tubular endoscope, etc.

[0079] The endoscope may include tubular and / or flexible medical devices configured for insertion into a patient's anatomy to capture images of the anatomical structures. In some embodiments, the endoscope may house wires and / or optical fibers to transmit signals to / from an optical assembly and a distal end of the endoscope, the distal end of which may include an imaging device, such as an optical camera. The camera / imaging device may be used to capture images of internal anatomical spaces, such as target calyces / papillaries of the kidney. The endoscope may be further configured to house optical fibers to transmit light from a proximal light source (such as a light-emitting diode) to the distal end of the endoscope. The distal end of the endoscope may include a port for the light source to illuminate the anatomical space when the camera / imaging device is used. In some embodiments, the endoscope is configured to be controlled by a robotic system (such as robotic system 110). The imaging device may include optical fibers, fiber arrays, and / or lenses. The optical components may move with the tip of the endoscope such that movement of the tip causes a change in the image captured by the imaging device.

[0080] The endoscope can be articulated, such as articulating relative to at least its distal end portion, allowing it to be manipulated within the human anatomy. In some embodiments, the endoscope is configured to articulate with, for example, five or six degrees of freedom, including X, Y, and Z coordinate movement, as well as pitch, yaw, and roll. The endoscope's position sensors may also have similar degrees of freedom relative to the positional information they generate / provide. The endoscope may include telescopic components, such as an internal guide portion and an external sheath portion, which can be manipulated to telescopically extend the endoscope. In some instances, the endoscope may include a rigid or flexible tube, and its dimensions may be configured to pass through an external sheath, catheter, guide, or other lumen-like device, or may be used without such devices. In some embodiments, the endoscope includes a working channel for deploying medical instruments (e.g., lithotripters, basket devices, forceps, etc.), irrigation, and / or suction to the surgical area at the distal end of the endoscope.

[0081] The robotic system 110 can be configured to at least partially facilitate the execution of medical procedures. The robotic system 110 can be configured in various ways, depending on the specific procedure. The robotic system 110 may include one or more robotic arms 112 configured to engage with and / or control a endoscopic endoscope 120 to execute procedures. As shown, each robotic arm 112 may include multiple arm segments coupled to joints, which provide multiple degrees of mobility. Figure 1 In the example, the robotic system 110 is positioned near the patient 130's leg, and the robotic arm 112 is actuated to engage and position the endoscope 120 to access a point of entry, such as the patient 130's urethra. When the robotic system 110 is correctly positioned, the endoscope 120 can be inserted into the patient 130 using the robotic arm 112, manually by a physician 160, or a combination thereof. The robotic arm 112 can also be connected to an EM field generator 180, which can be positioned near the treatment site, such as within proximity to the patient 130's kidney.

[0082] The robot system 110 may also include a support structure 114 coupled to one or more robot arms 112. The support structure 114 may include control electronics / circuit, one or more power supplies, one or more pneumatic devices, one or more light sources, one or more actuators (e.g., motors for moving one or more robot arms 112), memory / data storage devices, and / or one or more communication interfaces. In some embodiments, the support structure 114 includes an input / output (I / O) device 116 configured to receive input (such as user input for controlling the robot system 110) and / or provide output (such as a graphical user interface (GUI), information about the robot system 110, information about procedures, etc.). The I / O device 116 may include a display, touchscreen, touchpad, projector, mouse, keyboard, microphone, speaker, etc. In some embodiments, the robot system 110 is movable (e.g., the support structure 114 includes wheels), allowing the robot system 110 to be positioned suitable or desired for use in procedures. In other embodiments, the robot system 110 is a fixed system. In addition, in some implementations, the robot system 112 is integrated into the workbench 150.

[0083] Robotic system 110 may be coupled to any component of medical system 100, such as control system 140, worktable 150, EM field generator 180, endoscope 120, and / or needle 170. In some embodiments, the robotic system is communicatively coupled to control system 140. In one example, robotic system 110 may be configured to receive control signals from control system 140 to perform operations such as positioning robotic arm 112 in a specific manner, manipulating endoscope 120, etc. In response, robotic system 110 may control its components to perform operations. In another example, robotic system 110 is configured to receive images depicting the internal anatomy of patient 130 from endoscope 120 and / or send such images to control system 140, where they can then be displayed on display 142. Furthermore, in some embodiments, robotic system 110 is coupled to components of medical system 100 (such as control system 140) to allow the reception of fluids, optics, power, etc. References are made below. Figure 11 Exemplary details of the robot system 110 are discussed in further detail.

[0084] The control system 140 can be configured to provide various functions to assist in the execution of medical procedures. In some embodiments, the control system 140 may be coupled to and operate in cooperation with the robotic system 110 to perform medical procedures on the patient 130. For example, the control system 140 may communicate with the robotic system 110 via a wireless or wired connection (e.g., to control the robotic system 110 and / or endoscope 120, receive images captured by the endoscope 120, etc.), supply fluid to the robotic system 110 via one or more fluid channels, supply power to the robotic system 110 via one or more electrical connections, supply optics to the robotic system 110 via one or more optical fibers or other components, etc. Furthermore, in some embodiments, the control system 140 may communicate with the needle 170 and / or endoscope 170 to receive sensor data from the needle 170 and / or endoscope 120 (receiving sensor data via the robotic system 110 and / or directly from the needle 170 and / or endoscope 120). Furthermore, in some embodiments, the control system 140 may communicate with the worktable 150 to position the worktable 150 in a particular orientation or otherwise control the worktable 150. Additionally, in some embodiments, the control system 140 may communicate with the EM field generator 180 to control the generation of the EM field around the patient 130.

[0085] The control system 140 includes various I / O devices configured to assist physician 160 or other personnel in performing medical procedures. In this example, the control system 140 includes an I / O device 146 used by physician 160 or other users to control endoscope 120 to navigate it within patient 130. For example, physician 160 may provide input via I / O device 146, and in response, control system 140 may send control signals to robotic system 110 to manipulate endoscope 120. Although in Figure 1 In the example, I / O device 146 is shown as a controller, but I / O device 146 can be implemented as various types of I / O devices, such as touch screen, touchpad, mouse, keyboard, etc.

[0086] Other examples Figure 1 As shown, the control system 140 may include a display 142 to provide various information about the procedure. As described above, the display 142 may present an instrument alignment interface 144 to assist the physician 160 in manipulating the needle 170. The display 142 may also provide (e.g., via the instrument alignment interface 144 and / or another interface) information about the endoscope 120. For example, the control system 140 may receive real-time images captured by the endoscope 120 and display the real-time images via the display 142. Figure 2 An exemplary device alignment interface is shown. In addition or alternatively, the control system 140 may receive signals (e.g., analog signals, digital signals, electrical signals, acoustic / sound signals, pneumatic signals, tactile signals, hydraulic signals, etc.) from medical monitors and / or sensors associated with the patient 130, and the display 142 may present information about the patient 130's health or environment. Such information may include information displayed via medical monitors, including, for example, heart rate (e.g., ECG, HRV, etc.), blood pressure / rate, muscle biosignals (e.g., EMG), body temperature, blood oxygen saturation (e.g., SpO2), CO2, brain waves (e.g., EEG), environmental and / or local or core body temperature, etc.

[0087] To facilitate the functionality of the control system 140, the control system 140 may include various components (sometimes referred to as "subsystems"). For example, the control system 140 may include control electronics / circuit, and one or more power supplies, pneumatic devices, light sources, actuators, memory / data storage devices, and / or communication interfaces. In some embodiments, the control system 140 includes control circuitry that includes a computer-based control system configured to store executable instructions that, when executed, cause various operations to be performed. In some embodiments, the control system 140 is mobile, such as... Figure 1As shown, in other embodiments, the control system 140 is a fixed system. Although various functions and components implemented by the control system 140 have been discussed, any of these functions and / or components can be integrated into and / or performed by other systems and / or devices, such as the robot system 110, the workbench 150, and / or the EM generator 180 (or even the hysteroscope 120 and / or the needle 170). References below... Figure 12 Exemplary details of the control system 140 are discussed in further detail.

[0088] Imaging device 190 can be configured to capture / generate one or more images of patient 130 during procedures, such as one or more X-ray or CT images. In this example, images from imaging device 190 can be provided in real time to view anatomical structures and / or medical instruments (such as endoscope 120 and / or needle 170) within patient 130 to assist physician 160 in performing procedures. Imaging device 190 can be used to perform fluoroscopy (e.g., using contrast dyes within patient 130) or other types of imaging techniques. Although in Figure 1 As shown, but in many embodiments, the imaging device 190 is not implemented to perform procedures and / or remove the imaging device 190 (including the C-arm).

[0089] The various components of the medical system 100 can be communicatively coupled to each other via a network, which may include wireless and / or wired networks. Exemplary networks include one or more personal area networks (PANs), local area networks (LANs), wide area networks (WANs), Internet local area networks (IANs), cellular networks, the Internet, etc. Furthermore, in some embodiments, the components of the medical system 100 are connected via one or more support cables, conduits, etc., for data communication, fluid / gas exchange, power exchange, etc.

[0090] Medical system 100 offers a variety of benefits, such as providing guidance to assist physicians in performing procedures (e.g., instrument tracking, instrument alignment information, etc.), enabling physicians to perform procedures from an ergonomic position without awkward arm movements and / or positioning, enabling a single physician to perform procedures using one or more medical instruments, avoiding radiation exposure (e.g., associated with fluoroscopy), enabling procedures to be performed in a single surgical setting, and providing continuous aspiration for more efficient removal of axial structures (e.g., removal of kidney stones). For example, medical system 100 can provide guidance to help physicians use various medical instruments to access target anatomical features while minimizing bleeding and / or damage to anatomical structures (e.g., critical organs, blood vessels, etc.). Furthermore, medical system 100 can provide radiation-based navigation and / or positioning technologies to reduce radiation exposure for physicians and patients and / or reduce the amount of equipment in the operating room. Additionally, medical system 100 can provide functionality distributed between at least control system 140 and robotic system 110, which can move independently. Such a distribution of functionality and / or mobility allows the control system 140 and / or the robotic system 110 to be placed in an optimal location for a particular medical procedure, which maximizes the work area around the patient and / or provides an optimal location for the physician to perform the procedure.

[0091] While various technologies and systems are discussed for implementation as robot-assisted procedures (e.g., procedures that at least partially utilize medical system 100), these technologies and systems can be implemented in other procedures, such as fully robotic medical procedures, human-only procedures (e.g., non-robotic systems), etc. For example, medical system 100 can be used to perform procedures without a physician holding / manipulating medical instruments (e.g., fully robotic procedures). That is, medical instruments used during the procedure (such as endoscope 120 and needle 170) can be held / controlled by components of medical system 100 (such as the robotic arm 112 of robotic system 110).

[0092] Exemplary Interface

[0093] Figure 2An exemplary device alignment interface 200 for providing information about the location and / or orientation of a medical device and / or other information about medical procedures, according to one or more embodiments, is illustrated. As shown, the device alignment interface 200 (sometimes referred to as "device alignment graphical user interface (GUI) 200") may include: a hysteroscope segment 210 for providing an image 212 captured by a first medical device (such as a hysteroscope); and an alignment segment 220 for providing information about the orientation of a second medical device (such as a needle). Although the hysteroscope segment 210 and the alignment segment 220 are shown as being included in the same device alignment interface 200, in some embodiments, the device alignment interface 200 includes only one of segments 210 and 220. For example, the alignment segment 220 may be included as part of the device alignment interface 200, and the hysteroscope segment 210 may be included in an additional interface. Furthermore, in some examples, the peephole segment 210 and / or alignment segment 220 may be implemented within an augmented or virtual reality interface, such as alignment segment 220 covering at least a portion of peephole segment 212, or peephole segment 212 presenting alignment information in a different form than alignment segment 220. Figure 2 In the example, the instrument alignment interface 200 provides information for procedures using a endoscopic endoscope and another medical device. However, the instrument alignment interface 200 can be used for other types of procedures, such as those performed without an endoscope. In such cases, image 212 may not be displayed and / or the endoscope segment 210 may be removed.

[0094] As described above, the endoscope segment 212 provides an image 212 for an endoscope configured to navigate within a lumen or other anatomical structure. In this example, image 212 depicts the internal portion of the kidney, including a lumen 214 and a kidney stone 216 located within one of the lumens 214. Here, the kidney stone 216 is located within a calyx near the papilla. However, image 212 can depict any human anatomical structure depending on the position of the endoscope within the patient's body. Image 212 may include real-time images, such as video.

[0095] Alignment section 220 includes an alignment progress visualization unit 230 to indicate the alignment of the medical device's orientation with the target trajectory and / or the proximity of the medical device to the target position. As shown, the alignment progress visualization unit 230 includes: a device alignment element 232 (sometimes referred to as "device alignment icon 232" or "alignment icon 232") representing the orientation of the medical device; and an alignment mark 234 associated with the target trajectory. In this example, the device alignment element 232 can move within an area defined by the alignment mark 234(C) (also referred to as "boundary mark 234(C)") based on changes in the medical device's orientation. For example, as the medical device tilts, the device alignment element 232 can change its position within this area. In this example, the alignment mark 234(A) can represent the target position / trajectory / attitude.

[0096] In some embodiments, tilting the medical device in one direction will cause the device alignment element 232 to move in the opposite direction, similar to a bullseye-type level. For example, if the medical device tilts to the right, the device alignment element 232 may move to the left. In other embodiments, tilting the medical device will cause the device alignment element 232 to move in the same direction as the tilt. For example, if the medical device tilts to the right, the device alignment element 232 may move to the right. In any case, when the orientation of the medical device is aligned with the target trajectory, the arrangement of the device alignment element 232 aligned with the alignment mark 234 can be shown (e.g., centered on the alignment mark 234, such as within or centered on the alignment mark 234(A)).

[0097] In some implementations, for each unit of orientation change of the medical device, the amount of positional change of the device alignment element 232 (e.g., the sensitivity of the device alignment element 232) is based on the proximity of the medical device to the target location. For example, as the medical device moves closer to the target location, the device alignment element 232 can be moved by a larger or smaller amount to achieve the same amount of orientation change of the medical device. To illustrate, when the medical device is at a first distance from the target location, the device alignment interface 200 may change the position of the device alignment element 232 by a first amount in response to a unit of orientation change of the medical device. When the medical device is at a second distance from the target location (e.g., closer to the target location), the device alignment interface 200 may change the position of the device alignment element 232 by a second amount (e.g., a larger or smaller amount) in response to the same unit of orientation change of the medical device.

[0098] In some implementations, altering the sensitivity of the instrument alignment element 232 can further assist the physician in using the medical device to reach a target location. For example, in some cases, lower precision may be required to orient the medical device as it moves further from the target. As the medical device moves closer to the target location, higher precision may be required to orient it. In other words, as the medical device moves closer to the target, the physician may need to adjust the orientation of the medical device more precisely to actually reach the target. Therefore, by altering the sensitivity of the instrument alignment element 232, the physician can more precisely manipulate the medical device to reach a target location, which may be relatively small.

[0099] The alignment progress visualization unit 230 may also include a progress bar 236 to indicate the proximity of the medical device to the target location. Figure 2 In the example, progress bar 236 is presented around boundary marker 234(C). However, progress bar 236 can be presented anywhere within the device alignment interface 200, such as on one side of the alignment progress visualization section 230. Progress bar 236 provides information about the current position of the medical device relative to the target position. For example, progress bar 236 can be filled as the medical device moves closer to the target position, as discussed in the examples below. In some embodiments, if the medical device has reached the target position, the device alignment interface 200 can provide an indication that the medical device has reached the target position, such as an indication on progress bar 236. Similarly, in some embodiments, if the medical device is inserted outside the target position, the device alignment interface 200 can provide an indication that the medical device has been inserted outside the target position, such as an indication on progress bar 236. Figure 2 In the example, progress bar 236 indicates that the medical device has not yet been inserted into the patient's body (e.g., the medical device is on the patient's skin or outside the patient's body).

[0100] In some implementations, the alignment progress visualization unit 230 includes a single visualization unit for viewing information about the orientation and progress of the medical device. For example, information about the orientation of the medical device and its progress toward the target location can be displayed in a combined visualization unit. This combined visualization unit allows physicians or other users to maintain visual contact with a single item while manipulating the medical device and avoids accidental movement of the medical device due to movement of the physician's eyes or body, while viewing multiple displays, interfaces, visualization units, etc. Therefore, the combined visualization unit allows physicians or other users to more accurately manipulate the medical device to reach the target location inside the patient's body.

[0101] exist Figure 2In the example, the various components of the alignment progress visualization unit 230 are presented as circles. However, any component of the alignment progress visualization unit 230 can take many forms, such as any other shape. For example, the alignment mark 234, the instrument alignment element 232, and / or the progress bar 236 can be presented as rectangles or any other shape. In some specific embodiments, the instrument alignment element 232 includes a bubble representation used to indicate bubbles.

[0102] Other examples Figure 2 As shown, the device alignment interface 200 may include a navigation representation 240 for navigating between different visualizations associated with different stages / steps of a procedure. For example, a procedure associated with the removal of kidney stones may include multiple stages / steps, one of which involves aligning the medical device and inserting it into the patient to reach a target location. For this stage / step, the device alignment interface 200 may display... Figure 2 The information shown is intended to assist physicians in performing this stage / step. Physicians can move to different visualizations or interfaces of the previous or next stage / step by selecting the "back" or "next" text within the navigation representation 240. Furthermore, the instrument alignment interface 200 may include a visual representation 250 for accessing menus that allow access to interfaces / information associated with other types of procedures or other information.

[0103] Although many embodiments are discussed and shown in the context of instrument alignment interfaces including two-dimensional (2D) representations, in some embodiments, the instrument alignment interface may include a three-dimensional (3D) representation. For example, the instrument alignment interface may present a plane and twisted lines on that plane to indicate misalignment, the shape / form of the plane being configured to be twisted / changed to indicate the misaligned plane, etc.

[0104] In some implementations, the device alignment interface 200 and / or any other interface discussed herein are based on target determination data and / or interface data. For example, target determination / interface data indicating the projected position of a medical device on a plane relative to a target location on that plane can be generated. The target / interface data can be used to display the alignment progress visualization 230 to locate the device alignment element 232, which represents the orientation of the medical device relative to an alignment mark 234 associated with a target trajectory.

[0105] Example procedures performed using a medical system

[0106] Figures 3 to 5 This illustrates arrangements for performing transdermal procedures according to one or more embodiments. Figure 1A top view of the medical system 100. In these examples, the medical system 100 is positioned in an operating room to remove kidney stones from a patient 130 using a speculum 120 and a needle 170. In many embodiments of this procedure, the patient 130 is positioned in a modified supine position, slightly tilted to one side to approach the patient's back or side, such as... Figure 1 As shown. However, patient 130 can also be positioned in other ways, such as supine or prone. To facilitate viewing the anatomical structures of patient 130, Figures 3 to 5 The patient 130 is shown in a supine position with legs spread apart. Furthermore, for ease of illustration, the imaging device 190 (including the C-arm) has been removed.

[0107] although Figures 3 to 5 The illustration shows the use of medical system 100 to perform a percutaneous procedure to remove kidney stones from patient 130, but medical system 100 can be used to remove kidney stones in other ways and / or perform other procedures. Furthermore, patient 130 can be positioned in other locations as required by the procedure. Figures 3 to 5 The disclosure describes various actions performed by the physician 160. It should be understood that these actions can be performed directly by the physician 160, a user under the physician's guidance, another user (e.g., a technician), a combination thereof, and / or any other user.

[0108] As in Figures 3 to 5 The anatomy of the kidneys is shown in at least part here, and is described herein for reference in certain medical protocols relating to aspects of this concept. The kidneys typically consist of two bean-shaped organs located on either side of each other in the retroperitoneal space. An adult kidney is generally about 11 cm long. The kidneys receive blood from paired renal arteries; blood flows into paired renal veins. Each kidney is attached to a ureter, which is the tube that transports excreted urine from the kidney to the bladder. The bladder is attached to the urethra.

[0109] The kidneys are typically located relatively high in the abdominal cavity, retroperitoneally, at a slightly angled position. Intra-abdominal asymmetry caused by the location of the liver usually results in the right kidney being slightly lower than the left and slightly more centrally positioned than the left. The adrenal gland is located at the top of each kidney. The upper part of the kidney is partially protected by the 11th and 12th ribs. Each kidney and its adrenal gland are surrounded by two layers of fat: perirenal fat between the renal fascia and renal capsule, and pararenal fat higher than the renal fascia.

[0110] The kidneys are involved in controlling the volume of various fluid compartments, fluid osmotic pressure, acid-base balance, electrolyte concentrations, and toxin removal. The kidneys perform their filtering function by secreting certain substances and reabsorbing others. Examples of substances secreted into urine include hydrogen, ammonium, potassium, and uric acid. In addition, the kidneys perform various other functions, such as hormone synthesis.

[0111] The indented area on the renal rim is the renal hilum, where the renal artery enters the kidney, and the renal vein and ureter exit. The kidney is surrounded by tough fibrous tissue and the renal capsule, which itself is surrounded by perirenal fat, renal fascia, and pararenal fat. The anterior surface of these tissues is the peritoneum, while the posterior surface is the transverse fascia.

[0112] The functional substance or parenchyma of the kidney is divided into two main structures: the extrarenal cortex and the intrarenal medulla. These structures are shaped like multiple cone-shaped lobes, each containing a portion of the renal cortex surrounding a medullary body called a renal pyramid. Between the renal pyramids are cortical projections called renal columns. The nephron is the functional structure of the kidney that produces urine, spanning both the cortex and medulla. The initial filtration portion of the nephron is the renal corpuscle located in the cortex. Then come the renal tubules, which extend from the cortex into the medullary pyramids. As part of the renal cortex, the medullary rays are collections of renal tubules that drain into individual collecting ducts.

[0113] The apex or papilla of each cone empties urine into the corresponding calyx; calyxes enter calyces, calyces enter the renal pelvis, and then transition into the ureter. At the umbilicus, the ureter and renal vein leave the kidney, and the renal artery enters. Hilar fat and lymphatic tissue with lymph nodes surround these structures. The hilar fat is adjacent to a fat-filled cavity called the renal sinus. Together, the renal sinus contains the renal pelvis and calyces, separating these structures from the renal medulla.

[0114] Figures 3 to 5 Various features of the anatomical structures of patient 130 are shown. For example, patient 130 includes a kidney 310 fluidly connected to a bladder 330 via a ureter 320, and a urethra 340 fluidly connected to the bladder 330. As shown in the enlarged view of kidney 310(A), kidney 310(A) includes calyces (including calyces 312), renal papillae (including renal papillae 314, also referred to as "papillary 314"), and renal pyramids (including renal pyramids 316). In these examples, a kidney stone 318 is located near papillae 314. However, kidney stone 318 may be located in other locations within kidney 310(A) or elsewhere.

[0115] like Figure 3 As shown, in order to remove kidney stone 318 in an exemplary percutaneous procedure, physician 160 may position robotic system 110 on the side / bottom of worktable 150 to begin inserting endoscope 120 ( Figure 3(Not shown) is delivered to patient 130. Specifically, the robotic system 110 may be positioned on one side of the workbench 150, near the feet of patient 130, and aligned for direct linear access into the urethra 340 of patient 130. In this example, the hip of patient 130 is used as a reference point for positioning the robotic system 110. Once positioned, one or more robotic arms 112 (such as robotic arms 112(B) and 112(C)) may extend outward to reach between the legs of patient 130. For example, robotic arm 112(B) may be controlled to extend and provide a linear pathway to the urethra 340, as... Figure 3 As shown. In this example, physician 160 inserts medical device 350 at least partially into urethra 340 along this direct linear access path (sometimes referred to as a “virtual track”). Medical device 350 may include an endoscope 130 configured to receive endoscope 120, thereby assisting in the insertion of endoscope 120 into the anatomy of patient 130. Friction and / or forces on sensitive anatomy in this area can be reduced by aligning robotic arm 112(B) with patient 130’s urethra 340 and / or using medical device 350. Although medical device 350 in Figure 3 As shown, but in some embodiments, medical device 350 is not used (e.g., speculum 120 can be directly inserted into urethra 340).

[0116] The physician 160 may also position the robotic arm 112(A) near the treatment site of the surgery. For example, the robotic arm 112(A) may be positioned near the incision site of the patient 130 and / or near the kidney 310. The robotic arm 112(A) may be connected to an EM field generator 180 to help track the position of the endoscope 120 and / or needle 170 during the procedure. Although the robotic arm 112(A) is positioned relatively close to the patient 130, in some embodiments, the robotic arm 112(A) is positioned elsewhere and / or the EM field generator 180 is integrated into the workbench 150 (which allows the robotic arm 112(A) to be in a docked position). In this example, at this point in the procedure, the robotic arm 112(A) remains in the docked position, as... Figure 3 As shown. However, in some embodiments, the robotic arm 112(C) can be used to perform any of the functions described above for the robotic arms 112(A) and / or 112(C).

[0117] Once the robotic system 110 is properly positioned and / or the medical device 350 is at least partially inserted into the urethra 340, the endoscope 120 may be inserted into the patient 130 robotically, manually, or in a combination thereof, as follows: Figure 4As shown. For example, physician 160 may connect endoscope 120 to robotic arm 112(C) and / or position endoscope 120 at least partially within medical device 350 and / or patient 130. Endoscope 120 may be connected to robotic arm 112(C) at any time, such as before or during a procedure (e.g., after positioning robotic system 110). Physician 160 may then interact with control system 140 (such as I / O device 146) to navigate endoscope 120 within patient 130. For example, physician 160 may provide input via I / O device 146 to control robotic arm 112(C) to navigate endoscope 120 through urethra 340, bladder 330, ureter 320(A), and down to kidney 310(A).

[0118] As shown in the figure, the control system 140 can display the instrument alignment interface 410 (such as...) via the display 142. Figure 2 The instrument alignment interface 200 is used to view real-time images 412 captured by the endoscope 120, thereby assisting the physician 160 in controlling the endoscope 120. The physician 160 can navigate the endoscope 120 to locate the kidney stone 318, as shown in image 412. In some embodiments, the control system 140 may use positioning technology to determine the position and / or orientation of the endoscope 120, which can be determined by the physician 160 via a display 142 ( Figure 4 The endoscope 120 can be viewed (as shown on the display 142, which also helps in controlling the endoscope 120). In addition, in some embodiments, other types of information, such as X-ray images of the internal anatomy of the patient 130, can be presented on the display 142 to help the physician 160 control the endoscope 120.

[0119] When locating kidney stone 318, physician 160 may identify the position of needle 170 entering kidney 310(A) to ultimately remove kidney stone 318. For example, to minimize bleeding and / or avoid impact on kidney 310(A) and / or blood vessels or other undesirable anatomical structures surrounding kidney 310(A), physician 160 may attempt to align needle 170 with the axis of the calyx (e.g., attempt to reach the calyx head-on through the center of the calyx). To this end, physician 160 may identify the papilla as a target location. In this example, physician 160 uses endoscope 120 to locate papilla 314 near kidney stone 318 and designates papilla 314 as the target location. In some embodiments where papilla 314 is designated as the target location, physician 160 may navigate endoscope 120 to access papilla 314, control system 140 may use positioning techniques to determine the position of endoscope 120 (e.g., the position of the end of endoscope 120), and control system 140 may associate the position of endoscope 120 with the target location. In other embodiments, the physician 160 may navigate the endoscope 120 to a specific distance from the nipple 314 (e.g., parking it in front of the nipple 314) and provide input indicating that the target location is within the field of view of the endoscope 120. The control system 140 may perform image analysis and / or other positioning techniques to determine the location of the target location. In yet another embodiment, the endoscope 120 may transmit a reference to mark the nipple 314 as the target location.

[0120] like Figure 5 As shown, physician 160 can continue the procedure by inserting the positioning needle 170 to the target location. In some embodiments, physician 160 may use his or her best judgment to place the needle 170 at the incision site on patient 130, such as based on knowledge of patient 130's anatomy, experience from previous procedures, analysis of patient 130's CT / X-ray images or other preoperative information, etc. Furthermore, in some embodiments, control system 140 may provide information on the placement of the needle 170 on patient 130. Physician 160 may attempt to avoid key anatomical structures of patient 130, such as the lungs, pleura, colon, paraspinal muscles, ribs, intercostal nerves, etc. In some examples, control system 140 may use CT / X-ray / ultrasound images to provide information on the placement of the needle 170 on patient 130.

[0121] In any case, the control system 140 may determine a target trajectory 502 for inserting the needle 170 to assist the physician 160 in reaching a target location (i.e., nipple 314). The target trajectory 502 may represent a desired path for accessing the target location. The target trajectory 502 may be determined based on the location of a medical device (e.g., needle 170, endoscope 120, etc.), the target location within a human anatomy, the patient's location and / or orientation, the patient's anatomy (e.g., the location of organs within the patient relative to the target location), etc. In this example, the target trajectory 502 includes a straight line passing through the nipple 314 and the needle 170 (e.g., a point on the axis of the nipple 314 extending from the tip of the needle 170 through the nipple 314). However, the target trajectory 502 may take other forms (such as curves) and / or may be defined in other ways. In some examples, the needle 170 is implemented as a flexible beveled tip needle configured to bend when the needle 170 is inserted in a straight line. Such a needle can be used to deflect around a specific anatomical structure (such as a rib or other anatomical structure). Here, the control system 140 can provide information to guide the user, such as compensating for deviations in the needle trajectory or keeping the user on the target trajectory.

[0122] although Figure 5 The example shows a target trajectory 502 extending coaxially through the nipple 314, but the target trajectory 502 may have another location, angle, and / or form. For example, the target trajectory can be achieved using a lower pole entry point, such as through... Figure 5 The nipple shown is located below the kidney stone 318 and has a non-coaxial angle, which can be used to avoid the hip.

[0123] The control system 140 can use the target trajectory 502 to provide an alignment progress visualization 504 via the instrument alignment interface 410. For example, the alignment progress visualization 504 may include an instrument alignment element 506 that indicates the orientation of the needle 170 relative to the target trajectory 502. The physician 160 can view the alignment progress visualization 504 and orient the needle 170 to the appropriate orientation (i.e., the target trajectory 502). When aligned, the physician 160 may insert the needle 170 into the patient 130 to reach the target location. The alignment progress visualization 504 may provide a progress visualization 508 (also referred to as a "progress bar 508") that indicates the proximity of the needle 170 to the target location. Therefore, the instrument alignment interface 410 assists the physician 160 in aligning the needle 170 and / or inserting the needle to reach the target location.

[0124] Once the needle 170 reaches the target location, the physician 160 may insert another medical instrument (e.g., a powered catheter, vacuum, nephroscope, etc.) into and / or above the path formed by the needle 170. The physician 160 may use other medical instruments and / or a speculum 120 to crush and remove fragments of the kidney stone 318 from the kidney 310(A).

[0125] In some embodiments, the position of the medical device can be represented by a point / set of points, and / or the orientation of the medical device can be represented as an angle / offset relative to an axis / plane. For example, the position of the medical device can be represented by the coordinates of a point / set of points in a coordinate system (e.g., one or more X, Y, Z coordinates), and / or the orientation of the medical device can be represented by an angle relative to an axis / plane of the coordinate system (e.g., an angle relative to the X-axis / plane, Y-axis / plane, and / or Z-axis / plane). Here, a change in the orientation of the medical device can correspond to a change in the angle of the medical device relative to an axis / plane. Furthermore, in some embodiments, the orientation of the medical device is represented by yaw, pitch, and / or roll information.

[0126] In some implementations, a trajectory refers to an orientation. For example, the trajectory of a medical device may refer to the orientation of the medical device, including / indicating both the position and orientation of the medical device. Similarly, a target trajectory may refer to a target orientation, including / indicating the position and orientation of the desired path. However, in other implementations, a trajectory refers to orientation or position.

[0127] Despite the specific robotic arm of Robot System 110 in Figures 3 to 5 While shown in the context of performing a specific function, any robotic arm 112 can be used to perform these functions. Furthermore, any additional robotic arms and / or systems can be used to perform the procedure. Additionally, the robotic system 110 can be used to perform other parts of the procedure. For example, the robotic system 110 can be controlled to align the needle and / or insert the needle into the patient 130. For illustration, one of the robotic arms 112 can engage with and / or control the needle to position the needle 170 in place, align the needle 170 with a target trajectory, and / or insert the needle 170 into a target location. The control system 140 can use positioning techniques to perform this process. Therefore, in some embodiments, the medical system 100 can be used to perform a percutaneous procedure, either fully or partially (e.g., with or without the assistance of a physician 160).

[0128] Exemplary Device Visualization Section

[0129] Figures 6-1 to 6-11An exemplary interface, according to one or more embodiments, is shown for providing information about the alignment and / or progress of a medical device during a procedure. The exemplary interface is shown in the context of removing a kidney stone 662 from a patient 130 using medical system 100. Specifically, a visualization may be provided to assist physician 160 in inserting a needle 170 into the patient 130 to remove the kidney stone 662. However, the visualization may also be displayed for use with other medical systems and / or to perform other medical procedures. For ease of illustration, in Figures 6-1 to 6-11 Some features of the interface are not shown. For example, Figures 6-2 to 6-11 Alignment mark 634(B) is not shown.

[0130] Figure 6-1 An exemplary instrument alignment interface 600 is shown, which has a visualization section to assist physician 160 in aligning needle 170 with a target trajectory 670. As shown, the instrument alignment interface 600 may include: a endoscopic segment 610 to provide an image 612 captured by an endoscope 120 located within a kidney 660 of patient 130; and an alignment segment 620 to provide information about the orientation and / or position of needle 170. Here, image 612 depicts the internal portion of kidney 660 and a kidney stone 662 located within kidney 660. Alignment segment 620 includes an alignment progress visualization section 630 to indicate the alignment of needle 170 with target trajectory 670 and / or the proximity of needle 170 to target location 664 (e.g., position on a nipple). As shown, the alignment progress visualization 630 includes: an instrument alignment element 632 indicating the orientation of the needle 170 relative to the target trajectory 670; an alignment mark 634 indicating the target trajectory 670; and a progress bar 636 (also referred to as "progress indicator 636") indicating the proximity of the needle 170 to the target position 664. The instrument alignment interface 600 may also include: a navigation indicator 640 for navigation between different visualizations associated with different stages / steps of the procedure; and / or a visual indicator 650 for accessing menus and / or other options. Although the instrument alignment element 632 and alignment mark 634 are shown in a specific shape and size, they may have other shapes and / or sizes.

[0131] exist Figure 6-1In the example, the physician positions needle 170 on patient 130 and attempts to align needle 170 with target trajectory 670 using instrument alignment interface 600. Specifically, physician 160 may use one or more hands 680 to hold needle 170 and adjust its orientation (e.g., tilt) while viewing instrument alignment interface 600 via control system 140. Here, the orientation of needle 170 is misaligned with target trajectory 670. Therefore, instrument alignment interface 600 shows instrument alignment element 632 misaligned with center alignment mark 634(A) (e.g., instrument alignment element 632 is not within center alignment mark 634(A)).

[0132] In some implementations, the instrument alignment element 632 can move within the area of ​​the boundary alignment mark 634(C) (e.g., within the limits of the boundary alignment mark 634(C)). As the needle 170 becomes less aligned with the target trajectory 670, the instrument alignment element 632 can move closer to the boundary alignment mark 634(C), and as the needle 170 becomes more aligned with the target trajectory 670, the instrument alignment element can move closer to the center alignment mark 634(A). Figure 6-1 In the example, the instrument alignment interface 600 also provides the text "Tilt the needle to move the bubble to the center," indicating that the needle 170 is misaligned with the target trajectory 670. The visualization of the instrument alignment interface 600 helps the physician 160 tilt the needle 170 to align the needle 170 with the appropriate orientation so that the needle 170 can be inserted into the target position 664.

[0133] In some implementations, if needle 170 is substantially misaligned with target trajectory 670, instrument alignment element 632 can provide an indication of this misalignment configuration, such as... Figure 6-2 As shown. For example, if needle 170 is misaligned from target trajectory 670 by more than a threshold amount (e.g., misalignment threshold), progress bar 636 can be highlighted, outlined, and / or partially / fully filled with a specific color / fill pattern to provide an indication of this misalignment, as shown. For illustration, progress bar 636 can be filled with red (e.g., a closed red ring). In addition or alternatively, in some embodiments, instrument alignment element 632 may be shown as contacting boundary marker 634(C) with a deformed shape, such as Figure 6-2As shown. Here, as the instrument alignment element 632 moves within proximity to the boundary mark 634(C), the instrument alignment element 632 can be displayed in its initial circular form, and as the needle 170 moves further out of alignment and exceeds the misalignment threshold, the instrument alignment element transforms into a deformed shape. This transformation visualization may resemble a bubble in contact with a surface in liquid. Furthermore, in some embodiments, text or another indication may be provided within the instrument alignment interface 600 to indicate that the needle 170 is misaligned from the target trajectory by a threshold amount. In any case, this misalignment indication helps the physician 160 observe that the needle 170 is substantially off-axis from the target trajectory 670. Although the progress bar 636 is... Figure 6-2 The progress bar 636 is shown in the diagram with specific highlighting, outlining, and / or fill patterns to provide a substantially misalignment indication, but in some embodiments, the progress bar 636 may be implemented without such variations. Here, the instrument alignment element 632 may be shown with a deformed shape to provide a substantially misalignment indication.

[0134] When needle 170 is aligned with target trajectory 670, the instrument alignment element 632 can be displayed in a manner that aligns with alignment mark 634, such as... Figure 6-3 As shown. For example, instrument alignment element 632 may be displayed within and / or concentric with the center alignment mark 634(A). Alternatively, the center alignment mark 634(A) may be highlighted (e.g., using glow visualization, a specific color, etc.) to indicate that the needle 170 is aligned with the target trajectory 670. Furthermore, the instrument alignment interface 600 may display text to indicate alignment, such as the text "alignment," etc. Figure 6-3 As shown. Although Figure 6-3 The alignment mark 634(A) and text are highlighted, but in some embodiments only one of these visualizations is shown. Alternatively, other visualizations may be used to indicate this alignment.

[0135] In this example, when needle 170 is aligned with target trajectory 670, physician 160 inserts needle 170, as... Figures 6-4 to 6-6 As shown. Here, progress bar 636 provides an indication of the proximity (e.g., distance) of needle 170 relative to target position 664. Specifically, progress bar 636 may fill clockwise around boundary mark 634(C). Control system 140 can determine the proximity of needle 170 to target position 664 by tracking the position of needle 170 and / or the position of target position 664 / viewer 120 using positioning technology.

[0136] In some implementations, the amount of movement of the instrument alignment element 632 can vary (e.g., the sensitivity of the instrument alignment element 632 can vary) as the needle 170 moves closer to the target position 664. For example, when the needle 170 is relatively far from the target position 664 (e.g., beyond the distance to the target position 664), the control system 140 can initially set a position change parameter of the instrument alignment element 632 to a first value. The position change parameter indicates the amount of position change of the instrument alignment element 632 relative to a unit of movement of the needle 170. As the needle 170 moves closer to the target position 664, the position change parameter can be updated to a second value, such as a value associated with a position change of the same unit of movement of the needle 170 that is larger or smaller than the first value.

[0137] In one illustration of updating the position change parameter, the position change parameter can be set to an initial value when the needle 170 is positioned on the patient's skin 130. The initial value causes the instrument alignment element 632 to move a first number of pixels in response to, for example, a 5-degree change in the orientation of the needle 170. As the needle 170 moves closer to the target location 664, the position change parameter can be updated to a larger value, causing the instrument alignment element 632 to move a second number of pixels in response to a 5-degree change in the orientation of the needle 170, where the second number of pixels is greater than the first number of pixels. The position change parameter can be updated any number of times as the needle 170 moves closer to the target location 664. In some embodiments, this can help a physician align the needle 170 to reach relatively small targets, such as cups with a diameter of 4 mm to 8 mm.

[0138] When needle 170 has reached the target position 664, the instrument alignment interface 600 can display an indication that the target position 664 has been reached, such as... Figure 6-7 As shown. For example, progress bar 636 may be completely filled around the perimeter of boundary marker 634(C). In some embodiments, progress bar 636 may be highlighted, outlined, and / or partially / fully filled with a specific color / fill pattern to indicate that target position 664 has been reached. For example, progress bar 636 may be filled with green (e.g., a closed green ring). In addition or alternatively, instrument alignment interface 600 may provide text indicating that target position 664 has been reached, such as providing the text "Target reached," also as shown. In some embodiments, such as Figure 6-7 As shown, image 612 depicting the internal portion of kidney 660 also provides visual confirmation that needle 170 has reached target location 664.

[0139] In some implementations, if the needle 170 is inserted outside the target position 664, the instrument alignment interface 600 can provide an indication that the needle 170 is inserted outside the target position 664, such as... Figure 6-8As shown. For example, the progress bar 636 can be highlighted, outlined, and / or partially / fully filled with a specific color / fill pattern to indicate that the needle 170 is inserted outside the target position 664. For illustration, the progress bar 636 can be filled with red (e.g., a closed red ring) and / or in cases where it is substantially misaligned with the needle 170 (e.g., ...). Figure 6-2 (The progress bar is filled with different colors depending on the situation.) In addition or alternatively, the instrument alignment interface 600 may provide text indicating that the needle 170 has been inserted outside the target position 664, such as providing the text "Inserted outside the target. Please retract." In some embodiments, the control system 140 may determine that the needle 170 has been inserted outside the target position 664 when the needle 170 is beyond a threshold distance outside the target position 664 and / or when the needle 170 is within a specific distance from the endoscope 120.

[0140] In some implementation schemes, such as Figure 6-8 As shown, image 612, depicting the internal portion of kidney 660, also provides visual confirmation that needle 170 has been inserted beyond target location 664. In other embodiments, the field of view of endoscope 120 may not include the location where needle 170 enters kidney 660 (e.g., needle 170 may be inserted above or below the field of view of endoscope 120). Here, progress bar 636 can be particularly helpful in notifying physician 160 that needle 170 has been inserted beyond target location 664.

[0141] In some procedures, once the needle 170 has reached the target position 664, the medical device 638 can be inserted over and / or replace the needle 170, such as... Figure 6-9 As shown. Medical device 638 may include means for assisting in the removal of kidney stone 662 from kidney 660. For example, medical device 638 may include a catheter (e.g., a powered catheter), a vacuum tube, a nephroscope, or any other medical device. In some embodiments, one or more dilating instruments (e.g., wires, tubes, sheaths, etc.) may be used to dilate the path to target location 664 to provide sufficient space for insertion of medical device 638.

[0142] Medical device 638 and / or endoscope 120 (and / or needle 170, in some cases) facilitate the removal of kidney stone 662 from kidney 660. For example, endoscope 120 may deploy tools (e.g., laser, cutting instruments, etc.) to cut kidney stone 662 into fragments, and medical device 638 may aspirate the fragments from kidney 660, such as... Figure 6-10 As shown. In some embodiments, the endoscope 120 (and / or medical device 638) can provide flushing to help remove debris from the kidney 660. Figure 6-10 In the example, image 612 provides visual confirmation that kidney stone 662 is being removed from kidney 660.

[0143] In some implementations, when returning to the alignment of the needle 170 on the patient's skin (e.g., Figure 6-2 If needle 170 is inserted when it is substantially misaligned with the target trajectory 670, the instrument alignment interface 600 can provide an indication to retract needle 170, such as... Figure 6-11 As shown. For example, the control system 140 can determine that the needle is misaligned with the target trajectory 670 by more than a threshold amount (similar to a reference). Figure 6-2 (As discussed). Furthermore, when needle 170 is substantially misaligned with target trajectory 670, control system 140 can determine that needle 170 has been inserted into patient 130 beyond a specific distance. In some embodiments, progress bar 636 can be highlighted, outlined, and / or partially / fully filled with a specific color / fill pattern to indicate that needle 170 has been inserted and is substantially misaligned. For illustration, progress bar 636 can be filled with red (e.g., a closed red ring) and / or in cases where needle 170 is exactly substantially misaligned (e.g., ...). Figure 6-2 (The progress bar is filled with different colors depending on the situation.) In addition, or alternatively, in some embodiments, the instrument alignment interface 600 may provide text indicating that the needle 170 is substantially misaligned and needs to be retracted, such as providing the text "Retract and reinsert the needle in the proper orientation." In some embodiments, when it is determined that the needle 170 cannot be adjusted to reach the target position 664, a message may be displayed. Figure 6-11 Instructions.

[0144] although Figures 6-1 to 6-11 While specific indicators are provided for alignment and progress information, other indicators, including auditory, visual, and tactile, may also be available. For example, control system 140 may provide audible and / or tactile feedback via I / O devices associated with control system 140 to indicate the alignment and / or progress of needle 170 (e.g., a first sound when needle 170 is aligned with target trajectory 670, a second sound when needle 170 is initially inserted, a third sound when needle 170 is midway along target trajectory 664, a third sound when needle 170 has reached target position 664, etc.). Furthermore, any of the indicators discussed may be shown and / or presented in different forms (e.g., different shapes, sizes, colors, etc.) at different locations within instrument alignment interface 600.

[0145] In some embodiments, the progress bar 636 may comprise a straight progress bar instead of a circular bar shown around the boundary marker 634, which may be positioned anywhere within the instrument alignment interface 600. Furthermore, in some embodiments, instead of filling the progress bar 636 to indicate the proximity of the needle 170 to the target position 664, an icon may be used on the progress bar 636 to display the current position of the needle 170 (e.g., the icon is at the top, indicating that the needle 170 has not yet been inserted into the patient 130 and / or has reached the target position 664). Additionally, in some embodiments, the percentage of progress to the target position 664 may be presented via the instrument alignment interface 600.

[0146] Furthermore, in some embodiments, the size of the center alignment mark 634(A), boundary mark 634(C), and / or instrument alignment element 632 may vary to indicate the progress of inserting the needle 170 into the target position 664. For example, the diameter of the center alignment mark 634(A) may decrease as the needle 170 is inserted until the center alignment mark 634(A) reaches the same diameter as the instrument alignment element 632.

[0147] Exemplary flowchart

[0148] Figures 7 to 10 An exemplary flowchart is shown for performing one or more of the techniques discussed herein. Various operations associated with these processes may be performed by control circuitry or combinations thereof implemented in any device / system discussed herein, such as control system 140, robot system 110, workbench 150, EM field generator 180, peephole 120, and / or needle 170.

[0149] Figure 7 An exemplary flowchart of a process 700 for determining the alignment of a medical device relative to a target trajectory and presenting information about the alignment of the medical device with the target trajectory, according to one or more embodiments, is shown. At block 702, process 700 may include receiving sensor data from one or more medical devices. For example, depending on certain use cases, control circuitry of the device / system (such as a control system) may receive sensor data from one or more medical devices (such as a hysteroscope, needle, or any other medical device) via a communication interface. The sensor data may indicate the position and / or orientation of one or more medical devices.

[0150] At block 704, process 700 may include determining a target location within the human anatomy. For example, depending on certain use cases, control circuitry may determine a target location within a patient's body, such as an anatomical landmark, the location of a medical device, or any other location / target. In some embodiments, control circuitry may determine the target location based at least in part on sensor data from a medical device at least partially disposed within the patient's body.

[0151] At block 706, process 700 may include determining the position and / or orientation of one or more medical devices. For example, depending on certain use cases, control circuitry may determine the position and / or orientation of one or more medical devices based at least in part on sensor data from one or more medical devices. In some embodiments, control circuitry may use one or more positioning techniques to determine the position and / or orientation of one or more medical devices.

[0152] At block 708, process 700 may include determining a target trajectory for entry into a target location. For example, depending on certain use cases, control circuitry may determine a target trajectory for percutaneous entry into a target location within a patient. In some embodiments, control circuitry may determine the target trajectory based at least in part on sensor data from a medical device at least partially disposed within the patient, sensor data from a medical device located outside the patient (or partially inserted), the location of the target location, etc. Alternatively or additionally, the target trajectory may be determined based on input provided by a user through an interface to specify the target trajectory. In the example, the target trajectory may be defined relative to one or more anatomical planes / axis lines.

[0153] At block 710, process 700 may include generating user interface data representing an interface that includes device alignment elements indicating the alignment of the medical device's orientation with a target trajectory. For example, depending on certain use cases, control circuitry may generate user interface data representing an interface (e.g., a device alignment interface) that includes device alignment elements indicating the orientation of the medical device, such as needle alignment icons indicating needle orientation. In some embodiments, the positioning of the device alignment elements within the interface may indicate the alignment of the medical device's orientation with a target trajectory.

[0154] At box 712, process 700 may include causing the interface to be displayed. For example, depending on certain use cases, the control circuitry may display the interface via a display device, such as by sending user interface data to a display device associated with the control system. Furthermore, depending on certain use cases, the display device may display the interface at least in part based on the user interface data. In any case, the interface may include device alignment elements indicating the orientation of the medical device.

[0155] At block 714, process 700 may include updating the position of the instrument alignment element based at least in part on a change in the orientation of the medical device. For example, depending on certain use cases, control circuitry may determine a change in the orientation of the medical device and update the position of the instrument alignment element associated with the medical device based at least in part on the change in the orientation of the medical device.

[0156] In some embodiments of block 714, the control circuitry may update the position of the instrument alignment element based at least in part on the proximity of the medical device to the target location. For example, in response to determining that the orientation of the medical device has changed by a unit of measurement and that the medical device is outside a predetermined proximity to the target location, the control circuitry may update the position of the instrument alignment element within the interface by a first amount. Furthermore, in response to determining that the orientation of the medical device has changed by that unit of measurement and that the medical device is within a predetermined proximity to the target location, the control circuitry may update the position of the instrument alignment element within the interface by a second amount.

[0157] Figure 8 An exemplary flowchart of a process 800 for presenting information about the orientation of a medical device, according to one or more embodiments, is shown. At block 802, process 800 may include determining the orientation of the medical device. For example, depending on certain use cases, control circuitry may determine the orientation of a medical device configured for percutaneous insertion into a human anatomy based at least in part on sensor data from the medical device. In some embodiments, control circuitry may use one or more positioning techniques to determine the orientation of the medical device.

[0158] At block 804, process 800 may include determining whether the orientation of the medical device is aligned with a target trajectory. For example, depending on certain use cases, control circuitry may determine whether the orientation of the medical device is aligned with a target trajectory determined for percutaneous access to a target location based at least in part on sensor data of the medical device. In some embodiments, control circuitry may compare one or more coordinates and / or angles of the medical device's orientation with one or more coordinates and / or angles of the target trajectory and determine whether one or more thresholds are met (e.g., one or more coordinates and / or angles of the medical device's orientation are within a specific number of coordinates and / or degrees of one or more coordinates and / or angles of the target trajectory). In the example, alignment may be determined relative to positional and / or angular errors (e.g., X, Y, Z, yaw, pitch, roll) and / or relative to any coordinate system.

[0159] If it is determined that the orientation of the medical device is aligned with the target trajectory, process 800 can proceed to box 806. Conversely, if it is determined that the orientation of the medical device is not aligned with the target trajectory, process 800 can proceed to box 808.

[0160] At block 806, process 800 may include causing an indication of alignment of the medical device with a target trajectory to be displayed. For example, depending on certain use cases, control circuitry may cause an indication of alignment of the medical device with a target trajectory to be displayed within an interface, such as by sending data to a display device associated with a control system. Furthermore, depending on certain use cases, the display device may display an indication of alignment of the medical device with a target trajectory within an interface. In some embodiments, the device alignment element is displayed in an alignment arrangement having one or more alignment marks (e.g., centered on the marks) to indicate alignment of the medical device with a target trajectory.

[0161] At block 808, process 800 may include determining whether the orientation of the medical device is misaligned with the target trajectory by more than a threshold amount. For example, depending on certain use cases, the control circuitry may determine whether the orientation of the medical device is misaligned with the target trajectory by more than a threshold amount based at least in part on sensor data of the medical device. In some embodiments, the control circuitry may compare one or more coordinates and / or angles of the medical device's orientation with one or more coordinates and / or angles of the target trajectory.

[0162] If it is determined that the orientation of the medical device is misaligned from the target trajectory by more than a threshold amount, process 800 may proceed to box 810. In contrast, if it is determined that the orientation of the medical device is not misaligned from the target trajectory by more than a threshold amount, process 800 may proceed to box 812.

[0163] At block 810, process 800 may include causing the instrument alignment element to be displayed at the boundary mark and / or in a deformed form. For example, depending on certain use cases, control circuitry may cause the instrument alignment element to be displayed within a predetermined proximity of the boundary mark and / or in a deformed shape, such as by sending data to a display device associated with the control system. Furthermore, depending on certain use cases, the display device may display the instrument alignment element within a predetermined proximity of the boundary mark and / or in a deformed shape within the interface.

[0164] At block 812, process 800 may include an instrument alignment element that causes misalignment of the display position. For example, depending on certain use cases, control circuitry may cause the instrument alignment element to be displayed at a position not aligned with alignment marks, such as by sending data to a display device associated with a control system. Furthermore, depending on certain use cases, the display device may display the instrument alignment element at a position not aligned with alignment marks.

[0165] At block 814, process 800 may include determining whether a medical device is inserted into a human anatomical structure. For example, depending on certain use cases, control circuitry may determine whether the medical device is at least partially disposed within the patient's body based at least in part on sensor data from the medical device and / or information about the patient's position and / or orientation. In some embodiments, control circuitry may determine whether the medical device is inserted into the patient's body in a specific amount.

[0166] If it is determined that the medical device is inserted into a human anatomical structure, process 800 may proceed to box 816. Conversely, if it is determined that the medical device is not inserted into a human anatomical structure, process 800 may proceed to box 818.

[0167] At block 816, process 800 may include causing an instruction to retract the medical device to be displayed. For example, depending on certain use cases, the control circuitry may cause the instruction to retract the medical device to be displayed, such as by sending data to a display device associated with the control system. Furthermore, depending on certain use cases, the display device may display the instruction to retract the medical device. In some embodiments, the control circuitry may maintain the display of information associated with blocks 810 and / or 812 (e.g., device alignment elements) and provide the instruction to retract the medical device.

[0168] At box 818, process 800 may include maintaining the display of information. For example, depending on certain use cases, the control circuitry may maintain the display of text or other visual elements regarding the current orientation and / or location of the medical device (e.g., information presented at boxes 810 and / or 812). Although box 818 is shown, in some embodiments, another operation or process may be performed.

[0169] Figure 9 An exemplary flowchart of a process 900 for presenting information about the proximity of a medical device to a target location, according to one or more embodiments, is shown. At block 902, process 900 may include determining the proximity of the medical device to the target location. For example, depending on certain use cases, control circuitry may determine the proximity of the medical device to the target location within the patient's body based at least in part on sensor data from the medical device. In some embodiments, control circuitry may use one or more positioning techniques to determine the location of the medical device.

[0170] At block 904, process 900 may include causing an indication of the proximity of the medical device to a target location to be displayed. For example, depending on certain use cases, control circuitry may cause an indication of the proximity of the medical device to a target location to be displayed within an interface, such as by sending data to a display device associated with a control system. Furthermore, depending on certain use cases, the display device may display an indication of the medical device approaching a target location within the interface.

[0171] At block 906, process 900 may include determining whether the medical device has reached the target location. For example, depending on certain use cases, the control circuitry may determine whether the medical device has reached the target location inside the patient's body based at least in part on sensor data from the medical device.

[0172] If it is determined that the medical device has reached the target position, process 900 can proceed to box 908. Conversely, if it is determined that the medical device has not reached the target position, process 900 can return to box 902.

[0173] At box 908, process 900 may include causing an indication that the medical device has reached a target location to be displayed. For example, depending on certain use cases, the control circuitry may cause an indication that the medical device has reached a target location to be displayed on the interface, such as by sending data to a display device associated with the control system. Furthermore, depending on certain use cases, the display device may display an indication that the medical device has reached a target location on the interface.

[0174] At block 910, process 900 may include determining whether a medical device has been inserted outside a target location. For example, depending on certain use cases, control circuitry may determine whether a medical device has been inserted outside a target location based at least in part on sensor data from the medical device.

[0175] If it is determined that the medical device is inserted outside the target location, process 900 may proceed to box 912. Conversely, if it is determined that the medical device is not inserted outside the target location, process 900 may return to box 902. Although in Figure 9 In the example, process 900 is shown to return to box 902, but in some implementations, process 900 may return to box 906, box 908 or another box.

[0176] At block 912, process 900 may include causing an indication to be displayed that the medical device has been inserted outside the target position. For example, depending on certain use cases, the control circuitry may cause an indication to be displayed within the interface that the medical device has been inserted outside the target position, such as by sending data to a display device associated with the control system. Furthermore, depending on certain use cases, the display device may display an indication within the interface that the medical device has been inserted outside the target position. Process 900 may then return to block 902.

[0177] Figure 10An exemplary flowchart of a process 1000 for setting and / or updating a positional change parameter associated with an instrument alignment element, according to one or more embodiments, is shown. At block 1002, process 1000 may include setting a positional change parameter associated with a movement unit of the medical device. For example, depending on certain use cases, control circuitry may set the positional change parameter to an initial value that indicates a specific amount of positional change of the instrument alignment element. The positional change parameter may indicate the amount of positional change of an instrument alignment element within an interface relative to a movement unit (e.g., an orientation change unit) of the medical device. In some embodiments, the initial value includes a predetermined or default value associated with the medical device located outside the patient's body and / or at a predetermined proximity to a target location. For example, the positional change parameter may be set to an initial value when aligning the medical device before inserting it into the patient's body.

[0178] At block 1004, process 1000 may include using a position change parameter to change the position of an instrument alignment element based at least in part on a change in the orientation of the medical device. For example, depending on certain use cases, control circuitry may determine that the orientation of the medical device has changed, and in response, use the position change parameter to change the position of the instrument alignment element (e.g., use the value of the position change parameter to identify the amount of position change applied to the instrument alignment element).

[0179] In block 1006, process 1000 may include determining whether a medical device is closer to a target location. For example, depending on certain use cases, control circuitry may determine whether the medical device is closer to the target location compared to the last location of the medical device at the control circuitry. This determination may be based at least in part on sensor data from the medical device. In some embodiments, control circuitry may determine whether the medical device is within a predetermined proximity to the target location.

[0180] If it is determined that the medical device is closer to the target location, process 1000 can proceed to box 1008. Conversely, if it is determined that the medical device is not closer to the target location, process 1000 can return to box 1004 and continue using the previously set position change parameters.

[0181] At block 1008, process 1000 may include updating the position change parameter. For example, depending on certain use cases, the control circuitry may update the position change parameter to another value associated with a more or less position change for a unit of movement of the medical device. In some embodiments, block 1008 may be implemented any number of times to update the position change parameter once or more as the medical device moves closer to the target location. Furthermore, in some embodiments, block 1008 may be implemented once when the medical device is within a predetermined proximity to the target location. Here, process 1000 may not return to block 1004 after implementing block 1008.

[0182] Exemplary robot system

[0183] Figure 11 Exemplary details of a robot system 110 according to one or more embodiments are shown. In this example, the robot system 110 is shown as a movable cart-type robot-enabled system. However, the robot system 110 may be implemented as a fixed system, integrated into a workbench, etc.

[0184] The robot system 110 may include a support structure 114 comprising an elongated segment 114(A) (sometimes referred to as “post 114(A)”) and a base 114(B). Post 114(A) may include one or more brackets, such as bracket 1102 (alternatively referred to as “arm support 1102”), for supporting one or more robot arms 112 ( Figure 11 The diagram shows three of the components. The bracket 1102 may include individually configurable arm mounts that rotate along a vertical axis to adjust the base of the robotic arm 112 for positioning relative to the patient. The bracket 1102 also includes a bracket interface 1104 that allows vertical translation of the bracket 1102 along the post 114(A). The bracket interface 1104 is connected to the post 114(A) via slots (such as slot 1106) positioned on opposite sides of the post 114(A) to guide the vertical translation of the bracket 1102. Slot 1106 includes a vertical translation interface to position and hold the bracket 1102 at various vertical heights relative to the base 114(B). The vertical translation of the bracket 1102 allows the robotic system 110 to adjust the reach of the robotic arm 112 to accommodate various worktable heights, patient sizes, physician preferences, etc. Similarly, the individually configurable arm mounts on the bracket 1102 allow the robot arm base 1108 of the robot arm 112 to be configured at various angles. The column 114(A) may internally include mechanisms (such as gears and / or motors) designed to mechanically translate the bracket 1102 in response to control signals generated in response to user input (such as input from I / O device 116).

[0185] In some embodiments, slot 1106 may be supplemented with a slot cover flush with and / or parallel to the slot surface to prevent dust and / or fluid from entering the internal chambers and / or vertical translation interfaces of column 114(A) as the bracket 1102 translates vertically. The slot cover can be unfolded via a pair of spring reels positioned near the vertical top and bottom of slot 1106. As the bracket 1102 translates vertically upward and downward, the cover can be wound within the reels until unfolded to extend and retract from its wound state. The spring loading of the reels provides a force to retract the cover into the reels as the bracket 1102 translates toward the reels, while maintaining a tight seal as the bracket 1102 translates away from the reels. The cover can be attached to the bracket 1102 using, for example, a bracket in bracket interface 1104 to ensure proper extension and retraction of the cover as the bracket 1102 translates.

[0186] The base 114(B) balances the weight of the column 114(A), bracket 1102, and / or arm 112 on a surface such as a floor. Therefore, the base 114(B) can accommodate heavier components, such as one or more electronic devices, motors, power supplies, etc., as well as components that enable the robot system 110 to move and / or remain stationary. For example, the base 114(B) may include rolling wheels 1116 (also referred to as “casters 1116”) that allow the robot system 110 to move within a room for procedures. Once in place, the casters 1116 can be secured using wheel locks to hold the robot system 110 in place during the procedure. As shown, the robot system 110 also includes a handle 1118 to aid in manipulating and / or stabilizing the robot system 110.

[0187] A robotic arm 112 typically includes a robotic arm base 1108 and an end effector 1110, separated by a series of links 1112 connected by a series of joints 1114. Each joint 1114 may include an independent actuator, and each actuator may include an independently controllable motor. Each independently controllable joint 1114 represents an independent degree of freedom available to the robotic arm 112. For example, each arm 112 may have seven joints, thus providing seven degrees of freedom. However, any number of joints can be implemented with any degree of freedom. In this example, multiple joints can generate multiple degrees of freedom, thus allowing “redundant” degrees of freedom. Redundant degrees of freedom allow the robotic arm 112 to position its corresponding end effector 1110 in a specific location, orientation, and / or trajectory in space using different link positions and / or joint angles. In some embodiments, the end effector 1110 may be configured to engage and / or control medical devices, apparatuses, axes, etc. The degrees of freedom of movement of arm 112 allow robotic system 110 to locate and / or guide medical devices from desired points in space, and / or allow physicians to move arm 112 to a clinically advantageous location away from the patient to create access while avoiding arm collisions.

[0188] like Figure 11 As shown, the robotic system 110 may also include an I / O device 116. The I / O device 116 may include a display, touchscreen, touchpad, projector, mouse, keyboard, microphone, speaker, controller, camera (e.g., for receiving gesture input), or another I / O device for receiving input and / or providing output. The I / O device 116 may be configured to receive touch, voice, gesture, or any other type of input. The I / O device 116 may be located at the vertical end of column 114(A) (e.g., the top of column 114(A)) and / or provide a user interface for receiving user input and / or providing output. For example, the I / O device 116 may include a touchscreen (e.g., a dual-purpose device) to receive input and provide preoperative and / or intraoperative data to the physician. Exemplary preoperative data may include preoperative planning, navigation, and / or mapping data obtained from preoperative computed tomography (CT) scans, and / or records obtained from preoperative patient interviews. Exemplary intraoperative data may include optical information from tools / instruments, sensors, and / or coordinate information from sensors, as well as important patient statistics such as respiration, heart rate, and / or pulse. The I / O device 116 may be positioned and / or tilted to allow physician access to the I / O device 116 from various locations, such as the side of column 114(A) opposite the bracket 1102. From this position, the physician can view the I / O device 116, the robotic arm 112, and / or the patient while operating the I / O device 116 from behind the robotic system 110.

[0189] The robot system 110 may include a variety of other components. For example, the robot system 110 may include one or more control electronics / circuits, a power supply, pneumatic devices, a light source, actuators (e.g., motors for moving the robot arm 112), memory, and / or a communication interface (e.g., for communicating with another device). In some embodiments, the memory may store computer-executable instructions that, when executed by the control circuitry, cause the control circuitry to perform any of the operations discussed herein. For example, the memory may store computer-executable instructions that, when executed by the control circuitry, cause the control circuitry to receive input and / or control signals regarding the manipulation of the robot arm 112, and in response, control the robot arm 112 to position and / or navigate a medical device connected to the end effector 1110 in a specific arrangement.

[0190] In some embodiments, the robotic system 110 is configured to engage and / or control a medical device, such as a speculum 120. For example, a robotic arm 112 may be configured to control the position, orientation, and / or tip joint movement of the speculum (e.g., the sheath and / or guide of the speculum). In some embodiments, the robotic arm 112 may be configured to / be configured to manipulate the speculum 120 using an elongated moving member. The elongated moving member may include one or more drawwires (e.g., draw wires or push wires), cables, fibers, and / or flexible shafts. For illustration, the robotic arm 112 may be configured to actuate multiple drawwires coupled to the speculum 120 to deflect the tip of the speculum 120. The drawwires may include any suitable or desired material, such as metallic and / or non-metallic materials, such as stainless steel, Kevlar, tungsten, carbon fiber, etc. In some embodiments, the speculum 120 is configured to exhibit non-linear behavior in response to forces applied by the elongated moving member. Nonlinear behavior can be based on the stiffness and compressibility of the peephole 120, as well as the variability of relaxation or stiffness between different slender moving components.

[0191] Exemplary control system

[0192] Figure 12 Exemplary details of a control system 140 according to one or more embodiments are shown. As shown, the control system 140 may include, individually and / or in combination / commonly, one or more of the following components, devices, modules, and / or units (referred to herein as “components”): control circuitry 1202, data storage device / memory 1204, one or more communication interfaces 1206, one or more power supply units 1208, one or more I / O components 1210, and / or one or more wheels 1212 (e.g., casters or other types of wheels). In some embodiments, the control system 140 may include a housing / enclosure configured and / or sized to accommodate or contain at least a portion of one or more components of the control system 140. In this example, the control system 140 is shown as a trolley-type system that can move with one or more wheels 1212. In some cases, after reaching a suitable position, one or more wheels 1212 may be secured using wheel locks to hold the control system 140 in place. However, the control system 140 may be implemented as a fixed system, integrated into another system / device, etc.

[0193] although Figure 12 Some components of the control system 140 are shown, but it should be understood that additional components not shown may be included in embodiments according to this disclosure. Furthermore, in some embodiments, certain components shown may be omitted. Although in Figure 12In the illustration, control circuit 1202 is shown as a separate component, but it should be understood that any or all remaining components of control system 140 may be at least partially embodied in control circuit 1202. That is, control circuit 1202 may include various devices (active and / or passive), semiconductor materials and / or areas, layers, regions and / or portions thereof, conductors, leads, vias, connectors, etc., wherein one or more other components and / or portions thereof of control system 140 may be formed and / or implemented at least partially by such circuit components / devices.

[0194] Various components of the control system 140 may be electrically and / or communicatively coupled using certain connection circuits / devices / features, which may or may not be part of the control circuitry 1202. For example, connection features may include one or more printed circuit boards configured to facilitate the mounting and / or interconnection of at least some of the various components / circuits of the control system 140. In some embodiments, two or more of the control circuitry 1202, data storage device / memory 1204, communication interface 1206, power supply unit 1208, and / or input / output (I / O) components 1210 may be electrically and / or communicatively coupled to each other.

[0195] As shown in the figure, memory 1204 may include positioning unit 1214, target / track unit 1216, and user interface unit 1218, which are configured to facilitate the various functions discussed herein. In some embodiments, positioning unit 1214, target / track unit 1216, and / or user interface unit 1218 may include one or more instructions executable by control circuitry 1202 to perform one or more operations. Although many embodiments are discussed in the context of components 1214-1218 including one or more instructions executable by control circuitry 1202, any component 1214-1218 may be at least partially implemented as one or more hardware logic components, such as one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), one or more program-specific standard products (ASSPs), one or more complex programmable logic devices (CPLDs), etc. Furthermore, although components 1214-1218 are shown as included within the control system 140, any component of 1214-1218 may be implemented at least partially within another device / system, such as robot system 110, workbench 150, or another device / system. Similarly, any other component of the control system 140 may be implemented at least partially within another device / system.

[0196] The positioning component 1214 can be configured to perform one or more positioning techniques to determine and / or track the position and / or orientation of an object (such as a medical device). For example, the positioning component 1214 can process input data (e.g., sensor data from the medical device, model data about the patient's anatomy, patient position data, preoperative data, robot commands, and / or kinematic data, etc.) to generate position / orientation data 1220 for one or more medical devices. The position / orientation data 1220 can indicate the position and / or orientation of one or more medical devices relative to a reference frame. The reference frame can be relative to the patient's anatomy, a known object (e.g., an EM field generator), a coordinate system / space, etc. In some embodiments, the position / orientation data 1220 can indicate the position and / or orientation of the distal end (and / or proximal end, in some cases) of the medical device.

[0197] In some implementations, the positioning component 1214 can process preoperative data to determine the location and / or orientation of an object. Preoperative data (sometimes referred to as “mapped data”) can be generated by performing computed tomography (CT) scans (such as low-dose CT scans). Preoperative CT images from the scans can be reconstructed into three-dimensional images that are visualized as “slices” of cross-sectional views of, for example, the patient’s internal anatomy. When aggregated analysis is performed, image-based models of the anatomical cavities, spaces, and / or structures of the patient’s anatomy (such as the patient’s lung network) can be generated. Centerline geometry can be determined and / or estimated from the CT images to form a three-dimensional volume of the patient’s anatomy, referred to as model data (also referred to as “preoperative model data” when generated using only preoperative CT scans). Exemplary use of centerline geometry is discussed in U.S. Patent Application No. 14 / 523,760, the entire contents of which are incorporated herein by reference. Network topology models can also be derived from CT images.

[0198] Furthermore, in some embodiments, the positioning component 1214 may perform vision-based techniques to determine the position and / or orientation of an object. For example, the medical device may be equipped with a camera, a distance sensor (sometimes called a "depth sensor"), radar, etc., to provide sensor data in the form of visual data. The positioning component 1214 may process the visual data to facilitate vision-based position tracking of the medical device. For example, preoperative model data may be used in conjunction with the visual data to enable computer vision-based tracking of the medical device (e.g., an endoscope). In an example, using preoperative model data, the control system 140 may generate a library of expected endoscope images based on the expected path of travel of the endoscope, where each image is linked to a position within the model. During surgery, the control system 140 may refer to this library to compare real-time images and / or other visual data captured at the endoscope (e.g., a camera at the distal end of the endoscope) with those images and / or other visual data in the library to aid in positioning.

[0199] Furthermore, in some implementations, other types of vision-based techniques can be performed to determine the position and / or orientation of an object. For example, the positioning component 1214 can use feature tracking to determine the motion of an image sensor (e.g., a camera or other sensor), and thus the motion of a medical device associated with the image sensor. In some cases, the positioning component 1214 can identify circular geometries corresponding to anatomical cavities in preoperative model data and track changes in these geometries to determine which anatomical cavity was selected, as well as the relative rotational and / or translational motion of the medical device. The use of topology maps can also enhance vision-based algorithms or techniques. Additionally, the positioning component 1214 can use optical flow (another computer vision-based technique) to analyze the displacement and / or translation of image pixels in a video sequence within visual data to infer camera movement. Examples of optical flow techniques may include motion detection, object segmentation computation, brightness, motion compensation coding, stereo parallax measurement, etc. By comparing multiple frames in multiple iterations, the positioning component 1214 can determine the movement and position of the image sensor (and the endoscope).

[0200] Furthermore, in some embodiments, the positioning component 1214 may use electromagnetic tracking to determine the position and / or orientation of an object. For example, the positioning component 1214 may use real-time EM tracking to determine the real-time position of a medical device in a coordinate system / space that can be registered to the patient's anatomy and can be represented by a preoperative model or other model. In EM tracking, an EM sensor (or tracker) including one or more sensor coils may be embedded in one or more locations and / or orientations of the medical device (e.g., endoscope, needle, etc.). The EM sensor may measure changes in the EM field generated by one or more static EM field generators positioned at a known location. The positional information detected by the EM sensor may be stored as EM data. The positioning component 1214 may process the EM data to determine the position and / or orientation of an object (such as a medical device). An EM field generator (or transmitter) may be placed near the patient (e.g., within a predetermined distance) to generate a low-intensity magnetic field detectable by the EM sensor. The magnetic field may induce a small current in the sensor coil of the EM sensor, which may be analyzed to determine the distance and / or angle between the EM sensor and the EM field generator. These distances and / or orientations can be “registered” to the patient’s anatomy during surgery (e.g., a preoperative model) to determine the geometric transformations that align individual locations in the coordinate system with their locations in the preoperative model of the patient’s anatomy. Once registered, EM sensors (e.g., embedded EM trackers) located at one or more locations of the medical device (e.g., the distal tip of an endoscope, a needle, etc.) can provide real-time indication of the position and / or orientation of the medical device as it traverses the patient’s anatomy.

[0201] In addition to or alternatively, in some embodiments, the positioning component 1214 may use robot commands and / or kinematic data to determine the position and / or orientation of the object. Robot commands and / or kinematic data may indicate pitch and / or yaw (e.g., pitch and / or yaw of a robot arm) generated by joint movement commands, such as those used during preoperative calibration and / or during procedural procedures. During surgery, calibration measurements may be used in conjunction with known insertion depth information to estimate the position and / or orientation of the medical device. Alternatively or in addition, these calculations may be analyzed in conjunction with EM, vision, and / or topology modeling to estimate the position and / or orientation of the medical device.

[0202] Furthermore, in some embodiments, the positioning component 1214 may use other types of data to determine the position and / or orientation of an object. For example, the positioning component 1214 may analyze sensor data from shape-sensing optical fibers embedded in the medical device (e.g., which can provide shape data about the position / shape of the medical device), accelerometers, gyroscopes, satellite-based positioning sensors (e.g., Global Positioning System (GPS)), radio frequency transceivers, etc. Such data can indicate the position and / or orientation of the medical device.

[0203] In some implementations, the positioning component 1214 may use the input data in combination. For example, the positioning component 1214 may use a probabilistic method in which confidence weights are assigned to the position / orientation determined from multiple forms of input data. To illustrate, if the EM data is unreliable (in cases where EM interference may exist), the EM data may be associated with relatively low confidence values ​​and may rely on other forms of input data, such as visual data, robot commands, and kinematic data.

[0204] The target / tracking component 1216 can be configured to determine the location of a target position within a human anatomical structure and / or coordinate space / system. The target position may represent a point / set of points within a human anatomical structure and / or coordinate space / system. For example, the target / tracking component 1216 may identify one or more points of the target position within a coordinate system, identify the coordinates of those points (e.g., the X, Y, and Z coordinates of each point), and associate these coordinates with the target position. In some embodiments, the target / tracking component 1216 may use the position and / or orientation of a medical device to determine the location of the target position. For example, a scope may be navigated to contact a target position or within proximity to the target position (e.g., parked in front of the target position). The positioning component 1214 may use positioning techniques to determine the position of the scope (e.g., the position of the end of the scope) and / or the position of objects within the scope's field of view. The target / tracking component 1216 may associate the position of the scope (e.g., the coordinates of the scope) with the target position. In addition or alternatively, in some implementations, the endoscopic viewer can provide a reference point to mark the target location and determine the location of the reference point.

[0205] The target location can represent a fixed or movable point within a human anatomical structure and / or coordinate space / system. For example, if the nipple is initially designated as the target location, the coordinates of the target location can be determined and updated as the procedure progresses and the nipple moves (e.g., due to the insertion of a medical device). Here, the position of the endoscope (which may be within proximity to the nipple) can be tracked over time and used to update the coordinates of the target location. In some embodiments, the target / trajectory component 1216 can estimate / predict the location of the target location. Here, the target location can be represented by a predicted location. For example, the target / trajectory component 1216 can use an algorithm to predict the coordinates of the target location as the human anatomical structure moves. The predicted coordinates can be used to determine the target trajectory.

[0206] The target / trajectory component 1216 can also be configured to determine a target trajectory for a medical device or another object. The target trajectory may represent a desired path to a target location. The target trajectory may be determined based on various information, such as: the location of the medical device (e.g., a needle, endoscope, etc.), the target location within a human anatomy, the patient's location and / or orientation, the patient's anatomical structures (e.g., the location of organs within the patient relative to the target location), etc. For example, the target trajectory may include a line extending from the location of the medical device and / or its location on the patient's skin to / through the target location within the patient. In examples, a physician may analyze images or models of human anatomy and provide input to specify the target trajectory, such as by drawing a line on an image of the patient's internal anatomy. In some embodiments, the target / trajectory component 1216 may initially calculate the target trajectory and / or update it throughout the procedure. For example, as the target location moves during the procedure, the target trajectory may be updated due to changes in the location of the target location. In examples where the target location is estimated, the target trajectory may represent an estimated path to the target location.

[0207] In some implementations, the trajectory of the target and / or the trajectory of the medical device can be defined / represented relative to one or more anatomical planes / axises. For example, the trajectory can be defined / represented as an angle relative to the coronal / sagittal / lateral plane or another plane / axis (e.g., a head-to-tail angle of 20 degrees, an interior-exterior angle of 10 degrees, etc.). For illustration, the control system 140 can determine the orientation of the medical device relative to the EM field generator and / or the position of the target relative to the EM field generator. The control system 140 can also determine the orientation of the EM field generator relative to the robotic system based on robot kinematics. In some cases, the control system 140 can infer / determine that the robotic system is parallel to the bed. Based on such information, the control system 140 can determine the trajectory of the target and / or the trajectory of the medical device relative to anatomical planes, such as the angle of the patient on the bed relative to the anatomical plane.

[0208] User interface component 1218 may be configured to facilitate one or more user interfaces (also referred to as "one or more graphical user interfaces (GUIs)"). For example, user interface component 1218 may generate user interface data 1222 representing an instrument alignment interface 1224, which includes one or more visualizations to indicate the orientation and / or position of a medical device. User interface component 1228 may use position / orientation data 1220 of the medical device, information about the target position, and / or information about the target trajectory to present one or more visualizations within the instrument alignment interface 1224, which indicate the alignment of the medical device's orientation relative to the target trajectory and / or the proximity of the medical device to the target position. Furthermore, user interface component 1228 may use visual data (such as images captured by a hysteroscope) to present information within the instrument alignment interface 1224. In an example, information may be overlaid on an image from a hysteroscope (e.g., an enhanced image view). User interface component 1228 may provide user interface data 1222 or other data to one or more displays 142 and / or another display for displaying the instrument alignment interface 1224.

[0209] One or more communication interfaces 1206 may be configured to communicate with one or more devices / sensors / systems. For example, one or more communication interfaces 1206 may transmit / receive data wirelessly and / or via a network. Networks according to embodiments of this disclosure may include local area networks (LANs), wide area networks (WANs) (e.g., the Internet), personal area networks (PANs), body area networks (BANs), etc. In some embodiments, one or more communication interfaces 1206 may implement wireless technologies such as Bluetooth, Wi-Fi, near field communication (NFC), etc.

[0210] One or more power supply units 1208 may be configured to manage the power of the control system 140 (and / or the robot system 110, in some cases). In some embodiments, one or more power supply units 1208 include one or more batteries, such as lithium-based batteries, lead-acid batteries, alkaline batteries, and / or other types of batteries. That is, one or more power supply units 1208 may include one or more devices and / or circuits configured to provide a power source and / or provide power management functions. Furthermore, in some embodiments, one or more power supply units 1208 include a main power connector configured to be coupled to an alternating current (AC) or direct current (DC) main power supply.

[0211] One or more I / O components 1210 may include various components to receive input and / or provide output for user interaction. One or more I / O components 1210 may be configured to receive touch, voice, gesture, or any other type of input. In examples, one or more I / O components 1210 may be used to provide input regarding control of a device / system to control a robotic system 110, a navigation endoscope or other medical device attached to the robotic system 110, a control table 150, a control fluoroscopy device 190, etc. As shown, one or more I / O components 1210 may include one or more displays 142 (sometimes referred to as "one or more display devices 142") configured to display data. One or more displays 142 may include one or more liquid crystal displays (LCDs), light-emitting diode (LED) displays, organic LED displays, plasma displays, electronic paper displays, and / or any other type of technology. In some embodiments, one or more displays 142 include one or more touchscreens configured to receive input and / or display data. In addition, one or more I / O components 1210 may include one or more I / O devices 146, which may include touchscreens, touchpads, controllers, mice, keyboards, wearable devices (e.g., optical head-mounted displays), virtual or augmented reality devices (e.g., head-mounted displays), etc. Furthermore, one or more I / O components 1210 may include: one or more speakers 1226 configured to output sound based on audio signals; and / or one or more microphones 1228 configured to receive sound and generate audio signals. In some embodiments, one or more I / O components 1210 include a console or are implemented as a console.

[0212] although Figure 12 Not shown, but control system 140 may include and / or controllable other components, such as one or more pumps, flow meters, valve controllers, and / or fluid inlet components, to provide controlled flushing and / or suction capabilities to medical devices (e.g., endoscopes), devices deployable through medical devices, etc. In some embodiments, flushing and suction capabilities may be delivered directly to the medical device via a separate cable. Furthermore, control system 140 may include voltage and / or surge protectors designed to provide filtered and / or protected power to another device (such as robotic system 110), thereby avoiding the need for power transformers and other auxiliary power components in robotic system 110, resulting in a smaller, more mobile robotic system 110.

[0213] The control system 140 may also include support equipment for sensors deployed throughout the medical system 100. For example, the control system 140 may include optoelectronic equipment for detecting, receiving, and / or processing data received from optical sensors and / or cameras. Such optoelectronic equipment can be used to generate real-time images for display in any number of devices / systems, including within the control system 140. Similarly, the control system 140 may include electronic subsystems for receiving and / or processing signals received from deployed electromagnetic (EM) sensors. In some embodiments, the control system 140 may also be used to house and position an EM field generator for detection by EM sensors in or on the medical device.

[0214] In some embodiments, the control system 140 can be coupled to the robotic system 110, the worktable 150, and / or medical instruments (such as the endoscope 120 and / or needle 170) via one or more cables or connectors (not shown). In some embodiments, support functions from the control system 140 can be provided via a single cable, thereby simplifying and eliminating clutter in the operating room. In other embodiments, specific functions can be coupled in separate cables and connectors. For example, while power can be provided via a single power cable, support for control, optics, fluid, and / or navigation can be provided via separate cables.

[0215] The term "control circuitry" is used herein in its broad and general sense and can refer to one or more processors, processing circuitry, processing modules / units, chips, dies (e.g., semiconductor dies that include one or more active devices and / or passive devices and / or connection circuitry), microprocessors, microcontrollers, digital signal processors, microcomputers, central processing units, graphics processing units, field-programmable gate arrays, programmable logic devices, state machines (e.g., hardware state machines), logic circuits, analog circuits, digital circuits, and / or any means of manipulating signals (analog and / or digital) based on hard-coded circuit and / or operating instructions. Control circuitry may also include one or more storage devices, which may be embodied in a single memory device, multiple memory devices, and / or embedded circuitry. Such data storage devices may include read-only memory, random access memory, volatile memory, non-volatile memory, static memory, dynamic memory, flash memory, cache memory, data storage registers, and / or any means of storing digital information. It should be noted that in implementations where the control circuit includes a hardware state machine (and / or implements a software state machine), analog circuits, digital circuits, and / or logic circuits, a data storage device / register storing any associated operation instructions may be embedded within or outside the circuit that includes the state machine, analog circuits, digital circuits, and / or logic circuits.

[0216] The term "memory" is used herein in its broad and general sense and may refer to any suitable or desired type of computer-readable medium. For example, a computer-readable medium may include one or more volatile data storage devices, non-volatile data storage devices, removable data storage devices, and / or non-removable data storage devices implemented using any technology, layout, and / or data structure / protocol, including any suitable or desired computer-readable instructions, data structures, program modules, or other types of data.

[0217] Computer-readable media that can be implemented according to embodiments of this disclosure include, but are not limited to, phase-change memory, static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, optical disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage devices, magnetic tape, magnetic tape, disk storage devices or other magnetic storage devices, or any other non-transitory medium that can be used to store information for access by a computing device. As used in certain contexts herein, computer-readable media generally may not include communication media such as modulated data signals and carrier waves. Therefore, computer-readable media should generally be understood as non-transitory media.

[0218] Exemplary target plane and projection position

[0219] The various techniques described herein provide / generate information about the orientation / position and / or target trajectory / location of a medical device. In many examples, such techniques are discussed in the context of a endoscopic device that meets with another medical device (such as a needle, catheter, etc.) to facilitate the removal of kidney stones or another object from a patient. However, these techniques can be implemented in other contexts. For the sake of discussion, many examples will involve needles; although other medical devices, as described above, may be used.

[0220] In one illustration, as discussed similarly in the examples above, a physician can navigate a speculum to the location of a kidney stone within the kidney. The speculum can be navigated to contact the nipple and / or otherwise positioned near it. The physician can then provide input to record the speculum's position as the target location for percutaneous needle insertion into the kidney. The needle may be equipped with an EM sensor or another type of sensor to provide sensor data that can be used to determine the needle's orientation / position. The needle's position / or orientation and / or the location of the target (i.e., the nipple) can be mapped onto / projected onto a representation (such as a 2D representation / plane, 3D representation, etc.).

[0221] Mapping can be used to estimate / determine the error / difference between the current needle posture and the target position (sometimes referred to as "target determination error" or "trajectory alignment error"). Target determination error can include angular target determination error, representing the angular error relative to the target trajectory / posture (e.g., the desired / ideal trajectory / posture of a medical device), and / or positional target determination error, representing the positional error relative to the target position. In the example, angular target determination error and / or positional target determination error can be represented / indicated in the target determination data. Target determination data can be used to generate / provide information about the alignment / progress of the needle relative to the target position. This helps the user insert the needle to reach the target position. For example, target determination data can be used to render any user interface discussed herein.

[0222] In some implementations, one or more of the techniques discussed herein can be implemented for moving targets (sometimes referred to as “active targets”), such as target locations that may undergo movement during the procedure. In examples, target locations set and / or based on the position of the endoscope and / or the patient’s anatomy may shift due to anatomical movement and / or tissue deformation as the needle is inserted. In some instances, the position of the target (e.g., the nipple) can be updated in real time as the needle is inserted, such as using tracked endoscope posture and / or performing other positioning techniques.

[0223] Figures 13 to 16 illustrate exemplary techniques for mapping needle pose onto a representation and using such mapping to provide interface data according to one or more embodiments. In these examples, the left-hand image depicts the patient's anatomical structures (including kidneys 1302 / 1402 / 1502 / 1602), the target location 1304 / 1404 / 1504 / 1604, and the pose of the needles 1306 / 1406 / 1506 / 1606. Meanwhile, the right-hand image depicts one or more interface elements 1308 / 1408 / 1508 / 1608, which can be provided via an interface to indicate the pose of the needles 1302 / 1402 / 1502 / 1602 relative to the target location 1304 / 1404 / 1504 / 1604. For example, as... Figures 13-1 to 13-3As shown, one or more interface elements 1308 may include an instrument alignment element 1308(A) representing a needle 1306 and an alignment mark 1308(B) representing a target location 1304. Generally, when the needle 1306 is aligned with the target location 1304, the instrument alignment element 1308(A) may be arranged in an alignment with the alignment marks 1308(B) / 1308(C) (e.g., centered within the alignment mark 1308(B)). In Figures 13-16, the target locations 1304 / 1404 / 1504 / 1604 may be specified by a speculum 1310 / 1410 / 1510 / 1610, which can be used to remove kidney stones 1312 / 1412 / 1512 / 1612.

[0224] Figures 13-1 to 13-3 An exemplary technique is shown to map the orientation of needle 1306 onto a representation / plane 1320 (sometimes referred to as "target plane 1320") that remains fixed as the orientation of needle 1306 changes. Figure 13-1 As shown, the user can position the needle at an initial position 1322, such as at an insertion site on the patient's skin, or otherwise secure the distal end of the needle 1306 in place. A line 1324 can then be defined between the distal end of the needle 1306 (e.g., the needle tip) and the target position 1304. A plane 1320 can be defined such that the line 1324 is perpendicular to the plane 1320, and the plane 1320 includes the target position 1304. The projected position 1326 of the needle 1306 on the plane 1320 can then be determined. For example, the position of the needle 1306 can be projected onto the plane 1320 based on the current orientation / direction of travel of the needle 1306. If the needle 1306 is inserted into the plane 1320 with its current orientation / direction of travel, the projected position 1326 can represent the intersection of the needle 1306 and the plane 1320. As shown, the target trajectory / pose of the needle 1306 can include lines 1324 and / or 1328. The target trajectory / attitude may represent the trajectory / attitude to be followed to successfully reach / align with the target position 1304. Furthermore, in some instances, the target trajectory may include the direction of travel of the peephole 1310, which may be the same as or different from line 1324.

[0225] The projection position 1326 can be used to display one or more interface elements 1308, allowing a user to view the alignment of the needle 1306 relative to the target position 1304. For example, alignment mark 1308(B) can represent the target position 1304 on plane 1320, while instrument alignment element 1308(A) can represent the projection position 1326 of the needle 1306 on plane 1320. The position of instrument alignment element 1308(A) relative to alignment mark 1308(B) can indicate the alignment of the needle 1306 with the target position 1304. For example, as... Figures 13-1 to 13-3As shown, as the orientation of needle 1306 changes (e.g., the user tilts needle 1306) to align with target pose 1328, instrument alignment element 1308(A) moves closer to alignment mark 1308(B) or moves within alignment mark. Figure 13-3 The needle 1306 is shown aligned with the target pose 1328 (i.e., the projected position 1326 aligned with the target position 1304). In some examples, the positioning of one or more interface elements 1308 represents the angular target determination error; that is, the difference between the orientation of the needle 1306 and the target pose 1328.

[0226] In some cases, to facilitate one or more interface elements 1308 in Figures 13-1 to 13-3 Positioning within the interface allows plane 1320 to be mapped onto the interface. In other words, a position on plane 1320 can be mapped to a position within the interface. For example, a specific position on plane 1320 can be associated with an external alignment mark 1308(C), such that if the projected position 1326 is located near a specific position on plane 1320, then the instrument alignment element 1308(A) can be located near the alignment mark 1308(C).

[0227] As described above, the orientation of plane 1320 can remain fixed as the posture of needle 1306 changes. For example, during the needle insertion phase of a procedure (e.g., when inserting needle 1306 to attempt to reach target position 1304), the orientation of plane 1320 can remain the same even if the orientation / position of needle 1306 changes. Therefore, plane 1320 generally represents a fixed plane formed based on the initial fixed position of needle 1306. In some cases, plane 1320 can be determined when the distal end of needle 1306 is initially positioned at a relatively fixed position, such as when positioned on the patient's skin. The fixed position can be detected based on user input indicating that the position is fixed (e.g., switching to a targeting interface, selecting a specific icon, etc.), an amount of orientation movement of needle 1320 less than a threshold amount when the needle tip is in a relatively fixed position, etc. However, in an example, plane 1320 can be updated based on detecting a new fixed position of needle 1306 (e.g., orientation movement around a fixed tip point), input provided by the user indicating a new fixed needle position, etc.

[0228] Figures 14-1 to 14-3 Exemplary techniques for mapping the orientation of a needle 1406 onto a representation / plane 1420, according to one or more embodiments, are illustrated, the representation / plane being updated as the orientation of the needle 1406 changes. In this example, the user changes the orientation of the needle 1406 without changing the position of the tip of the needle 1406. Figure 14-1As shown, the user can position the needle at an initial position 1422. A plane 1420 can be defined such that the forward direction 1424 of the needle 1406 is perpendicular to the plane 1420 and the plane 1420 includes the target position 1404. Then, the projected position 1426 of the needle 1406 on the plane 1420 can be determined based on the current orientation / forward direction of the needle 1406.

[0229] The target trajectory / attitude of needle 1406 can be determined in several ways. In one illustration, the target trajectory / attitude may be or may include a line 1428 between the distal end of needle 1406 and the target position 1404. When using line 1428 as a reference, one or more interface elements 1408 may represent the angular error of needle 1406 (e.g., the angular difference between the attitude of needle 1406 and line 1428). In another illustration, the target trajectory / attitude may be or may include a line 1430 parallel to the initial direction of travel of needle 1406 and passing through the target position 1404 (e.g., line 1430 may be perpendicular to plane 1420). When using line 1430 as a reference, one or more interface elements 1408 may represent the positional error of needle 1406 (e.g., the positional difference between the projected position 1426 of needle 1406 on plane 1420 and the target position). In the example, the target trajectory / attitude includes the direction of travel of the peephole 1410 / alignment with the direction of travel of the peephole. In any case, the target trajectory can be updated as the attitude of needle 1406 changes.

[0230] Projection position 1426 can be used to display one or more interface elements 1408, allowing a user to view the alignment of needle 1406 relative to target position 1404. For example, alignment mark 1408(B) can represent target position 1404, while instrument alignment element 1408(A) can represent the projected position 1426 of needle 1406. The position of instrument alignment element 1408(A) relative to alignment mark 1408(B) indicates the alignment of needle 1406 with target position 1404. In this example, the user fixes the tip of needle 1406 in initial position 1422 and changes the orientation of needle 1406. Figures 14-1 to 14-3 As shown, as the orientation of needle 1406 changes to align with target trajectory 1428 / 1430 (which changes as needle 1406 moves), instrument alignment element 1408(A) moves closer to alignment mark 1408(B) / within that alignment mark. Figure 14-3 The needle 1406 is shown aligned with the target orientation 1428 / 1430 (i.e., the projected position 1426 is aligned with the target position 1404). Therefore, although the interface typically provides both the angular and positional errors of the needle 1406 relative to the target trajectory, the user is using the interface to view the angular error because the user is changing the orientation of the needle 1406 without changing the tip position.

[0231] As referenced above Figures 13-1 to 13-3 Similarly, plane 1420 can be mapped to an interface to display one or more interface elements 1408 at appropriate locations within the interface and / or relative to each other.

[0232] Figures 15-1 to 15-3 Another example is shown where the pose of needle 1506 is mapped onto representation / plane 1520, which updates as the pose of needle 1506 changes. In this example, the user changes the position of the tip of needle 1506 without changing the orientation of needle 1506. Figure 15-1 As shown, plane 1520 can be compared with the reference. Figures 14-1 to 14-3 The method discussed is similar to the one used to determine this. Specifically, the user can position the needle at an initial position 1522. Then, a plane 1520 can be defined such that the forward direction 1524 of the needle 1506 is perpendicular to the plane 1520, and the plane 1520 includes the target position 1504. The projected position 1526 of the needle 1506 on the plane 1520 can be determined based on the current orientation / forward direction of the needle 1506.

[0233] As mentioned above Figures 14-1 to 14-3 Similarly discussed in the text, the target trajectory / orientation of the needle 1506 may be determined as / may include (i) a line 1528 between the distal end of the needle 1506 and the target position 1504; and / or (ii) a line 1530 parallel to the initial orientation of the needle 1506 and passing through the target position 1504. The target trajectory may be updated as the orientation of the needle 1506 changes.

[0234] Projection position 1526 can be used to present one or more interface elements 1508. For example, the position of instrument alignment element 1508(A) relative to alignment mark 1508(B) can indicate the alignment of needle 1506 with target position 1504. In this example, the user changes the position of the tip of needle 1506 without changing the orientation of needle 1506. Figures 15-1 to 15-3 As shown, as the position of needle 1506 changes to align with target trajectory 1530, instrument alignment element 1508(A) moves within alignment mark 1508(B). Therefore, although the interface typically provides both the angular and positional errors of needle 1506 relative to target trajectory 1528 / 1530, the user is using this interface to view the positional error of needle 1506 relative to target trajectory 1530 because the user is changing the position without changing the orientation. Here, the user can use needle 1506 to scan target position 1504.

[0235] As referenced above Figures 14-1 to 14-3 Similarly, plane 1520 can be mapped to an interface to display one or more interface elements 1508 at appropriate locations within the interface and / or relative to each other.

[0236] Figures 16-1 to 16-3 Another example is shown where the pose of needle 1606 is mapped onto representation / plane 1620, which updates as the pose of needle 1606 changes. In this example, the user changes the position of the tip of needle 1606 and alters its orientation. Figure 16-1 As shown, plane 1620 can be compared with the reference. Figures 14-1 to 14-3 The method discussed is similar to the one used to determine this. Specifically, the user can position the needle at an initial position 1622. Then, a plane 1620 can be defined such that the forward direction 1624 of the needle 1606 is perpendicular to the plane 1620, and the plane 1620 includes the target position 1604. The projected position 1626 of the needle 1606 on the plane 1620 can be determined based on the current orientation / forward direction of the needle 1606.

[0237] As mentioned above Figures 14-1 to 14-3 Similarly discussed in the text, the target trajectory / orientation of the needle 1606 may be determined as / may include (i) a line 1628 between the distal end of the needle 1606 and the target position 1604; and / or (ii) a line 1630 parallel to the orientation of the needle 1606 and passing through the target position 1604. The target trajectory may be updated as the orientation of the needle 1606 changes.

[0238] Projection position 1626 can be used to present one or more interface elements 1608. For example, the position of instrument alignment element 1608(A) relative to alignment mark 1608(B) can indicate the alignment of needle 1606 with target position 1604. In this example, the user changes the position of the tip of needle 1406 and changes the orientation of needle 1606. Figures 16-1 to 16-3 As shown, as the position of needle 1606 changes to align with the target trajectory 1628 / 1630, the instrument alignment element 1608(A) moves within the alignment mark 1608(B). Therefore, the user typically uses this interface to view both the positional and angular errors of needle 1406 as the user changes the position and orientation.

[0239] As referenced above Figures 14-1 to 14-3 Similarly, plane 1620 can be mapped to an interface to display one or more interface elements 1608 at appropriate locations within the interface and / or relative to each other.

[0240] Although various examples have been discussed in the context of projecting the needle's position onto a plane using the needle's tip as a reference point, in some examples, other parts of the needle can be projected onto the plane, such as the proximal end / tip of the needle, the middle portion of the needle, etc. For example, in returning to... Figure 13-1In an example, the proximal end of needle 1306 can be projected onto plane 1320 (instead of the tip of needle 1306), such that the projection line includes the proximal end of needle 1306 and is parallel to the target trajectory 1324. Here, the projection position of the proximal end of needle 1306 on plane 1320 will be higher than the target position 1304 (and the interface element 1308(A) will be located at...). Figure 13-1 (In the interface at the top left corner). In some implementations, one or more projection geometry techniques can be used to determine the projected position of the needle or another medical device on the plane. Furthermore, although various 2D representations / planes are depicted, other types of representations, such as 3D representations, can also be implemented.

[0241] As discussed in this article, various techniques can be used to determine the target plane. For example, the target plane can be based on the direction of travel of the endoscope, such that the direction of travel of the endoscope is perpendicular to that plane. In some cases, the direction of travel of the endoscope can also represent the target trajectory. Therefore, in some instances, this can help the user align a needle or other medical device to enter the target position coaxially with the direction of travel of the endoscope.

[0242] Exemplary coordinate system

[0243] As described above, medical devices can be implemented to reach a target location. In some solutions using needles, it may be difficult to define specific types of movement of the medical device well. This could be due to the characteristics of the medical device (e.g., the cylindrical shape of the needle), the type of sensors used, and / or the characteristics of the system tracking the attitude of the medical device. Therefore, it may be difficult to correlate the movement of the needle with the movement of interface elements within the interface in an effective / intuitive way. For example, when the needle tilts to the right relative to the user, such solutions may cause the needle indicator to move upwards within the interface. This can disorient the user when viewing the interface to manipulate the needle.

[0244] Figure 17 Exemplary techniques for establishing one or more coordinate systems to correlate the movement of a medical device with the movement of interface elements within an interface are illustrated. Generally, these techniques can use information about the robotic system to determine the coordinate system of the medical device and use this coordinate system to provide information about the alignment / progress of the medical device. For example, these techniques can help achieve instinctive hand-eye coordination by mapping the movement of the medical device to the user interface in a manner related to how the user holds the device (e.g., moving the interface icon of the needle left / right within the interface when the needle is tilted left / right relative to the user, moving the interface icon up / down within the interface when the needle is tilted up / down relative to the user, etc.). This helps the user align the medical device.

[0245] For illustration, as described above, a physician can implement the robotic system 110 to perform medical procedures. The robotic system 110 can be coupled to an EM field generator 180, such as being coupled to a robotic arm 112(A). The EM field generator 180 can provide power from medical devices (such as needles). Figure 17 The EM field detected by the EM sensor on the medical device (not shown). Based on sensor data from the medical device, the orientation of the medical device relative to coordinate system 1702 of the EM field generator 180 can be determined. Furthermore, coordinate system 1704 (also referred to as "world coordinate system 1704") can be determined for the robot system 110 (such as the bracket interface 1104 / base attached to the robot arm 112(A)) or any other component of the robot system 110. The orientation of the medical device can then be represented within world coordinate system 1704. In this example, based on a transformation (e.g., homogeneous transformation) between coordinate system 1702 and world coordinate system 1704, the orientation of the medical device can be represented in world coordinate system 1704 using the positive kinematics of the robot system 110 (e.g., robot arm 112(A)).

[0246] Furthermore, any of the techniques discussed herein can be used to determine a target position 1706 within the patient's (not shown) body relative to coordinate system 1702 of the EM field generator 180. A plane 1708 (also referred to as the "target plane 1708") can then be determined based on the target position 1706. The plane 1708 can then be represented within world coordinate system 1704, such as in a manner similar to representing the orientation of a medical device in world coordinate system 1704. Coordinate axes 1710 can then be determined for the target plane 1708. For example, it can be assumed / inferred that the medical device will not be parallel to the z-axis of world coordinate system 1704, which can be used to define coordinate axes (e.g., the y-axis) on the target plane 1708. A cross product can be performed between the z-axis of world coordinate system 1704 and the forward direction (z-axis) of the medical device to define another coordinate axis (e.g., the y-axis) on the target plane 1708. Once two axes are defined, a third axis can be defined by performing a cross product between the newly acquired / determined axis and the forward direction of the medical device. Figure 17 The image shows a 3D representation (middle image) and a 2D representation (bottom image) of plane 1708. As shown, the projected position 1712 of the medical device is displayed on plane 1708 as the origin of coordinate axis 1710. Furthermore, the target position 1706 is represented by bubbles on the target plane 1706.

[0247] Once coordinate system 1710 is determined for target plane 1708, the coordinate system of the medical device can be determined. For example, the coordinate system of the medical device can be directly translated from target plane 1708 because the direction of travel of the medical device and the normal to the target plane can be mirror images of each other. Therefore, the coordinate system of the medical device can be the inverse coordinate system of the coordinate system of target plane 1708.

[0248] In the example, one or more of the following mathematical steps can be performed to generate one or more of the coordinate system, axes, and / or other elements:

[0249] world Z = [0, 0, 1]

[0250]

[0251]

[0252]

[0253] world R TP =[ world X TP , world Y TP , world Z TP ]

[0254] world t TP = world P target

[0255] [ BE P bubble ,1]=( world T TP ) -1 [ world P bubble ,1]

[0256] if world R needle [0∶2,2]= world Z→ world Z+ = eps

[0257] Exemplary Adaptive Target Determination

[0258] Figures 18 and 19 illustrate various exemplary adaptive target determination techniques (also referred to as “adaptive scaling”) according to one or more embodiments. In some embodiments, a certain amount of target determination error can be tolerated when the medical device is relatively far from the target location (e.g., beyond a threshold). However, as the medical device moves closer to the target location, the target determination error may need to be smaller (e.g., the size of the anatomical target (such as the size of a nipple), smaller than the size of the anatomical target, or another size) to facilitate access to the target location. Therefore, in the examples, adaptive target determination techniques can be implemented such that as the medical device moves closer to (or further away from) the target location (e.g., with insertion of the medical device), one or more interface features within the interface can function in different ways. For example, when the tip of the medical device is positioned relatively close to the target location, a relatively slight deviation from the target trajectory can result in a relatively larger movement of the device position feature from the target center in the interface compared to when the medical device is further away from the target location. This can help the physician align the medical device with the target location with a larger tolerance when the medical device is farther from the target location, and / or help the physician align the medical device with the target location more accurately as the medical device approaches the target location.

[0259] Figure 18-1 shows a user holding the needle 1820 or otherwise maintaining the needle in a posture 1821, wherein the needle 1820 is oriented in an axial orientation associated with the posture 1821. Figure 18-1A Image 1824 relates to a medical procedure in which a user inserts a needle 1820, attempting to advance the tip of the needle 1820 through the renal cone 1825 to a target location 1826 (e.g., a calyx puncture site). For example, as described in detail herein, the control circuitry of the relevant system may be configured to determine a target needle trajectory 1822 based on the current position 1828 of the needle tip and / or the orientation of the needle 1820. In the example, the target trajectory 1822 may be associated with a direct (or indirect / zigzag) path between the needle tip position 1828 and the target location 1826, as illustrated. Therefore, an ideal implementation of the relevant medical procedure may involve the user inserting the needle 1820 into the target location 1826 with minimal deviation from the target needle trajectory 1822.

[0260] Figure 18-1B Certain user interface features that can be implemented in conjunction with the medical protocols described herein are illustrated. For example, the control circuitry of the relevant system may be configured to generate interface data representing certain visual features as shown and described herein, wherein certain characteristics of such features are determined and / or at least in part based on detected needle position, determined needle target trajectory, determined target position, and / or other information derived using one or more position sensors or other position determination devices / mechanisms.

[0261] Figure 18-1B The user interface features shown may include alignment visualization element 1830, which includes certain features that can be dynamically adjusted relative to one or more of its properties in response to changes in needle position, trajectory, etc. Figure 18-1B In the example shown, alignment visualization element 1830 includes instrument position feature 1832 (also referred to as "instrument alignment element 1832"), which may have a bubble shape or form, or any other shape or form. For example, instrument position feature 1832 may be configured to visually evoke an image similar to a bubble level or other leveling instrument. Alignment visualization element 1830 may also include alignment target feature 1831 (also referred to as "alignment mark 1831"), which may be represented as a shape or form defining a boundary around a central target location 1836, which may generally be associated with the axial center of alignment visualization element 1830 and / or one or more of its features. Although shown as a full circle boundary, alignment target feature 1831 may be a missing circle or any other shape or form that visually indicates a radial boundary around the target center 1836.

[0262] In some embodiments, the alignment visualization element 1830 further includes one or more crosshair features 1833, which can be used to indicate to the user and / or visually guide the user to the target center 1836. In an example, the crosshair features 1833 intersect at the target center 1836. In some embodiments, the alignment visualization element 1830 also includes an outer boundary feature 1834 (also referred to as “boundary marker 1834”), wherein the outer boundary can provide a visual and / or enforced boundary for the instrument position feature 1832. Control circuitry can be configured to limit the radial distance of the instrument position feature 1832 from the target center 1836 to the outer or inner boundary of the outer boundary feature 1834. However, in some specific embodiments, the instrument position feature 1832 may be allowed to deviate from the outer boundary 1834, wherein the outer boundary provides the user with a visual indication of the degree to which the instrument position feature 1832 has deviated from the target center 1836 and / or the alignment target boundary 1831.

[0263] According to some specific embodiments of this disclosure, the instrument position feature 1832 can be located at a distance d from the target center 1836. t The distance is presented as a distance based on and / or representing the distance D between determined needle projection points 1829 located in a plane 1827 intersecting with the target location 1826. P For example, plane 1827 could be the target entry plane that is substantially perpendicular to the forward direction / attitude of needle 1820. However, it should be understood that the distance D between the projected position 1829 and the target position 1826... PThis can be relative to any plane intersecting the target position 1826. If the needle 1820 is inserted along the current trajectory 1823 until it intersects the target plane 1827, the projected needle position 1829 can represent the positioning / location of the tip projection of the needle 1820. However, it should be understood that aspects related to the projection deviation distance from the target position in this disclosure can represent any projection distance of the needle or other instrument from the target position. Reference Figure 18-1B The distance d between instrument position feature 1832 and target center 1836 t It can be based at least in part on the projection deviation distance D P .

[0264] Based on some of the implementation schemes disclosed in this article, such as references Figure 18-1B And / or other figures in this disclosure, the distance d between the reference device position feature 1832 and the target center relative to the alignment visualization element of the user interface. t It should be understood that the distance d t This can be interpreted / represented as a relative size with respect to the size of one or more other visual features of the aligned visualization element 1830. That is, in this paper, it describes the distance d between the instrument position feature and the target center 1836. t In the event of changes, it should be understood that such reference distance may refer to changes in the actual distance represented on the user interface display, and / or may refer to changes in the distance between the instrument position feature 1832 and the target center 1836 relative to the size of one or more other visual features of the alignment visualization element 1830. For example, this can be achieved by changing the actual distance d between the instrument position feature 1832 and the target center 1836. t And / or by changing the dimensions associated with one or more features of the alignment visualization element 1830 (such as the diameter / size d1 of the alignment target feature 1831, the diameter / size d2 of the instrument position feature 1832, the diameter / size d3 of the outer boundary feature 1834, and / or the length and / or position of the crosshair feature 1833) to visually represent the distance d between the instrument position feature 1832 and the target center 1836. t Therefore, the description of the change / modification of the distance between the instrument position feature and the target center of the alignment visualization user interface element in this paper can be understood as including changing the distance d between the instrument position feature 1832 and the target center 1836. t And / or such as shrinking or enlarging in a proportional manner Figure 18-1B Any other size shown.

[0265] Figure 18-2A This illustrates how the user holds needle 1820 in a changing position / pose 1821b. For example, Figure 18-2AImage 1837 can be represented by pose 1821b, which indicates needle 1821 and is associated with further insertion of needle 1820 toward target position 1826. That is, in Figure 18-2A In the diagram, the distance D between the needle tip position 1828b and the target position 1826 is... I-2 comparable Figure 18-1A The distance D shown I short. Figure 18-2B A modified version of the alignment visualization element 1830 is shown, where the distance d of the instrument position feature 1832 is... tb It has been modified to indicate the current position / attitude of needle 1820 1821b.

[0266] like Figure 18-2A As shown, the control circuit can determine the target needle trajectory 1822b and / or the current insertion distance D between the needle tip position 1828b and the target position 1826. I-2 The control circuit can be configured to determine the target deviation distance D. P-2 The target deviation distance represents the current projection deviation between the projection pin position 1829b and the target position 1826.

[0267] about Figure 18-2B In some examples, the image can be aligned with the visualization element 1830 at a distance d from the central target position 1836. tb The instrument's position features are shown at location 1832, where this distance d tb It can be based at least in part on the current projection deviation distance D P-2 In some specific implementations, the distance d tb Can be with Figure 18-1B distance d t Yes, there is a relationship, and this relationship is related to the projection deviation distance D. P-2 Relative to the projection deviation distance D P The difference between them is proportional. That is to say, relative to the change in the projection deviation distance between attitude 1821 and attitude 1821b, the change in the distance between the instrument position feature 1832 and the target center 1836 can usually be linear.

[0268] In some specific implementations, the change in distance between instrument position feature 1832 and target center 1836 relative to the change in projection deviation distance between posture 1821 and posture 1821b can often be non-linear. For example, the extent to which the change in projection deviation distance causes a change in the distance between instrument position feature 1832 and target center 1836 can be at least partially based on the insertion distance D. I / D I–2 In other words, when the distance d is determined... t / d tb At that time, the control circuit can be based at least in part on the distance D.I / D I-2 This type of determination is performed. For example, distance d. tb Based on the projection deviation distance D P-2 and insertion distance D I-2 The product of the factors and possible other multipliers is used to determine the location. In some specific implementations, this device position feature distance d is used. tb The determination can be performed by the control circuit using one or more insertion distance parameters. For example, for relatively short insertion distances, such parameters can have large values ​​(e.g., causing a large increase / decrease in the instrument position characteristic distance). That is, the insertion distance parameter can be approximately inversely proportional to the insertion distance, such that as the insertion distance decreases, the parameter has a smaller effect on the instrument position characteristic distance d. t The determination of this has a greater impact. Here, when the user inserts the needle tip to a position relatively close to the target position 1826, a slight deviation from the target needle trajectory can cause a relatively large movement of the instrument position feature 1832 from the target center 1836. It should be understood that... Figure 18-1A and Figure 18-2A The various distances shown can be direct vector distances, or vector component dimensions / distances relative to any suitable or desired reference plane or axis.

[0269] In some examples, instrument position feature 1832 is presented within alignment visualization element 1830, but not outside of it. For example, the size represented by alignment visualization element 1830 can vary linearly as a function of the distance between needle 1820 and target position 1826, where the position of instrument position feature 1832 can be located within alignment visualization element 1830 even for relatively large alignment errors. To illustrate, the position of instrument position feature 1832 can be the same for different distances when the alignment error is relatively large, because instrument position feature 1832 is only displayed within alignment visualization element 1830 (in some cases), and if the instrument position feature were located outside alignment visualization element 1830, it would be clipped. Therefore, in some instances, as needle 1820 moves closer to target position 1826, the physical representation associated with alignment visualization element 1830 becomes smaller, even if alignment visualization element 1830 looks the same. In the examples, the user interface (e.g., alignment visualization element 1830 and / or other elements) remains unchanged. As the needle 1820 moves closer to the target position 1826, the error or distance in the pixel space can correspond to a smaller error. To illustrate, when the needle 1820 is relatively far from the target position 1826, the 2mm error of the needle 1820 relative to the target trajectory may not be noticeable, but as the needle 1820 moves closer to the target position 1826, the error becomes noticeable.

[0270] Figures 19-1 to 19-3 An exemplary technique for scaling a target plane based on the insertion distance of a medical device is illustrated. Figure 19-1 and Figure 19-2 In the diagram, images 1902 and 1904 on the left depict the patient's anatomy (including the kidney 1906), as well as the orientation of the target location 1908 and the needle 1910. Furthermore, images 1912 and 1914 in the middle depict the target representation (e.g., corresponding to physical dimensions scaled with distance from the target) on plane 1916 (also referred to as the "target plane 1916"). The target plane 1916 may be associated with / include a circle with a defined diameter, where the circle may be centered at the target location 1908. Images 1912 and 1914 are not typically presented to the user but are provided herein for ease of discussion. Additionally, images 1918 and 120 on the right depict one or more interface elements 1922 (e.g., corresponding to fixed user interface dimensions) to indicate the current orientation of the needle 1910. Interface element 1922(A) may represent the needle 1910, interface element 1922(B) may represent the target location 1908, and interface element 1922(C) may represent a boundary. Figure 19-1 and 19-2 Also shown is a speculum 1924, used to help remove kidney stones 1926.

[0271] like Figure 19-1 As shown, when the tip of needle 1910 is farther from the target position 1908, the target representation on the target plane 1916 can be relatively large. The target representation may represent one or more interface elements 1922 (e.g., the size of one or more interface elements 1922). For ease of discussion, the target representation is shown as associated / projected onto the target plane 1916; however, the characteristics of the target representation can also be implemented in other ways without associating / projecting the target representation onto the target plane 1916.

[0272] like Figure 19-2 As shown, the size of the target representation can change as the tip of needle 1910 moves closer to the target location 1908, as illustrated in image 1914. For example, one or more dimensions of the target representation on target plane 1916 can be reduced such that a smaller area on target plane 1916 corresponds to one or more interface elements 1922. Therefore, the scaling ratio between the dimensions represented in target plane 1916 and the dimensions represented in the interface can be adjusted. In some embodiments, the central element 1928 of the target representation can be changed to be the same as or smaller than the size of the anatomical target (such as the size of a nipple). For example, when the tip of needle 1910 is within a threshold distance to the target location 1908, the central element 1928 can be updated to be the same as (or smaller than) the size of the anatomical target.

[0273] Figure 19-3An exemplary graph 1930 illustrates the scaling of the target plane 1916 relative to the distance (e.g., insertion distance) from the needle 1910 to the target position 1908. Specifically, the y-axis represents the distance between the tip of the needle 1910 and the target position 1908, while the x-axis represents the size of the target representation on the target plane 1916 (i.e., the scaling ratio of the distance on the target plane 1916 to the distance on the interface). As shown, the relationship (e.g., the scaling ratio) between the distance on the target plane 1916 and the corresponding distance on the interface can be updated as the needle 1910 moves closer to the target position 1908. Although in Figure 19-3 The diagram shows a linear relationship, but another type of relationship (e.g., nonlinear, step, etc.) can be achieved.

[0274] In some instances, Figures 19-1 to 19-3 The scaling techniques discussed in the context of this paper (and elsewhere) enable adaptive target determination to help the user accurately insert the needle 1910 to reach the target position 1908 (which may be relatively small). For example, when the needle 1910 is further from the target position, such scaling techniques may cause a relatively slight deviation from the target trajectory, resulting in a small movement of the instrument position feature from the target center in the interface, and when the needle 1910 is positioned relatively close to the target position, they may cause a relatively slight deviation from the target trajectory, resulting in a relatively large movement of the instrument position feature from the target center. In the example, as the needle 1910 is inserted, the size of one or more interface elements 1922 may not change within the interface (e.g., each interface element 1922 may maintain the same size / shape). However, in other examples, the size of one or more interface elements 1922 may change within the interface.

[0275] In some implementations, one or more of the techniques discussed herein can reduce the complexity of percutaneous access to a target location, such as by providing an interface that includes elements indicating the alignment / position of the medical device relative to a target trajectory and / or target location. In an example, this interface can assist a physician or other user in manually manipulating the medical device. Compared to other techniques such as fluoroscopy or ultrasound, such techniques enhance the ability of a physician or another user to accurately reach the target location using the medical device.

[0276] Exemplary Anatomical Visualization Section

[0277] Figure 20An exemplary anatomical visualization unit 2000, provided via an interface to assist a user in navigating a medical device according to one or more embodiments, is illustrated. In this example, the medical device is implemented as a needle; however, the medical device may be implemented as other devices. In some instances, the visualization unit 2000 may help the user to orient the needle appropriately to approach a target location coaxially, such as coaxially aligning it with the renal pyramid / nipple, which avoids damage to surrounding anatomical structures (e.g., this could occur due to the needle extending beyond its limits) and / or provides greater flexibility for inserting devices into the access pathway (e.g., catheters or other devices).

[0278] As shown in the figure, the visualization unit 2000 includes a needle representation 2002 and an anatomical representation 2004 of the patient's anatomical structures, such as the calyx network and / or other anatomical structures associated with the patient. The anatomical representation 2004 can be presented as a 2D or 3D representation within an interface, such as any interface discussed herein. The visualization unit 2000 also includes a target region representation 2006, which represents the target area in which the needle should stop to reach the target position (represented by the target position representation 2008). The target region representation 2006 can be presented as a 2D or 3D representation, such as a 2D cone (as shown), a 3D cone, etc. The target position and target region can be determined based on the orientation / position of the endoscope and / or another element (represented by element 2010 in the visualization unit 200). In this example, the target area is aligned with the distal end of the endoscope, which is positioned within the kidney to specify the target location, such that a line 2012 perpendicular to the distal end of the endoscope (e.g., a line indicating the direction of travel of the endoscope) passes through the center of the target area and the target location. In some instances, line 2012 may represent a target trajectory, which may or may not be presented in the visualization unit 2000. In this example, as the endoscope moves, the target area representation 2006 and / or the target location representation 2008 may be updated to maintain alignment with the distal end of the endoscope. The target location representation 2008 may be presented as a 2D or 3D representation, such as a 2D line (as shown), a 3D sphere / surface, etc. In this example, the target location representation 2008 is associated with an anatomical target (e.g., scaled to / representing the size of a nipple, which may be 4 mm or 6 mm). Furthermore, the visualization unit 2000 may include a line 2014 indicating the direction of needle travel.

[0279] Although various elements are discussed in the context of the visualization section, in some instances, these elements may not be presented; instead, only these elements may be determined / generated for evaluating the current pose of the needle 2000 relative to the target position 2008. For example, the target region and / or anatomical representation / model may be determined and used to evaluate the current pose of the needle without displaying the target region visualization section 2006 and / or the anatomical representation 2004.

[0280] Exemplary device status and area

[0281] Figure 21 An exemplary region 2102, which can be implemented / generated according to one or more embodiments to determine the state (sometimes referred to as "progress state") of a medical device, is shown. As illustrated, region 2102 is discussed in the context of a needle 2104 entering the kidney 2106 to aid in the removal of a kidney stone 2108. Here, a speculum 2110 is implemented to specify the target location 2112 of the percutaneous insertion of the needle 2104. However, region 2102 can also be implemented in the context of other situations / medical procedures. In the example, the state of the needle 2104 can indicate the position / progress of the needle 2104 relative to the target location 2112. For example, the state may include: a state in progress, in which the indicator 2104 is being inserted and has not yet reached / passed the target position 2112; a state in progress and aligned, in which the indicator 2104 is being inserted and aligned with the target trajectory / attitude; a state where the target is unreachable, indicating that the target position 2112 is unreachable given the current orientation of the needle 2104; a state where the target has been reached, in which the indicator 2104 has reached the target position 2112; a state where the target has passed, in which the indicator 2104 has been inserted outside the target position 2112, and so on. The state of the needle 2104 can be used for various purposes, such as providing indication via an interface and / or performing other processing.

[0282] In some implementations, region 2102 is determined based on the target location 2112 and / or the position / or orientation of the endoscope 2110. For example, a first region 2102(A) may be determined based on the target plane and / or the position / or orientation of the endoscope 2110. Here, region 2102(A) includes a tapered region aligned with line 2114 perpendicular to the distal end of the endoscope 2110. However, region 2102(A) may include other forms / shapes. In this example, needle 2104 is also aligned with line 2114. Region 2102(A) may be associated with a state in which target location 2112 may be reached (e.g., in progress, in progress and aligned, etc.). For example, when needle 2104 is positioned within region 2102(A) (e.g., relative to the tip and / or any other portion of needle 2104), an indication may be provided via an interface that indicates that needle 2104 can be aligned with target location 2112. Furthermore, if the needle is aligned with the line 2114 (e.g., target posture / trajectory), the needle 2104 can be associated with a state indicating that the needle 2104 is aligned with the target position 2112 (e.g., in progress and aligned). In the example, when the needle 2104 is positioned within region 2102(A), the progress indicator can indicate the proximity of the needle 2104 relative to the target position 2112 and / or indicate that the needle 2104 is in the process of being inserted.

[0283] Furthermore, the second region 2102(B) can be aligned with and extends from the distal end of the endoscope 2110. Here, region 2102(B) includes a specific form; that is, a semicircle whose straight line is aligned with the target plane (in Figure 21 (Partially shown). However, region 2102(B) may include other forms / shapes. Region 2102(B) may be associated with a state of reaching target position 2112. For example, reaching target position 2112 can be determined when the tip of needle 2104 is within region 2102(B). In the example, an indication of this state may be displayed via an interface indicating this state. In one illustration, a progress indicator indicating that needle 2104 has reached target position 2112 may be shown. In the example, region 2102(B) may allow a certain amount of overshoot at target position 2112. Furthermore, in the example, the size and shape of region 2102(B) may be determined based on the size and / or shape of human anatomy structures associated with the target position, such as the average size of a nipple (with a certain amount of tolerance), the size / shape of another organ / anatomical feature, etc. In the example, regions 2102(A) and / or 2102(B) represent target areas (e.g., areas in which needle 2104 should remain to reach target position 2112).

[0284] In addition, as well as Figure 21 As shown, the third region 2102(C) may encompass the area outside the first region 2102(A) and the second region 2102(B). Region 2102(C) may be associated with states that the target position 2112 is unreachable, the target position 2112 has been passed, or the needle 2104 is not within the target region. For example, when the tip of the needle 2104 is within region 2102(C), it can be determined that, given the current posture of the needle 2104, the target position 2112 is unreachable, the target position 2112 has been passed (e.g., the needle 2104 has substantially exceeded the target position 2112), or the needle 2104 is not within the target region. In the example, an indication may be displayed via an interface indicating such a state. This may instruct the physician to stop inserting and / or withdrawing the needle 2104 because the physician may be unable to correct the needle trajectory at that point and / or the physician has exceeded the target position 2112 by a specific amount. In one illustration, if needle 2104 crosses into area 2102(C), the deformable state can display the instrument alignment element and / or provide another indication.

[0285] Exemplary adaptive progress indicator

[0286] As discussed in the various examples herein, the proximity / progress of a medical device to a target location can be indicated in the user interface or otherwise. In some implementations, such information is indicated linearly, such that the amount of progress indicated for each unit of movement closer to / further from the target location can be consistently represented by a specific amount of progress variation. For illustration, every 10 mm change in the distance between the tip of the medical device and the target location can correspond to a 5% progress change displayed in the interface (e.g., indicating the progress of inserting the medical device into the target location). Here, this progress variation can be implemented for every 10 mm change until the medical device reaches the target location.

[0287] In other implementations, one or more adaptive progress indication techniques may be implemented to adjust the amount of progress indicated for each unit of movement. In one illustration, as shown... Figure 22 As shown in graph 2200, the amount of progress change indicated for a unit of positional change can be based on the distance between the medical device and the target location. In this graph, an insertion distance in the range of 150 mm to 75 mm can be associated with a non-linear progress indication. For example, every 1 mm change in proximity to the target location within the range of 150 mm to 75 mm can result in an increase in the indicated progress (e.g., a change from 150 mm to 140 mm can be associated with a 1% progress change, a change from 140 mm to 130 mm can be associated with a 2% progress change, and so on). Furthermore, in this graph, the progress change can be linear for changes from 75 mm to and / or past the target location. By implementing such adaptive techniques, the progress change within the interface can be minimized when the medical device is relatively far from the target location, which may allow the physician to reposition or otherwise select the insertion site without seeing a large progress change. Furthermore, this can cause the physician to slow down the insertion speed as the medical device approaches the target location, thereby achieving more precise alignment before reaching the target location.

[0288] Figure 22 The above diagram illustrates one of many exemplary techniques for adaptive schedule indication. Therefore, other linear or nonlinear schedule indications, such as piecewise linear functions, exponential functions, etc., can be implemented.

[0289] While various techniques for determining / presenting progress / alignment information are discussed herein, a variety of other techniques may be used in addition to or as alternatives to such techniques. In some embodiments, the endoscope may be positioned near a target location, such as parking in front of the nipple of interest. Here, images captured by the endoscope may be presented via an interface to provide visual confirmation that the target location has been reached, such as when anatomical movement is minimal and / or the endoscope is parked in the appropriate field of view. In an example, images from the endoscope may be analyzed to determine the progress status of the medical device. Furthermore, in some embodiments, any progress information discussed herein may be presented in addition to or as an alternative to the images captured by the endoscope to help confirm that the target location has been reached. Such progress information is particularly useful when there is substantial anatomical movement, the nipple or other anatomical structures are pressed against the endoscope's camera, etc. In an example, such progress information may augment the image obtained by the endoscope. Furthermore, such progress information may avoid navigating the endoscope into the kidney to search for confirmation that the medical device has been inserted into the kidney. In addition to or alternatively, in some embodiments, the current position of the medical device (e.g., a needle) may be overlaid on the endoscope's image view. In addition, or alternatively, in some embodiments, the medical device may be equipped with impedance sensing electrodes, and the impedance signal can be used to classify the medium in contact with the medical device electrodes. In an example, if the impedance signal is classified as urine, it can be determined that a target has been reached (e.g., the target arrival status can be determined).

[0290] Exemplary flowchart

[0291] Figures 23 to 28 An exemplary flowchart is shown for performing one or more techniques discussed herein. Various operations / blocks associated with these processes may be performed by control circuitry or combinations thereof implemented in any device / system discussed herein, such as control system 140, robot system 110, workbench 150, EM field generator 180, peephole 120, and / or needle 170.

[0292] Figure 23 An exemplary flowchart of a process 2300 for generating data indicating the distance between a projected position of a medical device on a plane and a target position on the plane, according to one or more embodiments, is shown.

[0293] At box 2302, process 2300 may include receiving sensor data from one or more medical devices. For example, depending on certain use cases, the control circuitry of the device / system (such as a control system) may receive sensor data from one or more medical devices (such as a hysteroscope, needle, or any other medical device) via a communication interface. The sensor data may indicate and / or be used to determine the position and / or orientation of one or more medical devices.

[0294] At box 2304, process 2300 may include determining a target location within a human anatomical structure. For example, control circuitry may determine a target location within a patient's body, such as an anatomical landmark, the location of a medical device, or any other location / target. In some embodiments, the control circuitry may determine the target location based on sensor data from a medical device (e.g., a endoscopic endoscope or another medical device) that is at least partially disposed within the patient's body.

[0295] At block 2306, process 2300 may include determining the position and / or orientation of one or more medical devices. For example, control circuitry may determine the position and / or orientation of one or more medical devices based on sensor data from one or more medical devices. In some embodiments, the control circuitry may use one or more positioning techniques to determine the position and / or orientation. In this example, the position and orientation of the medical devices may be referred to as the orientation of the medical devices.

[0296] At 2308, process 2300 may include determining a plane including the target location. In one example, control circuitry may determine the plane such that the direction of travel of the medical device is perpendicular to the plane based on sensor data from the medical device. In another example, control circuitry may determine a line between the distal end of the medical device and the target location based on sensor data from the medical device, and determine a plane such that the line is perpendicular to the plane. In yet another example, control circuitry may determine the plane such that the direction of travel of the endoscope is perpendicular to the plane based on sensor data from the endoscope.

[0297] At 2310, process 2300 may include updating or maintaining the orientation of the plane as the orientation of the medical device changes. In one example, the control circuitry may update the orientation of the plane as the orientation of the needle changes, such that the direction of needle movement remains perpendicular to the plane. In another example, the control circuitry may maintain a constant / fixed orientation of the plane as the orientation of the needle changes. In yet another example, the control circuitry may update the orientation of the plane as the orientation of the endoscope changes, such that the direction of endoscope movement remains perpendicular to the plane.

[0298] At 2312, process 2300 may include determining the projected position of the medical device on a plane. For example, control circuitry may determine the projected position of the medical device on the plane based on the orientation / position of the medical device. In this example, the projected position may be relative to the tip of the medical device; however, the projected position may be relative to other parts of the medical device, such as the rear / proximal end of the medical device, the middle portion of the medical device, or another portion.

[0299] At block 2314, process 2300 may include generating interface data representing one or more interface elements that indicate the distance (e.g., projection offset distance) between a projected position on a plane and a target position. For example, control circuitry may generate user interface data representing: an instrument alignment element indicating the orientation of a medical device; and / or alignment marks indicating the target position. In some embodiments, the positioning of the instrument alignment element relative to the alignment marks and / or other features in the interface may indicate alignment of the medical device with the target trajectory.

[0300] At box 2316, process 2300 may include causing the display of one or more interface elements based on interface data. For example, control circuitry may cause the interface to be displayed via a display device, such as by sending user interface data to a display device associated with a control system. Furthermore, the display device may display the interface based on interface data. In some cases, the interface presents a first interface element at a distance from the center of a second interface element, wherein this distance is based on the distance between a projected position on a plane and a target position and / or the insertion distance between the tip of a medical device and the target position on a plane.

[0301] Figure 24 An exemplary flowchart of a process 2400 for updating the progress amount of a movement unit indication for a medical device, according to one or more embodiments, is shown.

[0302] At block 2402, process 2400 may include causing a progress indicator to be displayed, which indicates the proximity of the tip of the medical device to a target position and / or another part of the medical device to a target position. For example, control circuitry may display the progress indicator via a display device, such as by sending interface data to a display device associated with the control system. Alternatively, the display device may display the progress indicator based on interface data.

[0303] At block 2404, process 2400 may include setting a progress change parameter of the progress indicator to a first value. For example, a control circuit system may set the progress change parameter to a first value, wherein the progress change parameter indicates the amount of progress change of the progress indicator relative to a unit of movement of the medical device (e.g., a change in the proximity of the tip of the medical device to a target position).

[0304] At box 2406, process 2400 may include determining that the tip of the medical device has moved closer to the target position. For example, control circuitry may determine (at a first moment / instance) that the tip of the medical device has moved closer to the target position based on sensor data from the medical device. In this example, the control circuitry may determine that the medical device has moved closer to the target position by a first amount.

[0305] At box 2408, process 2400 may include updating a progress indicator. For example, control circuitry may update (first instance / instance) the progress indicator based on progress change parameters and / or determining that the tip of the medical device has moved closer to the target position.

[0306] At block 2410, process 2400 may include setting a progress change parameter to a second value. For example, control circuitry may set the progress change parameter to a second value based on determining that the tip of the medical device has moved closer to a target position. This second value may be associated with a progress change amount of the progress indicator for that unit of movement of the medical device that is larger than the first value.

[0307] At box 2412, process 2400 may include determining that the tip of the medical device has moved closer to the target position. For example, control circuitry may determine (a second time / instance) that the tip of the medical device has moved closer to the target position based on sensor data from the medical device. In this example, the control circuitry may determine that the medical device has moved closer to the target position by a first amount.

[0308] At box 2414, process 2400 may include updating a progress indicator. For example, control circuitry may update (second time / instance) the progress indicator based on progress change parameters and / or determining that the tip of the medical device has moved closer to the target position. In the example, the progress indicator may be updated to indicate a different amount of progress change than indicated in box 2408.

[0309] Figure 25 An exemplary flowchart of a process 2500 for updating a position change parameter according to one or more embodiments is shown, the position change parameter indicating the amount of position change of an interface element for a unit of orientation / movement change of a medical device.

[0310] At box 2502, process 2500 may include setting a position change parameter to a first value. For example, control circuitry may set the position change parameter to a first value, wherein the position change parameter may indicate the amount of position change of one or more interface elements within the interface relative to a unit of movement / orientation change of the medical device.

[0311] At box 2504, process 2500 may include determining that the medical device has moved closer to the target location. For example, control circuitry may determine that the tip (e.g., a needle) of the medical device has moved closer to the target location based on sensor data from the medical device.

[0312] At block 2506, process 2500 may include setting a position change parameter to a second value. For example, control circuitry may set the position change parameter to a second value, wherein the second value is associated with an amount of position change of one or more interface elements within the interface that is larger than a first value for a unit of movement / orientation change for the medical device. In the example, the position change parameter may be set to a second value based on determining that the medical device has moved closer to a target location.

[0313] Figure 26 This is an exemplary flowchart of a process 2600 for updating the scaling ratio of a plane (e.g., a target plane) to an interface, according to one or more embodiments.

[0314] At block 2602, process 2600 may include identifying the scaling ratio of dimensions represented on the plane to dimensions represented on the interface. For example, control circuitry may determine the scaling ratio of the target plane's dimensions to the interface's dimensions.

[0315] At box 2604, process 2600 may include determining that the medical device has moved closer to the target location. For example, control circuitry may determine that the tip of the medical device (e.g., a needle) has moved closer to the target location based on sensor data from the medical device.

[0316] At box 2606, process 2600 may include updating the scaling ratio. For example, control circuitry may update the scaling ratio based on determining that the medical device has moved closer to the target position. The scaling ratio may decrease or increase. The scaling ratio can be used to move / represent one or more interface elements within the interface.

[0317] Figure 27 An exemplary flowchart of process 2700 for determining the coordinate system of a medical device and / or associating the movement of interface elements with the movement of the medical device, according to one or more embodiments, is shown.

[0318] At box 2702, process 2500 may include determining the world coordinate system of the robotic system associated with the robotic arm. For example, a medical system may include a robotic arm coupled to an electromagnetic field generator, and may include needles or other medical devices configured to generate sensor data based on the detection of electromagnetic fields from the electromagnetic field generator. Control circuitry may determine the coordinate system of the robotic system.

[0319] At box 2704, process 2700 may include representing the orientation of the medical device in a world coordinate system. For example, control circuitry may determine the orientation of the needle based on sensor data from the needle and / or represent the orientation of the needle in a world coordinate system.

[0320] At box 2706, process 2700 may include representing a plane in a world coordinate system. For example, control circuitry may determine the target plane and / or represent the target plane in a world coordinate system based on the target position and / or the needle's attitude.

[0321] At box 2708, process 2700 may include determining a target coordinate system for the plane represented in the world coordinate system. For example, control circuitry may determine the target coordinate system for the target plane based on the needle's attitude in the world coordinate system.

[0322] At box 2710, process 2700 may include determining the coordinate system of the medical device. For example, control circuitry may determine the coordinate system of the needle based on a target coordinate system.

[0323] At box 2712, process 2700 may include moving one or more interface elements within the interface in a direction related to the direction of movement of the medical device relative to the medical device's coordinate system. For example, control circuitry may cause one or more interface elements to move / update within the interface such that one or more interface elements move in a direction related to the direction of movement of the needle relative to the needle's coordinate system. In the example, this allows the user to see the interface elements moving left / right within the interface when the needle is tilted left / right relative to the user, and allows the user to see the interface elements moving up / down within the interface when the needle is tilted up / down relative to the user, and so on.

[0324] Figure 28 An exemplary flowchart of a process 2800 for determining the status of a medical device based on the region where the medical device is located, according to one or more embodiments, is shown.

[0325] At block 2802, process 2800 may include generating progress region data indicating multiple areas of the environment in which the medical device is located. For example, control circuitry may generate the progress region data based on sensor data from a needle and / or sensor data from a trocar. This progress region data may indicate multiple areas within the environment, each of which may be associated with a state.

[0326] At box 2804, process 2800 may include determining the area where the medical device is located from multiple regions. For example, control circuitry may determine the area where the needle is located based on sensor data and / or progress area data from the needle.

[0327] At box 2806, process 2800 may include determining the state of the medical device based on the region. For example, control circuitry may determine the state of the needle based on the region where the needle is located. This state may indicate whether the needle is within the target area (e.g., associated with reaching a target location), whether the needle is aligned with a target trajectory / attitude, whether the needle is outside the target area, whether the needle has reached the target location, whether the needle has passed the target location, etc.

[0328] At box 2808, process 2800 may include an indication causing a display status. For example, control circuitry may display a progress / alignment indication via a display device, such as by sending user interface data to a display device associated with the control system. Furthermore, the display device may display the progress / alignment indication based on the interface data. In this example, the status of the medical device and / or the indication of such status provided via the interface may be updated as the medical device changes position / or orientation, such as by moving to a different area.

[0329] Additional Implementation Plan

[0330] Depending on the implementation, certain behaviors, events, or functions of any process or algorithm described herein may be performed in different sequences, added, combined, or omitted entirely. Therefore, in some implementations, not all described behaviors or events are necessary for process practice.

[0331] Unless otherwise specifically stated or understood in the context in which they are used, conditional language used herein, such as “may,” “can,” “possibly,” “can,” “e.g.,” etc., is intended in its ordinary sense and is generally intended to convey that certain embodiments include certain features, elements, and / or steps while other embodiments do not. Therefore, such conditional language is not generally intended to imply that features, elements, and / or steps are necessary in any way for one or more embodiments, or to imply that one or more embodiments must, with or without the author’s input or prompting, include logic for determining whether such features, elements, and / or steps are included in any particular embodiment or will be performed in any particular embodiment. The terms “comprising,” “including,” “having,” etc., are synonyms used in their ordinary sense and are used inclusively in an open-ended manner, without excluding additional elements, features, behaviors, operations, etc. Additionally, the term “or” is used in its inclusive sense (rather than its exclusive sense) such that when used, for example, to connect a series of elements, the term “or” refers to one, some, or all of the elements in that series. Unless otherwise specified, conjunctions such as “at least one of X, Y, and Z” are generally understood to convey that items, terms, elements, etc., may be X, Y, or Z. Therefore, such conjunctions are generally not intended to imply that certain implementations require the existence of at least one of X, at least one of Y, and at least one of Z.

[0332] It should be understood that in the above description of the embodiments, various features are sometimes combined in a single embodiment, drawing, or description thereof in order to simplify the disclosure and aid in understanding one or more aspects of the invention. However, this method of disclosure should not be construed as reflecting an intention that any claim requires more features than those expressly set forth in that claim. Furthermore, any component, feature, or step shown and / or described in a particular embodiment herein may be applied to or used with any other embodiment. Moreover, for each embodiment, no component, feature, step, or group of components, features, or steps is necessary or indispensable. Therefore, the scope of this disclosure should not be limited to the particular embodiments described above, but should be determined solely by a fair reading of the following claims.

[0333] It should be understood that certain ordinal terms (e.g., "first" or "second") may be provided for ease of reference and do not necessarily imply physical properties or order. Therefore, as used herein, ordinal terms used to modify elements such as structures, components, operations (e.g., "first," "second," "third," etc.) do not necessarily indicate the element's priority or order relative to any other element, but rather generally distinguish the element from another element with a similar or identical name (but using ordinal terms). Furthermore, as used herein, the term "a / kind" may indicate "a / kind or more / kinds" rather than "a / kind." Additionally, operations performed "based on" conditions or events may also be performed based on one or more other conditions or events that are not explicitly stated.

[0334] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which the exemplary embodiments pertain. It should also be understood that terms (such as those defined in common dictionaries) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and not as having an idealized or overly formal meaning, unless expressly defined herein.

[0335] For ease of description, the spatial relative terms “external,” “internal,” “upper,” “lower,” “below,” “above,” “vertical,” “horizontal,” and similar terms may be used herein to describe the relationship between one element or component and another, as illustrated in the accompanying drawings. It should be understood that the spatial relative terms are intended to cover different orientations in use or operation other than those shown in the drawings. For example, in the case where the device shown in the drawings is flipped, a device positioned “below” or “below” another device may be placed “above” another device. Therefore, the illustrative term “below” may include both a lower position and an upper position. The device may also be oriented in another direction, and thus the spatial relative terms may be interpreted differently depending on the orientation.

[0336] Unless otherwise explicitly stated, comparative and / or quantitative terms (such as “less,” “more,” “greater,” etc.) are intended to encompass the concept of equivalence. For example, “less” can mean not only “less” in the strictest mathematical sense, but also “less than or equal to.”

Claims

1. A method for generating an alignment interface for a medical device, comprising: Control circuitry of the medical system: Receive sensor data, the sensor data indicating the position of at least a portion of the medical device; The orientation of the medical device is determined, at least in part, based on the sensor data; Determine the target location within the patient's organs; Determine the plane that includes the target location; The projected position of the medical device on the plane is determined at least in part based on the orientation of the medical device; Generate interface data, wherein the interface data represents one or more interface elements, and the one or more interface elements indicate a first distance between the projected position on the plane and the target position; as well as The one or more interface elements are displayed, at least in part, based on the interface data.

2. The method according to claim 1, wherein the first distance is the planar distance between the projected position on the plane and the target position, the method further comprising: Determine the planar distance.

3. The method of claim 2, wherein determining the plane includes determining the plane such that the direction of travel of the medical device is perpendicular to the plane, the method further comprising: As the orientation of the medical device changes, the orientation of the plane is updated so that the forward direction of the medical device remains perpendicular to the plane.

4. The method of claim 1, wherein determining the plane comprises: Determine a line between the distal end of the medical device and the target location, and determine a plane such that the line is perpendicular to the plane.

5. The method according to claim 4, further comprising: The orientation of the plane is maintained as the orientation of the medical device changes.

6. The method of claim 1, wherein determining the target location is based at least in part on additional sensor data from the endoscopic mirror, and determining the plane includes determining the plane such that the forward direction of the endoscopic mirror is perpendicular to the plane.

7. The method of claim 1, wherein displaying the one or more interface elements includes displaying a first interface element of the one or more interface elements at a second distance from the center of a second interface element of the one or more interface elements, the second distance being at least partially based on the first distance and an insertion distance between the tip of the medical device and the target location on the plane.

8. The method according to claim 1, further comprising: Display a progress indicator that indicates the proximity of the tip of the medical device to the target position; The progress change parameter of the progress indicator is set to a first value, the progress change parameter indicating the amount of progress change of the progress indicator relative to the movement unit of the medical device; Determine that the tip of the medical device has moved closer to the target position; as well as The progress change parameter is set to a second value, which is at least in part based on the determination that the tip of the medical device has moved closer to the target position. This second value is associated with a progress change in the progress indicator for the unit of movement of the medical device that is larger than the first value.

9. A medical system comprising: A communication interface configured to receive sensor data from a medical device configured to penetrate the human anatomy. as well as Control circuitry, which is communicatively coupled to the communication interface and configured to: The orientation of the medical device is determined, at least in part, based on the sensor data; Determine the target location within the human anatomical structure; Determine the plane that includes the target location; The projected position of the medical device on the plane is determined at least in part based on the orientation of the medical device; as well as Generate interface data, which represents one or more interface elements, and the one or more interface elements indicate the distance between the projected position on the plane and the target position.

10. The medical system according to claim 9, further comprising: An endoscope configured to enter the target location via the lumen of the human anatomy, the endoscope including sensors configured to provide additional sensor data to the communication interface; The control circuitry is configured to determine the target location based at least in part on the additional sensor data.

11. The medical system of claim 9, wherein the control circuit is configured to determine the plane such that the forward direction of the medical device is perpendicular to the plane.

12. The medical system of claim 11, wherein the control circuit is further configured to: As the orientation of the medical device changes, the orientation of the plane is updated so that the forward direction of the medical device remains perpendicular to the plane.

13. The medical system of claim 9, wherein the control circuitry is configured to determine the plane by: determining a line between the distal end of the medical device and the target location, and determining the plane such that the line is perpendicular to the plane.

14. The medical system of claim 13, wherein the control circuit is further configured to: The orientation of the plane is maintained as the orientation of the medical device changes.

15. The medical system of claim 9, wherein the medical system includes a robotic system coupled to a device configured to generate a signal, the sensor data being at least partially based on the signal, and the control circuitry is further configured to: The coordinate system of the medical device is determined at least in part based on the coordinate system of the robotic system; and This causes the one or more interface elements to move within the interface in a direction related to the direction of movement of the medical device relative to the coordinate system of the medical device.

16. The medical system of claim 9, wherein the control circuit is further configured to: The position change parameter is set to a first value, wherein the position change parameter indicates the amount of position change of the one or more interface elements within the interface relative to the movement unit of the medical device; It was determined that the medical device had moved closer to the target location; and The position change parameter is set to a second value, which is associated with a position change amount of one or more interface elements within the interface of the moving unit of the medical device that is larger than the first value.

17. The medical system of claim 9, wherein the control circuit is further configured to: This causes a progress indicator to be displayed, which indicates the proximity of the medical device to the target position; The progress change parameter of the progress indicator is set to a first value, the progress change parameter indicating the amount of progress change of the progress indicator relative to the movement unit of the medical device; Determine that the medical device has been moved closer to the target location; as well as The progress change parameter is set to a second value, which is associated with a progress change amount of the progress indicator for the moving unit of the medical device that is larger than the first value.

18. One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by control circuitry, cause the control circuitry to perform operations including: Determine the orientation of a medical device, which is configured to enter the human anatomy percutaneously; Determine the target location within the human anatomical structure; Determine the plane that includes the target location; The projected position of the medical device on the plane is determined at least in part based on the orientation of the medical device; as well as Generate interface data, which represents one or more interface elements, and the one or more interface elements indicate a first distance between the projected position on the plane and the target position.

19. One or more non-transitory computer-readable media according to claim 18, wherein the direction of travel of the medical device is perpendicular to the plane.

20. The one or more non-transitory computer-readable media of claim 19, wherein the operation further comprises: As the orientation of the medical device changes, the orientation of the plane is updated so that the forward direction of the medical device remains perpendicular to the plane.

21. One or more non-transitory computer-readable media of claim 18, wherein the plane is perpendicular to the line between the distal end of the medical device and the target location.

22. The one or more non-transitory computer-readable media of claim 21, wherein the operation further comprises: The orientation of the plane is maintained as the orientation of the medical device changes.

23. The one or more non-transitory computer-readable media of claim 18, wherein the operation further comprises: The first interface element is displayed at a second distance from the center of the second interface element among the one or more interface elements, the second distance being at least partially based on the first distance.

24. One or more non-transitory computer-readable media of claim 23, wherein the second distance is further based at least in part on the insertion distance between the tip of the medical device and the target location on the plane.

25. The one or more non-transitory computer-readable media of claim 18, wherein the operation further comprises: This causes a progress indicator to be displayed, which indicates the proximity of the medical device to the target position; The progress indicator is updated by a second amount, at least in part, based on the fact that the medical device moves closer to the target position by a first amount for the first time. as well as The progress indicator is updated by a third amount greater than the first amount, based at least in part on the fact that the medical device moves a second time closer to the target position by the first amount.

26. The one or more non-transitory computer-readable media of claim 18, wherein the operation further comprises: Generate progress area data, which indicates multiple areas of the environment of the medical device; The region where the medical device is located is determined from the plurality of regions, at least in part, based on the progress region data; The state of the medical device is determined at least in part based on the region; as well as This causes an indication to be displayed regarding the status.

27. A medical system comprising: Control circuit, the control circuit being configured to: The needle receives data from a first sensor and is configured for percutaneous insertion into the patient. The orientation of the needle is determined at least in part based on the data from the first sensor. The endoscope receives data from a second sensor, the endoscope being at least partially disposed within the anatomical cavity of the patient; The target location within the patient's organ is determined, at least in part, based on the data from the second sensor. Determine the plane that includes the target location; The projected position of the needle on the plane is determined at least in part based on the needle's posture; as well as Generate interface data, which represents one or more interface elements, and the one or more interface elements indicate the distance between the projected position on the plane and the target position.

28. The medical system of claim 27, wherein the control circuit is further configured to: Display one or more interface elements in the interface; Identify the scaling ratio between the dimensions represented on the plane and the dimensions represented on the interface; Determine that the needle has moved closer to the target position; and The scaling ratio is updated based at least in part on the determination that the needle has moved closer to the target position.

29. The medical system of claim 27, wherein the control circuit is further configured to: Set the position change parameter to a first value, the position change parameter indicating the amount of position change of the one or more interface elements in the interface relative to the position or orientation change unit of the needle; Determine that the needle has moved closer to the target position; and The position change parameter is set to a second value, at least in part, based on the determination that the needle has moved closer to the target position. The second value is associated with a position change of one or more interface elements within the interface that is larger than the first value for the position or orientation change of the needle.

30. The medical system of claim 27, wherein the medical system includes a robotic arm coupled to an electromagnetic field generator, and the first sensor data is at least partially based on an electromagnetic field from the electromagnetic field generator, and the control circuitry is further configured to: Determine the world coordinate system of the robot system associated with the robot arm; The orientation of the needle is represented in the world coordinate system; The plane is represented in the world coordinate system; The target coordinate system of the plane represented in the world coordinate system is determined at least in part based on the orientation of the needle in the world coordinate system. The coordinate system of the needle is determined at least in part based on the target coordinate system; and The one or more interface elements are moved within the interface in a direction related to the direction of movement of the needle relative to the coordinate system of the needle.

Images