Generating imaging pose recommendations

Abstract

A method for generating a pose recommendation for an external imaging device may include obtaining position data from one or more position sensors embedded in a medical instrument, the position data relating to a position of the medical instrument in a coordinate system over a period of time; determining a medical device location and a medical device tip orientation using the location data; and determining a pose recommendation for the external imaging device using the medical device location and the medical device tip orientation. A system may be configured to perform the method.

Description

[0001] Related Applications

[0002] This patent application claims priority to U.S. Provisional Application No. 63 / 347,009, filed on May 30, 2022, entitled “GENERATING IMAGING POSERE COMMENDATIONS,” the disclosure of which is hereby incorporated by reference in its entirety. Background Art

[0003] Various medical procedures involve the use of one or more devices configured to penetrate the human anatomy to reach the treatment site. Certain procedures may involve positioning a medical device within a patient and visualizing an area of ​​interest within the patient. To this end, many medical devices may include visual capabilities, such as embedded cameras or are compatible with visual probes. External imaging modalities (such as fluoroscopic / x-ray scanning devices) may provide additional contextual information of the robotic bronchoscope within the patient. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] For the purpose of illustration, various embodiments are depicted in the accompanying drawings, and should never be interpreted as limiting the scope of the present disclosure. In addition, the various features of the different disclosed embodiments may be combined to form additional embodiments as part of the present disclosure. Throughout the accompanying drawings, reference numerals may be reused to indicate the corresponding relationship between reference elements.

[0005] Figure 1 Example pose recommendation systems for performing various medical procedures in accordance with aspects of the present disclosure are illustrated.

[0006] Figure 2 is an illustration of an exemplary embodiment according to Figure 1 A diagram of the components and subsystems of the control system shown in .

[0007] Figure 3 is a flow chart illustrating a method for generating pose recommendations for an external imaging device according to an example embodiment.

[0008] Figure 4 The distal end of a medical device within an anatomical lumen is illustrated according to an exemplary embodiment.

[0009] Figure 5 is a diagram illustrating imaging vectors based on medical device positioning and medical device orientation according to an example embodiment.

[0010] Figure 6 FIG. 1 is a diagram showing projection onto the XY plane of a patient coordinate system according to an example embodiment. Figure 5 Schematic diagram of the imaging vector shown in .

[0011] Figure 7 is a diagram illustrating left-right motion based on different angle values ​​created by projected imaging vectors according to example embodiments.

[0012] Figure 8 is an illustration of a user interface including suggested imaging angles according to an example embodiment. DETAILED DESCRIPTION

[0013] The titles provided herein are for convenience only and do not necessarily affect the scope or meaning of the disclosure. Although certain exemplary embodiments are disclosed below, the subject matter extends beyond the specifically disclosed embodiments to other alternative embodiments and / or uses and modifications and equivalents thereof. Therefore, the scope of the claims that may appear herein is not limited by any of the specific embodiments described below. For example, in any method or process disclosed herein, the actions or operations of the method or process may be performed in any suitable sequence and are not necessarily limited to any specific disclosed sequence. The various operations may then be described as multiple discrete operations in a manner that may help understand certain embodiments; however, the order of description should not be interpreted as implying that these operations are dependent on the order. In addition, the structures, systems and / or devices described herein may be embodied as integrated components or separate components. For the purpose of comparing various embodiments, certain aspects and advantages of these embodiments are described. All such aspects or advantages may not necessarily be achieved by any particular embodiment. Therefore, for example, various embodiments may be performed by achieving or optimizing an advantage or a set of advantages taught herein without necessarily achieving other aspects or advantages that may also be taught or proposed herein.

[0014] Overview

[0015] The present disclosure relates to systems, devices and methods for generating pose recommendations for external imaging modalities based on position data obtained from the operation of a medical device. It should be understood that as used herein, "pose recommendation" may refer to any individual aspect or combination of aspects of a pose. Thus, a pose recommendation may include a position, an orientation (e.g., an angle), or a combination thereof.

[0016] Many medical procedures rely on an accurate representation of the patient's anatomy to navigate and control medical instruments within the anatomy. For example, in a bronchoscope, an accurate and safe biopsy may depend on accurate alignment between a steerable bronchoscope and a biopsy site such as a nodule or lesion. A robotic bronchoscope may include a navigation system to facilitate navigation of the bronchoscope to the biopsy site and provide information that helps align the end of the bronchoscope with the biopsy site. The navigation system may include a three-dimensional model of the anatomical structure. In the case of a bronchoscope, the three-dimensional model may include data about the structure of the luminal network formed by the airways of the lungs. The three-dimensional model may be generated by a preoperative computed tomography (CT) scan of the patient. During the procedure, the coordinate system of the three-dimensional model is aligned with the coordinate system of a position sensor incorporated into the bronchoscope so that the navigation system can provide an estimate of the position of the bronchoscope within the luminal network of the lungs. Examples of position sensors include automated sensors, inertial measurement units (IMUs), fiber optic shape sensors, electromagnetic (EM) sensors, and camera sensors.

[0017] Position sensors have their drawbacks when used to provide navigational functionality. For example, the accuracy of automated sensors will be affected by their miniaturized size, the accuracy of IMUs will be affected by cumulative errors, the accuracy of fiber optic shape sensors will be affected by ambient temperature, the accuracy of EM sensors will be affected by ferromagnetic objects, and the positioning accuracy of camera sensors will be affected by poor quality images. In this regard, interventional imaging modalities such as fluoroscopic / x-ray scanning devices (referred to herein as "external imaging devices") can provide additional positioning information for the robotic bronchoscope within the patient.

[0018] However, if the external imaging device (such as a fluoroscopic / x-ray scanning device) is not properly aligned with the region of interest, the resulting fluoroscopic image may not be useful for identifying and locating the target. For example, if the C-arm is looking into the face of the scope, it may be difficult for the physician to discern how much the biopsy tool has been inserted. The embodiments discussed herein may provide a solution for aligning an external imaging device with a region of interest by generating a pose recommendation for the external imaging device. Images acquired by the external imaging device at a recommended pose may facilitate a relatively improved image based on a visualization plane that contains a bird's-eye view of the medical device associated with the region of interest.

[0019] Pose Recommendation System

[0020] Figure 1An example pose recommendation system 100 for performing various medical procedures in accordance with aspects of the present disclosure is illustrated. The pose recommendation system 100 may be used, for example, for endoscopic procedures. Compared to strictly manual procedures, robotic medical solutions may provide relatively higher precision, superior control, and / or superior hand-eye coordination with respect to certain instruments. Although Figure 1 The system 100 is presented in the context of a ureteroscopic procedure, but it should be understood that the principles disclosed herein may be embodied in any type of endoscopic procedure.

[0021] The pose recommendation system 100 includes a robotic system 10 (e.g., a mobile robotic cart) configured to engage and / or control a medical device 40 (e.g., a bronchoscope) to perform a procedure on a patient 7, the medical device including a proximal handle and an axis coupled to the handle at its proximal portion. It should be understood that the medical device 40 can be any type of axis-based medical device, including an endoscope (such as a ureteroscope or bronchoscope), a catheter (such as a steerable or non-steerable catheter), a needle, a nephroscope, a laparoscope, or other type of medical device. The medical device 40 can access the patient's internal anatomical structure by direct entry (e.g., through a natural orifice) and / or by percutaneous entry via skin / tissue puncture.

[0022] The posture recommendation system 100 includes a control system 50 configured to interact with the robotic system 10, provide information about the procedure, and / or perform a variety of other operations. For example, the control system 50 may include one or more displays 56 configured to present certain information to assist the physician 5 and / or other technicians or individuals. The posture recommendation system 100 may include a table 15 configured to hold a patient 7. The system 100 may also include an electromagnetic (EM) field generator, such as a robot-mounted EM field generator 80 or EM field generator 85 mounted to the table 15 or other structure.

[0023] Although various robotic arms 12 are shown in various positions and coupled to various tools / devices, it should be understood that such configurations are shown for purposes of convenience and illustration, and that such robotic arms may have different configurations over time and / or at different points during a medical procedure. Figure 1The device / instrument shown in the figure is different from the device / instrument shown, and in some cases or time periods, one or more of these arms may not be utilized or coupled to the medical device. The instrument can be coupled to the robotic system 10 via a robotic end effector associated with the distal end of the corresponding arm 12. The term "end effector" is used herein in accordance with its broad and ordinary meaning, and may refer to any type of robotic manipulator device, component and / or assembly. The terms "robotic manipulator" and "robotic manipulator assembly" are used in accordance with their broad and ordinary meanings, and may refer to a robotic end effector and / or a sterile adapter or other adapter components coupled to the end effector (together or separately). For example, a "robotic manipulator" or "robotic manipulator assembly" may refer to an instrument device manipulator (IDM) including one or more drive outputs, whether embodied in a robotic end effector, adapter and / or other components.

[0024] In some embodiments, the physician 5 may interact with the control system 50 and / or the robotic system 10 to cause / control the robotic system 10 to advance and navigate a medical instrument shaft 40 (e.g., a scope) through the patient's anatomy to a target site and / or perform certain operations using associated instruments. The control system 50 may provide information associated with the medical instrument 40 and / or other instruments of the system 100 via the display 56, such as real-time endoscopic images captured by the medical instrument, to assist the physician 5 in navigating / controlling such instrumentation. The control system 50 may provide the physician 5 with imaging / position information based on certain positioning modalities, such as fluoroscopy, ultrasound, optical / camera imaging, EM field positioning, or other modalities, as described in detail herein.

[0025] The various scopes / shaft-type instruments disclosed herein (such as the medical device 40 of the system 100) can be configured to navigate within human anatomy, such as within a natural orifice or lumen of a human anatomy. The terms "scope" and "endoscope" are used herein according to their broad and ordinary meanings, and can refer to any type of elongated (e.g., shaft-type) medical instrument having image generation, viewing, and / or capture functionality and configured to be introduced into any type of organ, cavity, lumen, chamber, or space of the body. Scopes can include, for example, ureteroscopes (e.g., for accessing the urinary tract), laparoscopes, nephroscopes (e.g., for accessing the kidneys), bronchoscopes (e.g., for accessing airways such as bronchi), colonoscopes (e.g., for accessing the colon), arthroscopes (e.g., for accessing joints), cystoscopes (e.g., for accessing the bladder), colonoscopes (e.g., for accessing the colon and / or rectum), laparoscopes, and the like. In some cases, the scope / endoscope may include at least partially rigid and / or flexible tubes and may be sized to pass within an outer sheath, catheter, introducer, or other lumen-type device, or may be used without such a device. The endoscopes and other instruments described herein may have certain markers / sensors associated with their distal ends or other portions that are configured to be visible / detectable in a field / space associated with one or more localization (e.g., imaging) systems / modalities.

[0026] The system 100 is illustrated as including an external imaging device (e.g., a fluoroscopic system) 70, which includes an X-ray generator 75 and an image detector 74 (referred to as an "image intensifier" in some contexts; either of the components 74, 75 may be referred to as a "source" herein), which may be mounted on a movable C-arm 71. The control system 50 or other system / device may be used to store and / or manipulate images generated using the external imaging device 70. In some embodiments, the bed 15 is radiolucent so that radiation from the generator 75 can pass through the bed 15 and a target area of ​​the patient's anatomy, wherein the patient 7 is located between the ends of the C-arm 71. The structure / arm 71 of the fluoroscopic system 70 may be rotatable or fixed. The external imaging device 70 may be specifically implemented to allow viewing of real-time images to facilitate image-guided surgery. The structure / arm 71 may be selectively movable to allow the fluoroscopic panel source 74 to capture various images of the patient 7 and / or the surgical field.

[0027] exist Figure 1In the example bronchoscope configuration shown, the field generator 67 is mounted to the bed. In other example embodiments, the field generator 67 may be mounted to a robotic arm. Because the electric field generated by the electric field generator 67 may be distorted by the presence of metal or other conductive components therein, it may be desirable to position the electric field generator 67 in a manner such that other components of the system do not substantially interfere with the electric field. For example, it may be desirable to position the electric field generator 67 at least 8" or more from a support arm 71 associated with the fluoroscopy system.

[0028] System 100 (like other systems disclosed herein) can include an optical imaging source (not shown), such as a camera device (e.g., a stereo camera assembly). The optical imaging source can be configured to / used to observe a field in a surgical environment to identify certain markers disposed in the field of view. For example, in some embodiments, the imaging source can emit infrared (IR) or other frequencies of electromagnetic radiation and / or detect reflections of such radiation to identify markers including surfaces that reflect such radiation. Such optical deflections can indicate the position and / or orientation of a marker associated with a particular optical modality. System 100 can have certain markers / references that can be detected / located in one or more reference frames / coordinate systems / spaces associated with a corresponding positioning modality.

[0029] Figure 2 is an illustration of an exemplary embodiment according to Figure 1 . As described above, the control system 50 can be configured to provide various functions to assist in the execution of medical procedures. The control system 50 can communicate with the robotic system 10 via a wireless or wired connection (e.g., to control the robotic system 10). In some embodiments, the control system 50 can communicate with the robotic system 10 to receive position data related to the position of the distal end of the endoscope 40 or other instrument from the robotic system. Such positioning data can be derived using one or more sensors associated with the corresponding instrument (e.g., electromagnetic sensors, shape sensing fibers, accelerometers, gyroscopes, satellite-based positioning sensors (e.g., global positioning systems (GPS)), radio frequency transceivers, etc.) and / or at least in part based on robotic system data (e.g., arm position / pose data, known parameters or dimensions of various system components, etc.) and vision-based algorithms. In some embodiments, the control system 50 can communicate with an EM field generator to control the generation of an EM field around the patient 7 and / or in the area around the tracked instrument.

[0030] As referenced above, system 100 may include certain control circuits configured to perform certain functions described herein, including control circuit 251 of control system 50. That is, the control circuit of system 100 may be the robotic system 10, the control system 50, or some combination thereof. Thus, any reference herein to control circuitry may refer to a control circuit embodied in the robotic system, the control system, or the pose recommendation system (such as in Figure 1 The term "control circuit" is used herein according to its broad and ordinary meaning, and may refer to any collection of the following: a processor, a processing circuit, a processing module / unit, a chip, a die (e.g., a semiconductor die including one or more active and / or passive devices and / or connectivity circuits), a microprocessor, a microcontroller, a digital signal processor, a microcomputer, a central processing unit, a field programmable gate array, a programmable logic device, a state machine (e.g., a hardware state machine), a logic circuit, an analog circuit, a digital circuit, and / or any device that manipulates signals based on hard coding of circuits and / or operating instructions. The control circuit mentioned herein may also include one or more circuit substrates (e.g., printed circuit boards), conductive traces and through holes and / or mounting pads, connectors and / or components. The control circuit mentioned herein may also include one or more storage devices, which may be embodied in a single memory device, multiple memory devices, and / or embedded circuits of a device. Such data storage devices may include read-only memory, random access memory, volatile memory, non-volatile memory, static memory, dynamic memory, flash memory, cache, data storage registers and / or any device that stores digital information. It should be noted that in embodiments where the control circuit includes a hardware and / or software state machine, analog circuit, digital circuit and / or logic circuit, the data storage device / register storing any associated operating instructions may be embedded within the circuit including the state machine, the analog circuit, the digital circuit and / or the logic circuit or external to the circuit.

[0031] The control circuit 251 may include a computer-readable medium that stores and / or is configured to store hard-coded and / or operational instructions corresponding to at least some of the steps and / or functions illustrated in one or more of the figures in the present figure and / or described herein. In some cases, such a computer-readable medium may be included in an article of manufacture. The control circuit 251 may be completely maintained / set up locally or may be at least partially remotely located (e.g., indirectly communicatively coupled via a local area network and / or a wide area network). Any of the control circuits 251 may be configured to perform any aspect of the various processes disclosed herein.

[0032] Further references Figure 2, the control system 50 may include various I / O components 258 that are configured to assist the physician 5 or others in performing medical procedures. For example, the input / output (I / O) components 258 may be configured to allow user input to control / navigate the scope 40 and / or instruments within the patient 7. In some embodiments, for example, the physician 5 may provide input to the control system 50 and / or the robotic system 10, wherein in response to such input, control signals may be sent to the robotic system 10 to manipulate the scope 40 and / or other robotically controlled instruments. As discussed below, for example, with reference to Figure 8 , I / O component 258 may include circuitry that causes display 56 to present a user interface showing a recommended imaging angle as may be calculated according to the methods described in this disclosure.

[0033] The control system 50 and / or the robotic system 10 may include certain user controls (e.g., controls 55), which may include any type of user input (and / or output) device or device interface, such as one or more buttons, keys, joysticks, handheld controllers (e.g., video game type controllers), computer mice, trackpads, trackballs, control pads, and / or sensors that capture gestures and finger postures (e.g., motion sensors or cameras), touch screens, and / or interfaces / connectors therefor. Such user controls are communicatively and / or physically coupled to corresponding control circuits. The control system may include a structural tower 51 and one or more wheels 58 supporting the tower 51. The control system 50 may also include certain communication interfaces 254 and / or power interfaces 259.

[0034] The control circuit 251 may include a data repository 260 storing various types of data, such as positioning data 220. Positioning data 220 may be data representing the position of the endoscope and may be derived from data generated by the navigation system 242. Positioning data 220 may include raw data collected from and / or processed by an input device (e.g., control system 50, optical sensor, EM sensor, IDM) to generate estimated state information for the instrument and output from the navigation system 242. By way of example and not limitation, positioning data 220 may include image data 222, position sensor data 224, and robotic data 226. Image data 222 may include one or more image frames captured by a medical instrument (e.g., an imaging device at the distal end of an endoscope), and information that allows the time elapsed between pairs of frames to be determined, such as a frame rate or a timestamp. Robotic data 226 includes data related to the physical movement of a medical instrument or a portion of a medical instrument (e.g., an instrument tip or sheath) within a tubular network. Example robotic data includes command data instructing the instrument tip to reach a specific anatomical site and / or change its orientation within the tubular network (e.g., using a specific pitch, roll, yaw, insertion, and retraction of one or both of the leader and sheath), insertion data representing the insertion motion of a portion of the medical device (e.g., the instrument tip or sheath), IDM data, and mechanical data representing the mechanical motion of the slender member of the medical device (e.g., the motion of one or more pull wires, bundles, or shafts of the endoscope that drive the actual motion of the medical device within the tubular network). Position sensor data 224 may include data collected by a position sensor of the instrument (such as an EM sensor, a shape sensing fiber, etc.). The position sensor data may include data characterizing the pose of the medical device in multiple degrees of freedom, such as three degrees of freedom relative to position (e.g., x, y, z) and three degrees of angular freedom (e.g., yaw, pitch, roll) measuring the orientation of the medical device tip. The data characterizing the pose may be defined relative to a coordinate system defined by a sensor system, such as a pose defined based on an EM field generator or by a known position.

[0035] Various modules of the control circuit 251 may process the data stored in the data repository 260. For example, the control circuit 251 may include a pose recommendation module 240, a navigation system 242, and an external imaging device controller 244. The pose recommendation module 240 may be a control circuit configured to provide a navigation system 242 to an external imaging device such as a Figure 1 The operation of the pose recommendation module 240 is discussed in more detail below, but generally may involve receiving data indicating the position of the medical device and generating an imaging angle for the external imaging device based on the data.

[0036] The navigation system 242 may include control circuitry that processes the various data in the data repository to generate navigation information to an operator of the system 100. In some embodiments, the navigation system 242 may provide pose information of a medical device relative to a three-dimensional model, a region of interest for a procedure, and the like. In addition, the navigation system 242 may provide other information that may be helpful to a user of the system 100. As explained in more detail below, the navigation system 242 may provide suggested pose data for an external imaging device to an operator of the system 100 via the display 56. Doing so may allow the operator to conform the external imaging device to the suggested pose data, particularly in those embodiments where the control system 50 does not have direct control over the operation of the external imaging device 70.

[0037] The imaging device controller 244 may include control circuitry that provides feedback and / or control to the external imaging device 70. In some embodiments, the imaging device controller 244 may provide input commands to the imaging device 70 to cause the external imaging device 70 to adjust its positioning to conform to the recommended imaging angle provided by the pose recommendation module 240.

[0038] 3D model reconstruction methods and operations

[0039] The details of the operation of the exemplary pose recommendation system are now discussed. Figure 1 The pose recommendation system 100 shown in FIG. Figure 2 The methods and operations disclosed herein are described with reference to the modules and other components shown in FIG. However, it should be understood that the methods and operations may be performed by any of the components discussed herein, alone or in combination.

[0040] Figure 3 is a flow chart illustrating a method 300 for generating pose recommendations for an external imaging device according to an example embodiment. Figure 3 As shown, method 300 may start at box 310, where the posture recommendation system 100 obtains position data derived from one or more position sensors embedded in a first medical device. As described above, the medical device may include a position sensor (e.g., an automation sensor, an IMU, a fiber shape sensor, an EM sensor, and a camera sensor) that generates sensor data. The sensor data generated by the position sensor may utilize a coordinate space different from the coordinate space used to control the external imaging device. For example, in some embodiments, the position data of box 310 may be relative to the coordinate system used by the sensor system to which the position sensor belongs. In an embodiment where the position sensor is an EM sensor, the position data may be expressed according to a coordinate system defined by an EM field generator. As another example, in an embodiment where the position sensor is a shape sensing fiber or an IMU-type position sensor, the position data may be expressed according to a coordinate system relative to a known position.

[0041] Block 310 may involve transforming the position data relative to a coordinate system different from the coordinate system used by the position sensor. For example, some embodiments may transform the sensor data derived by the position sensor to the coordinate system of the patient's body. Some embodiments are capable of performing a direct transformation from the position sensor coordinate system to the coordinate system of the patient's body based on a transformation generated as part of the process of registering the position sensor to the patient's body. Other embodiments may infer a mapping between the patient coordinate space and the position sensor coordinate space (and vice versa) based on the registration between the coordinate system of the position sensor and the coordinate system of the virtual model of the patient's anatomy. The mapping may be performed by the navigation system 242. In an embodiment where the position sensor data is mapped to the coordinate system of the virtual model, block 310 may involve the navigation system performing a transformation of the position data from the coordinate system of the sensor system to the coordinate system of the virtual model.

[0042] At block 320, based on the position data obtained at block 310, the system determines the medical device position and the medical device tip orientation. Figure 1 As discussed, the robotic system 10 controls the motion of the medical device 40 and can cause linear movement of the medical device based on the linear movement of the end effector of the robotic arm that controls the medical device. Figure 1 , the robotic system 10 can control the medical device 40 and cause the joint movement at the distal segment of the medical device 40 to affect the angle change at the distal tip. The movement of the medical device (linear movement, such as insertion / retraction and joint movement) can be based on Figure 2 Furthermore, as discussed, the representation can be based on any number of coordinate systems, such as the coordinate system of a sensor system, an anatomical model, a patient, a robot, an external imaging device, etc.

[0043] therefore, Figure 3 Some embodiments of block 320 of relate to using position data over a period of time to characterize a location in which a medical device has been moved laterally within an anatomical structure and the orientation of the tip of the medical device within the anatomical structure. It should be understood that "orientation" indicates angular information of the tip of the instrument and may include overall roll, pitch, and yaw relative to the anatomical model, as well as pitch, roll, yaw within the identified branch. This angular information may be derived from a position sensor of the medical device or determined from robotic data and, if necessary, transformed to a coordinate system consistent with the patient's anatomy, such as via a transformation to a virtual model of the anatomical structure, as discussed above.

[0044] At block 330, based on the medical device position and the medical device tip orientation (as determined at block 320), the system 100 determines a pose recommendation for the external imaging device. In some embodiments, block 330 involves the system 100 determining a plane formed by the medical device position and the medical device tip orientation, and projecting a vector orthogonal to the plane to represent an external imaging device angle that produces a visualization plane containing the endoscope position and constrained by the orientation of the medical device tip. The visualization plane will then include both the endoscope position and the orientation that the medical device is pointing forward, as well as providing an axis that enables insertion of the medical device (or working channel instrument).

[0045] Medical device positioning and end-point orientation

[0046] As mentioned above Figure 3 As discussed in blocks 320 and 330 of , system 100 determines a pose recommendation for an external imaging device based on data indicating the location of a medical device within the patient's anatomy and the orientation of a tip of the medical device within the patient's anatomy (e.g., angular information such as yaw, pitch, roll). Concepts related to the generation of pose recommendations are now described in more detail. Figure 4 The distal end of a medical device within an anatomical lumen is illustrated according to an exemplary embodiment. Figure 4 , a medical device 400 including a shaft 401 is shown being navigated through an anatomical lumen 402 toward a surgical site 403 at a position 405. In some embodiments, the system achieves movement of the medical device 400 at the position 405 based on insertion movement of a distal end of a robotic arm that controls the medical device.

[0047] Upon reaching the surgical site 403, the control system 10 can bend the articulation segment 404 in the orientation marked by arrow 406 in order to guide a tool (e.g., a working channel instrument, such as, for example, a needle, brush, clamp, etc.) toward the surgical site 403. In some embodiments, the system achieves articulation of the medical device 400 toward arrow 406 based on actuation of a motor in the end effector of the robotic arm, which can, for example, induce tension on a pull wire extending through the medical device. Such tension on the pull wire can cause the articulation segment 404 to bend.

[0048] The system 100 can collect position data regarding the movement of a medical device through an anatomical lumen 402. The system can use the position data to determine the medical device position 405 and relative to the anatomical lumen 402. Figure 4406. For example, the position data may provide 6 DOF in the sensor coordinate space, which may then be mapped to the patient coordinate space based on direct registration to the patient coordinate space or based on registration to a virtual model of the anatomy and a correspondence (implicit or otherwise) between the virtual model and the patient's coordinate space.

[0049] As reference Figure 3 As described in block 330 of , as part of determining a pose recommendation, the system may determine an imaging vector relative to a medical device position and a medical device orientation. Figure 5 5 is a diagram 500 illustrating an imaging vector 506 based on a medical device position 502 and a medical device orientation 504 according to an example embodiment. Consistent with the embodiments discussed herein, the medical device position 502 and the medical device orientation 504 are vectors representing position data acquired from one or more sensors associated with the medical device over a period of time. The imaging vector 506 is a vector representing a plane orthogonal to the plane created by the medical device position 502 and the medical device orientation 504. Figure 5 As shown, the imaging vector 506 may be determined based on the cross product of the medical device position 502 and the medical device orientation 504 .

[0050] Figure 6 FIG. 1 is a diagram showing projection onto the XY plane of an external imaging device coordinate system according to an example embodiment. Figure 5 Graph 600 of imaging vector 506 shown in . Figure 6 The system 100 according to some embodiments is illustrated projecting the imaging vector 506 onto the XY plane of the imaging device coordinate system, thereby defining a projected imaging vector 606. In some embodiments, the gantry moves in a plane perpendicular to the patient's body, which is the same as the X'-Y' plane of the patient coordinate system shown in annotation 620. Since the coordinate system of the medical instrument and the external imaging device / patient coordinate system can be registered together, the systems discussed herein can determine the medical instrument tip position / orientation in the CT / patient coordinate system based on the medical instrument tip position / orientation in the instrument coordinate system.

[0051] Angle 608 created by projected imaging vector 606 and the Y-axis defines a suggested angle for an external imaging device.

[0052] In some embodiments, a suggested angle (such as may be displayed to an operator of system 100 or sent as a command to an external imaging device) may be modified based on projected angle 608 to reflect operable movement of the external imaging device to achieve a desired imaging plane. Figure 7 7 is a diagram illustrating left-right motion based on different angle values ​​(eg, angle 702 and angle 704) created by projected imaging vectors according to an example embodiment. Figure 7 As shown, for projected angle values ​​less than 90 degrees, such as angle 702 which can be measured as A degrees, the system 100 can move the external imaging device to the left by A degrees (angle 702') to achieve a right front oblique view. Moreover, for angle values ​​greater than 90 degrees, such as angle 704 which can be measured as B degrees, the system can move the external imaging device to the right by 180-B degrees (180 degrees minus B degrees; angle 704') to achieve a left front oblique view.

[0053] It should be understood that by using the medical device position and the medical device orientation to generate the imaging vector, the visualization plane will contain the medical device position. In addition, by constraining the visualization plane with the medical device orientation (e.g., the yaw, pitch, and roll of the medical device tip), the image captured by the external imaging device will contain both the medical device position and the orientation that the scope is pointing. This in turn can provide an axis that enables tool insertion.

[0054] Displays recommended imaging angles

[0055] As previously described, in embodiments where the external imaging device is integrated with the control system 50, the recommended imaging angle for the external imaging device may be sent to the external imaging device via a communication interface. However, some embodiments may include a control system that lacks the ability to actuate controls for the external imaging device. In any of these embodiments, the control system 50 may generate a notification of the recommended imaging angle to the operator. For example, some embodiments may cause a display device to present a user interface that displays the recommended imaging angle.

[0056] For embodiments lacking an integrated external imaging device, notifying the operator of the recommended imaging angle can provide actionable information to the operator of system 100 so that they can then move the external imaging device in a manner to achieve the recommended imaging angle and subsequently be in a position to obtain relatively improved imaging of the surgical site.

[0057] Additionally, for embodiments that include an integrated external imaging device, presenting the suggested imaging angle may provide information to the operator that they may confirm before proceeding, or provide contextual information about the operation of a component of the system. For example, in embodiments where the control system controls movement of the external imaging device, notification of the suggested imaging angle may allow the operator to troubleshoot whether the control system is moving the external imaging device to the suggested imaging angle as intended.

[0058] Figure 8802 is an illustration of a user interface 800 including a suggested imaging angle 802 according to an example embodiment. The user interface 800 may be presented by the control system 50 described above. By way of example and not limitation, the suggested imaging angle 802 may provide an angle and direction relative to a patient coordinate space. The suggested imaging angle 802 may be determined by any of the components and methods described herein. Figure 8 As shown, the suggested imaging angle 802 indicates that the external imaging device (eg, C-arm) is expected to move 20 degrees to the right.

[0059] It should be understood that although a graphical user interface for notifying an operator of a suggested imaging angle is discussed above, other embodiments contemplated by the present disclosure are not limited thereto. For example, some embodiments may use any other suitable notification, including sound (e.g., voice), tactile feedback, other visual notifications (e.g., visual depiction of the angle), or any combination thereof.

[0060] Implementation systems and terminology

[0061] Implementations disclosed herein provide systems, methods, and apparatus for generating pose recommendations for external imaging devices in a medical system. Various implementations described herein provide improved visualization of anatomical structures during medical procedures using a medical robot.

[0062] The system 100 may include a variety of other components. For example, the system 100 may include one or more control electronics / circuits, a power source, pneumatics, a light source, an actuator (e.g., a motor for moving a robotic arm), a memory, and / or a communication interface (e.g., for communicating with another device). In some embodiments, the memory may store computer-executable instructions that, when executed by the control circuit, cause the control circuit to perform any of the operations discussed herein. For example, the memory may store computer-executable instructions that, when executed by the control circuit, cause the control circuit to receive input and / or control signals regarding manipulation of the robotic arm, and in response, control the robotic arm to be positioned in a particular arrangement.

[0063] The various components of the system 100 may be electrically connected and / or communicatively coupled using certain connection circuits / devices / features, which may or may not be part of the control circuitry. For example, the connection features may include one or more printed circuit boards configured to facilitate the mounting and / or interconnection of at least some of the various components / circuits of the system 100. In some embodiments, two or more of the control circuitry, data storage device / memory, communication interface, power supply unit, and / or input / output (I / O) components may be electrically and / or communicatively coupled to each other.

[0064] The term "memory" is used herein according to its broad and ordinary meaning and may refer to any suitable or desired type of computer-readable medium. For example, a computer-readable medium may include one or more volatile data storage devices, non-volatile data storage devices, removable data storage devices, and / or non-removable data storage devices, which are implemented using any technology, layout and / or data structure / protocol, including any suitable or desired computer-readable instructions, data structures, program modules or other types of data.

[0065] Computer-readable media that may be implemented according to the embodiments of the present disclosure include, but are not limited to, phase change memory, static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage device, cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, or any other non-transient medium that can be used to store information for access by a computing device. As used in certain contexts herein, computer-readable media generally may not include communication media, such as modulated data signals and carrier waves. Therefore, computer-readable media should generally be understood to refer to non-transient media.

[0066] Additional Implementation

[0067] Depending on the implementation, some actions, events or functions of any one of the processes or algorithms described herein may be performed in a different order, added, merged or completely ignored. Therefore, in some embodiments, not all described actions or events are necessary for the practice of the process.

[0068] Unless otherwise specifically stated or understood in the context of use, conditional language used herein, such as "may", "can", "might", "could", "for example", etc., refers to its ordinary meaning and is generally intended to convey that certain embodiments include and other embodiments do not include certain features, elements and / or steps. Therefore, such conditional language is generally not intended to imply that one or more embodiments require features, elements and / or steps in any way, or that one or more embodiments must include logic for determining whether these features, elements and / or steps are included in any particular embodiment or whether they will be performed in any particular embodiment with or without author input or prompting. The terms "include", "comprise", "have", etc. are synonymous and are used in their ordinary sense and are used inclusively in an open-ended manner and do not exclude additional elements, features, actions, operations, etc. Moreover, the term "or" is used in its inclusive sense (rather than in its exclusive sense) so that when used, for example, to connect a series of elements, the term "or" refers to one, some, or all of the elements in the series. Unless specifically stated otherwise, transitional terms such as the phrase "at least one of X, Y, and Z" are understood in the context of general usage to convey that an item, term, element, etc. can be X, Y, or Z. Thus, such transitional terms are generally not intended to imply that certain embodiments require that at least one of X, at least one of Y, and at least one of Z each be present.

[0069] It should be understood that in the above description of the embodiments, in order to simplify the present disclosure and assist in understanding one or more of the various inventive aspects, various features are sometimes grouped together in a single embodiment, figure, or description thereof. However, this method of the present disclosure should not be understood to reflect the following intention: any claim requires more features than the features explicitly recited in that claim. In addition, any component, feature, or step illustrated and / or described in the specific embodiments herein may be applied to any other embodiment or used in conjunction with any other embodiment. In addition, for each embodiment, no component, feature, step, or group of components, features, or steps is necessary or indispensable. Therefore, the scope of the present disclosure should not be limited to the above-mentioned specific embodiments, but should only be determined by a fair reading of the following claims.

[0070] It should be understood that certain ordinal terms (e.g., "first" or "second") may be provided for ease of reference and do not necessarily imply physical characteristics or ordering. Therefore, as used herein, ordinal terms (e.g., "first," "second," "third," etc.) used to modify elements such as structures, components, operations, etc. do not necessarily indicate the priority or order of the element relative to any other element, but may generally distinguish the element from another element with a similar or identical name (but used to use the ordinal term). In addition, as used herein, the indefinite article ("a") may indicate "one or more" rather than "one." In addition, an operation performed "based on" a condition or event may also be performed based on one or more other conditions or events that are not explicitly stated.

[0071] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the example embodiments belong. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and not interpreted in an idealized or overly formal sense unless expressly defined as such herein.

[0072] For ease of description, spatially relative terms "external," "internal," "upper," "lower," "below," "above," "vertical," "horizontal," and similar terms may be used herein to describe the relationship between one element or component and another element or component illustrated in the drawings. It should be understood that the spatially relative terms are intended to cover different orientations of the device in use or operation other than the orientation depicted in the drawings. For example, where the device shown in the drawings is flipped, a device that is "below" or "under" another device may be placed "above" the other device. Thus, the illustrative term "below" may include both a lower position and an upper position. The device may also be oriented in another direction, and thus the spatially relative terms may be interpreted differently depending on the orientation.

[0073] Unless otherwise expressly stated, comparative and / or quantitative terms such as "less", "more", "greater", etc., are intended to encompass equivalent concepts. For example, "less" may mean not only "less" in the strictest mathematical sense, but also "less than or equal to".

Claims

1. A method for generating a pose recommendation for an external imaging device, the method comprising: obtaining position data from one or more position sensors embedded in the medical device, the position data relating to a position of the medical device in a coordinate system over a period of time; using the position data to determine medical device positioning and medical device tip orientation; as well as The pose recommendation is determined for the external imaging device using the medical device positioning and the medical device tip orientation. 2 . The method of claim 1 , wherein the pose recommendation comprises an angle at which the external imaging is to be pointed relative to a patient coordinate space.

3. The method of claim 1 or claim 2, further comprising generating a notification of the posture recommendation to an operator of the medical system. The method of claim 3 , wherein the notification includes a graphical representation of the pose recommendation. 5 . The method according to claim 1 , further comprising commanding the external imaging device to move according to the posture recommendation.

6. The method of claim 1 or claim 2, wherein determining the pose recommendation for the external imaging device comprises determining an imaging plane using the medical device positioning and the medical device tip orientation.

7. The method of claim 6, further comprising determining an imaging vector orthogonal to the imaging plane. The method of claim 7 , wherein the pose recommendation is based on the imaging vector relative to an XY plane of the external imaging device.

9. The method of claim 1 or claim 2, wherein the position data comprises at least one of: position sensor data, robotic data, or image data.

10. The method of claim 1 or claim 2, further comprising using the positioning of the movement of the external imaging device to adjust the recommended pose.

11. A system for generating pose recommendations for an external imaging device, the system comprising: A medical device having a distal tip; one or more electromagnetic (EM) sensors embedded in the medical device; Control circuit; and A computer readable medium having instructions which, when executed, cause the control circuit to: obtaining position data from the one or more EM sensors, the position data relating to a position of the medical device in a coordinate system over a period of time; using the position data to determine a medical device location and a medical device tip orientation, the medical device tip orientation corresponding to an angular orientation of the distal tip; and The pose recommendation is determined for the external imaging device using the medical device positioning and the medical device tip orientation.

12. The system of claim 11, wherein the medical device tip orientation corresponds to a coordinate system associated with the patient's anatomy.

13. The system of claim 11 or claim 12, further comprising generating a notification of the posture recommendation to an operator of the medical system.

14. The system of claim 11 or claim 12, wherein the external imaging device comprises an imaging device configured to capture fluoroscopic images.

15. The system of claim 11 or claim 12, further comprising commanding the external imaging device to move according to the pose recommendation.

16. The system of claim 11 or claim 12, wherein determining the pose recommendation for the external imaging device comprises determining an imaging plane using the medical device position and the medical device tip orientation.

17. The system of claim 16, further comprising determining an imaging vector orthogonal to the imaging plane.

18. The system of claim 17, wherein the pose recommendation is based on the imaging vector relative to an XY plane of the external imaging device.

19. The system of claim 11 or claim 12, wherein the position data comprises at least one of: position sensor data, robotic data, or image data.

20. The system of claim 11 or claim 12, further comprising using the direction of movement of the external imaging device to adjust the recommended pose.

21. A system for generating pose recommendations for a fluoroscopic imaging device, the system comprising: A medical device having a distal tip; one or more electromagnetic (EM) sensors embedded in the medical device; Control circuit; and A computer readable medium having instructions which, when executed, cause the control circuit to: obtaining position data from the one or more EM sensors, the position data relating to a position of the medical device in a coordinate system associated with an EM field generator over a period of time; using the position data to determine a medical device position and a medical device tip orientation corresponding to one or more angular directions of the distal tip relative to a coordinate system associated with the patient's anatomy; and The pose recommendation is determined for the fluoroscopic imaging device using the medical device positioning and the medical device tip orientation.

22. The system of claim 21, further comprising generating a notification of the posture recommendation to an operator of the medical system.

23. The system of claim 21 or claim 22, further comprising commanding the fluoroscopic imaging device to move according to the posture recommendation.

24. The system of claim 21 or claim 22, wherein determining the pose recommendation for the fluoroscopic imaging device comprises determining an imaging plane using the medical device positioning and the medical device tip orientation.

25. The system of claim 24, further comprising determining an imaging vector orthogonal to the imaging plane.

26. The system of claim 25, wherein the pose recommendation is based on the imaging vector relative to a plane of an external imaging device.

27. The system of claim 21 or claim 22, wherein the position data further comprises at least one of: robot data or image data.

28. The system of claim 21 or claim 22, further comprising using a direction of movement of the fluoroscopic imaging device to adjust the recommended pose.

Images