Generation of recommended imaging poses

Abstract

A method for generating pose recommendations for an external imaging device may include obtaining position data from one or more position sensors embedded in a medical device, where the position data relates to the position of the medical device over a period of time in a coordinate system, determining the medical device positioning and the orientation of the tip of the medical device using the position data, and determining pose recommendations for the external imaging device using the medical device positioning and the orientation of the tip of the medical device. The system may be configured to perform this method.

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application is filed on May 30, 2022, and claims the priority of U.S. Provisional Patent Application No. 63 / 347,009 entitled "GENERATING IMAGING POSE RECOMMENDATIONS", the disclosure of which is hereby incorporated by reference in its entirety.

Background Art

[0002] Various medical procedures involve the use of one or more devices configured to reach a treatment site through the human anatomical structure. Certain operative processes can include identifying the position of a medical instrument within a patient and visualizing regions of interest within the patient. To do so, many medical instruments can include visual capabilities such as an embedded camera or compatibility with a visual probe. External imaging modalities such as fluoroscopy / X - ray scanning devices can provide additional contextual information about a robotic bronchoscope within a patient's body.

Brief Description of the Drawings

[0003] Various embodiments are shown in the accompanying drawings for purposes of illustration and should in no way be construed as limiting the scope of the present disclosure. Additionally, various features of the disclosed different embodiments can be combined to form further embodiments that are part of the present disclosure. Throughout the drawings, reference numerals may be reused to indicate corresponding relationships between reference elements.

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Mode for Carrying Out the Invention

[0004] The directions provided in this specification are for convenience only and do not necessarily affect the scope or meaning of the disclosure. While certain exemplary embodiments are disclosed below, the subject matter extends to other alternative embodiments and / or uses, as well as modifications and their equivalents, beyond the specifically disclosed embodiments. Accordingly, the claims that may arise from this specification are not limited by any of the specific embodiments described below. For example, in any method or process disclosed herein, the acts or operations of the method or process may be performed in any suitable order and are not necessarily limited to any particular disclosed order. Various operations may be described sequentially as a plurality of distinct operations in a way that may be helpful in understanding a particular embodiment, however, the order of description should not be construed as implying that these operations are order-dependent. Further, the structures, systems, and / or devices described herein may be embodied as integrated components or as separate components. For purposes of comparing various embodiments, certain aspects and advantages of these embodiments are described. Not all such aspects or advantages are necessarily realized by any particular embodiment. Thus, for example, various embodiments may be carried out in a way that realizes or optimizes one advantage or group of advantages taught herein without necessarily realizing other aspects or advantages that may be taught or suggested herein as well.

[0005] Summary The present disclosure relates to systems, devices, and methods for generating pose recommendations for an external imaging modality based on position data obtained from the operation of a medical instrument. As used herein, it should be understood that "pose recommendation" may refer to any individual aspect of a pose or combinations thereof. Accordingly, a pose recommendation may include a position, orientation (e.g., angle), or a combination thereof.

[0006] Many medical procedures rely on an accurate representation of a patient's anatomical structure to navigate and control medical devices within that anatomical structure. For example, in bronchoscopy, an accurate and safe biopsy may depend on the precise alignment of a maneuverable bronchoscope with a biopsy site, such as a nodule or lesion. Robotic bronchoscopy can include a navigation system to facilitate the navigation of the bronchoscope to the biopsy site and to provide useful information when aligning the tip of the bronchoscope with the biopsy site. The navigation system may include a three-dimensional model of the anatomical structure. In the case of bronchoscopy, the three-dimensional model may include data regarding the structure of the lumen network formed by the airways of the lungs. This three-dimensional model can be generated from a patient's preoperative Computerized Tomography (CT) scan. During the procedure, the coordinate system of the three-dimensional model is aligned with the coordinate system of a position sensor incorporated into the bronchoscope, whereby the navigation system can provide an estimated position of the bronchoscope within the lung's lumen network. Examples of position sensors include robotic sensors, Inertial Measurement Units (IMUs), fiber optic shape sensors, Electromagnetic (EM) sensors, and camera sensors.

[0007] Position sensors have limitations when used to provide a navigation function. For example, the accuracy of robotic sensors is reduced by their miniaturized size, the accuracy of IMUs is reduced by cumulative error, the accuracy of fiber optic shape sensors is affected by environmental temperature, the accuracy of EM sensors is reduced by ferromagnetic materials, and the position determination accuracy of camera sensors is reduced by low-quality images. In this regard, interventional imaging modalities such as fluoroscopy / X-ray scanning devices (referred to herein as "external imaging devices") can provide additional position determination information of a robotic bronchoscope within a patient's body.

[0008] However, if an external imaging device, such as a fluoroscopy / X-ray scanning device, is not properly aligned with the region of interest, the resulting fluoroscopy image may not be useful for identifying and localizing the target. For example, when the C-arm is looking at the plane of the scope, it can be difficult for the physician to identify how far the biopsy tool has been inserted. The embodiments described herein can provide a solution for aligning an external imaging device with the region of interest by generating pose recommendations for the external imaging device. Images acquired by the external imaging device in the recommended pose can facilitate relatively improved images based on a visualization plane that includes an aerial view of the medical instrument related to the region of interest.

[0009] Pose Recommendation System FIG. 1 shows an exemplary pose recommendation system 100 for performing various medical procedures, according to aspects of the present disclosure. The pose recommendation system 100 can be used, for example, in endoscopic procedures. Robotic medical solutions can provide relatively higher accuracy, better control, and / or better visual and hand coordination for certain instruments compared to procedures performed by human hands alone. Although the system 100 of FIG. 1 is presented in the context of a bronchoscopy procedure, it should be understood that the principles disclosed herein can be implemented in any type of endoscopic procedure.

[0010] The pose recommendation system 100 includes a robotic system 10 (e.g., a mobile robotic cart) configured to engage and / or control a medical instrument 40 (e.g., a bronchoscope) that includes a proximal handle and a shaft coupled to the handle at its proximal portion, to perform a procedure on a patient 7. It should be understood that the medical instrument 40 can be any type of shaft-based medical instrument, including an endoscope (such as a ureteroscope or bronchoscope), a catheter (such as a steerable or non-steerable catheter), a needle, a nephroscope, a laparoscope, or other types of medical instruments. The medical instrument 40 may access the internal patient anatomical structure through direct access (e.g., through a natural orifice) and / or through percutaneous access via skin / tissue puncture.

[0011] The pose recommendation system 100 includes a control system 50 configured to cooperate with the robot system 10, provide information regarding a treatment, and / or perform various other operations. For example, the control system 50 can include one or more displays 56 configured to present specific information to assist the doctor 5 and / or other technicians or individuals. The pose recommendation system 100 can include a table 15 configured to hold the patient 7. The system 100 can further include an electromagnetic (EM) field generator such as a robot-mounted EM field generator 80 and / or an EM field generator 85 attached to the table 15 or other structure.

[0012] Various robotic arms 12 are shown at various positions and coupled to various tools / devices, such a configuration being shown for convenience and illustrative purposes, it being understood that such robotic arms may have different configurations over time and / or at different points during a medical procedure. Further, robotic arm 12 may be coupled to a device / instrument different from that shown in FIG. 1, and in some cases or periods, one or more of the arms may not be utilized or coupled to a medical instrument. The coupling of an instrument to the robotic system 10 may be via a robotic end effector associated with the distal end of each arm 12. The term "end effector" is used herein in its broad and ordinary sense and may refer to any type of robotic manipulator device, component, and / or assembly. The terms "robotic manipulator" and "robotic manipulator assembly" are used in their broad and ordinary sense and may refer collectively or individually to a robotic end effector and / or a sterilization adapter or other adapter component coupled to the end effector. For example, a "robotic manipulator" or "robotic manipulator assembly" may refer to an instrument device manipulator (IDM) that includes one or more drive output portions, whether embodied in a robotic end effector, an adapter, and / or other components.

[0013] In some embodiments, the physician 5 can interact with the control system 50 and / or the robotic system 10 to cause the robotic system 10 to advance and navigate a medical instrument shaft 40 (e.g., a scope) through a patient's anatomical structure to a target site and / or to perform / control a specific action using associated instruments. The control system 50 can provide information associated with the medical instrument 40 and / or other instruments of the system 100, such as real-time endoscopic images captured thereby, to the physician 5 via a display 56 to assist the physician 5 in navigating / controlling such instruments. The control system 50 may provide the physician 5 with imaging / position information based on a particular positioning modality, such as fluoroscopy, ultrasound, optical / camera imaging, EM field positioning, or other modalities, as detailed herein.

[0014] Various scope / shaft-type instruments disclosed herein, such as the medical instrument 40 of the system 100, can be configured to navigate within the human anatomical structure, such as through a natural opening or within a lumen of the human anatomical structure. The terms "scope" and "endoscope" are used herein in their broad and ordinary sense and refer to any type of elongated (e.g., shaft-type) medical instrument having an image generation, viewing, and / or capture function and configured to be introduced into any type of organ, cavity, lumen, chamber, or space of the body. A scope can include, for example, a ureteroscope (e.g., for accessing the urinary tract), a laparoscope, a nephroscope (e.g., for accessing the kidney), a bronchoscope (e.g., for accessing the airways such as the bronchi), a colonoscope (e.g., for accessing the colon), an arthroscope (e.g., for accessing a joint), a cystoscope (e.g., for accessing the bladder), a sigmoidoscope (e.g., for accessing the colon and / or rectum), a borescope, and the like. A scope / endoscope can, in some instances, comprise at least partially rigid and / or flexible tubes and can be sized to pass within an outer sheath, catheter, introducer, or other lumen-type device or can be used without such a device. The endoscopes and other instruments described herein can have specific markers / sensors associated with the distal end or other portions thereof configured to be visible / detectable within a field of view / space associated with one or more positioning (e.g., imaging) systems / modalities.

[0015] System 100 is illustrated as including an external imaging device (e.g., a fluoroscopy system) 70, which includes an X-ray generator 75 and an image detector 74 (referred to as an “image intensifier” in some contexts). Either component 74, 75 may be referred to herein as a “source” and both may be mounted on a movable C-arm 71. A control system 50 or other system / device may be used to store and / or manipulate images generated using the external imaging device 70. In some embodiments, the bed 15 is radiation transmissive such that radiation from the generator 75 can pass through the bed 15 and the target region of the patient's anatomical structure, and the patient 7 is positioned between the ends of the C-arm 71. The structure / arm 71 of the fluoroscopy system 70 may be rotatable or fixed. The external imaging device 70 may be implemented to enable viewing live images to facilitate image-guided surgery. The structure / arm 71 can be selectively movable such that various images of the patient 7 and / or the surgical field can be captured by the fluoroscopy panel source 74.

[0016] In the exemplary bronchoscopy configuration shown in FIG. 1, the field generator 67 is mounted on the bed. In other exemplary embodiments, the field generator 67 may be attached to a robotic arm. Since the electric field generated by the electric field generator 67 can be distorted by the presence of metal or other conductive components therein, it may be desirable to position the electric field generator 67 such that other components of the system do not substantially interfere with the electric field. For example, it may be desirable to position the electric field generator 67 at least 8 inches or more away from the support arm 71 associated with the fluoroscopy system.

[0017] System 100 (as with other systems disclosed herein) can include an optical imaging source (not shown), such as a camera device (e.g., a stereo camera assembly). The optical imaging source may be configured / used to view a field of view within the surgical environment and identify certain markers disposed within the field of view. For example, in some embodiments, the imaging source can emit infrared (IR) or other frequency electromagnetic radiation and / or detect the reflection of such radiation to identify markers that include surfaces that reflect such radiation. Such optical deflection can indicate the position and / or orientation of markers associated with a particular optical modality. System 100 can have particular markers / references that can be detectable / positionable in one or more reference / coordinate frames / spaces associated with each positioning modality.

[0018] FIG. 2 is a diagram showing the components and subsystems of the control system 50 shown in FIG. 1 according to an exemplary embodiment. As described above, the control system 50 can be configured to provide various functions to assist in the performance of a medical procedure. For example, the control system 50 can communicate with the robotic system 10 via a wireless or wired connection (e.g., to control the robotic system 10). In some embodiments, the control system 50 can communicate with the robotic system 10 to receive position data from the robotic system regarding the position of the distal end of the scope 40 or other instruments. Such positioning data may be derived using one or more sensors associated with each instrument (e.g., electromagnetic sensors, shape sensing fibers, accelerometers, gyroscopes, satellite-based positioning sensors (e.g., Global Positioning System (GPS)), radio frequency transceivers, etc.) and / or based at least in part on robotic system data (e.g., arm position / pose data, known parameters or dimensions of various system components, etc.) and vision-based algorithms. In some embodiments, the control system 50 can communicate with an EM field generator to control the generation of an EM field in the region around the patient 7 and / or around the instruments being tracked.

[0019] As mentioned above, system 100 can include a specific control circuit configured to perform the specific functions described herein, including control circuit 251 of control system 50. That is, the control circuit of system 100 can be part of robot system 10, control system 50, or some combination thereof. Thus, all references herein to a control circuit can refer to circuits embodied in a robot system, a control system, or any other component of a pose recommendation system such as pose recommendation system 100 shown in FIG. 1. The term "control circuit" is used herein in its broad and ordinary sense and can refer to a processor, processing circuit, processing module / unit, chip, die (e.g., a semiconductor die including one or more active and / or passive devices and / or connectivity circuits), microprocessor, microcontroller, digital signal processor, microcomputer, central processing unit, field programmable gate array, programmable logic device, state machine (e.g., a hardware state machine), logic circuit, analog circuit, digital circuit, and / or any collection of devices that operate on signals (analog and / or digital) based on hard coding and / or operational instructions of the circuit. The control circuits referred to herein may further include one or more circuit boards (e.g., printed circuit boards), conductive traces and vias, and / or mounting pads, connectors, and / or components. The control circuits referred to herein may further comprise one or more storage devices that may be embodied in a single memory device, multiple memory devices, and / or embedded circuits of the device. Such data storage devices may comprise read only memory, random access memory, volatile memory, non-volatile memory, static memory, dynamic memory, flash memory, cache memory, data storage registers, and / or any device for storing digital information.Note that in an embodiment where the control circuit includes a hardware and / or software state machine, an analog circuit, a digital circuit, and / or a logic circuit, a data storage device / register that stores any associated operation instructions may be embedded within or external to a circuit that includes a state machine, an analog circuit, a digital circuit, and / or a logic circuit.

[0020] The control circuit 251 may comprise a hard-coded and / or operation instruction storing computer-readable medium that corresponds to at least some of the steps and / or functions illustrated in one or more of this figure and / or described herein. Such a computer-readable medium may, in some instances, be included in a manufactured article. The control circuit 251 may be maintained / located entirely locally or may be located at least partially remotely (e.g., communicatively coupled indirectly via a local area network and / or a wide area network). Any of the control circuits 251 may be configured to perform any aspect of the various processes disclosed herein.

[0021] Referring still to FIG. 2, the control system 50 can include various I / O components 258 configured to assist the physician 5 or others during the execution of a medical procedure. For example, the input / output (I / O) components 258 can be configured to enable user input to control / navigate the scope 40 and / or instruments within the patient 7. In some embodiments, for example, the physician 5 can provide input to the control system 50 and / or the robotic system 10, and in response to such input, transmit a control signal to the robotic system 10 to operate the scope 40 and / or other robotic control instruments. As will be described later with reference to FIG. 8, for example, the I / O component 258 may include a circuit that renders a user interface that displays on the display 56 a recommended imaging angle that can be calculated according to the methods described in this disclosure.

[0022] The control system 50 and / or the robot system 10 may include any type of user input (and / or output) device or device interface, such as one or more buttons, keys, joysticks, handheld controllers (e.g., video game type controllers), computer mice, trackpads, trackballs, control pads, and / or sensors that capture hand gestures and finger gestures (e.g., motion sensors or cameras), touch screens, and / or interfaces / connectors therefor, and may include specific user controls (e.g., control 55). Such user controls are communicatively and / or physically coupled to respective control circuits. The control system may include a structural tower 51 and one or more wheels 58 that support the tower 51. The control system 50 may further include a specific communication interface 254 and / or a power interface 259.

[0023] The control circuit 251 may include a data store 260 that stores various types of data such as the location identification data 220. The location identification data 220 may be data representing the location within the scope, and may be derived from data generated by the navigation system 242. The location identification data 220 is collected from input devices (such as the control system 50, optical sensors, EM sensors, IDM) for generating the estimated state information of the instrument and / or processed by the input devices, and may also include the output from the navigation system 242. By way of non-limiting example, the location identification data 220 may include image data 222, position sensor data 224, and robot data 226. The image data 222 may include one or more image frames captured by a medical instrument (such as an imaging device at the distal end of an endoscope), and information such as a frame rate or time stamp that enables determination of the elapsed time between pairs of frames. The robot data 226 includes data related to the physical movement of a medical instrument or a part of the medical instrument (such as the instrument tip or sheath) within the tubular network. Exemplary robot data includes command data for instructing the instrument tip to reach a specific anatomical site within the tubular network and / or change its orientation (such as using specific pitch, roll, yaw, insertion, and retraction of one or both of the leader and sheath), insertion data representing the insertion movement of a part of the medical instrument (such as the instrument tip or sheath), IDM data, and mechanical data representing the mechanical movement of an elongate member of the medical instrument, such as the operation of one or more pull wires, tendons, or shafts of an endoscope that drive the actual movement of a medial instrument within the tubular network. The position sensor data 224 may include data collected by position sensors of the instrument such as EM sensors, shape sensing fibers. The position sensor data may include data characterizing the pose of the medical instrument in several degrees of freedom such as three degrees of freedom for position (e.g., x, y, z) and three angular degrees of freedom for the orientation of the tip of the medical instrument (e.g., yaw, pitch, roll).The data characterizing the pose can be relative to the coordinate system defined by the sensor system such that it can be defined based on the pose defined by the EM field generator or a known position.

[0024] The various modules of the control circuit 251 may process the data stored in the data store 260. For example, the control circuit 251 may include a pose recommendation module 240, a navigation system 242, and an external imaging device controller 244. The pose recommendation module 240 may be a control circuit configured to generate a proposed pose (e.g., position and / or angle) for an external imaging device such as the external imaging device 70 shown in FIG. 1. The operation of the pose recommendation module 240, which will be described in more detail below, generally may include receiving data indicating the position of the medical instrument and generating from that an imaging angle for the external imaging device.

[0025] The navigation system 242 can include a control circuit that processes various data in the data store to generate navigation information for the operator of the system 100. In some embodiments, the navigation system 242 can provide pose information of the medical instrument with respect to a three-dimensional model, a region of interest of the procedure, etc. Further, the navigation system 242 can provide other information that may be useful to the user of the system 100. As will be described in more detail below, the navigation system 242 can provide proposed pose data regarding the external imaging device to the operator of the system 100 via the display 56. This may be done so that the operator can adapt the external imaging device to the proposed pose data, particularly in embodiments where the control system 50 does not directly control the operation of the external imaging device 70.

[0026] The imaging device controller 244 may include a control circuit that provides feedback and / or control to the external imaging device 70. In some embodiments, the imaging device controller 244 may provide an input command to the imaging device 70 to adjust its positioning to conform to the recommended imaging angle provided by the pose recommendation module 240 for the external imaging device 70.

[0027] 3D Model Reconstruction Method and Operation Here, the details of the operation of an exemplary pose recommendation system will be described. The methods and operations disclosed herein will be described in relation to the pose recommendation system 100 shown in FIG. 1 and the modules and other components shown in FIG. 2. However, it should be understood that the methods and operations may be performed alone or in combination by any of the components considered herein.

[0028] FIG. 3 is a flowchart showing a method 300 for generating pose recommendations for an external imaging device according to an exemplary embodiment. As shown in FIG. 3, the method 300 may start at block 310, where the pose recommendation system 100 obtains position data derived from one or more position sensors embedded in the first medical device. As described above, the medical device may include position sensors (e.g., robotic sensors, IMUs, fiber optic shape sensors, EM sensors, and camera sensors) that generate sensor data. The sensor data generated by the position sensors may utilize a coordinate space different from the coordinate space used to control the external imaging device. For example, the position data at block 310 may, in some embodiments, be with respect to the coordinate frame used by the sensor system to which the position sensor belongs. In embodiments where the position sensor is an EM sensor, the position data may be represented according to the coordinate frame defined by the EM field generator. As another example, in embodiments where the position sensor is a shape sensing fiber or an IMU-type position sensor, the position data may be represented according to a coordinate frame with respect to a known position.

[0029] Block 310 may include converting position data with respect to a coordinate frame different from the coordinate frame used by the position sensor. For example, some embodiments can convert sensor data derived by the position sensor into the coordinate frame of the patient's body. Some embodiments may be able to perform a direct conversion from the position sensor coordinate frame to the coordinate frame of the patient's body based on a transformation generated as part of the process of aligning the position sensor with the patient's body. Other embodiments can infer a mapping between the patient coordinate space and the position sensor coordinate space (and vice versa) based on an alignment between the coordinate frame of the position sensor and the coordinate frame of the virtual model of the patient's anatomical structure. This mapping can be performed by the navigation system 242. In embodiments that map position sensor data to the coordinate frame of the virtual model, block 310 may include the navigation system performing a conversion of the position data from the coordinate system of the sensor system to the coordinate system of the virtual model.

[0030] In block 320, based on the position data obtained in block 310, the system determines the medical instrument positioning and the orientation of the distal end of the medical instrument. As described above with reference to FIG. 1, the robotic system 10 can control the movement of the medical instrument 40 and cause the linear movement of the medical instrument based on the linear movement of the end effector of the robotic arm that controls the medical instrument. Further, still referring to FIG. 1, the robotic system 10 can control the medical instrument 40 and cause articulation movement in the distal portion of the medical instrument 40 so as to affect the angular change at the distal tip. The movement of the medical instrument (linear movements such as insertion / retraction and articulation movement) can be characterized according to the data store shown in FIG. 2. Further, as will be explained, this characterization may be according to any number of coordinate systems such as the coordinate systems of the sensor system, the anatomical model, the patient, the robot, the external imaging device, etc.

[0031] Accordingly, some embodiments of block 320 in FIG. 3 involve using position data over a period of time to characterize the positioning of a medical instrument as it moves laterally within an anatomical structure and the orientation of the tip of the medical instrument within the anatomical structure. "Orientation" indicates the angular information of the instrument tip and may include the overall roll, pitch, and yaw relative to the anatomical model, as well as the pitch, roll, and yaw within the identified branch. It should be understood that this angular information may be derived from the position sensors of the medical instrument or determined from robotic data and, if necessary, may be converted to a coordinate system that matches the patient's anatomical structure, such as through conversion to a virtual model of the anatomical structure as described above.

[0032] In block 330, based on the medical instrument positioning and the orientation of the tip of the medical instrument (determined in block 320), system 100 determines a pose recommendation for the external imaging device. In some embodiments, block 330 involves system 100 determining a plane formed by the medical instrument positioning and the orientation of the tip of the medical instrument, including scope positioning, and projecting a vector perpendicular to that plane to represent the external imaging device angle for generating a visualization plane constrained by the orientation of the tip of the medical instrument. This visualization plane includes both the position of the scope and the direction the medical instrument is facing and provides an axis along which the medical instrument (or working channel instrument) is inserted.

[0033] Medical Instrument Positioning and Tip Orientation As described above with reference to blocks 320 and 330 of FIG. 3, system 100 determines pose recommendations for an external imaging device based on data indicating the positioning of a medical device within a patient's anatomical structure and the orientation of the distal end of the medical device within the patient's anatomical structure (e.g., angular information such as yaw, pitch, roll, etc.). The concept including the generation of pose recommendations will be described in more detail herein. FIG. 4 shows the distal end of a medical device within an anatomical lumen according to an exemplary embodiment. In FIG. 4, a medical device 400 comprising a shaft 401 is shown navigating through an anatomical lumen 402 towards a surgical site 403 at a positioning 405. In some embodiments, the system achieves the movement of the medical device 400 at the positioning 405 based on the insertion movement of the distal end of a robotic arm that controls the medical device.

[0034] Upon reaching the surgical site 403, the control system 10 may bend the articulation portion 404 in the direction marked by the arrow 406 in order to direct a tool (e.g., a working channel instrument such as a needle, brush, forceps, etc.) towards the surgical site 403. In some embodiments, the system achieves the articulation movement of the medical device 400 towards the arrow 406 based on the actuation of a motor within the end effector of the robotic arm, which may cause tension in a pull wire extending through the medical device, for example. This tension on the pull wire may cause the bending at the articulation portion 404.

[0035] System 100 can collect position data regarding the movement of a medical device through the anatomical lumen 402. The system can use the position data to determine the positioning 405 of the medical device and the orientation of the medical device with respect to the arrow 406 shown in FIG. 4. For example, the position data can provide 6DOF in a sensor coordinate space, which can then be mapped to a patient coordinate space based on a direct alignment to the patient coordinate space, or based on an alignment to a virtual model of the anatomical structure and a correspondence (implied or otherwise) between the coordinate space of the virtual model and the patient's coordinate space.

[0036] As described with reference to block 330 of FIG. 3, the system can determine an imaging vector with respect to the medical instrument positioning and the orientation of the medical instrument as part of determining pose recommendations. FIG. 5 is a diagram 500 showing an imaging vector 506 based on a medical instrument positioning 502 and an orientation 504 of the medical instrument, according to an exemplary embodiment. The medical instrument positioning 502 and the medical instrument orientation 504 are vectors representing position data obtained from one or more sensors associated with the medical instrument over a period of time, consistent with the embodiments described herein. The imaging vector 506 is a vector representing the normal to the plane created by the medical instrument positioning 502 and the medical instrument orientation 504. As shown in FIG. 5, the imaging vector 506 may be determined based on the cross product of the medical instrument positioning 502 and the medical instrument orientation 504.

[0037] FIG. 6 is a diagram 600 showing the imaging vector 506 shown in FIG. 5 projected onto the X-Y plane of the external imaging device coordinate system, according to an exemplary embodiment. FIG. 6 shows that in some embodiments, the system 100 projects the imaging vector 506 onto the X-Y plane of the imaging device coordinate system, thereby defining a projected imaging vector 606. In some embodiments, the gantry moves within a plane perpendicular to the patient's body, which is the same as the X'-Y' plane of the patient coordinate system shown in the callout 620. Since the coordinate system of the medical instrument and the external imaging device / patient coordinate system can be aligned together, the system described herein can determine the position / orientation of the medical instrument tip in the CT / patient coordinate system based on the position / orientation of the medical instrument tip in the instrument coordinate system.

[0038] The angle 608 created from the projected imaging vector 606 and the Y-axis defines the angle proposed to the external imaging device.

[0039] In some embodiments, the proposed angle (which can be surfaced to the operator of system 100 or sent as a command to an external imaging device) can be based on a modification of the projection angle 608 to reflect the executable movement of the external imaging device to achieve a desired imaging plane. FIG. 7 is a diagram showing left and right movements based on different values (e.g., angle 702 and angle 704) of the angle created from the projected imaging vectors according to an exemplary embodiment. As shown in FIG. 7, for a projection angle value less than 90 degrees, such as angle 702 that can be measured as A degrees, system 100 may move the external imaging device A degrees (angle 702') to the left to achieve a right anterior oblique image. Also, for an angle value greater than 90 degrees, such as angle 704 that can be measured as B degrees, the system may move the external imaging device 180 - B degrees (180 degrees - B degrees, angle 704') to the right to achieve a left anterior oblique image.

[0040] It should be understood that by using the medical instrument positioning and the orientation of the medical instrument to generate the imaging vector, the visualization plane includes the medical instrument positioning. Further, by constraining this visualization plane with the orientation of the medical instrument (e.g., yaw, pitch, and roll of the tip of the medical instrument), the image captured by the external imaging device will include both the position of the medical instrument and the direction the scope is pointing. This, in turn, can provide the axis along which tool insertion will occur.

[0041] Surfacing of the proposed imaging angle As described above, in embodiments where the external imaging device is integrated with the control system 50, the proposed imaging angle for the external imaging device may be sent to the external imaging device via the communication interface. However, some embodiments may include a control system that lacks the ability to actuate the control of the external imaging device. In any of these embodiments, the control system 50 can generate a notification of the proposed imaging angle to the operator. For example, some embodiments may render a user interface on a display device to display the proposed imaging angle.

[0042] In embodiments lacking an integrated external imaging device, notifying the operator of the proposed imaging angle may provide actionable information to the operator of system 100, such that the operator can move an external imaging device to achieve the proposed imaging angle, and thus position it to obtain a relatively improved image of the surgical site.

[0043] Furthermore, in embodiments including an integrated external imaging device, surfacing the proposed imaging angle can provide information that the operator can confirm before proceeding, or can provide context information regarding the operation of components of the system. For example, in embodiments where a control system controls the movement of an external imaging device, notification of the proposed imaging angle may enable the operator to troubleshoot whether the control system is moving the external imaging device as intended to the proposed imaging angle.

[0044] FIG. 8 is a diagram of a user interface 800 including a proposed imaging angle 802, according to an exemplary embodiment. The user interface 800 may be rendered by the control system 50 described above. By way of example and not limitation, the proposed imaging angle 802 can provide an angle and direction with respect to the patient coordinate space. The proposed imaging angle 802 may be determined by any of the components and methods described herein. As shown in FIG. 8, the proposed imaging angle 802 indicates that for external imaging, an external imaging device (e.g., a C-arm) is expected to move 20 degrees to the right.

[0045] Although the graphical user interface for notifying an operator of the proposed imaging angle has been described above, it should be understood that other embodiments contemplated by the present disclosure are not so limited. For example, some embodiments may use any other suitable notification, including sound (e.g., voice), tactile feedback, other visual notifications (e.g., visual depiction of the angle), or any combination thereof.

[0046] Implementation System and Terms The implementation forms disclosed in this specification provide a system, method, and apparatus for generating pose recommendations for an external imaging device in a medical system. The various implementation forms described herein provide improved visualization of anatomical structures during medical procedures using a medical robot.

[0047] System 100 can include various other components. For example, System 100 can include one or more control electronics / circuits, a power source, a pneumatic source, a light source, an actuator (e.g., a motor for moving a robotic arm), a memory, and / or a communication interface (e.g., for communicating with another device). In some embodiments, the memory can store computer-executable instructions that, when executed by the control circuit, cause the control circuit to perform any of the operations contemplated herein. For example, the memory can store computer-executable instructions that, when executed by the control circuit, cause the control circuit to receive input and / or control signals regarding the operation of the robotic arm and, in response, control the robotic arm to be arranged in a specific arrangement.

[0048] The various components of System 100 can be electrically and / or communicatively coupled using a certain specific connection circuit / device / feature, but those components may or may not be part of the control circuit. For example, the connection feature(s) can include one or more printed circuit boards configured to facilitate the attachment and / or interconnection of at least some of the various components / circuits of System 100. In some embodiments, two or more of the control circuit, the data storage device / memory, the communication interface, the power supply unit, and / or the input / output (I / O) components can be electrically and / or communicatively coupled to each other.

[0049] As used herein, the term "memory" is used in its broad and ordinary sense and can refer to any suitable or desirable type of computer-readable medium. For example, computer-readable media can include one or more volatile data storage devices, non-volatile data storage devices, removable data storage devices, and / or non-removable data storage devices, implemented using any technology, layout, and / or data structure / protocol, which can contain any suitable or desirable computer-readable instructions, data structures, program modules, or other types of data.

[0050] Examples of computer-readable media that can be implemented according to embodiments of the present disclosure include, without limitation, the following. Phase change memory, static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other non-transitory medium that can be used to store information for access by a computing device. As used in a particular context herein, computer-readable media may generally exclude communication media such as modulated data signals and carrier waves. Thus, computer-readable media should generally be understood to refer to non-transitory media.

[0051] Further Embodiments Depending on the embodiment, any particular act, event, or function of any of the algorithms or processes described herein may be performed in a different order, may be added, merged, or completely excluded. Thus, in certain embodiments, not all of the acts or events described are necessary for the execution of the process.

[0052] In particular, conditional language used herein such as "can", "could", "might", "may", "e.g.", and equivalents is intended in its ordinary sense, unless specifically described otherwise or understood otherwise within the context in which it is used, and generally is not intended to convey that a particular embodiment includes a particular feature, element, and / or step while other embodiments do not. Thus, such conditional language generally is not intended to imply that a feature, element, and / or step is required in any way for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether these features, elements, and / or steps are included in or implemented in any particular embodiment, whether or not user input or prompting is present. Terms such as "comprising", "including", "having", and the like are synonyms and are used in their ordinary sense and inclusively in a non-limiting manner and do not exclude additional elements, features, acts, operations, and the like. Also, the term "or", when used, for example, to connect a list of elements, is used in its inclusive sense (and not in its exclusive sense) so as to mean one, some, or all of the listed elements. Unless specifically described otherwise, connective language such as the phrase "at least one of X, Y, and Z" is understood in the context in which it is generally used to convey that an item, term, element, etc. can be any of X, Y, or Z. Thus, such connective language generally is not intended to imply that a particular embodiment requires the presence of at least one of each of X, at least one of each of Y, and at least one of each of Z.

[0053] In the above description of the embodiments, it should be understood that various features are sometimes grouped in a single embodiment, figure, or description thereof for the purpose of streamlining the present disclosure and assisting in the understanding of one or more of the various aspects of the invention. However, the methods of the present disclosure should not be construed as reflecting an intention that any claim requires more features than are expressly recited in that claim. Further, any component, feature, or step illustrated and / or described in a particular embodiment herein may be applied to or used in conjunction with any other embodiment. Further, no component, feature, step, or group of components, features, or steps is necessary or essential for each embodiment. Accordingly, it is intended that the scope of the present disclosure not be limited by the above specific embodiments, but rather be determined only by a fair reading of the following claims.

[0054] It should be understood that specific ordinal terms (e.g., "first" or "second") may be provided for ease of reference and do not necessarily imply a physical characteristic or ranking. Thus, as used herein, ordinal terms (e.g., "first," "second," "third," etc.) used to modify elements such as structures, components, operations, etc. do not necessarily indicate a priority or order of the element with respect to any other element, but rather, generally, serve to distinguish the element (apart from the use of the ordinal term) from another element having a similar or identical name. Additionally, as used herein, the indefinite articles ("a" and "an") may indicate "one or more" rather than "one." Further, an operation performed "based on" a condition or event may also be performed based on one or more other conditions or events not expressly recited.

[0055] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiments belong. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and it should be further understood that they should not be interpreted in an idealized or overly formal sense unless explicitly so defined herein.

[0056] Spatially relative terms such as "outer", "inner", "upper", "lower", "downward", "upward", "vertical", "horizontal", and similar terms may be used herein to facilitate descriptions of the relationship between one element or component and another element or component as illustrated in the drawings. It should be understood that spatially relative terms are intended to encompass different orientations of the device during use or operation in addition to the orientation depicted in the drawings. For example, if the device shown in the drawings is inverted, a device positioned "below" or "beneath" another device may be positioned "above" the other device. Thus, the exemplary term "downward" can include both lower and upper positions. The device may also be oriented in other directions, and thus, spatially relative terms may be interpreted differently depending on the orientation.

[0057] Unless otherwise specified, comparative and / or quantitative terms such as "less", "more", "greater", etc. are intended to encompass the concept of equality. For example, "less" can mean not only "less" in a strict mathematical sense but also "less than or equal to".

[0058] 〔Embodiments〕 (1) A method for generating pose recommendations for an external imaging device, comprising: obtaining position data from one or more position sensors embedded in a medical device, the position data relating to the position of the medical device over a period of time in a coordinate system; and using the position data to determine the positioning of the medical instrument and the orientation of the distal tip of the medical instrument; determining the pose recommendation for the external imaging device using the positioning of the medical instrument and the orientation of the distal tip of the medical instrument, the method comprising: (2) The method according to embodiment 1, wherein the pose recommendation includes an angle at which the external imaging should be directed with respect to the patient coordinate space. (3) The method according to embodiment 1 or 2, further comprising generating a notification of the pose recommendation to an operator of the medical system. (4) The method according to embodiment 3, wherein the notification includes a graphical representation of the pose recommendation. (5) The method according to embodiment 1 or 2, further comprising instructing the external imaging device to move according to the pose recommendation.

[0059] (6) The method according to embodiment 1 or 2, wherein determining the pose recommendation for the external imaging device includes determining an imaging plane using the positioning of the medical instrument and the orientation of the distal tip of the medical instrument. (7) The method according to embodiment 6, further comprising determining an imaging vector perpendicular to the imaging plane. (8) The method according to embodiment 7, wherein the pose recommendation is based on the imaging vector with respect to the X-Y plane of the external imaging device. (9) The method according to embodiment 1 or 2, wherein the position data includes at least one of position sensor data, robot data, or image data. (10) The method according to embodiment 1 or 2, further comprising adjusting the recommended pose using the positioning of the movement of the external imaging device.

[0060] (11) A system for generating a pose recommendation for an external imaging device, comprising: a medical instrument having a distal tip; one or more electromagnetic (EM) sensors embedded in the medical instrument; a control circuit; A computer-readable medium, and the computer-readable medium has instructions that, when executed, cause the control circuit to obtain position data from the one or more EM sensors, the position data relating to the position of the medical instrument over a period of time in a coordinate system, and use the position data to determine the positioning of the medical instrument and the orientation of the distal tip of the medical instrument, the orientation of the distal tip of the medical instrument corresponding to the angular direction of the distal tip, and use the positioning of the medical instrument and the orientation of the distal tip of the medical instrument to determine the pose recommendation for the external imaging device. A system that causes (12) The system according to embodiment 11, wherein the orientation of the distal tip of the medical instrument corresponds to a coordinate frame associated with the anatomical structure of the patient. (13) The system according to embodiment 11 or embodiment 12, further comprising generating a notification of the pose recommendation to an operator of the medical system. (14) The system according to embodiment 11 or embodiment 12, wherein the external imaging device includes an imaging device configured to capture a fluoroscopic image. (15) The system according to embodiment 11 or embodiment 12, further comprising instructing the external imaging device to move according to the pose recommendation.

[0061] (16) Determining the pose recommendation for the external imaging device includes determining an imaging plane using the positioning of the medical instrument and the orientation of the distal tip of the medical instrument. The system according to embodiment 11 or embodiment 12. (17) The system according to embodiment 16, further comprising determining an imaging vector perpendicular to the imaging plane. (18) The system according to embodiment 17, wherein the pose recommendation is based on the imaging vector with respect to the X-Y plane of the external imaging device. (19) The system according to embodiment 11 or embodiment 12, wherein the position data includes at least one of position sensor data, robot data, or image data. (20) The system according to embodiment 11 or embodiment 12, further comprising adjusting the recommended pose using a moving direction of the external imaging device.

[0062] (21) A system for generating a pose recommendation for a fluoroscopic imaging device, a medical instrument having a distal tip, one or more electromagnetic (EM) sensors embedded in the medical instrument, a control circuit, a computer-readable medium, wherein the computer-readable medium has instructions that, when executed, cause the control circuit to acquire position data from the one or more EM sensors, the position data relating to the position of the medical instrument over a period in a coordinate system associated with an EM field generator, use the position data to determine a medical instrument positioning and an orientation of a distal tip of the medical instrument, the orientation of the distal tip of the medical instrument corresponding to one or more angular directions of the distal tip relative to a coordinate system associated with an anatomical structure of a patient, and use the medical instrument positioning and the orientation of the distal tip of the medical instrument to determine the pose recommendation for the fluoroscopic imaging device. (22) The system according to embodiment 21, further comprising generating a notification of the pose recommendation to an operator of a medical system. (23) The system according to embodiment 21 or embodiment 22, further comprising instructing the fluoroscopic imaging device to move in accordance with the pose recommendation. (24) Determining the pose recommendation for the fluoroscopic imaging device includes determining an imaging plane using the medical instrument positioning and the orientation of the distal tip of the medical instrument, according to the system of embodiment 21 or embodiment 22. The system according to embodiment 24, further comprising determining an imaging vector perpendicular to the imaging plane.

[0063] (26) The system according to embodiment 25, wherein the pose recommendation is based on the imaging vector with respect to the plane of the external imaging device. (27) The system according to embodiment 21 or embodiment 22, wherein the position data further comprises at least one of robot data or image data. (28) The system according to embodiment 21 or embodiment 22, further comprising adjusting the recommended pose using the moving direction of the fluoroscopic imaging device.

Claims

1. A system for generating pose recommendations for an external imaging device, A medical device having a distal tip, One or more position sensors associated with the aforementioned medical device, A control circuit, Location data is acquired from one or more of the aforementioned location sensors. Based on the position data, the change in the position of the medical device over a certain period of time and the orientation of the tip of the medical device relative to the first coordinate frame are determined. The imaging plane formed by the change in the position of the medical device and the orientation of the tip is determined. Based on a vector perpendicular to the imaging plane, a recommended pose for the external imaging device is determined. A system comprising a control circuit configured in such a manner.

2. The first coordinate frame is associated with the anatomical structure of the patient, The system according to claim 1, wherein the recommended pose includes the angle at which the external imaging device should be directed with respect to the second coordinate frame.

3. The system according to claim 1, wherein the control circuit is further configured to generate a notification to an operator of a medical system indicating the recommended pose, the notification including a graphical representation of the recommended pose.

4. The system according to claim 1, wherein the external imaging device is configured to capture a fluorescence-detectable image.

5. The system according to claim 1, wherein the control circuit is further configured to instruct the external imaging device to move according to the recommended pose.

6. The system according to claim 1, wherein determining the recommended pose further comprises projecting the vector perpendicular to the imaging plane onto a second coordinate frame associated with the external imaging device.

7. The system according to claim 1, wherein the position data includes position sensor data, robot data, image data, or any combination thereof.

8. The system according to claim 1, wherein the control circuit is further configured to adjust the recommended pose based on the range of movement that can be performed by the external imaging device, the direction of movement of the external imaging device, the positioning of the movement of the external imaging device, or any combination thereof.

9. The external imaging device is a fluorescence imaging device, The one or more position sensors are electromagnetic (EM) sensors. The aforementioned position data relates to the position in the coordinate system associated with the EM field generator. The system according to claim 1, wherein the first coordinate frame is associated with the anatomical structure of the patient.

10. A method for generating pose recommendations for an external imaging device, Acquiring location data from one or more location sensors associated with medical devices, Based on the position data, the change in the position of the medical device over a certain period of time and the orientation of the tip of the medical device relative to the first coordinate frame are determined. The imaging plane formed by the change in the position of the medical device and the orientation of the tip is determined, A method comprising determining a recommended pose for the external imaging device based on a vector perpendicular to the imaging plane.

Images