Offset reticle for target selection in anatomical images

Abstract

Systems and method for providing a selection reticle that is offset from a touched region such that the selected portion is not obscured by line of sight can involve: displaying an image of an anatomy on a touch-enabled display; receiving user input associated with touch of the touch-enabled display at a first location; displaying a selection indicator on the touch-enabled display superimposed on a portion of the image based at least in part on the received user input, where the selection indicator is displayed at an offset relative to the first location; and determining a location of one or more features associated with the image based at least in part on a relative alignment between the selection indicator and the superimposed portion of the image.

Description

OFFSET RETICLE FOR TARGET SELECTION IN ANATOMICAL IMAGESCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority and benefit under 35 U.S.C. § 119(e) to U.S. Provisional Patent Application No. 63 / 572,082, filed March 29, 2024, which is incorporated herein by reference in its entirety.BACKGROUNDField

[0002] The present disclosure relates to the field of medical procedures, specifically focusing on identification of a displayed object. The displayed object may relate to a medical instrument, an anatomical feature, or any portion thereof.Description of Related Art

[0003] During a medical procedure, physicians are presented with various views of a subject’s anatomy. The views can include, as few examples, preoperatively generated model views, fluoroscopic image views, endoscopic image views, or the like. As part of the medical procedure, physicians may need to accurately identify an object in the views by selecting a portion depicting the object.SUMMARY

[0004] This Summary is provided to introduce in a simplified form a selection of concepts that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0005] One innovative aspect of the subject matter of this disclosure can be implemented in a controller for a medical system, including a processing system and a memory. The memory stores instructions that, when executed by the one or more processors, cause the controller to display an image of an anatomy on a touch-enabled display; receive user input associated with touch of the touch-enabled display at a first location; display a selection indicator on the touch-enabled display superimposed on a portion of the image based at least in part on the received user input, where the selection indicator is displayed at an offset relative to the first location; and determine a location of one or more features associated with the image based at least in part on a relative alignment between the selection indicator and the superimposed portion of the image.

[0006] Another innovative aspect of the subject matter of this disclosure can be implemented in a method for selecting features in images. The method includes steps of displaying an image of an anatomy on a touch-enabled display; receiving user input associated with touch of the touch-enabled display at a first location; displaying a selection indicator on the touch-enabled display superimposed on a portion of the image based at least in part on the received user input, where the selection indicator is displayed at an offset relative to the first location; and determining a location of one or more features associated with the image based at least in part on a relative alignment between the selection indicator and the superimposed portion of the image.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Various embodiments are depicted in the accompanying drawings for illustrative purposes and should in no way be interpreted as limiting the scope of the inventions. In addition, various features of different disclosed embodiments can be combined to form additional embodiments, which are part of this disclosure. Throughout the drawings, reference numbers may be reused to indicate correspondence between reference elements.

[0008] Figure 1 illustrates an example medical system, in accordance with one or more examples.

[0009] Figure 2 illustrates a schematic view of different components of the medical system of Figure 1, in accordance with one or more embodiments.

[0010] Figure 3 illustrates a block diagram depicting various positioning and / or imaging systems / modalities, in accordance with one or more examples.

[0011] Figure 4 illustrates a flow diagram illustrating an offset reticle presentation process, in accordance with one or more embodiments.

[0012] Figures 5A and 5B illustrate a first implementation of the offset reticle, in accordance with one or more embodiments.

[0013] Figure 6 illustrates a second implementation of the offset reticle, in accordance with one or more embodiments.

[0014] Figure 7 illustrates an example system including an offset reticle framework, in accordance with one or more embodiments.

[0015] Figures 8A-8E illustrate example reticle visuals, in accordance with one or more embodiments.

[0016] Figure 9 shows a block diagram of an example controller for a medicalsystem, according to some implementations.

[0017] Figure 10 shows an illustrative flowchart depicting an example operation for selecting features in images, according to some implementations.DETAILED DESCRIPTION

[0018] The headings provided herein are for convenience only and do not necessarily affect the scope or meaning of the claimed invention. Although certain preferred embodiments and examples are disclosed below, inventive subject matter extends beyond the specifically disclosed embodiments to other alternative embodiments and / or uses and to modifications and equivalents thereof. Thus, the scope of the claims that may arise herefrom is not limited by any of the particular embodiments described below. For example, in any method or process disclosed herein, the acts or operations of the method or process may be performed in any suitable sequence and are not necessarily limited to any particular disclosed sequence. Various operations may be described as multiple discrete operations in turn, in a manner that may be helpful in understanding certain embodiments; however, the order of description should not be construed to imply that these operations are order dependent. Additionally, the structures, systems, and / or devices described herein may be embodied as integrated components or as separate components. For purposes of comparing various embodiments, certain aspects and advantages of these embodiments are described. Not necessarily all such aspects or advantages are achieved by any particular embodiment. Thus, for example, various embodiments may be carried out in a manner that achieves or optimizes one advantage or group of advantages as taught herein without necessarily achieving other aspects or advantages as may also be taught or suggested herein.

[0019] Although certain spatially relative terms, such as “outer,” “inner,” “upper,” “lower,” “below,” “above,” “vertical,” “horizontal,” “top,” “bottom,” “lateral,” “upwardly,” “side,” and similar terms, are used herein to describe a spatial relationship of one device / element or anatomical structure to another device / element or anatomical structure, it is understood that these terms are used herein for ease of description to describe the positional relationship between element(s) / structures(s), such as with respect to the illustrated orientations of the drawings. It should be understood that spatially relative terms are intended to encompass different orientations of the element(s) / structures(s), in use or operation, in addition to the orientations depicted in the drawings. For example, an element / structure described as “above” another element / structure may represent a positionthat is below or beside such other element / structure with respect to alternate orientations of a subject or element / structure, and vice-versa. It should be understood that spatially relative terms, including those listed above, may be understood relative to a respective illustrated orientation of a referenced figure.

[0020] Certain reference numbers are re-used across different figures of the figure set of the present disclosure as a matter of convenience for devices, components, systems, features, and / or modules having features that may be similar in one or more respects. However, with respect to any of the embodiments disclosed herein, re-use of common reference numbers in the drawings does not necessarily indicate that such features, devices, components, or modules are identical or similar. Rather, one having ordinary skill in the art may be informed by context with respect to the degree to which usage of common reference numbers can imply similarity between referenced subject matter. Use of a particular reference number in the context of the description of a particular figure can be understood to relate to the identified device, component, aspect, feature, module, or system in that particular figure, and not necessarily to any devices, components, aspects, features, modules, or systems identified by the same reference number in another figure. Furthermore, aspects of separate figures identified with common reference numbers can be interpreted to share characteristics or to be entirely independent of one another. In some contexts, features associated with separate figures that are identified by common reference numbers are not related and / or similar with respect to at least certain aspects. Overview

[0021] The present disclosure provides systems, devices, and methods for presenting an improved user interface for selecting a target on an image. Existing touchbased target selection involves a user aligning a target indicator (e.g., a reticle, crosshairs) by touching a target (e.g., a target object, a target region, a target anatomical feature, etc.) to select or dragging the target indicator onto the target, both of which can block the line of sight from the user to the target during selection as the user finger can obscure a portion of the target corresponding to the touch beneath the operator’s finger. That is, conventional approaches find it difficult to pick a particular point in an image without simultaneously obscuring that point on a touch screen, making the selection inconvenient at best and inaccurate at worst.

[0022] An improved approach rooted in computer and robotic technology overcomes the foregoing and other disadvantages associated with conventional approaches. The present disclosure provides for an offset reticle (e.g., a reticle that is offsetfrom a touched region) that enables the user to select the target using a touch-based interaction without obscuring the line of sight from the user to the target. The user can touch any region of the reticle to drag and align the reticle with the target. Thus, the offset reticle advantageously allows selection of the target without frustration.Example Medical System

[0023] Figure 1 illustrates an example medical system 100 (also referred to as “surgical medical system 100” or “robotic medical system 100”) in accordance with one or more examples. For example, the medical system 100 can be arranged for diagnostic and / or therapeutic bronchoscopy, as shown. The medical system 100 can include and utilize a robotic system 10, which can be implemented as a robotic cart, for example. Although the medical system 100 is shown as including various cart-based systems / devices, the concepts disclosed herein can be implemented in any type of robotic system / arrangement, such as robotic systems employing rail-based components, tablebased robotic end-effectors / manipulators, etc. The robotic system 10 can comprise one or more robotic arms 12 (also referred to as “robotic positioner(s)”) configured to position or otherwise manipulate a medical instrument, such as a medical instrument 32 (e.g., a steerable endoscope or another elongate instrument having a flexible elongated body). For example, the medical instrument 32 can be advanced through a natural orifice access point (e.g., the mouth 9 of a subject 7, positioned on a table 15 in the present example) to deliver diagnostic and / or therapeutic treatment. Although described in the context of a bronchoscopy procedure, the medical system 100 can be implemented for other types of procedures, such as gastro-intestinal (GI) procedures, renal / urological / nephrological procedures, etc. The term “subject” is used herein to refer to live patient as well as any subjects to which the present disclosure may be applicable. For example, the “subject” may refer to subjects including physical anatomic models (e.g., anatomical education model, anatomical model, medical education anatomy model, etc.) used in dry runs, models in computer simulations, or the like that covers non-live patients or test subjects.

[0024] With the robotic system 10 properly positioned, the medical instrument 32 can be inserted into the subject 7 robotically, manually, or a combination thereof. In examples, the one or more robotic arms 12 and / or instrument driver(s) 28 thereof can control the medical instrument 32. The instrument driver(s) 28 can be repositionable in space by manipulating the one or more robotic arms 12 into different angles and / or positions.

[0025] The medical system 100 can also include a control system 50 (also referred to as “control tower” or “mobile tower”), described in detail below with respect to Figure 2. The control system 50 can include one or more displays 212 to provide / display / present various information related to medical procedures, such as anatomical images. The control system 50 can additionally include one or more control mechanisms, which may be a separate directional input control 216 or a graphical user interface (GUI) presented on the displays 212.

[0026] In some embodiments, the display 212 can be a touch-capable display, as shown, that may present anatomical images and allow selection thereon. Few example anatomical images can include CT images, fluoroscopic images, images of an anatomical map, or the like. With the touch-capable display, an operator 5 reviewing the images may find it convenient to identify targets (e.g., target objects or a target region of interest) within the images using a touch-based selection instead of using the directional input control 216. For example, the operator 5 may select a scope tip and / or a nodule using a touchscreen.

[0027] The control system 50 can be communicatively coupled (e.g., via wired and / or wireless connection(s)) to the robotic system 10 to provide support for controls, electronics, fluidics, optics, sensors, and / or power to the robotic system 10. Placing such functionality in the control system 50 can allow for a smaller form factor of the robotic system 10 that may be more easily adjusted and / or re-positioned by an operator 5. Additionally, the division of functionality between the robotic system 10 and the control system 50 can reduce operating room clutter and / or facilitate efficient clinical workflow.

[0028] The medical system 100 can include an electromagnetic (EM) field generator 120, which is configured to broadcast / emit an EM field that is detected by EM sensors, such as a sensor associated with the medical instrument 32. The EM field can induce small currents in coils of EM sensors (also referred to as “position sensors”), which can be analyzed to determine a pose (position and / or angle / orientation) of the EM sensors relative to the EM field generator 120. In some embodiments, the EM sensors may be positioned at a distal end of the medical instrument 32 and a pose of the distal end may be determined in connection with the pose of the EM sensors. Although EM fields and EM sensors are described in many examples herein, position sensing systems and / or sensors can be any type of position sensing systems and / or sensors, such as optical position sensing systems / sensors, image-based position sensing systems / sensors, etc.

[0029] The medical system 100 can further include an imaging system 122 (e.g., a fluoroscopic imaging system) configured to generate and / or provide / send imagedata (also referred to as “image(s)”) to another device / system. For example, the imaging system 122 can generate image data depicting anatomy of the subject 7 and provide the image data to the control system 50, robotic system 10, a network server, a cloud server, and / or another device. The imaging system 122 can comprise an emitter / energy source (e.g., X-ray source, ultrasound source, or the like) and / or detector (e.g., X-ray detector, ultrasound detector, or the like) integrated into a supporting structure (e.g., mounted on a C-shaped arm support 124), which may provide flexibility in positioning around the subject 7 to capture images from various angles without moving the subject 7. Use of the imaging system 122 can provide visualization of internal structures / anatomy, which can be used for a variety of purposes, such as navigation of the medical instrument 32 (e.g., providing images of internal anatomy to the operator 5), localization of the medical instrument 32 (e.g., based on an analysis of image data), etc. In examples, use of the imaging system 122 can enhance the efficacy and / or safety of a medical procedure, such as a bronchoscopy, by providing clear, continuous visual feedback to the operator 5.

[0030] In some examples, the imaging system 122 is a mobile device configured to move around within an environment. For instance, the imaging system 122 can be positioned next to the subject 7 (as illustrated) during a particular phase of a procedure and removed when the imaging system 122 is no longer needed. In other examples, the imaging system 122 can be part of the table 15 or other equipment in an operating environment. The imaging system(s) 122 can be implemented as a Computed Tomography (CT) machine / system, X-ray machine / system, fluoroscopy machine / system, Positron Emission Tomography (PET) machine / system, PET-CT machine / system, CT angiography machine / system, Cone-Beam CT machine / system, 3DRA machine / system, single-photon emission computed tomography (SPECT) machine / system, Magnetic Resonance Imaging (MRI) machine / system, Optical Coherence Tomography (OCT) machine / system, ultrasound machine / system, etc. In some cases, the medical system 100 includes multiple imaging system, such as a first type of imaging system and a second type of imaging system, wherein the different types of imaging systems can be used or positioned over the subject 7 during different phases / portions of a procedure depending on the needs at that time.

[0031] In some embodiments, the imaging system 122 can be configured to generate a three-dimensional (3D) model of an anatomy. For example, the imaging system 122 is configured to process multiple images (also referred to as “image data,” in some cases) to generate the 3D model. For example, the imaging device 122 can be implementedas a CT machine configured to capture / generate a series of images / image data (e.g., 2D images / slices) from different angles around the subject 7, and then use one or more algorithms to reconstruct these images / image data into a 3D model. The 3D model can be provided to the control system 50, robotic system 10, a network server, a cloud server, and / or another device, such as for processing, display, or otherwise.

[0032] In the interest of facilitating descriptions of the present disclosure, Figure 1 illustrates a respiratory system as an example anatomy. The respiratory system includes the upper respiratory tract, which comprises the nose / nasal cavity, the pharynx (i.e., throat), and the larynx (i.e., voice box). The respiratory system further includes the lower respiratory tract, which comprises the trachea 6, the lungs 4 (4r and 4i), and the various segments of the bronchial tree. The bronchial tree includes primary bronchi 71, which branch off into smaller secondary 78 and tertiary 75 bronchi, and terminate in even smaller tubes called bronchioles 77. Each bronchiole tube is coupled to a cluster of aveoli (not shown). During the inspiration phase of the respiratory cycle, air enters through the mouth and nose and travel down the throat into the trachea 6, into the lungs 4 through the right and left main bronchi 71, into the smaller bronchi airways 78, 75, into the smaller bronchiole tubes 77, and into the alveoli, where oxygen and carbon dioxide exchange takes place.

[0033] The bronchial tree is an example luminal network in which robotically- controlled instruments may be navigated and utilized in accordance with the inventive solutions presented here. However, although aspects of the present disclosure are presented in the context of luminal networks including a bronchial network of airways (e.g., lumens, branches) of a subject’s lung, some embodiments of the present disclosure can be implemented in other types of luminal networks, such as renal networks, cardiovascular networks (e.g., arteries and veins), gastrointestinal tracts, urinary tracts, etc.

[0034] In some embodiments, the imaging system 122 can be configured to capture / update / present images of the anatomy in real-time (or substantially in real-time). For example, the imaging system 122 can integrate a fluoroscopic imaging system. Fluoroscopy is a real-time imaging technique that uses X-rays to visualize moving structures inside the body. During fluoroscopy, the subject 7 may be positioned on the table 15 between an X-ray source and detector mounted on the C-shaped arm support 124 where X-ray beams are passed through a target anatomy in a continuous manner, and the resulting images are updated in real-time.

[0035] Figure 2 illustrates example components of the control system 50,robotic system 10, and medical instrument 32, in accordance with one or more examples. The control system 50 can be coupled to the robotic system 10 and operate in cooperation therewith to perform a medical procedure. For example, the control system 50 can include communication interface(s) 202 for communicating with communication interface(s) 204 of the robotic system 10 via a wireless or wired connection (e.g., to control the robotic system 10). Further, in examples, the control system 50 can communicate with the robotic system 10 to receive position / sensor data therefrom relating to the position of sensors associated with an instrument / member controlled by the robotic system 10. In some examples, the control system 50 can communicate with the EM field generator 120 to control generation of an EM field in an area around a subject 7. The control system 50 can further include a power supply interface(s) 206.

[0036] The control system 50 can include control circuitry 251 configured to cause one or more components of the medical system 100 to actuate and / or otherwise control any of the various system components, such as carriages, mounts, arms / positioners, medical instruments, imaging devices, position sensing devices, sensor, etc. Further, the control circuitry 251 can be configured to perform other functions, such as cause display of information, process data, receive input, communicate with other components / devices, and / or any other function / operation discussed herein.

[0037] The control system 50 can further include one or more input / out (I / O) components 210 configured to assist a physician or others in performing a medical procedure. For example, the one or more I / O components 210 can be configured to receive input and / or provide output to enable a user to control / navigate the medical instrument 32, the robotic system 10, and / or other instruments / devices associated with the medical system 100. The control system 50 can include one or more displays 212 to provide / display / present various information regarding a procedure. For example, the one or more displays 212 can be used to present navigation information including a virtual anatomical model of anatomy with a virtual representation of a medical instrument, image data, and / or other information. The one or more I / O components 210 can include a user input control(s) 214, which can include any type of user input (and / or output) devices or device interfaces, such as a directional input control(s) 216, touch-based input control(s) including gesture-based input control(s), motion-based input control(s), or the like. The user input control(s) 214 may include one or more buttons, keys, joysticks, handheld controllers (e.g., video-game-type controllers), computer mice, trackpads, trackballs, control pads, sensors (e.g., motion sensors or cameras) that capture hand gestures andfinger gestures, touchscreens, toggle (e.g., button) inputs, and / or interface s / connectors therefore. In examples, such input(s) can be used to generate commands for controlling medical instrument(s), robotic arm(s), and / or other components.

[0038] The control system 50 can also include data storage 218 configured to store executable instruments (e.g., computer-executable instructions) that are executable by the control circuitry 251 to cause the control circuitry 251 to perform various operations / functionality discussed herein. In examples, two or more of the components of the control system 50 can be electrically and / or communicatively coupled to each other.

[0039] The robotic system 10 can include the one or more robotic arms 12 configured to engage with and / or control, for example, the medical instrument 32 and / or other elements / components to perform one or more aspects of a procedure. As shown, each robotic arm 12 can include multiple segments 220 coupled to joints 222, which can provide multiple degrees of movement / freedom. The robotic system 10 can be configured to receive control signals from the control system 50 to perform certain operations, such as to position one or more of the robotic arms 12 in a particular manner, manipulate an instrument, and so on. In response, the robotic system 10 can control, using control circuitry 211 thereof, actuators 226 and / or other components of the robotic system 10 to perform the operations. For example, the control circuitry 211 can control insertion / retraction, articulation, roll, etc. of a shaft of the medical instrument 32 or another instrument by actuating a drive output(s) 228 of a manipulator(s) 230 (e.g., end-effectors) coupled to a base of a robotically-controllable instrument. The drive output(s) 228 can be coupled to a drive input on an associated instrument, such as an instrument base of an instrument that is coupled to the associated robotic arm 12. The robotic system 10 can include one or more power supply interfaces 232.

[0040] The robotic system 10 can include a support column 234, a base 236, and / or a console 238. The console 238 can provide one or more I / O components 240, such as a user interface for receiving user input and / or a display screen (or a dual-purpose device, such as a touchscreen) to provide the physician / user with preoperative and / or intraoperative data. The support column 234 can include an arm support 242 (also referred to as “carriage”) for supporting the deployment of the one or more robotic arms 12. The arm support 242 can be configured to vertically translate along the support column 234. Vertical translation of the arm support 242 allows the robotic system 10 to adjust the reach of the robotic arms 12 to meet a variety of table heights, subject sizes, and / or physician preferences. The base 236 can include wheel-shaped casters 244 (also referred to as“wheels 244”) that allow for the robotic system 10 to move around the operating room prior to a procedure. After reaching the appropriate position, the casters 244 can be immobilized using wheel locks to hold the robotic system 10 in place during the procedure.

[0041] The joints 222 of each robotic arm 12 can each be independently- controllable and / or provide an independent degree of freedom available for instrument navigation. In some examples, each robotic arm 12 has seven joints, and thus provides seven degrees of freedom, including “redundant” degrees of freedom. Redundant degrees of freedom can allow robotic arms 12 to be controlled to position their respective manipulators 230 at a specific position, orientation, and / or trajectory in space using different linkage positions and joint angles. This allows for the robotic system 10 to position and / or direct a medical instrument from a desired point in space while allowing the physician to move the joints 222 into a clinically advantageous position away from the patient to create greater access, while avoiding collisions.

[0042] The one or more manipulators 230 (e.g., end-effectors) can be couplable to an instrument base / handle, which can be attached using a sterile adapter component in some instances. The combination of the manipulator 230 and coupled instrument base, as well as any intervening mechanics or couplings (e.g., sterile adapter), can be referred to as a manipulator assembly, or simply a manipulator. Manipulator / manipulator assemblies can provide power and / or control interfaces. For example, interfaces can include connectors to transfer pneumatic pressure, electrical power, electrical signals, and / or optical signals from the robotic arm 12 to a coupled instrument base. Manipulator / manipulator assemblies can be configured to manipulate medical instruments (e.g., surgical tools / instruments) using techniques including, for example, direct drives, harmonic drives, geared drives, belts and / or pulleys, magnetic drives, and the like.

[0043] The robotic system 10 can also include data storage 246 configured to store executable instruments (e.g., computer-executable instructions) that are executable by the control circuitry 211 to cause the control circuitry 211 to perform various operations / functionality discussed herein. In example, two or more of the components of the robotic system 10 can be electrically and / or communicatively coupled to each other.

[0044] Data storage (including the data storage 218, data storage 246, and / or other data storage / memory) can include any suitable or desirable type of computer- readable media. For example, computer-readable media can include one or more volatile data storage devices, non-volatile data storage devices, removable data storage devices, and / or nonremovable data storage devices implemented using any technology, layout,and / or data structure(s) / protocol, including any suitable or desirable computer-readable instructions, data structures, program modules, or other types of data.

[0045] Computer-readable media that can include, but is not limited to, phase change memory, static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to store information for access by a computing device. As used in certain contexts herein, computer-readable media may not generally include communication media, such as modulated data signals and carrier waves. As such, computer-readable media should generally be understood to refer to non-transitory media.

[0046] Control circuitry (including the control circuitry 251, control circuitry 211, and / or other control circuitry) can include circuitry embodied in a robotic system, control system / tower, instrument, or any other component / device. Control circuitry can include any collection of processors, processing circuitry, processing modules / units, chips, dies (e.g., semiconductor dies including one or more active and / or passive devices and / or connectivity circuitry), microprocessors, micro-controllers, digital signal processors, microcomputers, central processing units, field-programmable gate arrays, programmable logic devices, state machines (e.g., hardware state machines), logic circuitry, analog circuitry, digital circuitry, and / or any device that manipulates signals (analog and / or digital) based on hard coding of the circuitry and / or operational instructions. Control circuitry referenced herein can further include one or more circuit substrates (e.g., printed circuit boards), conductive traces and vias, and / or mounting pads, connectors, and / or components. Control circuitry can further comprise one or more storage devices, which may be embodied in a single device, a plurality of devices, and / or embedded circuitry of a device. Such data storage can comprise read-only memory, random access memory, volatile memory, non-volatile memory, static memory, dynamic memory, flash memory, cache memory, data storage registers, and / or any device that stores digital information. In examples in which control circuitry comprises a hardware and / or software state machine, analog circuitry, digital circuitry, and / or logic circuitry, data storage device(s) / register(s) storing any associated operational instructions can be embedded within, or external to, the circuitry comprising the state machine, analog circuitry, digital circuitry, and / or logiccircuitry.

[0047] Functionality described herein can be implemented by the control circuitry 251 of the control system 50 and / or the control circuitry 211 of the robotic system 10, such as by the control circuitry 251, 211 executing executable instructions to cause the control circuitry 251, 211 to perform the functionality.

[0048] The scope assembly / medical instrument 32 includes a handle or base 31 coupled to an endoscope shaft. For example, an endoscope 40 (also referred herein as “scope” or “shaft”) can include the elongate shaft including one or more lights 49 and one or more cameras 48 or other imaging devices. The medical instrument 32 can be powered through a power interface 36 and / or controlled through a control interface 38, each or both of which may interface with a robotic arm / component of the robotic system 10. The medical instrument 32 may further comprise one or more sensors 37, such as pressure sensors and / or other force-reading sensors, which may be configured to generate signals indicating forces experienced at / by one or more components of the medical instrument 32.

[0049] The medical instrument 32 includes certain mechanisms for causing the scope 40 to articulate / deflect with respect to an axis thereof. For example, the scope 40 may have been associated with a proximal portion thereof, one or more drive inputs 34 associated, and / or integrated with one or more pulleys / spools 33 that are configured to tension / untension pull wires / tendons 45 of the scope 40 to cause articulation of the shaft 40.

[0050] The scope 40 can further include one or more working channels 44, which may be formed inside the elongate shaft and run a length of the scope 40. The working channel 44 may serve for deploying therein a medical tool 35 or a component of the medical instrument 32 (e.g., a lithotripter, a basket 35, forceps, laser, or the like) or for performing irrigation and / or aspiration, out through a distal end of the scope 40, into an operative region surrounding the distal end. The medical instrument 32 may be used in conjunction with a medical tool 35 and include various hardware and control components for the medical tool 35 and, in some instances, include the medical tool 35 as part of the medical instrument 32. For example, as shown, the medical instrument 32 can comprise a basket formed of one or more wire tines but any medical tool 35 are contemplated.Mapping. Navigation, and Positioning Modalities / Systems

[0051] Figure 3 is a block diagram illustrating a system 300 including various positioning and / or imaging systems / modalities 302-312 (sometimes referred to as“subsystems”), which can be implemented to facilitate anatomical mapping, navigation, positioning, and / or visualization for procedures in accordance with one or more examples. For example, the various systems 302-312 can be configured to provide data for generating an anatomical map, determining a location of an instrument, determining a location of a target, and / or performing other techniques.

[0052] Each of the systems 302-312 can be associated with a respective coordinate frame (also referred to as “position coordinate frame’) and / or can provide data / information relating to instrument and / or anatomy locations, wherein registering the various coordinate frames to one another can allow for integration of the various systems to provide mapping, navigation, and / or instrument visualization. For example, registration of various modalities to one another can allow for determined positions in one modality to be tracked and / or superimposed on / in a reference frame associated with another modality, thereby providing layers of positional information that can be combined to provide a robust localization system.

[0053] In examples, the system 300 configured to implement one or more localization / localizing techniques (also referred to as “localization / localizing system 300”). Localization / localizing can refer to processes of determining a location and orientation / pose of an instrument or other element / component within a given space or environment.

[0054] In various examples, the anatomical space in which a medical instrument can be localized (i.e., where position and / or shape of the instrument is determined / estimated) is a 2D or 3D portion of a subject’s tracheobronchial airways, vasculature, urinary tract, gastrointestinal tract, or any organ or space accessed via lumens. Various modalities can be implemented to provide images / representations / models of the anatomical space using various imaging techniques described in relation to the imaging system 122 of Figure 1. One or both of preoperative and intraoperative images can be acquired in connection with a procedure.

[0055] The systems 302-312 can provide information for generating a 2D or 3D anatomical model / map 314 (e.g., airway model). In examples, the anatomical map 314 and / or other localization information can be displayed to a user, such as the operating user 5, during a procedure to assist the user in performing the procedure. For example, a visualization of a tracked instrument can be superimposed on the anatomical map 314 based on position / sensor data associated with the tracked medical instrument.

[0056] As shown, the system 300 can include a surgical bed or other subjectplatform or positioning / support structure 302 (e.g., the table 15 of Figure 1). The position of the support structure 302 can be known based on data maintained relating to the position of the support structure 302 within the surgical / procedure environment. Alternatively, or additionally, the position of the support structure 302 can be sensed or otherwise determined using one or more markers and / or an appropriate imaging / positioning modality.

[0057] The system 300 can further include a robotic system 304, such as the robotic system 10 (e.g., a robotic cart or other device or system including one or more robotic end effectors). Data relating to the position and / or state of robotic arms, actuators, and / or other components of the robotic system 304 can be known or derived from robotic command data or other robotic data relative to a coordinate frame of the robotic system 304. In some examples, reference frame registration 316 occurs between the support structure 302 and the robotic system 304, which can be a relatively coarse registration (in some cases) based on robotic system / cart-set-up procedure (which can have any suitable or desirable scheme).

[0058] The system 300 can further include an electromagnetic (EM) sensor system 306, which can include an EM field generator (e.g., the EM field generator 120) and one or more EM sensors. An EM sensor can be associated with a portion of an instrument that is tracked / controlled, such as along a length of the instrument and / or other elongate member disposed in the working channel of the instrument. In some implementations, the EM field generator can be mechanically coupled to either the support structure 302 orthe robotic system 304, in which case registration / association 318 between such systems can be known and / or determined. In some implementations, the registration 318 between the EM sensor system 306 and the robotic system 304 can be determined through forward kinematics and / or field generator mount transform information. For example, the field generator can be mounted to an end effector / manipulator of the robotic system 304, such that the position of the field generator can be known relative to the robotic system positioning frame based on the known relationship between the position of the robotic end effector and the robotic system 304. The EM sensor system 306 can provide instrument pose and / or path information based on sensor readings associated with the instrument.

[0059] The system 300 can further include an optical camera system 308 including one or more cameras or other imaging devices, wherein such device(s) is / are configured to generate images of subject anatomy within a visual field thereof, such asreal-time image data during a surgical procedure. In examples, registration 320 between the optical camera system 308 and the EM sensor system 306 can be achieved through identification of features having EM sensor data associated therewith, such as by a medical instrument tip, in images generated by the optical camera system 308. The registration 320 can further be based at least in part on hand-eye interaction of the physician when viewing real-time camera images while the EM-sensor-equipped endoscope is navigating in the subject anatomy.

[0060] The system 300 can further include a computed tomography (CT) imaging system 310 configured to generate CT images of the subject anatomy, which can be done preoperatively and / or intraoperatively. In examples, image processing can be implemented for registration 322 of the CT image data with the camera image data generated by the optical camera system 308. For example, common features identified in both camera image data and CT image data can be identified to relate the CT image frame to the camera image frame in space. In some examples, the CT imaging system 310 can be used to generate preoperative imaging data for producing the anatomical map 314 and / or for path navigation planning.

[0061] In examples, the fluoroscopy imaging system 312 can be registered 324 to the CT imaging system 310 using any image processing technique suitable for such registration. In examples, the CT imaging system 310 and / or the fluoroscopy imaging system 312 can be registered 326 to the EM sensor system 306 through various techniques, such as tool registration, a transformation function, etc. In one example, a mechanical structure of the C-arm instrumentation of the system 310, 312 can have a known physical transform / relationship with respect to a mounting position of the EM field generator of the EM sensor system 306. Such known relationship can be used to register the CT / fluoroscopy image space to the EM sensor image space. The connections 328, 330 represent registrations / relationships of the CT imaging system 310 and the fluoroscopy imaging system 312 to the anatomical map 314, respectively.

[0062] The position, shape, and / or orientation of an instrument, such as an endoscope, can be determined using any one or more of the systems 302-312, which can facilitate generation of graphical interface data representing the estimated position and / or shape of the instrument relative to the anatomical map 314. The position, shape, and / or orientation of the instrument and / or anatomical map 314 can be displayed on a display device, such as via the control system 50 and / or robotic system 10, or another device. In some examples, the anatomical map 314 also indicates a position(s) of atarget(s), such asa location within the anatomy that has been designated for further treatment.

[0063] Although the systems 302-312 are discussed in a specific order, the systems can be implemented in different orders. Moreover, the systems can be used in different ways. Further, registration can occur between different systems.

[0064] In some illustrations, one or more of the systems 302-312 can be implemented to generate the anatomical map 314 preoperatively and / or determine a location of one or more targets within the anatomical map 314. Intraoperatively, one or more of the systems 302-312 can also be implemented to determine a location of a medical instrument and / or position of a target relative to the anatomical map 314. As discussed herein, one or more of the systems 302-312 can also be implemented to update the anatomical map 314, location of the medical instrument, location of the target, etc.Fluoroscopy To Confirm / Correct Registrations

[0065] Referring back to the respiratory system example of Figure 1, it was described that the imaging system 122 may generate a 3D model of the lungs 4 for preoperative planning purposes. The anatomical map 314 of Figure 3 can correspond to a 3D model of a subject’s anatomy. The operator 5 may strategize how to navigate the medical instrument 32 to a nodule 89 based on the anatomical map 314. In addition to the preoperative planning, the generated anatomical map 314 may be used as an intraoperative reference model in connection with localization of a tip of the medical instrument 32. As previously described, a pose of the tip may be estimated using a position sensor. It is noted that a pose of the tip may also be estimated using other types of sensors, such as a shape sensor, or through use of various techniques, such as using visual odometry based on endoscopic images. The pose may be localized into the anatomical map 314 such that the operator 5 may intraoperatively refer to where the tip is located within the corresponding anatomy.

[0066] However, the use of the anatomical map 314 as a reference model may come with challenges. When the anatomical map 314 is generated based on preoperatively captured images, there may be discrepancies between the anatomical map 314 and actual anatomy of the subject 7. For example, respiration of the subject 7 under anesthesia may cause discrepancies in exact airway location between actual lung 4 and the anatomical map 314 to the extent that the pose of the tip is displayed at an incorrect location. Such discrepancies may at best confuse the operator 5 and at worst complicate execution of the planned medical procedure.

[0067] Many intraoperative challenges can be addressed with real-time imaging of the anatomy that simultaneously captures both the tip and its surroundings by an imaging device positioned external to the subject 7. For instance, the system 300 can include a fluoroscopy imaging system 312 (e.g., the imaging system 122 of Figure 1) configured to generate X-ray images (e.g., 2D real-time images) of the surgical site. Although the CT imaging system 310 and fluoroscopy imaging system 312 are illustrated as separated systems, in examples the same system may perform the functionality of the CT imaging system 310 and fluoroscopy imaging system 312.

[0068] Fluoroscopy can provide real-time (or near real-time), continuous X-ray imaging of moving structures inside the body. The images produced during fluoroscopy show a dynamic view of the area being examined. This real-time imaging can enable the operator 5 to visualize the movement and position of anatomical structures or medical instruments 32 during procedures.

[0069] The real-time nature of fluoroscopic images can aid the operator 5 in confirming or correcting registration 326 with respect to other positioning or imaging systems when registered to the same coordinate system. As an example confirmation, instrument pose provided by the EM sensor system 306 with respect to the anatomical map 314 can be compared against location of the tip in the fluoroscopic images to confirm registration or detect undesirable drifts. In some embodiments, the operator 5 may manually select the location of the tip (or any feature relating to anatomy or objects) on the fluoroscopic image presented on a display for comparison with known location of the tip provided by the EM sensor system 306. As an example correction, registration 326 can be corrected in the anatomical map 314 when the anatomical map 314 presents the tip in a first branch segment but the fluoroscopic image captured shows the tip in a second branch segment different from the first branch segment. The difference in locations can be resolved in favor of the real-time fluoroscopic image (or vice versa) and the registration 330 to the anatomical map 314 can be updated to present the tip in the second branch segment.

[0070] In some embodiments, the confirmation / correction of registration may be made manually, automatically, or semi-automatically (e.g., prompting for an approval of a proposed correction). Where manual confirmation / correction is involved, the operator 5 may be presented a fluoroscopic image and prompted to identify (e.g., identify through selection) one or more known features, as shown in Figures 5A, 5B, and 6. The features may relate to discernable objects (e.g., a tip of the medical instrument 32, kidney stones,etc.) or anatomical features (e.g., a nodule, a known branch junction, etc.). The identified features can be matched to known features in the anatomical map 314 or known pose and / or path information provided by the EM sensor system 306 for the confirmation / correction of registration.

[0071] Aspects of the present disclosure provide systems, devices, and methods for presenting an improved user interface for selecting features or objects in images. In some aspects, the user interface may include an offset reticle (e.g., a reticle that is offset from a touched region of a touch-enabled display) that enables the user to select the target using a touch-based interaction without obscuring the line of sight from the user to the target. The user can touch any region of the reticle to drag and align the reticle with the target.Offset Reticle Presentation Workflow

[0072] Figure 4 illustrates a flow diagram illustrating an offset reticle presentation process 400, in accordance with one or more embodiments. The process 400, when followed, can enable touch-selection of a feature on a touch-enabled display without simultaneously obscuring the feature.

[0073] At block 402, the process 400 can involve causing the touch-enabled display to present an image of a subject’s anatomy. The image can be an image captured by the imaging system 122 of Figure 1. In particular, the image can be a real-time fluoroscopic image captured intraoperatively, such as images shown in image views 512a- c of Figures 5 A, 5B, and 6.

[0074] At block 404, the process 400 can involve causing the touch-enabled display to present a reticle (e.g., a crosshair, an offset reticle). The reticle can indicate which target or target region on the image is to be considered as a selection (i.e., a selection 520a-b within an offset reticle 522a-b of a reticle boundary 518a-b).

[0075] At block 406, the process 400 can involve receiving a touch-based selection on the image. Referring again to Figures 5A and 5B, an operator has touched a touched region 516a-b with indicated with the operator’s finger 514a-b. It is noted that the touched region 516a-b does not overlap with the selection 520a-b targeted by the offset reticle 522a-b. In other words, the touched region 516a-b and the offset reticle 522a-b are offset (e.g., translated) such that the operator’s finger 514a-b does not obscure the selection 520a-b during the touch-based selection.

[0076] Conventional touch-based interactions rely on a touch-based selectionthat positions a selection reticle underneath a finger providing the touch-based selection, thereby placing the selection reticle, the finger, and eyes in a line (e.g., the touch-based selection is within a line of sight to the selection reticle, or vice versa). In contrast, the touch-based selection of the present disclosure places the offset reticle 522a-b, the operator’s finger 514a-b, and eyes out of a line (e.g., the selection 520a-b is outside the line of sight to the reticle 522a-b).

[0077] The touch-based selection can involve a drag interaction dragging the reticle along with a drag to the touched region. As shown, between Figures 5A and 5B, the operator’s finger 514a-b drags from 514a to 514b.

[0078] At block 408, the process 400 can involve determining an amount and a direction of the drag interaction during the drag interaction. Referring again to Figures 5A and 5B, the operator’s finger 514a-b drags from an initial location in Figure 5A to a final location that is relatively lower and to the left, thereby transitioning the touched region 516a-b from 516a to 516b. The amount and direction reflective of the transition can be determined.

[0079] At block 410, the process 400 can involve causing the touch-enabled display to move the reticle based on the determined amount and direction of the drag interaction during the drag interaction. Referring again to Figures 5A and 5B, the reticle boundary 518a-b transitions from 518a to 518b (and its offset reticle 522a-b transitions from 522a to 522b) in a manner consistent with the transition of the touched region 516a- b from 516a to 516b. Similarly, the selection 520a-b under the reticles 522a-b transitions from 520a to 520b. Initially, a medical instrument 32 is not selected by the offset reticle 522a in Figure 5A, but the medical instrument 32 becomes selected by the offset reticle 522b in Figure 5B. It is noted that the operator’s finger 514b does not obscure the selection of the medical instrument 32 in Figure 5B.Example Offset Reticle Interfaces

[0080] Figures 5A, 5B, and 6 illustrate example interfaces 500, 550, 600 (sometimes referred to as “graphical user interfaces (GUIs)”), respectively, to facilitate various functionality associated with updating navigation guidance in accordance with one or more examples. The example interfaces 500, 550, 600 can provide intraoperative information regarding an instrument / target and / or to receive input regarding a location of a target / instrument within anatomy.

[0081] As shown, the interfaces 500, 550, 600 can include an instrument view502 providing image data captured from a distal end of the instrument (e.g., endoscope) within the anatomy. For example, the instrument view 502 can display real-time image data from a camera / imaging device located at a distal end of the instrument. Here, the realtime image data depicts internal anatomy, such as an anatomical lumen in which the instrument is located.

[0082] Further, the interfaces 500, 550, 600 can include a map view 504 providing a model / representation of an anatomical map 506 (e.g., a virtual representation) and instrument / target location information. The map view 504 depicts the instrument within the anatomy with an instrument representation / indicator 32, which extends through the anatomical map 506 in this example. The map view 504 also depicts a location of a target (e.g., a nodule 89) with a target representation / indicator. In this example, the map view 504 depicts 3D information, such as the anatomical map 506 in 3D, and planes 510a- c associated with image views 512a-c.

[0083] The image views 512a-c each include image data captured by an imaging system, such as a CT imaging system, X-Ray imaging system, etc. The image views 512a-c can each include image data representing a slice / layer of the internal anatomy, such as a 2D slice / layer. In examples, image data from the imaging system includes certain characteristics that are different than other forms of image data. For example, image data from the imaging system can represent / present different tissue types with different grayscale values. The image data can also include other features common to the type of imaging system used.

[0084] In the interfaces 500, 550, 600, the image data can be of fluoroscopic images. The fluoroscopic images may be captured intraoperatively as the operator 5 deems helpful. For example, the operator 5 may select a plane 510a to update the image view 512a with a new image that corresponds to the plane 510a. In some embodiments, the resulting cross-sectional image associated with the plane 510a may be presented in a secondary view 632 for comparison, as shown in Figure 6.

[0085] In relation to Figure 3, it was described that the fluoroscopic images can help confirm or correct registration errors. As shown in the interfaces 500, 550, 600 of Figures 5A, 5B, and 6, the image views 512a-c depict / show various anatomical features that can be identified by qualified individuals, such as an operator 5 trained to examine fluoroscopic images. For example, the image views 512a-c show a distal end of an medical instrument 32. While not shown, other features, such as a nodule 89, can be depicted / represented. The interfaces 500, 550, 600 can also include control / interfaceelements 508 configured to adjust one of the image views 512a-c. For example, a user can move the selected image view 512a up and down (e.g., scroll) by using the control / interface elements 508. This can allow the user to position the desired portion of the image within the center of the image view 512a or otherwise view features within the image view 512a. In some embodiments, the control / interface elements 508 can allow the user to navigate through multiple image slices associated with the anatomy.

[0086] The interfaces 500, 550, 600 can enable a user to select locations of an instrument and / or target within the image views 512a-c. For example, the image views 512a-b of Figures 5A and 5B can present a selection indicator that includes a reticle boundary 518a-b and an offset reticle 522a-b. The reticle boundary 518a-b may encompass a region of interest that includes the offset reticle 522a-b to identify a selection 520a-b within the region of interest. As described in relation to Figure 4, the selection 520a-b can be presented away or at an offset from a touched region 516a-b in a manner that positions the selection 520a-b unobscured from an operator’s finger 514a-b. The reticle boundary 518a-b indicates an interactive region of the selection indicator, such as where a user can tap and drag or otherwise manipulate the offset reticle 522a-b. The reticle boundary 518a- b and / or the offset reticle 522a-b are exemplary and any type of indicator / user interface element (e.g., a cursor, pointer, arrows, cues, circle, dot, or any other shape or form) are contemplated so long as the selection 520a-b can be presented unobscured. It will be understood that the present disclosure contemplates multiple distinct implementations.

[0087] In some aspects, the region of interest within the reticle boundary 518a- b may be magnified so that details within the region of interest can be more readily discerned. In some implementations, the magnification or scale of the region of interest may be controlled or adjusted (such as by zooming in or zooming out) via user interaction with a magnification control feature 526. In some other implementations, the magnification can be adjusted in response to a multi-touch gesture (such as pinch or spread). For example, a “pinching” gesture (such as where a user pinches two or more fingers together on the touch-enabled display) can be used to zoom out or otherwise decrease the magnification of the region of interest. On the other hand, a “spreading” gesture (such as where a user spreads two or more fingers apart on the touch-enabled display) can be used to zoom in or otherwise increase the magnification of the region of interest.

[0088] Figures 5A and 5B illustrate a first implementation where its reticle boundary 518a-b is superimposed on an image with its offset reticle 522a-b centered on a target region corresponding to a selection 520a-b. As described in relation to Figure 4, thereticle boundary 518a-b may be repositioned by a touch interaction anywhere within the reticle boundary 518a-b followed by a drag interaction.

[0089] In some embodiments, the interfaces 500, 550 may be configured such that where the touch interaction initially begins in the reticle boundary 518a-b may alter sensitivity of the drag interaction that follows. For example, if the touch interaction initially touches closer to the edge of the reticle boundary 518a-b and further from the offset reticle 522a-b (e.g., center), sensitivity may be decreased for the drag interaction (e.g., the reticle boundary 518a-b moves a centimeter for two centimeters of drag distance, providing finer or more granular adjustments). In other words, decreasing the sensitivity of the drag interaction reduces the magnitude of movement by the offset reticle 522a-b relative to the magnitude of the drag interaction. In contrast, if the touch interaction initially touches closer to the offset reticle 522a-b (e.g., center) and further from the edge of the reticle boundary 518a-b, sensitivity may be increased for the drag interaction (e.g., the reticle boundary 518a-b moves a centimeter or more for a centimeter of drag distance, providing coarser or cruder adjustments). In other words, increasing the sensitivity of the drag interaction increases the magnitude of movement by the offset reticle 522a-b relative to the magnitude of the drag interaction. The dynamic adjustment of sensitivity can enable the operator 5 pinpoint the target.

[0090] Figure 6 illustrate a second implementation where its reticle boundary (referred herein as a “displaced reticle boundary 618” to differentiate from the reticle boundary 518a-b of Figures 5 A and 5B) presents a copy of a touched region 516c at a displaced location. For example, the displaced reticle boundary 618 is presenting a magnified version of the touched region 516c underneath the operator’s finger 514c as indicated by the scope 32 in the touched region 516c and the displaced reticle 622.

[0091] In some configurations, the displaced reticle boundary 618 may remain at a fixed location, for example, fixed at where it is shown during drag interactions. In some other configurations, the displaced reticle boundary 618 may move along with the touched region 516c as dragged while displaced by an X and Y offset. The displacement for the displaced reticle 622 may be adjusted based on X and Y offsets 628 such that the displaced reticle boundary 618 can be positioned anywhere in the interface 600. In some configurations, the movement of the displaced reticle 622 may be amplified or dampened based on a sensitivity control 624.

[0092] In some embodiments, the amplified or dampened sensitivity may correspond to lead or lag, respectively, of a selected region 620 in comparison to a draginteraction of the touched region 516c. For instance, when the sensitivity control 624 is set to 1 :2 ratio (e.g., amplified), one centimeter of drag interaction for the touched region 516c will move the selected region 620 by two centimeters. In contrast, when the sensitivity control 624 is set to 4: 1 ratio (e.g., dampened), four centimeters of drag interaction for the touched region 516c will move the selected region 620 by a centimeter. Such dampened sensitivity can aid granular selection of the selected region 620. In some alternate embodiments, the amplified or dampened sensitivity may correspond to a more immediate movement of the selected region 620 or a less urgent movement of the selected region 620. In both embodiments, the movements of the selected region 620 are described without taking into consideration magnification applied to the displaced reticle boundary 618. That is, relationships between movements of the touched region 516c and the selected region 620 are dependent on the sensitivity control 624 but independent of the magnification control 526.

[0093] The interface 600 may present a reticle view 630. The reticle view 630 may present a version of the touched region 516c underneath the operator’s finger 514c in a similar manner as the displaced reticle boundary 618. This embodiment may advantageously present the touched region 516c without crowding or obscuring other regions of interest within the image view 512c. In some embodiments, a magnification control 526 can be used to adjust a magnification scale for viewing the touched region 516c as presented inside the displaced reticle boundary 618 or the reticle view 630. In some other embodiments, the magnification can be adjusted in response to a multi-touch gesture (such as pinch or spread). For example, a pinching gesture can be used to zoom out or otherwise decrease the magnification of the region of interest. On the other hand, a spreading gesture can be used to zoom in or otherwise increase the magnification of the region of interest. Still further, in some embodiments, the magnification can be adjusted in response to a single-touch gesture (such as a single-finger drag to zoom in and / or out).

[0094] In some embodiments, a touch to a location on the image view 512c may trigger presentation of a preview of the touched region 516c within the displaced reticle 622 or the reticle view 630. The preview may be a magnified preview presenting a magnified version of the touched region 516c.Offset Reticle Framework

[0095] Figure 7 illustrates an example system 700 including an offset reticle framework 702, in accordance with one or more embodiments. The offset reticleframework 702 can be configured to configure an offset reticle and cause a display to present the configured reticle for selecting a portion of an image. For example, the offset reticle framework 702 can select a reticle visual, configure various interaction parameters related to the offset reticle, and manage interactions with GUI.

[0096] As shown, the offset reticle framework 702 can include a reticle selector module 710, a reticle configurator module 720, and a GUI manager module 730. Each of the modules can implement functionalities in connection with certain aspects of the offset reticle as described herein. It should be noted that the components (e.g., modules) shown in this figure and all figures herein are exemplary only, and other implementations may include additional, fewer, integrated or different components. Some components may not be shown so as not to obscure relevant details. Furthermore, it will be understood that the architecture of the offset reticle framework 702 is modular in design and performance may be improved by improving individual modules. For example, one can improve the reticle selector module 710, the reticle configurator module 720, the GUI manager module 730, or any component modules thereof for improved performance.

[0097] In some embodiments, the various modules and / or applications described herein can be implemented, in part or in whole, as software, hardware, or any combination thereof. In general, a module and / or an application, as discussed herein, can be associated with software, hardware, or any combination thereof. In some implementations, one or more functions, tasks, and / or operations of modules and / or applications can be carried out or performed by software routines, software processes, hardware, and / or any combination thereof. In some cases, the various modules and / or applications described herein can be implemented, in part or in whole, as software running on one or more computing devices or systems, such as on a user or client computing device, on a network server or cloud servers (e.g., Software-as-a-Service (SaaS)), or a control circuitry (e.g., the control circuitry 211, 251 of Figure 2). It should be understood that there can be many variations or other possibilities.

[0098] As shown with the example system 700, the offset reticle framework 702 can be configured to communicate with one or more data stores 704. The data store 704 can be configured to store, maintain, and provide access to various types of data to support the functionality of the offset reticle framework 702. For example, the data store 704 may store, maintain, and provide access to available or selected reticle visuals and reticle configurations, various interaction parameters related to the offset reticle, or the like.

[0099] The reticle selector module 710 can be configured to enable selectionof a reticle type, configuration of reticle visuals, and / or presentation location of an offset reticle. In connection with the functionalities, the reticle selector module 710 can include any of a reticle module 712, visual module 714, and / or reticle view module 716.

[0100] The reticle module 712 can be configured to select a reticle of any shape. Figures 8A-8E illustrate a few example reticle shapes without limitations. Each reticle may have associated advantages and disadvantages. For instance, a first reticle of Figure 8A has a hollow center to allow visibility of a selected target. In contrast, a second reticle of Figure 8B and a fifth reticle of Figure 8E allow more direct identification of the selected target. A third reticle of Figure 8C can allow for more ready determination of sensitivity level, described in relation to the reticle boundary 518a-b in Figures 5 A and 5B, based on which the third reticle can be dragged. A fourth reticle of Figure 8D provides the cleanest view of a region of interest surrounding a selection. Such reticle shapes may be stored at and accessed from the data store 704. Many variations are possible including non-circular reticles.

[0101] The visual module 714 can be configured to enable configuration of reticle visuals, which may include color, shading, transparency / opacity, pattern, line thickness, brightness, texture, size, etc. of a reticle selected by the reticle module 712. The visuals may relate to the reticle or any portions thereof. For example, the center dot of the second reticle of Figure 8B may be made with lighter shading or transparency to allow visibility through the center dot.

[0102] The reticle view module 716 can be configured to toggle on or off a reticle view (e.g., the reticle view 630 of Figure 6). In some embodiments, the reticle view module 716 may configure various characteristics, including a magnification level, associated with the reticle view.

[0103] The reticle configurator module 720 can be configured to enable modification of sensitivity, modification, and / or offset. In connection with the functionalities, the reticle configurator module 720 can include any of a sensitivity module 722, magnification module 724, and / or offset module 726.

[0104] The sensitivity module 722 can be configured to manage and adjust responsiveness of an offset reticle to drag interactions, as described in relation to Figures 5A, 5B, and 6.

[0105] The magnification module 724 can be configured to manage and adjust how much of a region of interest to present within a reticle boundary (e.g., the reticle boundary 518a-b of Figures 5A and 5B). When a region of interest is magnified, themagnification module 724 may generate a magnified visual representation surrounding a selection (e.g., the selection 520a-b) and present it within the reticle boundary. While the offset reticle is overlaid on the magnified visual representation, the selection is to be determined in relation to the original, unmagnified version and / or treated consistently with respect to coordinates of the image view 512a-c.

[0106] The offset module 726 can be configured to manage and adjust an X and Y offset for a displaced reticle (e.g., the displaced reticle 622 of Figure 6).

[0107] The GUI manager module 730 can be configured to present an offset reticle to a touch-enabled display and manage interactions. In connection with the functionalities, the GUI manager module 730 can include any of a presentation module 732, interaction module 734, and / or selection translation module 736.

[0108] The presentation module 732 can be configured to cause a touch- enabled display to present an offset reticle. In some embodiments, the presentation module 732 can generate or render the offset reticle in accordance with configurations of the reticle selector module 710, including reticle type and reticle visuals.

[0109] The interaction module 734 can be configured to receive user interactions (e.g., touch-based selections) and update the offset reticle. “Touch-based selection” as referred herein will be broadly interpreted to include any touch-based control such as a tap, swipe, pinch, rotate, press and hold, multi-tap, multi-touch gestures, double tap, swipe and hold, touch keyboard input, or the like as well as touch-like controls including gestures and motions. As an example interaction, user interactions can involve touch-and-drag interactions, described in relation to Figures 4, 5A, 5B, and 6, decoded to update position of the offset reticle. As another example, user interactions can involve determining a level of sensitivity to be applied to a drag interaction (e.g., a location of the initial touch is interpreted for sensitivity of a following drag interaction based on proximity of the touch to a center of the offset reticle) described in relation to Figures 5A and 5B is triggered. As yet another example, user interactions can involve multi-touch pinch gesture or spread gesture to change magnification (e.g., zoom out and zoom in, respectively) of the offset reticle. Many variations are possible.

[0110] The selection translation module 736 can be configured to receive a selection on a presented image and translate the selection into coordinates in one or more coordinate systems. For example, the selection can be originally within coordinates of a fluoroscopic imaging system. The coordinates can be translated into coordinates in a global coordinate system to which all coordinate systems are registered into based on respectiveregistrations, as described in relation to Figure 3. The medical system 100 can use the translated selection (e.g., coordinates in the global coordinate system) to confirm or correct registrations of other systems, as described in relation to Figure 3.

[0111] Figure 9 shows a block diagram of an example controller 900 for a medical system, according to some implementations. In some implementations, the controller 900 may be one example of the control system 50 of Figures 1 and 2 or any of the control circuitry 251 and / or 211 of Figure 2. More specifically, the controller 900 is configured to determine locations of features associated with images based on user input.

[0112] The controller 900 includes a communication interface 910, a processing system 920, and a memory 930. The communication interface 910 is configured to communicate with one or more components of the medical system. More specifically, the communication interface 910 includes an image interface (I / F) 912 for communicating with one or more imaging systems (such as the CT imaging system 310 and / or the fluoroscopy imaging system 312 of Figure 3) and a display interface 914 for communicating with a touch-enabled display (such as the display 212 and / or input controls 214 of Figure 2). In some implementations, the display interface 914 may receive user input associated with touch of the touch-enabled display at a first location.

[0113] The memory 930 may include a non-transitory computer-readable medium (including one or more nonvolatile memory elements, such as EPROM, EEPROM, Flash memory, or a hard drive, among other examples) that may store the following software (SW) modules: an image display SW module 932 to display an image of an anatomy on the touch-enabled display; and an indicator display SW module 934 to display a selection indicator on the touch-enabled display superimposed on a portion of the image based at least in part on the received user input, where the selection indicator is displayed at an offset relative to the first location; and a target selection SW module 936 to determine a location of one or more features associated with the image based at least in part on a relative alignment between the selection indicator and the superimposed portion of the image.

[0114] The processing system 920 may include any suitable one or more processors capable of executing scripts or instructions of one or more software programs stored in the controller 900 (such as in the memory 930). For example, the processing system 920 may execute the image display SW module 932 to display an image of an anatomy on the touch-enabled display. The processing system 920 also may execute the indicator display SW module 934 to display a selection indicator on the touch-enableddisplay superimposed on a portion of the image based at least in part on the received user input, where the selection indicator is displayed at an offset relative to the first location. The processing system 920 may further execute the target selection SW module 936 to determine a location of one or more features associated with the image based at least in part on a relative alignment between the selection indicator and the superimposed portion of the image.

[0115] Figure 10 shows an illustrative flowchart depicting an example operation 1000 for selecting features in images, according to some implementations. In some implementations, the example operation 1000 may be performed by a controller for a medical system such as the control system 50 of Figures 1 and 2 or the controller 900 of Figure 9.

[0116] The controller displays an image of an anatomy on a touch-enabled display (1002). The controller receives user input associated with touch of the touch- enabled display at a first location (1004). The controller displays a selection indicator on the touch-enabled display superimposed on a portion of the image based at least in part on the received user input, where the selection indicator is displayed at an offset relative to the first location (1006). In some implementations, the offset may represent a distance between the first location and a second location on the touch-enabled display, where the superimposed portion of the image represents a portion of the image centered around the second location. In some other implementations, the superimposed portion of the image represents a portion of the image centered around the first location. The controller further determines a location of one or more features associated with the image based at least in part on a relative alignment between the selection indicator and the superimposed portion of the image (1008).

[0117] In some aspects, the controller may further detect a drag interaction on the touch-enabled display based on the received user input and change the superimposed portion of the image responsive to the drag interaction. In some implementations, the changing of the superimposed portion of the image may include moving the selection indicator in the direction of the drag interaction and controlling a magnitude of the movement of the selection indicator relative to a magnitude of the drag interaction based at least in part on a distance between the first location and a center of the selection indicator. In some other implementations, the selection indicator may remain fixed relative to the touch-enabled display responsive to the drag interaction.

[0118] In some other aspects, the controller may further detect a multi-touchgesture on the touch-enabled display based on the received user input and adjust a magnification of the superimposed portion of the image based on the multi-touch gesture. In some aspects, the controller may further display the superimposed portion of the image in a separate view outside the image.Additional Embodiments

[0119] Depending on the embodiment, certain acts, events, or functions of any of the processes or algorithms described herein can be performed in a different sequence, may be added, merged, or left out altogether. Thus, in certain embodiments, not all described acts or events are necessary for the practice of the processes.

[0120] Conditional language used herein, such as, among others, “can,” “could,” “might,” “may,” “e.g.,” and the like, unless specifically stated otherwise, or otherwise understood within the context as used, is intended in its ordinary sense and is generally intended to convey that certain embodiments include, while other embodiments do not include, certain features, elements and / or steps. Thus, such conditional language is not generally intended to imply that features, elements and / or steps are in any way required for one or more embodiments or that one or more embodiments necessarily include logic for deciding, with or without author input or prompting, whether these features, elements and / or steps are included or are to be performed in any particular embodiment. The terms “comprising,” “including,” “having,” and the like are synonymous, are used in their ordinary sense, and are used inclusively, in an open-ended fashion, and do not exclude additional elements, features, acts, operations, and so forth. Also, the term “or” is used in its inclusive sense (and not in its exclusive sense) so that when used, for example, to connect a list of elements, the term “or” means one, some, or all of the elements in the list. Conjunctive language such as the phrase “at least one of X, Y and Z,” unless specifically stated otherwise, is understood with the context as used in general to convey that an item, term, element, etc. may be either X, Y or Z. Thus, such conjunctive language is not generally intended to imply that certain embodiments require at least one of X, at least one of Y and at least one of Z to each be present.

[0121] It should be appreciated that in the above description of embodiments, various features are sometimes grouped together in a single embodiment, Figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. This method of disclosure, however, is not to be interpreted as reflecting an intention that any claim requires morefeatures than are expressly recited in that claim. Moreover, any components, features, or steps illustrated and / or described in a particular embodiment herein can be applied to or used with any other embodiment(s). Further, no component, feature, step, or group of components, features, or steps are necessary or indispensable for each embodiment. Thus, it is intended that the scope of the inventions herein disclosed and claimed below should not be limited by the particular embodiments described above, but should be determined only by a fair reading of the claims that follow.

[0122] It should be understood that certain ordinal terms (e.g., “first” or “second”) may be provided for ease of reference and do not necessarily imply physical characteristics or ordering. Therefore, as used herein, an ordinal term (e.g., “first,” “second,” “third,” etc.) used to modify an element, such as a structure, a component, an operation, etc., does not necessarily indicate priority or order of the element with respect to any other element, but rather may generally distinguish the element from another element having a similar or identical name (but for use of the ordinal term). In addition, as used herein, indefinite articles (“a” and “an”) may indicate “one or more” rather than “one.” Further, an operation performed “based on” a condition or event may also be performed based on one or more other conditions or events not explicitly recited.

[0123] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which example embodiments belong. It be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0124] The spatially relative terms “outer,” “inner,” “upper,” “lower,” “below,” “above,” “vertical,” “horizontal,” and similar terms, may be used herein for ease of description to describe the relations between one element or component and another element or component as illustrated in the drawings. It be understood that the spatially relative terms are intended to encompass different orientations of the device in use or operation, in addition to the orientation depicted in the drawings. For example, in the case where a device shown in the drawing is turned over, the device positioned “below” or “beneath” another device may be placed “above” another device. Accordingly, the illustrative term “below” may include both the lower and upper positions. The device may also be oriented in the other direction, and thus the spatially relative terms may be interpreted differently depending on the orientations.

[0125] Unless otherwise expressly stated, comparative and / or quantitative terms, such as “less,” “more,” “greater,” and the like, are intended to encompass the concepts of equality. For example, “less” can mean not only “less” in the strictest mathematical sense, but also, “less than or equal to.”

Claims

WHAT IS CLAIMED IS:

1. A controller for a medical system, comprising: one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the controller to: display an image of an anatomy on a touch-enabled display; receive user input associated with touch of the touch-enabled display at a first location; display a selection indicator on the touch-enabled display superimposed on a portion of the image based at least in part on the received user input, the selection indicator being displayed at an offset relative to the first location; and determine a location of one or more features associated with the image based at least in part on a relative alignment between the selection indicator and the superimposed portion of the image.

2. The system of claim 1, wherein execution of the instructions further causes the controller to: detect a drag interaction on the touch-enabled display based on the received user input; and change the superimposed portion of the image responsive to the drag interaction.

3. The system of claim 2, wherein the changing of the superimposed portion of the image comprises moving the selection indicator in a direction of the drag interaction.

4. The system of claim 3, wherein execution of the instructions further causes the controller to: control a magnitude of the movement of the selection indicator relative to a magnitude of the drag interaction based at least in part on a distance between the first location and a center of the selection indicator.

5. The system of claim 2, wherein the selection indicator remains fixed relative to the touch-enabled display responsive to the drag interaction.

6. The system of claim 1, wherein execution of the instructions further causes the controller to magnify the superimposed portion of the image.

7. The system of claim 1 , wherein the offset represents a distance between the first location and a second location on the touch-enabled display, the superimposed portion of the image representing a portion of the image centered around the second location.

8. The system of claim 7, wherein the selection indicator comprises a reticle centered around the second location.

9. The system of claim 8, wherein the reticle is surrounded by a boundary indicating an interactive region of the selection indicator.

10. The system of claim 1, wherein the superimposed portion of the image represents a portion of the image centered around the first location.

11. The system of claim 1, wherein execution of the instructions further causes the controller to display the superimposed portion of the image in a separate view outside the image.

12. A method for selecting features in images, comprising: displaying an image of an anatomy on a touch-enabled display; receiving user input associated with touch of the touch-enabled display at a first location; displaying a selection indicator on the touch-enabled display superimposed on a portion of the image based at least in part on the received user input, the selection indicator being displayed at an offset relative to the first location; and determine a location of one or more features associated with the image based at least in part on a relative alignment between the selection indicator and the superimposed portion of the image.

13. The method of claim 12, further comprising: detecting a drag interaction on the touch-enabled display based on the received user input; and changing the superimposed portion of the image responsive to the drag interaction.

14. The method of claim 13, wherein the changing of the superimposed portion of the image comprises: moving the selection indicator in a direction of the drag interaction; and controlling a magnitude of the movement of the selection indicator relative to a magnitude of the drag interaction based at least in part on a distance between the first location and a center of the selection indicator.

15. The method of claim 13, wherein the selection indicator remains fixed relative to the touch-enabled display responsive to the drag interaction.

16. The method of claim 12, further comprising magnifying the superimposed portion of the image.

17. The method of claim 12, wherein the offset represents a distance between the first location and a second location on the touch-enabled display, the superimposed portion of the image representing a portion of the image centered around the second location.

18. The method of claim 17, wherein the selection indicator comprises a reticle centered around the second location, the reticle being surrounded by a boundary indicating an interactive region of the selection indicator.

19. The method of claim 12, wherein the superimposed portion of the image represents a portion of the image centered around the first location.

20. The method of claim 12, further comprising displaying the superimposed portion of the image in a separate view outside the image.

Images