Pose estimation using intensity thresholding and point cloud analysis
By generating a point cloud associated with the distal end of the instrument and determining the principal axis that maximizes the variance, the inaccuracy of pose estimation for flexible, slender bodies in medical procedures is addressed, thereby improving the alignment accuracy of the instrument's end effector.
Patent Information
- Application Number
- CN202580024535.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2025-03-24
- Filing Date
- 2025-03-25
- Publication Date
- 2026-11-17
AI Technical Summary
Existing end-effector pose estimation techniques cannot adequately address inaccuracies caused by the shape and curvature of the flexible, slender body, the geometry of the end-effector, various imaging artifacts, and the presence of medical instruments in the body's working channel.
By receiving image data representing a three-dimensional model of the instrument in the anatomical structure, a point cloud associated with the distal end of the instrument is generated, a first principal axis that maximizes the variance of the point cloud is determined, and the pose of the distal end of the instrument is estimated based on this axis.
It provides more accurate instrument end-effector pose estimation, improving alignment accuracy in medical procedures, such as aiding in the alignment of biopsy needles with nodules during robot-assisted bronchoscopy.
Smart Images

Figure CN123124943A_ABST
Abstract
Description
Cross-references to related applications
[0001] This patent application claims priority to U.S. Nonprovisional Patent Application No. 19 / 088,505, filed March 24, 2025, entitled “POSE ESTIMATION USING INTENSITY THRESHOLDING AND POINT CLOUD ANALYSIS,” which claims priority and benefit to U.S. Provisional Patent Application No. 63 / 572,032, filed March 29, 2024, entitled “POSE ESTIMATION USING INTENSITY THRESHOLDING AND POINT CLOUD ANALYSIS,” pursuant to 35 USC § 119(e). The disclosures of these earlier applications are considered part of this patent application and are incorporated herein by reference. Technical Field
[0002] This disclosure relates to pose estimation of objects, and more specifically to pose estimation using intensity thresholding and point cloud analysis. Background Technology
[0003] During medical procedures, various views of the subject's anatomy are presented to the physician. These views may include, for example, preoperative model views, 3D reconstructed model views, fluorescein images, endoscopic images, etc. As part of the medical procedure, the physician may need to accurately identify the location and orientation of the depicted object. Summary of the Invention
[0004] The present invention is presented in a simplified form to introduce some concepts, which will be further described in the detailed embodiments below. This summary is not intended to identify key or essential features of the subject matter protected by the claims, nor is it intended to limit the scope of the subject matter protected by the claims.
[0005] One innovative aspect of the subject matter of this disclosure can be implemented in a controller for a medical system, the controller including a processing system and a memory. The memory stores instructions that, when executed by the processing system, cause the controller to: receive image data representing a three-dimensional (3D) model of an anatomical structure in which an instrument is disposed; generate a point cloud associated with the distal end of the instrument based on the image data; determine a first principal axis that maximizes the variance of the point cloud; and determine the pose of the distal end of the instrument based at least in part on the first principal axis.
[0006] Another innovative aspect of the subject matter of this disclosure can be implemented in a pose estimation method. The method includes the steps of: receiving image data representing a 3D model of an anatomical structure in which an instrument is disposed; generating a point cloud associated with the distal end of the instrument based on the image data; determining a first principal axis that maximizes the variance of the point cloud; and determining the pose of the distal end of the instrument based at least in part on the first principal axis. Attached Figure Description
[0007] Various embodiments are depicted in the accompanying drawings for illustrative purposes and should in no way be construed as limiting the scope of the invention. Furthermore, various features of different disclosed embodiments may be combined to form additional embodiments that are part of this disclosure. Throughout the drawings, reference numerals may be repeated to indicate correspondences between reference elements.
[0008] Figure 1 An example medical system based on one or more examples is shown.
[0009] Figure 2 Examples are given based on one or more implementation schemes. Figure 1 A schematic diagram of the different components of a medical system.
[0010] Figure 3 Block diagrams illustrating various positions and / or imaging systems / modalities are shown based on one or more examples.
[0011] Figure 4 An example block diagram is shown for an end-effector pose estimation pipeline based on one or more implementations.
[0012] Figure 5A Figure 5C illustrates example images associated with a region extraction process according to one or more implementation schemes.
[0013] Figure 6 Example point clouds based on one or more implementation schemes are shown.
[0014] Figures 7A and 7B illustrate example renderings associated with the process of determining the forward direction of the endoscope tip according to one or more embodiments.
[0015] Figure 8 Example renderings are shown that are associated with the process of determining the position of the end of the endoscope according to one or more implementations.
[0016] Figure 9 A flowchart illustrating an exemplary device pose estimation process according to one or more embodiments is provided.
[0017] Figure 10 A block diagram of an example controller for a medical system, based on some specific implementations, is shown.
[0018] Figure 11 An exemplary flowchart illustrating pose estimation operations based on some specific implementations is shown. Detailed Implementation
[0019] The headings provided herein are for convenience only and do not necessarily affect the scope or meaning of the claimed invention. Although specific preferred embodiments and examples are disclosed below, the subject matter of the invention extends beyond the specifically disclosed embodiments to other alternative embodiments and / or uses, as well as modifications and equivalents thereof. Therefore, the scope of the claims that may appear herein is not limited to any particular embodiment of the specific embodiments described below. For example, in any method or process disclosed herein, the actions or operations of the method or process may be performed in any suitable sequence and are not necessarily limited to any particular disclosed sequence. Various operations may then be described as multiple discrete operations in a manner that may aid in understanding certain embodiments; however, the order of description should not be construed as implying that these operations depend on the order. Furthermore, the structures, systems, and / or apparatuses described herein may be embodied as integrated components or separate components. Certain aspects and advantages of these embodiments are described for the purpose of comparing the various embodiments. Not all such aspects or advantages are necessarily achieved by any particular embodiment. Therefore, for example, various embodiments may be performed by implementing or optimizing one or a set of advantages taught herein without necessarily achieving other aspects or advantages that may also be taught or proposed herein.
[0020] Although specific spatial relative terms such as “external,” “internal,” “upper,” “lower,” “below,” “above,” “vertical,” “horizontal,” “top,” “bottom,” “outer,” “upward,” “side,” and similar terms are used herein to describe the spatial relationship between one device / element or anatomical structure and another device / element or anatomical structure, it should be understood that these terms are used herein for descriptive convenience to describe the positional relationship between elements / structures, such as with respect to the illustrated orientations in the accompanying drawings. It should be understood that spatial relative terms are intended to cover different orientations of elements / structures in use or operation other than those depicted in the accompanying drawings. For example, an element / structure described as “above” another element / structure may indicate a position below or beside such other element / structures relative to an alternative orientation of the subject or element / structure, and vice versa. It should be understood that spatial relative terms, including those listed above, can be understood relative to the corresponding illustrated orientations in the reference drawings.
[0021] To facilitate the use of devices, components, systems, features, and / or modules having similar features in one or more aspects, specific reference numerals are repeated in different figures within the group of figures disclosed herein. However, the repeated use of common reference numerals in the figures with respect to any embodiment of the embodiments disclosed herein does not necessarily indicate that such features, devices, components, or modules are identical or similar. Rather, those skilled in the art will be informed by the context that the use of common reference numerals may imply the degree of similarity between the referenced subjects. The use of specific reference numerals in the context of the description of a particular figure may be understood to refer to the device, component, aspect, feature, module, or system identified in that particular figure, and not necessarily to any device, component, aspect, feature, module, or system identified by the same reference numeral in another figure. Furthermore, aspects of individual figures identified by common reference numerals may be interpreted as sharing characteristics or being completely independent of each other. In some contexts, features associated with individual figures identified by common reference numerals are unrelated and / or similar with respect to at least some aspects.
[0022] Overview
[0023] This disclosure provides systems, apparatus, and methods for estimating the pose (e.g., position and / or orientation) of the distal end of a medical device having a flexible, elongated body within the anatomical structure of a subject based on images captured outside the subject. As an example, the distal end may be the distal tip of an endoscope.
[0024] Existing end-effector pose estimation techniques may not be able to adequately address inaccuracies caused by variations in the shape and curvature of the flexible, slender body, the geometry of the end-effector, various imaging artifacts, and the presence of medical instruments in the working channel of the body.
[0025] This disclosure discloses an automated or semi-automated process for estimating the end-effector pose in three-dimensional (3D) space. The image can be a captured computed tomography (CT) image, including cone-beam computed tomography (CBCT) images. Such CT images provide a third-person perspective of the end-effector pose compared to a first-person perspective captured from an endoscopic imaging device inside the patient. The estimated end-effector pose can provide another reference point when aligning the end-effector with a target during medical procedures. For example, the estimated end-effector pose based on CT images can aid in aligning a biopsy needle with a nodule during robot-assisted bronchoscopy.
[0026] Example medical system
[0027] Figure 1An example medical system 100 (also referred to as "surgical medical system 100" or "robotic medical system 100") according to one or more examples is illustrated. For example, medical system 100 may be configured for diagnostic and / or therapeutic bronchoscopy, as shown. Medical system 100 may include and utilize robotic system 10, which may be implemented as, for example, a robotic cart. Although medical system 100 is shown as including various cart-based systems / devices, the concepts disclosed herein can be implemented in any type of robotic system / arrangement, such as robotic systems employing track-based components, stage-based robotic end effectors / manipulators, etc. Robotic system 10 may include one or more robotic arms 12 (also referred to as "robotic locators") configured to locate or otherwise manipulate medical instruments, such as medical instrument 32 (e.g., a steerable endoscope or another elongated instrument with a flexible, elongated body). For example, medical instrument 32 may be advanced through a natural orifice entry point (e.g., the mouth 9 of subject 7 positioned on stage 15 in this example) to deliver diagnostic and / or therapeutic treatments. Although described in the context of a bronchoscopy procedure, the medical system 100 can be implemented for other types of procedures, such as gastrointestinal (GI) procedures, renal / urinary / nephrology procedures, etc. The term "subject" is used herein to refer to a living patient as well as any subject to which this disclosure applies. For example, "subject" can refer to a subject including those in physical anatomical models used in dry runs (e.g., anatomical education models, anatomical models, medical education anatomical models, etc.), models in computer simulations, etc., which encompass non-living patients or test subjects.
[0028] With the robotic system 10 correctly positioned, the medical device 32 can be inserted into the subject 7 robotically, manually, or in combination thereof. In this example, one or more robotic arms 12 and / or their instrument actuators 28 can control the medical device 32. The instrument actuators 28 can reposition the device in space by manipulating one or more robotic arms 12 to different angles and / or positions.
[0029] Medical system 100 may also include control system 50 (also referred to as "control tower" or "mobile tower"), which will be referenced below. Figure 2 Detailed Description. The control system 50 may include one or more displays 212 to provide / display / present various information related to medical procedures, such as anatomical images. The control system 50 may additionally include one or more control mechanisms, which may be individual directional input controls 216 or graphical user interfaces (GUIs) presented on the displays 212.
[0030] In some implementations, display 212 may be a touch-sensitive display, as shown, that can present anatomical images and allow selection thereon. Some example anatomical images may include CT images, fluorescein images, images of anatomical mapping, etc. Using a touch-sensitive display, the operator 5 viewing the images may find it convenient to use touch-based selection instead of using directional input controls 216 to identify targets within the image (e.g., target objects or regions of interest). For example, operator 5 can use the touchscreen to select the endoscope tip and / or nodules.
[0031] The control system 50 can communicatively connect (e.g., via wired and / or wireless connections) to the robotic system 10 to provide support for controls, electronics, fluid dynamics, optics, sensors, and / or power. Placing such functionality within the control system 50 allows for easier adjustment and / or repositioning of the robotic system 10 by the operator 5, given its smaller form factor. Furthermore, the separation of functions between the robotic system 10 and the control system 50 can reduce operating room clutter and / or facilitate efficient clinical workflows.
[0032] Medical system 100 may include an electromagnetic (EM) field generator 120 configured to broadcast / transmit an EM field detected by an EM sensor, such as a sensor associated with medical device 32. The EM field may induce a small current in the coil of the EM sensor (also referred to as a “position sensor”), which may be analyzed to determine the pose (position and / or angle / or orientation) of the EM sensor relative to the EM field generator 120. In some embodiments, the EM sensor may be positioned at a distal end of medical device 32, and the pose of the distal end may be determined in conjunction with the pose of the EM sensor. Although EM fields and EM sensors are described in many examples herein, position sensing systems and / or sensors may be of any type, such as optical position sensing systems / sensors, image-based position sensing systems / sensors, etc.
[0033] The medical system 100 may also include an imaging system 122 (e.g., a fluorescence imaging system) configured to generate image data (also referred to as “images”) and / or provide / send such image data to / from another device / system. For example, the imaging system 122 may generate image data depicting the anatomical structures of the subject 7 and provide the image data to a control system 50, a robotic system 10, a web server, a cloud server, and / or another device. The imaging system 122 may include transmitters / energy sources (e.g., X-ray sources, ultrasound sources, etc.) and / or detectors (e.g., X-ray detectors, ultrasound detectors, etc.) integrated into a support structure (e.g., mounted on a C-arm support 124) that provides flexibility in positioning around the subject 7 to capture images from various angles without moving the subject 7. The use of the imaging system 122 can provide visualization of internal structures / anatomical structures for a variety of purposes, such as navigation of the medical device 32 (e.g., providing images of internal anatomy to the operator 5), positioning of the medical device 32 (e.g., based on an image data analysis system), etc. In the example, the use of imaging system 122 can enhance the effectiveness and / or safety of medical procedures such as bronchoscopy by providing clear, continuous visual feedback to operator 5.
[0034] In some examples, the imaging system 122 is a mobile device configured to move around within the environment. For example, the imaging system 122 may be positioned next to the subject 7 during a specific phase of the procedure (as illustrated in the figure) and removed when the imaging system 122 is no longer needed. In other examples, the imaging system 122 may be part of the table 15 or other equipment in the operating environment. The imaging system 122 may be implemented as a computed tomography (CT) machine / system, an X-ray machine / system, a fluoroscopic examination machine / system, a positron emission tomography (PET) machine / system, a PET-CT machine / system, a CT angiography machine / system, a cone-beam computed tomography (CBCT) machine / system, a 3DRA machine / system, a single-photon emission computed tomography (SPECT) machine / system, a magnetic resonance imaging (MRI) machine / system, an optical coherence tomography (OCT) machine / system, an ultrasound machine / system, etc. In some cases, medical system 100 includes multiple imaging systems, such as a first type of imaging system and a second type of imaging system, wherein different types of imaging systems may be used or positioned above subject 7 as needed at different stages / parts of the procedure.
[0035] In some implementations, the imaging system 122 may be configured to generate a three-dimensional (3D) model of the anatomical structure. For example, the imaging system 122 may be configured to process multiple images (also referred to in some cases as "image data") to generate a 3D model. For example, the imaging device 122 may be implemented as a CT scanner configured to capture / generate a series of images / image data (e.g., 2D images / slices) from different angles around the subject 7, and then reconstruct a 3D model from these images / image data using one or more algorithms. The 3D model may be provided to the control system 50, the robotic system 10, a web server, a cloud server, and / or another device, such as for processing, display, or other purposes.
[0036] For ease of description of this disclosure, Figure 1 The respiratory system is illustrated as an example anatomical structure. The respiratory system includes the upper respiratory tract, which comprises the nose / nasal cavity, pharynx (i.e., larynx), and larynx (i.e., vocal cord box). The respiratory system also includes the lower respiratory tract, which comprises the trachea (6), lungs (4) r and 4 l The bronchial tree consists of the main bronchus 71, which branches into smaller secondary bronchi 78 and tertiary bronchi 75, and terminates in even smaller tubes called bronchioles 77. Each bronchiole connects to a cluster of alveoli (not shown). During the inspiratory phase of the respiratory cycle, air enters through the mouth and nose and travels down the larynx into the trachea 6, through the right and left main bronchus 71 into the lungs 4, into the smaller bronchial airways 78 and 75, into the smaller bronchioles 77, and into the alveoli, where oxygen and carbon dioxide exchange occurs.
[0037] The bronchial tree is an example of an end-luminal network in which a robot-controlled instrument can be navigated and utilized according to the inventive solutions presented herein. However, although aspects of this disclosure are presented in the context of an end-luminal network including the airways (e.g., lumens, branches) of the lungs of a subject, some embodiments of this disclosure can be implemented in other types of end-luminal networks, such as renal networks, cardiovascular networks (e.g., arteries and veins), the gastrointestinal tract, the urinary tract, etc.
[0038] In some embodiments, imaging system 122 may be configured to capture / update / present images of anatomical structures intraoperatively using a CBCT imaging system. During CBCT imaging, subject 7 may be positioned on a stage 15 between an X-ray source and a detector mounted on a C-arm support 124, where the X-ray beam passes through the target anatomical structure and updates the acquired images intraoperatively. For example, regarding the subject's lung 4, one or more CBCT-captured images or reconstructed 3D models may be presented to operator 5 on a monitor 56. While CBCT has been described, it should be understood that this disclosure contemplates any other imaging techniques capable of providing 3D reconstruction, such as conventional CT imaging techniques.
[0039] Figure 2 Example components of a control system 50, a robotic system 10, and a medical device 32 according to one or more examples are illustrated. The control system 50 may be coupled to and cooperate with the robotic system 10 to perform medical procedures. For example, the control system 50 may include a communication interface 202 for communicating with (e.g., controlling) the robotic system 10 via a wireless or wired connection to a communication interface 204. Furthermore, in the examples, the control system 50 may communicate with the robotic system 10 to receive position / sensor data relating to the position of sensors associated with and controlled by the robotic system 10. In some examples, the control system 50 may communicate with an EM field generator 120 to control the generation of an EM field in the area surrounding the subject 7. The control system 50 may also include a power interface 206.
[0040] The control system 50 may include control circuitry 251 configured to actuate one or more components of the medical system 100 and / or otherwise control any of the various system components, such as brackets, mounts, arms / positioners, medical devices, imaging devices, position sensing devices, sensors, etc. Furthermore, control circuitry 251 may be configured to perform other functions, such as inducing the display of information, processing data, receiving input, communicating with other components / devices, and / or any other functions / operations discussed herein.
[0041] The control system 50 may also include one or more input / output (I / O) components 210 configured to assist a physician or other person in performing medical procedures. For example, one or more I / O components 210 may be configured to receive input and / or provide output to enable a user to control / navigate the medical device 32, the robotic system 10, and / or other instruments / devices associated with the medical system 100. The control system 50 may include one or more displays 212 to provide / display / present various information about the procedures. For example, one or more displays 212 may be used to present navigation information, including virtual anatomical models of anatomical structures with virtual representations of the medical devices, image data, and / or other information. One or more I / O components 210 may include user input controls 214, which may include any type of user input (and / or output) device or device interface, such as directional input controls 216, touch-based input controls including gesture-based input controls, motion-based input controls, etc. User input control 214 may include one or more buttons, keypads, joysticks, handheld controllers (e.g., video game controllers), computer mice, touchpads, trackballs, control panels, sensors for capturing gestures and finger gestures (e.g., motion sensors or cameras), touchscreens, toggle key (button) inputs, and / or interfaces / components for them. In the example, such input can be used to generate commands for controlling medical devices, robotic arms, and / or other components.
[0042] The control system 50 may also include a data storage device 218 configured to store executable instruments (e.g., computer-executable instructions) that can be executed by the control circuitry 251 to cause the control circuitry 251 to perform the various operations / functions discussed herein. In the example, two or more components of the control system 50 may be electrically and / or communicatively connected to each other.
[0043] Robotic system 10 may include one or more robotic arms 12 configured to engage and / or control, for example, a medical device 32 and / or other elements / components, to perform one or more aspects of a procedure. As shown, each robotic arm 12 may include multiple segments 220 coupled to joints 222, which provide multiple degrees of motion / freedom. Robotic system 10 may be configured to receive control signals from control system 50 to perform certain operations, such as positioning one or more of the robotic arms 12 in a particular manner, manipulating instruments, etc. In response, robotic system 10 may use its control circuitry 211 to control actuators 226 and / or other components of robotic system 10 to perform operations. For example, control circuitry 211 may control insertion / retraction, joint movement, rolling, etc., of the axis of medical device 32 or another instrument by actuating a drive output 228 of a manipulator 230 (e.g., an end effector) coupled to the base of a robotically controllable instrument. Drive output 228 may be coupled to a drive input on an associated instrument, such as the instrument base of an instrument coupled to an associated robotic arm 12. The robot system 10 may include one or more power interfaces 232.
[0044] The robotic system 10 may include a support column 234, a base 236, and / or a console 238. The console 238 may provide one or more I / O components 240, such as a user interface and / or display (or dual-purpose device, such as a touchscreen) for receiving user input to provide preoperative and / or intraoperative data to the physician / user. The support column 234 may include arm supports 242 (also referred to as “brackets”) for supporting the deployment of one or more robotic arms 12. The arm supports 242 may be configured to translate vertically along the support column 234. Vertical translation of the arm supports 242 allows the robotic system 10 to adjust the range of the robotic arms 12 to accommodate various table heights, patient body shapes, and / or physician preferences. The base 236 may include wheel-shaped casters 244 (also referred to as “wheels 244”) that allow the robotic system 10 to move around the operating room prior to the procedure. Once in place, the casters 244 may be secured using wheel locks to hold the robotic system 10 in place during the procedure.
[0045] Each joint 222 of the robotic arm 12 can be independently controllable and / or provide independent degrees of freedom for instrument navigation. In some examples, each robotic arm 12 has seven joints, thus providing seven degrees of freedom, including “redundant” degrees of freedom. Redundant degrees of freedom allow the robotic arms 12 to be controlled to position their respective manipulators 230 at specific locations, orientations, and / or trajectories in space using different link positions and joint angles. This allows the robotic system 10 to locate and guide medical instruments from desired points in space, while allowing physicians to move joints 222 to clinically advantageous positions away from the patient to achieve greater access while avoiding collisions.
[0046] One or more manipulators 230 (e.g., end effectors) may be coupled to an instrument base / handle, which in some cases may be attached using a sterile adapter component. The combination of the manipulator 230, the coupled instrument base, and any intermediate mechanisms or connectors (e.g., sterile adapters) may be referred to as a manipulator assembly, or simply as a manipulator. The manipulator / manipulator assembly may provide an electrical and / or control interface. For example, the interface may include connectors for transmitting pneumatic pressure, electrical, electrical signals, and / or optical signals from the robotic arm 12 to the coupled instrument base. The manipulator / manipulator assembly may be configured to manipulate medical devices (e.g., surgical tools / instruments) using techniques including, for example, direct drive, harmonic drive, gear drive, belt and / or pulley, magnetic drive, etc.
[0047] The robot system 10 may also include a data storage device 246 configured to store executable instruments (e.g., computer-executable instructions) that can be executed by control circuitry 211 to cause control circuitry 211 to perform the various operations / functions discussed herein. In the example, two or more components of the robot system 10 may be electrically and / or communicatively connected to each other.
[0048] Data storage devices (including data storage device 218, data storage device 246, and / or other data storage devices / memories) may include any suitable or desired type of computer-readable medium. For example, computer-readable media may include one or more volatile data storage devices, non-volatile data storage devices, removable data storage devices, and / or non-removable data storage devices, which are implemented using any technology, layout, and / or data structure / protocol, including any suitable or desired computer-readable instructions, data structures, program modules, or other types of data.
[0049] Computer-readable media may include, but is not limited to, phase-change memory, static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, optical disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage devices, magnetic tape, magnetic tape, disk storage devices or other magnetic storage devices, or any other non-transitory medium that can be used to store information for access by a computing device. As used in certain contexts herein, computer-readable media generally may not include communication media, such as modulated data signals and carrier waves. Therefore, computer-readable media should generally be understood as non-transitory media.
[0050] Control circuitry (including control circuitry 251, control circuitry 211, and / or other control circuitry) may include circuitry embodied in a robot system, control system / tower, machinery, or any other component / device. Control circuitry may include any collection of processors, processing circuitry, processing modules / units, chips, dies (e.g., semiconductor dies including one or more active devices and / or passive devices and / or connection circuitry), microprocessors, microcontrollers, digital signal processors, microcomputers, central processing units, field-programmable gate arrays, programmable logic devices, state machines (e.g., hardware state machines), logic circuitry, analog circuitry, digital circuitry, and / or any device that manipulates signals (analog and / or digital) based on hard-coded circuitry and / or operating instructions. Control circuitry as referred to herein may also include one or more circuit substrates (e.g., printed circuit boards), conductive traces and vias and / or mounting pads, connectors, and / or components. Control circuitry may also include one or more storage devices, which may be embodied in the embedded circuitry of a single device, multiple devices, and / or devices. Such data storage devices may include read-only memory, random access memory, volatile memory, non-volatile memory, static memory, dynamic memory, flash memory, cache memory, data storage registers, and / or any device for storing digital information. In examples where the control circuitry includes hardware and / or software state machines, analog circuits, digital circuits, and / or logic circuits, the data storage device / register storing any associated operational instructions may be embedded within or outside the circuitry including the state machine, analog circuitry, digital circuitry, and / or logic circuitry.
[0051] The functions described herein can be implemented by the control circuit 251 of the control system 50 and / or the control circuit 211 of the robot system 10, such as by the control circuits 251 and 211 executing executable instructions to enable the control circuits 251 and 211 to perform functions.
[0052] The endoscope assembly / medical device 32 includes a handle or base 31 coupled to an endoscope shaft. For example, the endoscope 40 (also referred to herein as an "endoscope" or "shaft") may include an elongated shaft comprising one or more lamps 49 and one or more cameras 48 or other imaging devices. The medical device 32 may be powered via a power interface 36 and / or controlled via a control interface 38, each or both of which may interface with a robotic arm / component of the robotic system 10. The medical device 32 may also include one or more sensors 37, such as pressure sensors and / or other force-reading sensors, which may be configured to generate signals indicating forces at or experienced by one or more components of the medical device 32.
[0053] The medical device 32 includes certain mechanisms for articulating / deflecting the endoscope 40 relative to its axis. For example, the endoscope 40 may already be associated with its proximal portion, and one or more drive inputs 34 may be associated with and / or integrated with one or more pulleys / spindles 33 configured to tension / untension the pull cable / traction line 45 of the endoscope 40 to induce articulation of the endoscope 40.
[0054] The endoscope 40 may also include one or more working channels 44, which may be formed within an elongated shaft and extend the length of the endoscope 40. The working channels 44 may be used to deploy components of the medical tool 35 or medical device 32 (e.g., lithotripter, basket, clamp, laser, etc.) therein, or to perform irrigation and / or suction, accessing the surgical area surrounding the distal end through the distal end of the endoscope 40. The medical device 32 may be used in conjunction with the medical tool 35 and includes various hardware and control components for the medical tool 35, and in some cases, the medical tool 35 may be included as part of the medical device 32. For example, as shown, the medical device 32 may include a basket formed of one or more wire teeth, but any medical tool 35 is contemplated.
[0055] Calibration, navigation and positioning modes / systems
[0056] Figure 3 This is a block diagram illustrating a system 300 comprising various localization and / or imaging systems / modalities 302-312 (sometimes referred to as "subsystems") according to one or more examples, which may be implemented to facilitate anatomical mapping, navigation, localization, and / or visualization for procedures. For example, the various systems 302-312 may be configured to provide data for generating anatomical mapping maps, determining the location of instruments, determining the location of targets, and / or performing other techniques.
[0057] Each system in systems 302-312 can be associated with a corresponding coordinate system (also referred to as a “position coordinate system”) and / or can provide data / information related to the position of instruments and / or anatomical structures, wherein registering various coordinate systems with each other allows for the integration of various systems to provide mapping, navigation, and / or instrument visualization. For example, registration of various modalities with each other can allow a determined position in one modality to be tracked and / or superimposed on / within a reference system associated with another modality, thereby providing a layer of positional information that can be combined to provide a robust positioning system.
[0058] In the example, system 300 is configured to implement one or more positioning techniques (also referred to as "positioning system 300"). Positioning can refer to the process of determining the location and orientation / pose of an instrument or other element / component within a given space or environment.
[0059] In various examples, the anatomical space in which a medical device can be positioned (i.e., where the location and / or shape of the device is determined / estimated) is a 2D or 3D portion of the subject's trachea, bronchus, airway, vascular system, urethra, gastrointestinal tract, or any organ or space accessed via an internal lumen. Various modalities can be implemented for use with respect to... Figure 1 The imaging system 122 describes various imaging techniques to provide images / representations / models of anatomical space. It can combine procedures to acquire one or both of preoperative and intraoperative images.
[0060] Systems 302-312 can provide information for generating 2D or 3D anatomical models / figures 314 (e.g., airway models). In the example, anatomical mapping 314 and / or other positioning information can be displayed to the user (such as operator 5) during procedures to assist the user in performing the procedures. For example, a visualization of the tracked medical device can be overlaid on the anatomical mapping 314 based on location / sensor data associated with the tracked medical device.
[0061] As shown in the figure, system 300 may include an operating table or other subject platform or positioning / support structure 302 (e.g., Figure 1 (Operating table 15). The position of the support structure 302 can be determined based on data maintained relating to its position within the surgical / procedural environment. Alternatively or in addition, one or more markers and / or appropriate imaging / positioning modalities can be used to sense or otherwise determine the position of the support structure 302.
[0062] System 300 may also include robot system 304, such as robot system 10 (e.g., a robot cart or other device or system including one or more robot end effectors). Data relating to the position and / or state of the robot arm, actuators, and / or other components of robot system 304 may be known or derived from robot command data or other robot data relative to the coordinate system of robot system 304. In some examples, reference system registration 316 occurs between support structure 302 and robot system 304, which may be a relatively coarse registration based on robot system / cart setup procedures (which may have any suitable or desired scheme) (in some cases).
[0063] System 300 may also include an electromagnetic (EM) sensor system 306, which may include an EM field generator (e.g., EM field generator 120) and one or more EM sensors. The EM sensors may be associated with a portion of the tracked / controlled apparatus, such as along the length of the apparatus and / or other elongated members disposed in the working channel of the apparatus. In some embodiments, the EM field generator may be mechanically coupled to a support structure 302 or a robot system 304, in which case registration / association 318 between such systems can be known and / or determined. In some embodiments, registration 318 between the EM sensor system 306 and the robot system 304 may be determined by forward kinematics and / or field generator mounting transformation information. For example, the field generator may be mounted to an end effector / manipulator of the robot system 304, such that the position of the field generator relative to the robot system positioning frame can be known based on a known relationship between the position of the robot end effector and the position of the robot system 304. The EM sensor system 306 may provide apparatus pose and / or path information based on sensor readings associated with the apparatus.
[0064] System 300 may also include an optical camera system 308 comprising one or more cameras or other imaging devices configured to generate images of a subject's anatomy within its field of view, such as real-time image data during surgery. In the example, registration 320 between the optical camera system 308 and the EM sensor system 306 can be achieved by identifying features (such as those via the medical device tip) in the image generated by the optical camera system 308 that have associated EM sensor data. Registration 320 may also be based, at least in part, on hand-eye interaction of a physician viewing real-time camera images as an endoscope equipped with EM sensors navigates through the subject's anatomy.
[0065] System 300 may also include a computed tomography (CT) imaging system 310 configured to generate CT images of the subject's anatomy, which can be performed preoperatively and / or intraoperatively. Specifically, CBCT is a CT imaging technique that uses a cone-shaped X-ray beam and a specialized detector to capture multiple two-dimensional (2D) X-ray images from different angles around the patient. In CBCT imaging, 2D images can be used to reconstruct a 3D volumetric representation.
[0066] In the example, image processing can be implemented for registration 322 between CT image data and camera image data generated by the optical camera system 308. For example, common features identified in both the camera image data and the CT image data can be recognized to spatially correlate CT image frames with camera image frames. In some examples, the CT imaging system 310 can be used to generate preoperative imaging data for producing anatomical mapping 314 and / or for path navigation planning.
[0067] In this example, the fluoroscopy imaging system 312 can be registered 324 to the CT imaging system 310 using any image processing technique suitable for such registration. Fluoroscopy can provide real-time, continuous X-ray imaging of moving structures inside the body. The images produced during fluoroscopy are typically two-dimensional and show a dynamic view of the area being examined. This real-time imaging allows the operator 5 to visualize the movement and position of anatomical structures or medical instruments 32 during the procedure. Although the CT imaging system 310 and the fluoroscopy imaging system 312 are illustrated as separate systems, in this example, the same system can perform the functions of both the CT imaging system 310 and the fluoroscopy imaging system 312.
[0068] In the example, the CT imaging system 310 and / or the fluoroscopy imaging system 312 can be registered 326 to the EM sensor system 306 using various techniques (such as tool registration, transformation functions, etc.). In one example, the mechanical structure of the C-arm instruments of systems 310, 312 may have a known physical transformation / relationship relative to the mounting position of the EM field generator of the EM sensor system 306. Such a known relationship can be used to register the CT / fluoroscopy image space to the EM sensor image space. Connections 328 and 330 represent the registration / relationship between the CT imaging system 310 and the fluoroscopy imaging system 312 and the anatomical mapping map 314, respectively.
[0069] The position, shape, and / or orientation of an instrument (such as an endoscope) can be determined using any one or more of systems 302-312, which facilitates the generation of graphical interface data representing the estimated position and / or shape of the instrument relative to anatomical mapping 314. The position, shape, and / or orientation of the instrument and / or anatomical mapping 314 can be displayed on a display device, such as via control system 50 and / or robotic system 10 or another device. In some examples, anatomical mapping 314 also indicates the location of a target, such as a location within an anatomical structure designated for further treatment.
[0070] Although systems 302-312 are discussed in a specific order, they can be implemented in different orders. Furthermore, they can be used in different ways. Additionally, registration can occur between different systems.
[0071] In some examples, one or more of systems 302-312 may be implemented to generate an anatomical mapping 314 preoperatively and / or determine the position of one or more targets within the anatomical mapping 314. During surgery, one or more of systems 302-312 may also be implemented to determine the position of a medical device and / or the position of a target relative to the anatomical mapping 314. As discussed herein, one or more of systems 302-312 may also be implemented to update the anatomical mapping 314, the position of the medical device, the position of the target, etc.
[0072] Intraoperative end-effector pose estimation using external imaging modalities
[0073] See again Figure 1 An example of the respiratory system is described, illustrating how an imaging system 122, including external imaging modalities such as CBCT, can generate a 3D model of the lung 4. Anatomical mapping 314 can correspond to a 3D model of the subject's anatomy. Traditionally, the use of external imaging has been primarily limited to generating 3D models for preoperative planning and / or as an intraoperative reference model related to end-effector pose estimation of the medical device 32.
[0074] Using anatomical mapping 314 as a reference model for EM-based or camera-based end-effector localization may be inaccurate. For example, there may be a difference between the anatomical mapping 314 generated based on preoperatively captured images and the actual anatomy of the subject 7 during the medical procedure. As another example, anatomical mapping 314 may not capture the movement of the object and / or anatomical structures (such as the respiratory system, gastrointestinal tract, blood vessels) or medical device 32. Re-reference Figure 1 The breathing of subject 7 can cause the shape of lung 4 to deform to the extent that the end of lung 4 may be displaced in an incorrect location in anatomical mapping 314 (e.g., secondary bronchus 78 instead of tertiary bronchus 75 or in the wrong bronchus).
[0075] This disclosure uses images captured during the procedure to provide end-effector pose estimation. CT imaging techniques, including CBCT and fluoroscopy, can capture images intraoperatively or otherwise provide them. These imaging techniques can effectively capture the relationship between the instrument's end-effector and its surrounding anatomical environment, thus providing valuable insights. Furthermore, using the disclosed CT images for pose estimation and localization can help resolve discrepancies between anatomical mapping 314 and the actual anatomical structures by providing additional reference for estimation / localization.
[0076] Specifically, this disclosure provides a time- and resource-efficient CT-based simplified end-effector pose estimation method. The simplified method helps address the stringent latency requirements and resource demands traditionally involved in CT-based end-effector pose estimation. Therefore, this disclosure envisions extending the use of CT-based systems to intraoperative end-effector pose estimation. CT-based end-effector pose estimation can assist the operator in adjusting the end-effector pose when aligning the end-effector with a target in 3D space. For example, CT-based end-effector pose estimation can help align the endoscope tip with the nodule (e.g., Figure 1 The nodules in the middle (89) are aligned. The end-effector pose estimation based on CT is described in more detail below.
[0077] End-effector pose estimation pipeline
[0078] Figure 4 An example block diagram is shown for an end-effector pose estimation pipeline 400 based on one or more implementations. Figure 4 Pipeline 400 visualizes the stages or sequence of stages involved in the end-effector pose estimation process. Specifically, Pipeline 400 focuses on data flow and data processing aspects.
[0079] As shown in the figure, in general, pipeline 400 may involve accessing image data 402 as a volumetric representation (e.g., a point cloud or a collection of 2D image slices that collectively provide a volumetric representation), extracting an end-effector from the image data, and analyzing the volumetric representation to determine the end-effector pose (position and / or orientation). In some embodiments, extracting the end-effector may involve thresholding the image data 402 to distinguish the portion corresponding to the medical device from its surrounding anatomical structures in the image data 402, and extracting the end-effector from the portion corresponding to the medical device. In some embodiments, analyzing the volumetric representation may involve performing principal component analysis (“PCA”) and / or calculating the centroid of the point cloud, and then determining the end-effector pose based on the principal axes and centroids. It should be understood that although this disclosure will be described based on the specific implementation techniques described above, other techniques and various variations are possible.
[0080] Pipeline 400 may involve accessing (e.g., receiving, retrieving, capturing, decompressing, etc.) image data 402. Image data 402 may correspond to data acquired by an external imaging device (e.g., Figure 1 Images captured by an imaging system 122 located outside the subject 7, which can be configured for CBCT imaging. In CBCT, a cone X-ray beam and a dedicated detector can capture multiple 2D X-ray images from different angles around the scanned area of the subject 7. A 3D volumetric representation or model of the scanned area can be reconstructed based on the multiple 2D X-ray images. In this document, depending on the context, the 2D images (which together form a 3D representation) or 3D representation may be referred to as image data 402.
[0081] Similar to other CT-based images, the radiodensity in image data 402 can be quantified based on Henle units (HU), a measurement scale used to quantify the radiodensity of tissues and materials. The Henle unit scale assigns values to different tissues and materials based on their X-ray attenuation characteristics, with metallic objects and dense structures that absorb more X-rays (more attenuation) having higher HU unit values compared to soft tissues and fluids (lower attenuation). In CT images, high-density material associated with higher HU unit values appears brighter (e.g., has greater luminance) than low-density material associated with lower HU unit values, and low-density material associated with lower HU unit values appears darker (e.g., has less luminance).
[0082] At thresholding box 404, pipeline 400 may involve filtering image data 402 to distinguish the endoscope and / or its tip (“endoscope tip”) from surrounding anatomical structures. For example, filtering image data 402 may include comparing voxels of image data 402 to at least one threshold. The threshold may be a HU unit value or an intensity value (e.g., a luminance intensity value), which may be a predefined value or a dynamically calculated value based on known (or estimated) properties of the endoscope. As an example of a predefined threshold HU unit value, a threshold level of +1000 may be associated with gold (Au), or a threshold level of +800 may be associated with titanium (Ti). As a dynamically calculated value, the HU unit value to be applied to a specific image data 402 may be calculated based on the global image luminance intensity, which may help distinguish materials with a wide range of varying HU unit values, such as ceramics.
[0083] Figure 5A Figure 5C illustrates example images associated with a region extraction process according to one or more implementation schemes. More specifically, Figure 5AFigures 500 to 510 illustrate pre-thresholding image 500 and post-thresholding image 510 before and after the thresholding process, respectively. Pre-thresholding image 500 depicts the surrounding tissue 502, the endoscope 504a, and the endpiece 506a on the endoscope. As shown in post-thresholding image 510, thresholding can distinguish the endoscope 504b and / or the endpiece 506b from the surrounding tissue 502 based on a threshold. In some specific implementations, post-thresholding image 510 may be a binary image representing the endoscope 504b, the endpiece 506b, and / or any other metallic object, where a first value (e.g., "1") is equal to or higher than the threshold, and a second value (e.g., "0") is lower than the threshold. Such thresholding would exclude all tissue 502 in post-thresholding image 510. Post-thresholding image 510 can be provided to region extraction box 406.
[0084] At region extraction box 406, pipeline 400 may involve extracting a region of interest or volume (also referred to herein as "VOI" or "region") including the end of the thresholded image. To accurately identify the VOI, in some embodiments, a trocar end selection 408 may be provided to the VOI extraction box 406. For example, an indication corresponding to end 506b (also referred to herein as "seed" or "seed location 508") may be received by the region extraction box 406 of the thresholded image 510. Seed location 508 may be provided as user input (such as by...). Figure 1 The approximate position of the end 506b provided by the operator 5). This can be relative to, for example, a touch-sensitive display (such as...). Figure 2 The presentation of any one of image data 402, image 500 before thresholding, and / or image 510 after thresholding on the display 212) provides for manual selection of seed position 508.
[0085] The VOI (Voice of Interest) surrounding the seed location can be extracted. Figure 5C illustrates an example VOI 520 including the end 506c extracted from the thresholded image 510 of Figure 5B based on the seed location 508. The size of the VOI 520 can be configured based on the geometry of the endoscope tip. For example, when the endoscope tip has a known size of less than 6 mm, the size of the VOI 520 can be predefined as a 10 mm × 10 mm region in 2D space.
[0086] The extracted VOI 520 in Figure 5C is an example of region extraction from a slice of a CT 2D image; however, it should be understood that region extraction can be performed on 3D representations (e.g., CBCT images reconstructed as image data 402) in a similar manner. For example, a 3D extraction region can be extracted based on the known size of the endoscope tip and the seed location 508, which can be a volume region of 10 mm × 10 mm × 10 mm surrounding the seed location 508.
[0087] At point cloud generation box 410, pipeline 400 may involve generating a point cloud based on the extracted VOI. For example, the point cloud may be generated based on the extracted VOI described with respect to region extraction box 406. In some examples, the points in the point cloud are uniformly distributed in 3D, and each point may appear identical to the others. For example, each point may correspond to pixels / voxels that are uniformly spaced in the thresholded image 510, without distinguishing between pixels / voxels corresponding to the end of a endoscopic endoscope and those corresponding to medical instruments. Figure 6 An example point cloud 602 is illustrated according to one or more implementation schemes. In some specific implementations, point cloud 602 may satisfy (or not satisfy) certain conditions. Figure 4 The threshold processing box 404 is the set of all voxels in the extracted VOI.
[0088] exist Figure 6 In the example, the outline 610 of a 3D model (e.g., a computer-aided design (CAD) model) of the endpiece is overlaid on top of the point cloud 602 to provide a sense of which points of the point cloud 602 correspond to which part of the endpiece. Specifically, the 3D model shows a working channel 612 (e.g., formed inside the endpiece and extending along the length of the endpiece) corresponding to the working channel 612 formed inside the endpiece and extending along the length of the endpiece. Figure 2 The structure of the working channel 44). The point cloud 602 may include medical tool points 604, which are mapped to a medical tool (e.g., medical tool 35) when the image data 402 from which the point cloud is generated reflects an endoscope having a medical tool at least partially housed within the working channel 612. That is, the medical tool points 604 are not part of the endoscope, but part of the medical tool. To help distinguish the endoscope tip from the medical tool, each medical tool point 604 is depicted with a "+". Note that the medical tool is made of a dense material (e.g., above the threshold described with respect to threshold processing box 440), which can be extracted as part of the extracted VOI from which the medical tool points 604 are generated.
[0089] At Principal Component Analysis (PCA) box 412, pipeline 400 may involve performing Principal Component Analysis (PCA) to determine one or more principal axes (also called “principal components”) of the point cloud that have the maximum variance. The PCA applied to the point cloud can find the principal axis that captures the direction of the highest variance in the point cloud. From the three possible principal axes of the point cloud in 3D space, the principal axis associated with the maximum variance (also called the “primary axis”) can be selected.
[0090] Referring to point cloud 602, performing PCA can find principal axis 606 (such as the principal axis providing the maximum variance in the point cloud). Principal axis 606 is an axis that extends infinitely relative to the reference point and has no sense of direction. PCA box 412 can further determine the centroid of point cloud 602, which is similar to the centroid, and principal axis 606 can be divided into two possible directions, each away from the centroid that can be used as a reference point. Either of the two possible directions can correspond to an end-effector direction (e.g., the directional component of a 3D vector). In this disclosure, the end-effector direction can be referred to as the "scope end-effector direction of travel" or "scope forward direction," either of which reflects the end-effector orientation.
[0091] Figures 7A and 7B illustrate example renders 700 and 750, respectively, associated with the process of determining the forward direction of the endoscope tip according to one or more embodiments. More specifically, Figure 7A shows a point cloud 702 with a centroid 712 to which a first possible direction 706a of the endoscope tip is mapped, while Figure 7B shows a second possible direction 706b of the endoscope tip, which is mapped to the centroid 712 of the point cloud 702. In some specific embodiments, the point cloud 702 may be... Figure 6 An example of point cloud 602. To determine the direction in which the end of the endoscope is pointing, directions 706a and / or 706b can be compared with a reference direction originating from centroid 712 and pointing towards seed 708 (shown by the arrows or vectors between centroid 712 and seed 708 in Figures 7A and 7B). In some specific implementations, seed 708 can be an example of seed position 508 in Figure 5. For example, seed 708 can be provided via user input.
[0092] Rendering 700 shows a first direction 706a with an offset angle (θ1) greater than 90 degrees from the reference direction. In contrast, rendering 750 shows a second direction 706b with an offset angle (θ2) less than 90 degrees from the reference direction. Therefore, the second direction 706b is better aligned with the reference direction than the first direction 706a. Therefore, the second direction 706b can be selected as the direction the end of the endoscope points. In a vector-based implementation, the determination of the endoscope direction can be performed based on whether the dot product of the first direction 706a (as the first vector) and the reference direction (as the second vector) is positive or negative. If the dot product is positive, each of the first and second vectors has components in the same direction. In this case, the first direction 706a can be selected as the endoscope direction. If the dot product is negative, one vector has a component in the opposite direction to the other. In this case, a direction opposite to the first direction 706a (e.g., the second direction 706b) can be selected as the endoscope direction.
[0093] When medical instruments are positioned within the working channel of a endoscopicoscope, the process for determining the orientation of the endoscope tip can be enhanced or improved. For example, regarding point cloud generation box 410 and... Figure 6 As described, a medical tool made of a dense material (e.g., a material with a radioactivity density above a threshold and retained in the threshold processing box 404) and positioned within the working channel increases the number of points in the point cloud associated with the endoscope. These additional points (e.g., in...) Figure 6 The medical tool point 604 (represented by a "+") can extend the point cloud 702 beyond the end of the endoscope, thus changing the position of the centroid 712. In some implementations, the working channel removal box 414 and the centroid recalculation box 416 can be used to refine the direction of travel of the endoscope end when the medical tool is inserted into the working channel.
[0094] At work channel removal box 414, pipeline 400 may involve filtering or removing medical tool points from the point cloud. (Reference) Figure 6 For example, medical tool points 604 (e.g., "+" points) can be removed from point cloud 602. In some implementations, the removal of medical tool points 604 can be based on the known geometry of the working channel 612. For example, when the working channel 612 has a known diameter, assuming the main axis is aligned with the centerline of the endoscope tip, the working channel removal box 414 can remove each medical tool point 604 from point cloud 602. After removing medical tool points 604, point cloud 602 can become a hollow cylindrical point cloud (e.g., leaving a hollow cylinder where medical tool points 604 were once located).
[0095] At centroid recalculation box 416, pipeline 400 may involve recalculating the altered centroid of the point cloud after filtering or removing medical tool points in the working channel. For example, after working channel removal box 414, the centroid may be recalculated for a hollow cylindrical point cloud. The recalculated centroid more accurately reflects the centroid of the endoscope tip compared to the centroid calculated at PCA box 412.
[0096] In some implementations, the direction of the endoscope's advance can be updated based on the recalculated centroid. Figure 4 The updated endoscope forward direction 418 is then described. In some embodiments, the updated endoscope forward direction 418 may be represented as a vector derived from a recalculated centroid aligned with the principal axis of the filtered point cloud (such as the principal axis with the maximum variance) and pointing towards a selected endoscope direction on the principal axis (as described with reference to Figures 7A and 7B). Thus, the recalculated centroid can help refine the endoscope position to provide an updated endoscope position 422.
[0097] At the farthest point projection frame 420, pipeline 400 may involve projecting the farthest point in the point cloud onto its principal axis. For example, Figure 8 An example rendering 800 is shown, illustrating a process associated with determining the position of the endoscope tip according to one or more implementations. More specifically, Figure 8 A point cloud 802 is shown with a centroid 812 and a farthest point 810 projected onto its principal axis. In some embodiments, the farthest point 810 in the point cloud 802 can be selected as the point farthest from the centroid 812 in the point cloud 802, which is close to the seed position 708 (e.g., within a threshold proximity of the seed position 708). The projection of the farthest point 810 onto the principal axis can be used to determine the updated end position 814 on the principal axis as the updated end position 422 of the endoscope.
[0098] The direction of travel of the endoscope can represent the end-effector orientation, and the end-effector position can represent the end-effector position. Therefore, pipeline 400 can use externally captured images to estimate the end-effector pose. It should be understood that the boxes depicted in pipeline 400 are merely exemplary, and fewer or more boxes, as well as boxes organized in different orders, may be involved in the end-effector pose estimation. For example, without involving the medical instrument, the work channel may be skipped, box 414 removed, and the centroid recalculated, box 416 recalculated. Many other variations are also possible.
[0099] Machine pose estimation workflow
[0100] Figure 9 A flowchart illustrating an exemplary device pose estimation process 900 according to one or more embodiments is provided. In some specific embodiments, the device may be a endoscopic endoscope (such as...). Figure 1 and Figure 2 (The endoscope 40). When followed, process 900 can determine the direction of advance of the endoscope and the position of the endoscope tip from the externally captured image.
[0101] At box 902, process 900 may involve accessing image data of multiple images captured by an imaging sensor positioned outside the subject's anatomical structures. The images may be CT images, including CBCT images. Each of the multiple images may depict an object within the anatomical structure.
[0102] At box 904, process 900 may involve receiving an indication corresponding to the distal end of an object depicted in at least one of a plurality of images. The object may be a hysteroscope, and the distal end may be the tip of the hysteroscope. In some embodiments, an operator may manually provide the indication by selecting one or more pixels of the presented image. This can be achieved by including, for example... Figure 1 Use the various input controls shown on the touchscreen to make selections.
[0103] At box 906, process 900 may involve extracting the VOI surrounding the indication from the image. The VOI may be extracted based on the known geometry of the distal end of the object. For example, the endoscope may have a diameter of less than 5.7 mm, and the extracted VOI may be configured to be 10 mm × 10 mm × 10 mm.
[0104] At box 908, process 900 may involve generating a 3D representation of the distal end based on the VOI. This 3D representation may be a point cloud of all points known to be associated with the object or its distal end. Whether a point is associated with the object can be determined based on various imaging processing techniques, including thresholding.
[0105] At box 910, process 900 may involve determining the position and orientation of the distal end based on the VOI. This orientation can be determined as a vector originating from a reference point and having a direction pointing outwards from the distal end. The reference point can be the centroid in 3D representation. Orientation can be determined using PCA in 3D representation, and directions can be filtered / selected based on indications. The position can be determined by projecting the farthest point (farthest from the centroid) onto the principal axis determined by PCA.
[0106] It should be understood that the boxes depicted in process 900 are exemplary, and fewer or more boxes, as well as boxes organized in different orders, may be involved in end-effector pose estimation. Many variations are possible.
[0107] Figure 10 Another block diagram of an example controller 1000 for a medical system according to some specific implementations is shown. In some specific implementations, controller 1000 may be... Figure 2An example of control circuitry 251 and / or control circuitry 211. More specifically, controller 1000 is configured to estimate the pose of an object within an anatomical structure. For example, see reference... Figure 4 The controller 100 can implement one or more stages of the end-effector pose estimation pipeline 400.
[0108] The controller 1000 includes a communication interface 1010, a processing system 1020, and a memory 1030. The communication interface 1010 is configured to communicate with one or more components of the medical system. More specifically, the communication interface 1010 includes features for communicating with one or more image sources (such as...) Figure 3 The image source interface (I / F) 1012 communicates with the CT imaging system 310 and / or the fluorescence fluoroscopy imaging system 312. In some embodiments, the image source I / F 1012 may receive image data representing a 3D model of an anatomical structure in which instruments are disposed.
[0109] The memory 1030 may include a non-transitory computer-readable medium (including one or more non-volatile memory elements, such as EPROM, EEPROM, flash memory, or hard disk drive, etc.) that stores the following software (SW) modules: a point cloud generation SW module 1032 for generating a point cloud associated with the distal end of the instrument based on image data; a principal axis determination SW module 1034 for determining a principal axis that maximizes the variance of the point cloud; and a pose determination SW module 1036 for determining the pose of the distal end of the instrument based at least in part on the principal axis.
[0110] Processing system 1020 may include any suitable one or more processors capable of executing scripts or instructions of one or more software programs stored in controller 1000, such as in memory 1030. For example, processing system 1020 may execute point cloud generation SW module 1032 to generate a point cloud associated with the distal end of the instrument based on image data. Processing system 1020 may also execute principal axis determination SW module 1034 for determining the principal axis that maximizes the variance of the point cloud. Processing system 1020 may also execute pose determination SW module 1036 to determine the pose of the distal end of the instrument, at least in part, based on the principal axis.
[0111] Figure 11 An exemplary flowchart depicting an example pose estimation operation 1100 according to some specific implementations is shown. In some specific implementations, example operation 1100 may be provided by a controller for a medical system (such as...) Figure 10 The controller 1000) executes.
[0112] The controller receives image data (1302) representing a 3D model of an anatomical structure in which an instrument is disposed. The controller generates a point cloud (1304) associated with the distal end of the instrument based on the image data. The controller determines a first principal axis (1306) that maximizes the variance of the point cloud. The controller also determines the pose of the distal end of the instrument based at least in part on the first principal axis (1308).
[0113] In some aspects, the controller may also filter image data based on one or more known characteristics of the instrument, such that the filtered image data includes voxels associated with the instrument and excludes voxels associated with anatomical structures, wherein the point cloud is generated based on the filtered image data. In some specific implementations, filtering the image data may include comparing the intensity of each voxel of the image data with a threshold intensity associated with one or more known characteristics of the instrument, and assigning a binary value to each voxel of the image data based on whether the voxel intensity exceeds the threshold intensity.
[0114] In some aspects, the controller can also receive a seed location associated with the distal end of the instrument in the 3D model and extract the volume of interest (VOI) around the seed location in the 3D model, where the point cloud is generated based on the VOI. In some implementations, the VOI can be configured based on the known geometry of the instrument.
[0115] In some aspects, the controller may also identify a portion of the point cloud associated with the instrument's working channel, remove the identified portion from the point cloud, and determine a second principal axis that maximizes the variance of the point cloud after removing the portion associated with the working channel, wherein the pose of the distal end of the instrument is determined at least in part based on the second principal axis. In some implementations, the portion of the point cloud associated with the working channel may be identified based on a first principal axis, the centroid of the point cloud, and the known diameter of the working channel.
[0116] In some aspects, determining the pose of the distal end of an apparatus may include determining a reference direction associated with the orientation of the distal end of the apparatus, comparing the reference direction with a first direction derived from the centroid of the point cloud along a first principal axis, comparing the reference direction with a second direction opposite to the first direction along the first principal axis, and determining the orientation of the distal end of the apparatus based on the comparison of the reference direction with the first and second directions. In some specific implementations, determining the reference direction may include receiving a seed position associated with the distal end of the apparatus in the 3D model, and determining a vector from the centroid of the point cloud to the seed position, the vector representing the reference direction.
[0117] In some aspects, determining the pose of the distal end of the instrument may include receiving a seed position associated with the distal end of the instrument in a 3D model, selecting a point in the point cloud that is farthest from the centroid of the point cloud based on the seed position, and projecting the selected point onto a first principal axis, wherein the projected point represents the position of the distal end of the instrument.
[0118] Additional Implementation Plan
[0119] Depending on the implementation, certain actions, events, or functions of any of the processes or algorithms described herein may be performed in different orders, added, combined, or completely ignored. Therefore, in some implementations, not all described actions or events are necessary for the practice of the process.
[0120] Unless otherwise specifically stated or otherwise understood in the context in which they are used, the conditional language used herein, such as “may,” “can,” “possibly,” “can,” “e.g.,” etc., refers in its ordinary sense and is generally intended to convey that certain embodiments include certain features, elements, and / or steps that are not included in other embodiments. Therefore, such conditional language is not generally intended to imply that one or more embodiments require features, elements, and / or steps in any way, or that one or more embodiments necessarily include logic for determining, with or without author input or prompting, whether such features, elements, and / or steps are included in any particular embodiment or whether they will be performed in any particular embodiment. The terms “comprising,” “including,” “having,” etc., are synonymous and used in their ordinary sense, and are used inclusively in an open-ended manner, without excluding additional elements, features, actions, operations, etc. Moreover, the term “or” is used in its inclusive sense (rather than in its exclusive sense) such that when used, for example, to connect a series of elements, the term “or” refers to one, some, or all of the elements in that series. Unless otherwise specified, combined language such as “at least one of X, Y, and Z” is understood in the context of general use to convey that an item, term, element, etc., can be X, Y, or Z. Therefore, such combined language is generally not intended to imply that a particular implementation requires the presence of at least one of X, at least one of Y, and at least one of Z.
[0121] It should be understood that in the above description of the embodiments, various features are sometimes grouped together in a single embodiment, figure, or description therein in order to simplify this disclosure and aid in understanding one or more aspects of the various inventive aspects. However, this approach of the disclosure should not be construed as reflecting an intention that any claim requires more features than those expressly recited in that claim. Furthermore, any component, feature, or step shown and / or described in the specific embodiments herein may be applied to or used with any other embodiment. Moreover, for each embodiment, no component, feature, step, or group of components, features, or steps is necessary or indispensable. Therefore, it is expected that the scope of the invention disclosed herein and claimed below is not limited to the specific embodiments described above, but should be determined solely by a fair reading of the appended claims.
[0122] It should be understood that certain ordinal terms (e.g., "first" or "second") may be provided for ease of reference and do not necessarily imply physical characteristics or ordering. Therefore, as used herein, ordinal terms used to modify elements such as structures, components, operations, etc. (e.g., "first," "second," "third," etc.) do not necessarily indicate the priority or order of that element relative to any other element, but rather serve to generally distinguish that element from another element with a similar or identical name (but used in ordinal terms). Additionally, as used herein, the indefinite article ("a") may indicate "one or more" rather than "one." Furthermore, an operation performed "based on" a certain condition or event may also be performed based on one or more other conditions or events not explicitly stated.
[0123] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the example embodiments pertain. It will be further understood that terms, such as those defined in common dictionaries, shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and shall not be interpreted in an idealized or overly formal sense unless explicitly defined herein. As used herein, the term “patient” may generally refer to a human being, an anatomical model, a simulator, a cadaver, and other living or non-living objects.
[0124] For ease of description, the spatial relative terms “external,” “internal,” “upper,” “lower,” “below,” “above,” “vertical,” “horizontal,” and similar terms are used herein to describe the relationship between one element or component shown in the accompanying drawings and another. It should be understood that the spatial relative terms are intended to cover different orientations of the device in use or operation other than those depicted in the accompanying drawings. For example, if the device shown in the accompanying drawings is flipped, a device located “below” or “under” another device may be positioned “above” another device. Therefore, the illustrative term “below” can include both a lower position and an upper position. The device may also be oriented in another direction, and therefore the spatial relative terms may be interpreted differently depending on the orientation.
[0125] Unless otherwise explicitly stated, comparative and / or quantitative terms, such as “less,” “more,” “greater,” etc., are intended to encompass the concept of equality. For example, “less” may mean not only “less” in the strictest mathematical sense, but may also mean “less than or equal to.”
Claims
1. A controller for a medical system, comprising: Processing system; and The memory stores instructions that, when executed by the processing system, cause the controller to: Receive image data representing a three-dimensional (3D) model of an anatomical structure in which instruments are set; A point cloud associated with the distal end of the device is generated based on the image data; Determine the first principal axis that maximizes the variance of the point cloud; and The pose of the distal end of the instrument is determined at least in part based on the first principal axis.
2. The controller according to claim 1, wherein, The execution of the instruction further enables the controller to: The image data is filtered based on one or more known characteristics of the instrument, such that the filtered image data includes voxels associated with the instrument and excludes voxels associated with the anatomical structure, and the point cloud is generated based on the filtered image data.
3. The controller according to claim 2, wherein, Filtering the image data includes: The intensity of each voxel of the image data is compared with a threshold intensity, the threshold intensity being associated with one or more known characteristics of the device; and A binary value is assigned to each voxel of the image data based on whether the intensity of the voxel exceeds the threshold intensity.
4. The controller according to claim 1, wherein, The execution of the instruction further enables the controller to: Receive the seed position associated with the distal end of the device in the 3D model; and Extract the volume of interest (VOI) around the seed location in the 3D model, and the point cloud is generated based on the VOI.
5. The controller according to claim 4, wherein, The VOI is configured based on the known geometry of the instrument.
6. The controller according to claim 1, wherein, The execution of the instruction further enables the controller to: Identify a portion of the point cloud associated with the working channel of the instrument; Remove the portion associated with the working channel from the point cloud; as well as A second principal axis is determined that maximizes the variance of the point cloud after removing the portion associated with the working channel, and the pose of the distal end of the instrument is determined at least in part based on the second principal axis.
7. The controller according to claim 6, wherein, The portion of the point cloud associated with the working channel is identified based on the first principal axis, the centroid of the point cloud, and the known diameter of the working channel.
8. The controller according to claim 1, wherein, Determining the pose of the distal end of the device includes: Determine a reference direction associated with the orientation of the distal end of the instrument; The reference direction is compared with a first direction along the first principal axis originating from the centroid of the point cloud; The reference direction is compared with a second direction originating from the centroid of the point cloud along the first principal axis; and The orientation of the distal end of the device is determined by comparing the reference direction with the first direction and the second direction.
9. The controller according to claim 8, wherein, Determining the reference direction includes: Receive the seed position associated with the distal end of the device in the 3D model; and Determine a vector pointing from the centroid of the point cloud to the seed location, the vector representing the reference direction.
10. The controller according to claim 1, wherein, Determining the pose of the distal end of the instrument includes: Receive the seed position associated with the distal end of the device in the 3D model; Based on the seed location, select the point in the point cloud that is farthest from the centroid of the point cloud; and The selected point is projected onto the first principal axis, and the projected point represents the position of the distal end of the instrument.
11. A pose estimation method, comprising: Receive image data representing a three-dimensional (3D) model of an anatomical structure in which instruments are set; A point cloud associated with the distal end of the device is generated based on the image data; Determine the first principal axis that maximizes the variance of the point cloud; and The pose of the distal end of the instrument is determined at least in part based on the first principal axis.
12. The method of claim 11, further comprising: The image data is filtered based on one or more known characteristics of the instrument, such that the filtered image data includes voxels associated with the instrument and excludes voxels associated with the anatomical structure, and the point cloud is generated based on the filtered image data.
13. The method according to claim 12, wherein, Filtering the image data includes: The intensity of each voxel of the image data is compared with a threshold intensity, the threshold intensity being associated with one or more known characteristics of the device; and A binary value is assigned to each voxel of the image data based on whether the intensity of the voxel exceeds the threshold intensity.
14. The method of claim 11, further comprising: Receive the seed position associated with the distal end of the device in the 3D model; as well as Extract the volume of interest (VOI) around the seed location in the 3D model, and the point cloud is generated based on the VOI.
15. The method according to claim 14, wherein, The VOI is configured based on the known geometry of the instrument.
16. The method of claim 11, further comprising: Identify a portion of the point cloud associated with the working channel of the instrument; Remove the portion associated with the working channel from the point cloud; as well as A second principal axis is determined that maximizes the variance of the point cloud after removing the portion associated with the working channel, and the pose of the distal end of the instrument is determined at least in part based on the second principal axis.
17. The method according to claim 16, wherein, The portion of the point cloud associated with the working channel is identified based on the first principal axis, the centroid of the point cloud, and the known diameter of the working channel.
18. The method according to claim 11, wherein, Determining the pose of the distal end of the device includes: Determine a reference direction associated with the orientation of the distal end of the instrument; The reference direction is compared with a first direction along the first principal axis originating from the centroid of the point cloud; Compare the reference direction with a second direction along the first principal axis that is opposite to the first direction; and The orientation of the distal end of the device is determined by comparing the reference direction with the first direction and the second direction.
19. The method according to claim 18, wherein, Determining the reference direction includes: Receive the seed position associated with the distal end of the device in the 3D model; and Determine a vector pointing from the centroid of the point cloud to the seed location, the vector representing the reference direction.
20. The method according to claim 11, wherein, Determining the pose of the distal end of the device includes: Receive the seed position associated with the distal end of the device in the 3D model; Based on the seed location, select the point in the point cloud that is farthest from the centroid of the point cloud; and The selected point is projected onto the first principal axis, and the projected point represents the position of the distal end of the instrument.
