Pose estimation using machine learning

Abstract

The present disclosure provides methods, apparatuses, and systems for pose estimation. The present implementations more specifically relate to techniques for determining a pose of a medical instrument within an anatomical structure. In some aspects, a controller for a medical system can receive image data representative of a three-dimensional (3D) model of an anatomical structure in which an instrument is disposed. The controller generates a point cloud associated with a distal end of the instrument based on the image data, and determines a pose of the distal end of the instrument based at least in part on the point cloud and a known geometry of the distal end of the instrument.

Description

Cross-references to related applications

[0001] This patent application claims priority to U.S. Non-Provisional Patent Application No. 19 / 085,919, filed March 20, 2025, entitled “POSE ESTIMATION USING MACHINE LEARNING,” which claims priority and benefit to U.S. Provisional Patent Application No. 63 / 572,063, filed March 29, 2024, entitled “POSE ESTIMATION USING MACHINE LEARNING,” pursuant to 35 USC § 119(e). The disclosure of the earlier patent application is considered part of this patent application and is incorporated herein by reference. Technical Field

[0002] This disclosure relates to pose estimation of objects, and more particularly to pose estimation using machine learning. Background Technology

[0003] During medical procedures, various views of the subject's anatomy are presented to the physician. These views may include, for example, preoperative model views, three-dimensional (3D) reconstructed model views, computed tomography (CT) image views, endoscopic image views, etc. As part of the medical procedure, the physician may need to accurately identify the location and orientation of the depicted object. Summary of the Invention

[0004] The present invention is presented in a simplified form to introduce some concepts, which will be further described in the detailed embodiments below. This summary is not intended to identify key or essential features of the subject matter protected by the claims, nor is it intended to limit the scope of the subject matter protected by the claims.

[0005] One innovative aspect of the subject matter of this disclosure can be implemented in a controller for a medical system, the controller including a memory and a processor. The memory stores instructions that, when executed by a processing system, cause the controller to: receive image data representing a three-dimensional (3D) model of an anatomical structure in which an instrument is disposed; generate a point cloud associated with a distal end of the instrument based on the image data; and determine the pose of the distal end of the instrument based at least in part on the point cloud and the known geometry of the distal end of the instrument.

[0006] Another innovative aspect of the subject matter of this disclosure can be implemented in a pose estimation method. The method includes the steps of: receiving image data representing a 3D model of an anatomical structure in which an instrument is disposed; generating a point cloud associated with the distal end of the instrument based on the image data; and determining the pose of the distal end of the instrument based at least in part on the point cloud and the known geometry of the distal end of the instrument. Attached Figure Description

[0007] Various embodiments are depicted in the accompanying drawings for illustrative purposes and should in no way be construed as limiting the scope of the invention. Furthermore, various features of different disclosed embodiments may be combined to form additional embodiments that are part of this disclosure. Throughout the drawings, reference numerals may be repeated to indicate correspondences between reference elements.

[0008] Figure 1 An example medical system based on one or more examples is shown.

[0009] Figure 2 Examples are given based on one or more implementation schemes. Figure 1 A schematic diagram of the different components of a medical system.

[0010] Figure 3 Block diagrams illustrating various positions and / or imaging systems / modalities are shown based on one or more examples.

[0011] Figure 4 An example block diagram illustrates an end-effector pose estimation pipeline based on one or more implementations.

[0012] Figure 5A and Figure 5B Example images associated with the segmentation mask generation process according to one or more implementation schemes are illustrated.

[0013] Figure 6 Example point clouds based on one or more implementation schemes are shown.

[0014] Figure 7A and Figure 7B Example renderings are shown that are associated with the process of determining the forward direction of the endoscope according to one or more implementations.

[0015] Figure 8A and Figure 8B Example renderings are shown that are associated with the end-of-scope alignment process according to one or more implementation schemes.

[0016] Figure 9 Example renderings are shown that are associated with the process of determining the position of the end of the endoscope according to one or more implementations.

[0017] Figure 10A flowchart illustrating an exemplary device pose estimation process according to one or more embodiments is provided.

[0018] Figure 11 A block diagram of an example controller for a medical system, based on some specific implementations, is shown.

[0019] Figure 12 An exemplary flowchart illustrating pose estimation operations based on some specific implementations is shown. Detailed Implementation

[0020] The headings provided herein are for convenience only and do not necessarily affect the scope or meaning of the claimed invention. Although specific preferred embodiments and examples are disclosed below, the subject matter of the invention extends beyond the specifically disclosed embodiments to other alternative embodiments and / or uses, as well as modifications and equivalents thereof. Therefore, the scope of the claims that may appear herein is not limited to any particular embodiment of the specific embodiments described below. For example, in any method or process disclosed herein, the actions or operations of the method or process may be performed in any suitable sequence and are not necessarily limited to any particular disclosed sequence. Various operations may then be described as multiple discrete operations in a manner that may aid in understanding certain embodiments; however, the order of description should not be construed as implying that these operations depend on the order. Furthermore, the structures, systems, and / or apparatuses described herein may be embodied as integrated components or separate components. Certain aspects and advantages of these embodiments are described for the purpose of comparing the various embodiments. Not all such aspects or advantages are necessarily achieved by any particular embodiment. Therefore, for example, various embodiments may be performed by implementing or optimizing one or a set of advantages taught herein without necessarily achieving other aspects or advantages that may also be taught or proposed herein.

[0021] Although specific spatial relative terms such as “external,” “internal,” “upper,” “lower,” “below,” “above,” “vertical,” “horizontal,” “top,” “bottom,” “outer,” “upward,” “side,” and similar terms are used herein to describe the spatial relationship between one device / element or anatomical structure and another device / element or anatomical structure, it should be understood that these terms are used herein for descriptive convenience to describe the positional relationship between elements / structures, such as with respect to the illustrated orientations in the accompanying drawings. It should be understood that spatial relative terms are intended to cover different orientations of elements / structures in use or operation other than those depicted in the accompanying drawings. For example, an element / structure described as “above” another element / structure may indicate a position below or beside such other element / structures relative to an alternative orientation of the subject or element / structure, and vice versa. It should be understood that spatial relative terms, including those listed above, can be understood relative to the corresponding illustrated orientations in the reference drawings.

[0022] To facilitate the use of devices, components, systems, features, and / or modules having similar features in one or more aspects, specific reference numerals are repeated in different figures within the group of figures disclosed herein. However, the repeated use of common reference numerals in the figures with respect to any embodiment of the embodiments disclosed herein does not necessarily indicate that such features, devices, components, or modules are identical or similar. Rather, those skilled in the art will be informed by the context that the use of common reference numerals may imply the degree of similarity between the referenced subjects. The use of specific reference numerals in the context of the description of a particular figure may be understood to refer to the device, component, aspect, feature, module, or system identified in that particular figure, and not necessarily to any device, component, aspect, feature, module, or system identified by the same reference numeral in another figure. Furthermore, aspects of individual figures identified by common reference numerals may be interpreted as sharing characteristics or being completely independent of each other. In some contexts, features associated with individual figures identified by common reference numerals are unrelated and / or similar with respect to at least some aspects.

[0023] Overview

[0024] This disclosure provides systems, apparatus, and methods for estimating the pose (e.g., position and / or orientation) of the distal end of a medical device having a flexible, elongated body within the anatomical structure of a subject based on images captured outside the subject. As an example, the distal end may be the distal tip of an endoscope.

[0025] Existing end-effector pose estimation techniques may not adequately address inaccuracies caused by variations in the shape and curvature of the flexible, slender body, the geometry of the end-effector, various imaging artifacts, and the presence of medical instruments within the body's working channel. Furthermore, these techniques typically involve one or more manual steps, during which errors may be introduced.

[0026] This disclosure discloses an automated process using machine learning to address the aforementioned drawbacks by estimating the end-effector pose in a three-dimensional (3D) space generated from externally captured images. The images can be computed tomography (CT) images, including cone-beam computed tomography (CBCT) images. Such CT images provide a third-person perspective of the end-effector pose compared to a first-person perspective captured from an endoscopic imaging device inside the patient. The estimated end-effector pose can provide another reference point when aligning the end-effector with a target during medical procedures. For example, the estimated end-effector pose based on CT images can aid in aligning a biopsy needle with a nodule during robot-assisted bronchoscopy.

[0027] Example medical system

[0028] Figure 1 An example medical system 100 (also referred to as "surgical medical system 100" or "robotic medical system 100") according to one or more examples is illustrated. For example, medical system 100 may be configured for diagnostic and / or therapeutic bronchoscopy, as shown. Medical system 100 may include and utilize robotic system 10, which may be implemented as, for example, a robotic cart. Although medical system 100 is shown as including various cart-based systems / devices, the concepts disclosed herein can be implemented in any type of robotic system / arrangement, such as robotic systems employing track-based components, stage-based robotic end effectors / manipulators, etc. Robotic system 10 may include one or more robotic arms 12 (also referred to as "robotic locators") configured to locate or otherwise manipulate medical instruments, such as medical instrument 32 (e.g., a steerable endoscope or another elongated instrument with a flexible, elongated body). For example, medical instrument 32 may be advanced through a natural orifice entry point (e.g., the mouth 9 of subject 7 positioned on stage 15 in this example) to deliver diagnostic and / or therapeutic treatments. Although described in the context of a bronchoscopy procedure, the medical system 100 can be implemented for other types of procedures, such as gastrointestinal (GI) procedures, renal / urinary / nephrology procedures, etc. The term "subject" is used herein to refer to a living patient as well as any subject to which this disclosure applies. For example, "subject" can refer to a subject including those in physical anatomical models used in dry runs (e.g., anatomical education models, anatomical models, medical education anatomical models, etc.), models in computer simulations, etc., which encompass non-living patients or test subjects.

[0029] With the robotic system 10 correctly positioned, the medical device 32 can be inserted into the subject 7 robotically, manually, or in combination thereof. In this example, one or more robotic arms 12 and / or their instrument actuators 28 can control the medical device 32. The instrument actuators 28 can reposition the device in space by manipulating one or more robotic arms 12 to different angles and / or positions.

[0030] Medical system 100 may also include control system 50 (also referred to as "control tower" or "mobile tower"), which will be referenced below. Figure 2 Detailed Description. The control system 50 may include one or more displays 212 to provide / display / present various information related to medical procedures, such as anatomical images. The control system 50 may additionally include one or more control mechanisms, which may be individual directional input controls 216 or graphical user interfaces (GUIs) presented on the displays 212.

[0031] In some implementations, display 212 may be a touch-sensitive display, as shown, that can present anatomical images and allow selection thereon. Some example anatomical images may include CT images, fluorescein images, images of anatomical mapping, etc. Using a touch-sensitive display, the operator 5 viewing the images may find it convenient to use touch-based selection instead of using directional input controls 216 to identify targets within the image (e.g., target objects or regions of interest). For example, operator 5 can use the touchscreen to select the endoscope tip and / or nodules.

[0032] The control system 50 can communicatively connect (e.g., via wired and / or wireless connections) to the robotic system 10 to provide support for controls, electronics, fluid dynamics, optics, sensors, and / or power. Placing such functionality within the control system 50 allows for easier adjustment and / or repositioning of the robotic system 10 by the operator 5, given its smaller form factor. Furthermore, the separation of functions between the robotic system 10 and the control system 50 can reduce operating room clutter and / or facilitate efficient clinical workflows.

[0033] Medical system 100 may include an electromagnetic (EM) field generator 120 configured to broadcast / transmit an EM field detected by an EM sensor, such as a sensor associated with medical device 32. The EM field may induce a small current in the coil of the EM sensor (also referred to as a “position sensor”), which may be analyzed to determine the pose (position and / or angle / or orientation) of the EM sensor relative to the EM field generator 120. In some embodiments, the EM sensor may be positioned at a distal end of medical device 32, and the pose of the distal end may be determined in conjunction with the pose of the EM sensor. Although EM fields and EM sensors are described in many examples herein, position sensing systems and / or sensors may be of any type, such as optical position sensing systems / sensors, image-based position sensing systems / sensors, etc.

[0034] The medical system 100 may also include an imaging system 122 (e.g., a fluorescence imaging system) configured to generate image data (also referred to as “images”) and / or provide / send such image data to / from another device / system. For example, the imaging system 122 may generate image data depicting the anatomical structures of the subject 7 and provide the image data to a control system 50, a robotic system 10, a web server, a cloud server, and / or another device. The imaging system 122 may include transmitters / energy sources (e.g., X-ray sources, ultrasound sources, etc.) and / or detectors (e.g., X-ray detectors, ultrasound detectors, etc.) integrated into a support structure (e.g., mounted on a C-arm support 124) that provides flexibility in positioning around the subject 7 to capture images from various angles without moving the subject 7. The use of the imaging system 122 can provide visualization of internal structures / anatomical structures for a variety of purposes, such as navigation of the medical device 32 (e.g., providing images of internal anatomy to the operator 5), positioning of the medical device 32 (e.g., based on an image data analysis system), etc. In the example, the use of imaging system 122 can enhance the effectiveness and / or safety of medical procedures such as bronchoscopy by providing clear, continuous visual feedback to operator 5.

[0035] In some examples, the imaging system 122 is a mobile device configured to move around within the environment. For example, the imaging system 122 may be positioned next to the subject 7 during a specific phase of the procedure (as illustrated in the figure) and removed when the imaging system 122 is no longer needed. In other examples, the imaging system 122 may be part of the table 15 or other equipment in the operating environment. The imaging system 122 may be implemented as a computed tomography (CT) machine / system, an X-ray machine / system, a fluoroscopic examination machine / system, a positron emission tomography (PET) machine / system, a PET-CT machine / system, a CT angiography machine / system, a cone-beam computed tomography (CBCT) machine / system, a 3DRA machine / system, a single-photon emission computed tomography (SPECT) machine / system, a magnetic resonance imaging (MRI) machine / system, an optical coherence tomography (OCT) machine / system, an ultrasound machine / system, etc. In some cases, medical system 100 includes multiple imaging systems, such as a first type of imaging system and a second type of imaging system, wherein different types of imaging systems may be used or positioned above subject 7 as needed at different stages / parts of the procedure.

[0036] In some implementations, the imaging system 122 may be configured to generate a three-dimensional (3D) model of the anatomical structure. For example, the imaging system 122 may be configured to process multiple images (also referred to in some cases as "image data") to generate a 3D model. For example, the imaging system 122 may be implemented as a CT scanner configured to capture / generate a series of images / image data (e.g., 2D images / slices) from different angles around the subject 7, and then reconstruct a 3D model from these images / image data using one or more algorithms. The 3D model may be provided to the control system 50, the robotic system 10, a web server, a cloud server, and / or another device, such as for processing, display, or other purposes.

[0037] For ease of description of this disclosure, Figure 1 The respiratory system is illustrated as an example anatomical structure. The respiratory system includes the upper respiratory tract, which comprises the nose / nasal cavity, pharynx (i.e., larynx), and larynx (i.e., vocal cord box). The respiratory system also includes the lower respiratory tract, which comprises the trachea (6), lungs (4) r and 4 l The bronchial tree consists of the main bronchus 71, which branches into smaller secondary bronchi 78 and tertiary bronchi 75, and terminates in even smaller tubes called bronchioles 77. Each bronchiole connects to a cluster of alveoli (not shown). During the inspiratory phase of the respiratory cycle, air enters through the mouth and nose and travels down the larynx into the trachea 6, through the right and left main bronchus 71 into the lungs 4, into the smaller bronchial airways 78 and 75, into the smaller bronchioles 77, and into the alveoli, where oxygen and carbon dioxide exchange occurs.

[0038] The bronchial tree is an example of an end-luminal network in which a robot-controlled instrument can be navigated and utilized according to the inventive solutions presented herein. However, although aspects of this disclosure are presented in the context of an end-luminal network including the airways (e.g., lumens, branches) of the lungs of a subject, some embodiments of this disclosure can be implemented in other types of end-luminal networks, such as renal networks, cardiovascular networks (e.g., arteries and veins), the gastrointestinal tract, the urinary tract, etc.

[0039] In some embodiments, imaging system 122 may be configured to capture / update / present images of anatomical structures intraoperatively using a CBCT imaging system. During CBCT imaging, subject 7 may be positioned on a stage 15 between an X-ray source and a detector mounted on a C-arm support 124, where the X-ray beam passes through the target anatomical structure and updates the acquired images intraoperatively. For example, regarding the subject's lung 4, one or more CBCT-captured images or reconstructed 3D models may be presented to operator 5 on a monitor 56. While CBCT has been described, it should be understood that this disclosure contemplates any other imaging techniques capable of providing 3D reconstruction, such as conventional CT imaging techniques.

[0040] Figure 2 Example components of a control system 50, a robotic system 10, and a medical device 32 according to one or more examples are illustrated. The control system 50 may be coupled to and cooperate with the robotic system 10 to perform medical procedures. For example, the control system 50 may include a communication interface 202 for communicating with the robotic system 10 via a wireless or wired connection (e.g., to control the robotic system 10). Furthermore, in the examples, the control system 50 may communicate with the robotic system 10 to receive position / sensor data relating to the position of sensors associated with and controlled by the robotic system 10. In some examples, the control system 50 may communicate with an EM field generator 120 to control the generation of an EM field in the area surrounding the subject 7. The control system 50 may also include a power interface 206.

[0041] The control system 50 may include control circuitry 251 configured to actuate one or more components of the medical system 100 and / or otherwise control any of the various system components, such as brackets, mounts, arms / positioners, medical devices, imaging devices, position sensing devices, sensors, etc. Furthermore, control circuitry 251 may be configured to perform other functions, such as inducing the display of information, processing data, receiving input, communicating with other components / devices, and / or any other functions / operations discussed herein.

[0042] The control system 50 may also include one or more input / output (I / O) components 210 configured to assist a physician or other person in performing medical procedures. For example, one or more I / O components 210 may be configured to receive input and / or provide output to enable a user to control / navigate the medical device 32, the robotic system 10, and / or other instruments / devices associated with the medical system 100. The control system 50 may include one or more displays 212 to provide / display / present various information about the procedures. For example, one or more displays 212 may be used to present navigation information, including virtual anatomical models of anatomical structures with virtual representations of the medical devices, image data, and / or other information. One or more I / O components 210 may include user input controls 214, which may include any type of user input (and / or output) device or device interface, such as directional input controls 216, touch-based input controls including gesture-based input controls, motion-based input controls, etc. User input control 214 may include one or more buttons, keypads, joysticks, handheld controllers (e.g., video game controllers), computer mice, touchpads, trackballs, control panels, sensors for capturing gestures and finger gestures (e.g., motion sensors or cameras), touchscreens, toggle key (button) inputs, and / or interfaces / components for them. In the example, such input can be used to generate commands for controlling medical devices, robotic arms, and / or other components.

[0043] The control system 50 may also include a data storage device 218 configured to store executable instruments (e.g., computer-executable instructions) that can be executed by the control circuitry 251 to cause the control circuitry 251 to perform the various operations / functions discussed herein. In the example, two or more components of the control system 50 may be electrically and / or communicatively connected to each other.

[0044] Robotic system 10 may include one or more robotic arms 12 configured to engage and / or control, for example, a medical device 32 and / or other elements / components, to perform one or more aspects of a procedure. As shown, each robotic arm 12 may include multiple segments 220 coupled to joints 222, which provide multiple degrees of motion / freedom. Robotic system 10 may be configured to receive control signals from control system 50 to perform certain operations, such as positioning one or more of the robotic arms 12 in a particular manner, manipulating instruments, etc. In response, robotic system 10 may use its control circuitry 211 to control actuators 226 and / or other components of robotic system 10 to perform operations. For example, control circuitry 211 may control insertion / retraction, joint movement, rolling, etc., of the axis of medical device 32 or another instrument by actuating a drive output 228 of a manipulator 230 (e.g., an end effector) coupled to the base of a robotically controllable instrument. Drive output 228 may be coupled to a drive input on an associated instrument, such as the instrument base of an instrument coupled to an associated robotic arm 12. The robot system 10 may include one or more power interfaces 232.

[0045] The robotic system 10 may include a support column 234, a base 236, and / or a console 238. The console 238 may provide one or more I / O components 240, such as a user interface and / or display (or dual-purpose device, such as a touchscreen) for receiving user input to provide preoperative and / or intraoperative data to the physician / user. The support column 234 may include arm supports 242 (also referred to as “brackets”) for supporting the deployment of one or more robotic arms 12. The arm supports 242 may be configured to translate vertically along the support column 234. Vertical translation of the arm supports 242 allows the robotic system 10 to adjust the range of the robotic arms 12 to accommodate various table heights, patient body shapes, and / or physician preferences. The base 236 may include wheel-shaped casters 244 (also referred to as “wheels 244”) that allow the robotic system 10 to move around the operating room prior to the procedure. Once in place, the casters 244 may be secured using wheel locks to hold the robotic system 10 in place during the procedure.

[0046] Each joint 222 of the robotic arm 12 can be independently controllable and / or provide independent degrees of freedom for instrument navigation. In some examples, each robotic arm 12 has seven joints, thus providing seven degrees of freedom, including “redundant” degrees of freedom. Redundant degrees of freedom allow the robotic arms 12 to be controlled to position their respective manipulators 230 at specific locations, orientations, and / or trajectories in space using different link positions and joint angles. This allows the robotic system 10 to locate and guide medical instruments from desired points in space, while allowing physicians to move joints 222 to clinically advantageous positions away from the patient to achieve greater access while avoiding collisions.

[0047] One or more manipulators 230 (e.g., end effectors) may be coupled to an instrument base / handle, which in some cases may be attached using a sterile adapter component. The combination of the manipulator 230, the coupled instrument base, and any intermediate mechanisms or connectors (e.g., sterile adapters) may be referred to as a manipulator assembly, or simply as a manipulator. The manipulator / manipulator assembly may provide an electrical and / or control interface. For example, the interface may include connectors for transmitting pneumatic pressure, electrical, electrical signals, and / or optical signals from the robotic arm 12 to the coupled instrument base. The manipulator / manipulator assembly may be configured to manipulate medical devices (e.g., surgical tools / instruments) using techniques including, for example, direct drive, harmonic drive, gear drive, belt and / or pulley, magnetic drive, etc.

[0048] The robot system 10 may also include a data storage device 246 configured to store executable instruments (e.g., computer-executable instructions) that can be executed by control circuitry 211 to cause control circuitry 211 to perform the various operations / functions discussed herein. In the example, two or more components of the robot system 10 may be electrically and / or communicatively connected to each other.

[0049] Data storage devices (including data storage device 218, data storage device 246, and / or other data storage devices / memories) may include any suitable or desired type of computer-readable medium. For example, computer-readable media may include one or more volatile data storage devices, non-volatile data storage devices, removable data storage devices, and / or non-removable data storage devices, which are implemented using any technology, layout, and / or data structure / protocol, including any suitable or desired computer-readable instructions, data structures, program modules, or other types of data.

[0050] Computer-readable media may include, but is not limited to, phase-change memory, static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, optical disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage devices, magnetic tape, magnetic tape, disk storage devices or other magnetic storage devices, or any other non-transitory medium that can be used to store information for access by a computing device. As used in certain contexts herein, computer-readable media generally may not include communication media, such as modulated data signals and carrier waves. Therefore, computer-readable media should generally be understood as non-transitory media.

[0051] Control circuitry (including control circuitry 251, control circuitry 211, and / or other control circuitry) may include circuitry embodied in a robot system, control system / tower, machinery, or any other component / device. Control circuitry may include any collection of processors, processing circuitry, processing modules / units, chips, dies (e.g., semiconductor dies including one or more active devices and / or passive devices and / or connection circuitry), microprocessors, microcontrollers, digital signal processors, microcomputers, central processing units, field-programmable gate arrays, programmable logic devices, state machines (e.g., hardware state machines), logic circuitry, analog circuitry, digital circuitry, and / or any device that manipulates signals (analog and / or digital) based on hard-coded circuitry and / or operating instructions. Control circuitry as referred to herein may also include one or more circuit substrates (e.g., printed circuit boards), conductive traces and vias and / or mounting pads, connectors, and / or components. Control circuitry may also include one or more storage devices, which may be embodied in the embedded circuitry of a single device, multiple devices, and / or devices. Such data storage devices may include read-only memory, random access memory, volatile memory, non-volatile memory, static memory, dynamic memory, flash memory, cache memory, data storage registers, and / or any device for storing digital information. In examples where the control circuitry includes hardware and / or software state machines, analog circuits, digital circuits, and / or logic circuits, the data storage device / register storing any associated operational instructions may be embedded within or outside the circuitry including the state machine, analog circuitry, digital circuitry, and / or logic circuitry.

[0052] The functions described herein can be implemented by the control circuit 251 of the control system 50 and / or the control circuit 211 of the robot system 10, such as by the control circuits 251 and 211 executing executable instructions to enable the control circuits 251 and 211 to perform functions.

[0053] The endoscope assembly / medical device 32 includes a handle or base 31 coupled to an endoscope shaft. For example, the endoscope 40 (also referred to herein as an "endoscope" or "shaft") may include an elongated shaft comprising one or more lamps 49 and one or more cameras 48 or other imaging devices. The medical device 32 may be powered via a power interface 36 and / or controlled via a control interface 38, each or both of which may interface with a robotic arm / component of the robotic system 10. The medical device 32 may also include one or more sensors 37, such as pressure sensors and / or other force-reading sensors, which may be configured to generate signals indicating forces at or experienced by one or more components of the medical device 32.

[0054] The medical device 32 includes certain mechanisms for articulating / deflecting the endoscope 40 relative to its axis. For example, the endoscope 40 may already be associated with its proximal portion, and one or more drive inputs 34 may be associated with and / or integrated with one or more pulleys / spindles 33 configured to tension / untension the pull cable / traction line 45 of the endoscope 40 to induce articulation of the endoscope 40.

[0055] The endoscope 40 may also include one or more working channels 44, which may be formed within an elongated shaft and extend the length of the endoscope 40. The working channels 44 may be used to deploy components of the medical tool 35 or medical device 32 (e.g., lithotripter, basket, clamp, laser, etc.) therein, or to perform irrigation and / or suction, accessing the surgical area surrounding the distal end through the distal end of the endoscope 40. The medical device 32 may be used in conjunction with the medical tool 35 and includes various hardware and control components for the medical tool 35, and in some cases, the medical tool 35 may be included as part of the medical device 32. For example, as shown, the medical device 32 may include a basket formed of one or more wire teeth, but any medical tool 35 is contemplated.

[0056] Calibration, navigation and positioning modes / systems

[0057] Figure 3 This is a block diagram illustrating a system 300 comprising various localization and / or imaging systems / modalities 302-312 (sometimes referred to as "subsystems") according to one or more examples, which may be implemented to facilitate anatomical mapping, navigation, localization, and / or visualization for procedures. For example, the various systems 302-312 may be configured to provide data for generating anatomical mapping maps, determining the location of instruments, determining the location of targets, and / or performing other techniques.

[0058] Each system in systems 302-312 can be associated with a corresponding coordinate system (also referred to as a “position coordinate system”) and / or can provide data / information related to the position of instruments and / or anatomical structures, wherein registering various coordinate systems with each other allows for the integration of various systems to provide mapping, navigation, and / or instrument visualization. For example, registration of various modalities with each other can allow a determined position in one modality to be tracked and / or superimposed on / within a reference system associated with another modality, thereby providing a layer of positional information that can be combined to provide a robust positioning system.

[0059] In the example, system 300 is configured to implement one or more positioning techniques (also referred to as "positioning system 300"). Positioning can refer to the process of determining the location and orientation / pose of an instrument or other element / component within a given space or environment.

[0060] In various examples, the anatomical space in which a medical device can be positioned (i.e., where the location and / or shape of the device is determined / estimated) is a 2D or 3D portion of the subject's trachea, bronchus, airway, vascular system, urethra, gastrointestinal tract, or any organ or space accessed via an internal lumen. Various modalities can be implemented for use with respect to... Figure 1 The imaging system 122 describes various imaging techniques to provide images / representations / models of anatomical space. It can combine procedures to acquire one or both of preoperative and intraoperative images.

[0061] Systems 302-312 can provide information for generating 2D or 3D anatomical models / figures 314 (e.g., airway models). In the example, anatomical mapping 314 and / or other positioning information can be displayed to the user (such as operator 5) during procedures to assist the user in performing the procedures. For example, a visualization of the tracked medical device can be overlaid on the anatomical mapping 314 based on location / sensor data associated with the tracked medical device.

[0062] As shown in the figure, system 300 may include an operating table or other subject platform or positioning / support structure 302 (e.g., Figure 1 (Operating table 15). The position of the support structure 302 can be determined based on data maintained relating to its position within the surgical / procedural environment. Alternatively or in addition, one or more markers and / or appropriate imaging / positioning modalities can be used to sense or otherwise determine the position of the support structure 302.

[0063] System 300 may also include robot system 304, such as robot system 10 (e.g., a robot cart or other device or system including one or more robot end effectors). Data relating to the position and / or state of the robot arm, actuators, and / or other components of robot system 304 may be known or derived from robot command data or other robot data relative to the coordinate system of robot system 304. In some examples, reference system registration 316 occurs between support structure 302 and robot system 304, which may be a relatively coarse registration based on robot system / cart setup procedures (which may have any suitable or desired scheme) (in some cases).

[0064] System 300 may also include an electromagnetic (EM) sensor system 306, which may include an EM field generator (e.g., EM field generator 120) and one or more EM sensors. The EM sensors may be associated with a portion of the tracked / controlled apparatus, such as along the length of the apparatus and / or other elongated members disposed in the working channel of the apparatus. In some embodiments, the EM field generator may be mechanically coupled to a support structure 302 or a robot system 304, in which case registration / association 318 between such systems can be known and / or determined. In some embodiments, registration 318 between the EM sensor system 306 and the robot system 304 may be determined by forward kinematics and / or field generator mounting transformation information. For example, the field generator may be mounted to an end effector / manipulator of the robot system 304, such that the position of the field generator relative to the robot system positioning frame can be known based on a known relationship between the position of the robot end effector and the position of the robot system 304. The EM sensor system 306 may provide apparatus pose and / or path information based on sensor readings associated with the apparatus.

[0065] System 300 may also include an optical camera system 308 comprising one or more cameras or other imaging devices configured to generate images of a subject's anatomy within its field of view, such as real-time image data during surgery. In the example, registration 320 between the optical camera system 308 and the EM sensor system 306 can be achieved by identifying features (such as those via the medical device tip) in the image generated by the optical camera system 308 that have associated EM sensor data. Registration 320 may also be based, at least in part, on hand-eye interaction of a physician viewing real-time camera images as an endoscope equipped with EM sensors navigates through the subject's anatomy.

[0066] System 300 may also include a computed tomography (CT) imaging system 310 configured to generate CT images of the subject's anatomy, which can be performed preoperatively and / or intraoperatively. Specifically, CBCT is a CT imaging technique that uses a cone-shaped X-ray beam and a specialized detector to capture multiple two-dimensional (2D) X-ray images from different angles around the patient. In CBCT imaging, 2D images can be used to reconstruct a 3D volumetric representation.

[0067] In the example, image processing can be implemented for registration 322 between CT image data and camera image data generated by the optical camera system 308. For example, common features identified in both the camera image data and the CT image data can be recognized to spatially correlate CT image frames with camera image frames. In some examples, the CT imaging system 310 can be used to generate preoperative imaging data for producing anatomical mapping 314 and / or for path navigation planning.

[0068] In this example, the fluoroscopy imaging system 312 can be registered 324 to the CT imaging system 310 using any image processing technique suitable for such registration. Fluoroscopy can provide real-time, continuous X-ray imaging of moving structures inside the body. The images produced during fluoroscopy are typically two-dimensional and show a dynamic view of the area being examined. This real-time imaging allows the operator 5 to visualize the movement and position of anatomical structures or medical instruments 32 during the procedure. Although the CT imaging system 310 and the fluoroscopy imaging system 312 are illustrated as separate systems, in this example, the same system can perform the functions of both the CT imaging system 310 and the fluoroscopy imaging system 312.

[0069] In the example, the CT imaging system 310 and / or the fluoroscopy imaging system 312 can be registered 326 to the EM sensor system 306 using various techniques (such as tool registration, transformation functions, etc.). In one example, the mechanical structure of the C-arm instruments of systems 310, 312 may have a known physical transformation / relationship relative to the mounting position of the EM field generator of the EM sensor system 306. Such a known relationship can be used to register the CT / fluoroscopy image space to the EM sensor image space. Connections 328 and 330 represent the registration / relationship between the CT imaging system 310 and the fluoroscopy imaging system 312 and the anatomical mapping map 314, respectively.

[0070] The position, shape, and / or orientation of an instrument (such as an endoscope) can be determined using any one or more of systems 302-312, which facilitates the generation of graphical interface data representing the estimated position and / or shape of the instrument relative to anatomical mapping 314. The position, shape, and / or orientation of the instrument and / or anatomical mapping 314 can be displayed on a display device, such as via control system 50 and / or robotic system 10 or another device. In some examples, anatomical mapping 314 also indicates the location of a target, such as a location within an anatomical structure designated for further treatment.

[0071] Although systems 302-312 are discussed in a specific order, they can be implemented in different orders. Furthermore, they can be used in different ways. Additionally, registration can occur between different systems.

[0072] In some examples, one or more of systems 302-312 may be implemented to generate an anatomical mapping 314 preoperatively and / or determine the position of one or more targets within the anatomical mapping 314. During surgery, one or more of systems 302-312 may also be implemented to determine the position of a medical device and / or the position of a target relative to the anatomical mapping 314. As discussed herein, one or more of systems 302-312 may also be implemented to update the anatomical mapping 314, the position of the medical device, the position of the target, etc.

[0073] Intraoperative end-effector pose estimation using external imaging modalities

[0074] See again Figure 1 An example of the respiratory system is described, illustrating that an imaging system 122, including external imaging modalities such as CBCT, can generate a 3D model of the lung 4 (e.g., anatomical mapping 314). Anatomical mapping 314 can correspond to a 3D model of the subject's anatomical structure. Traditionally, the use of external imaging has been primarily limited to generating 3D models for preoperative planning and / or as intraoperative reference models related to EM-based or camera-based end-effector pose estimation.

[0075] EM-based or camera-based end-effector pose estimation using anatomical mapping 314 as a reference model may be inaccurate. For example, there may be a difference between the anatomical mapping 314 generated based on preoperatively captured images and the actual anatomy of the subject 7 during medical procedures. As another example, anatomical mapping 314 may not capture movement of the object and / or anatomy, gastrointestinal tract, blood vessels, or endoscope 40. In some cases, breathing by the subject 7 can cause the shape of the lung 4 to deform to the extent that the end-effector may be displaced in an incorrect location in anatomical mapping 314 (e.g., secondary bronchus 78 instead of tertiary bronchus 75 or in the wrong bronchus).

[0076] This disclosure uses images captured during the procedure to provide end-effector pose estimation. CT imaging techniques, including CBCT and fluoroscopy, can capture images intraoperatively or otherwise provide them. These imaging techniques can effectively capture the end of the endoscope (also referred to herein as the “end-effector portion” or “endoscope tip”) and its surrounding anatomical environment, thus providing valuable insight into the relationship between them. Therefore, using CT images for end-effector pose estimation and / or localization can help resolve discrepancies between anatomical mapping diagram 314 and the actual anatomical structures by providing additional reference.

[0077] Specifically, this disclosure provides a time- and resource-efficient CT-based simplified end-effector pose estimation method. This disclosure envisions extending the use of CT-based systems to intraoperative end-effector pose estimation. CT-based end-effector pose estimation can assist the operator in adjusting the end-effector pose when aligning the end-effector with a target in 3D space. For example, CT-based end-effector pose estimation can help during biopsy positioning of the endoscope tip and the biopsy needle carried within the working channel of the endoscope tip relative to the nodule (e.g., ...). Figure 1 (89) Alignment of nodules in the ureteroscopy. As another example, CT-based end pose estimation can help align the basket within the working channel with the kidney stone during ureteroscopy. CT-based end pose estimation is described in more detail below.

[0078] End-effector pose estimation pipeline

[0079] Figure 4 An example block diagram is shown for an end-effector pose estimation pipeline 400 based on one or more implementations. Figure 4 Pipeline 400 visualizes the stages or sequence of stages involved in the end-effector pose estimation process. Specifically, Pipeline 400 focuses on data flow and data processing aspects.

[0080] At a high level, pipeline 400 may involve accessing externally captured image data, segmenting the endoscope tip from the image data, generating a volumetric representation of the endoscope tip, and analyzing the volumetric representation to determine the endoscope tip pose (position and / or orientation). In some embodiments, segmenting the endoscope tip may involve using a machine learning model, such as a deep learning model comprising one or more neural networks, configured to identify / segment the endoscope tip from the endoscope and its surrounding environment. In some embodiments, generating the volumetric representation may involve generating a point cloud, and analyzing the volumetric representation may involve determining the principal axes and / or calculating the centroids of the point cloud based on principal component analysis (“PCA”). The endoscope pose may be estimated based on the principal axes and centroids. It should be understood that while this disclosure will be described based on the specific implementation techniques described above, other techniques and various variations are possible.

[0081] "Endoscope segmentation model 408" will be understood as referring to a machine learning model. Suitable machine learning architectures for examples include convolutional neural networks (CNNs), recurrent neural networks (RNNs), long short-term memory (LSTM) networks, and transformer-based architectures (such as visual transformers). In some implementations, endoscope segmentation model 408 may have a U-Net architecture. It should be understood that the U-Net architecture is an example architecture for endoscope segmentation model 408, and any type or combination of machine learning models can be used depending on task requirements, available computing resources, and available data. Endoscope segmentation model 408 can be trained preoperatively and used intraoperatively for segmentation of the endoscope tip.

[0082] I. Training the spy endoscope segmentation model

[0083] During training, the endoscope tip segmentation model 408 learns to identify / segment the region corresponding to the endoscope tip. Training may involve preparing a subject image dataset 402, annotating the endoscope tip in the subject image dataset 402, and training the endoscope tip segmentation model 408 based on the subject image dataset 402.

[0084] During training, pipeline 400 may involve accessing (e.g., receiving, retrieving, capturing, decompressing, etc.) raw images depicting the endoscope, its endoscope tip, and anatomical structures (e.g., tissues, bones, etc.) surrounding the endoscope. Pipeline 400 may also involve preparing a subject image dataset 402 based on the raw images. The raw images may be externally captured images (e.g., images from sources located at...). Figure 1 The original image (CT image captured by the imaging system 122 outside the subject 7) or a simulated image. In some embodiments, the original image may be resized, normalized in some visual properties (e.g., brightness, gamma correction, etc.), converted to a common format, etc. In some embodiments, the original image may be altered by transformations (e.g., rotation, scaling, flipping, cropping, elastic deformation, etc.) to increase diversity.

[0085] In some implementations, the original image may be a CT image, including a CBCT image. In CT, an X-ray beam and detector can capture multiple 2D X-ray images from different angles around the scanned area of ​​the subject 7. A 3D volumetric representation or model of the scanned area can be reconstructed based on multiple 2D X-ray images. Therefore, when referring to CT, multiple 2D X-ray images can collectively form an image dataset, and the subject image dataset 402 may include multiple such CT image datasets. In this document, depending on the context, the 2D image or 3D representation may be referred to as the subject image dataset 402.

[0086] At the end-of-sight annotation box 404, pipeline 400 may involve annotating the end of the end-of-sight in the subject image dataset 402. Annotation involves creating a corresponding ground truth segmentation mask for each image. This ground truth segmentation mask is used as a ground truth label during training and is not confused with the inference segmentation mask generated during inference. In the ground truth segmentation mask, each pixel in the mask is assigned a label or category corresponding to an object, or a background label if it does not belong to any object.

[0087] Figure 5A An example raw image 500 depicting both a endoscopic endoscope 504 surrounded by tissue 502 and its endoscope tip 506 is shown. The example raw image 500 may be a 2D slice of a CT image dataset commonly associated with 3D reconstruction. Therefore, in some embodiments, annotation of the ground truth segmentation mask may involve assigning labels or categories to voxels of a 3D volumetric representation reconstructed from a CT image dataset, thereby creating a 3D segmentation mask or multiple 2D slices of the segmentation mask.

[0088] While some embodiments of annotation box 404 may involve manually identifying pixels / voxels in a segmented mask, this disclosure also envisions semi-automatic annotation using known geometry (e.g., physical dimensions and shape) of the endpiece. The known geometry can be provided by a known 3D model, which may be a computer-aided design (CAD) model of the endpiece. Figure 5B An example annotated image 550 is shown, depicting the identification of pixels / voxels in the original example image 500 using a CAD model 508 (or a simpler model generated from a CAD model, such as a convex hull model 510). The convex hull model 510 is the smallest convex shape or polygon enclosing all points in the CAD model 508, without any concave parts, and simplifies the annotation / creation of the segmentation mask. While the CAD model 508 and the convex hull model 510 are described as example 3D models, other 3D models, including bounding boxes, bounding spheres, alpha shapes, etc., can be envisioned.

[0089] As shown in the example annotated image 550, the convex hull model 510 can be positioned and oriented to align with the endoscope tip 506. For example, the convex hull model 510 can be positioned to align with the endoscope tip 506 in a 2D CT image slice or a 3D reconstruction generated based on a 2D CT image slice. In some embodiments, the convex hull model 510 can be converted into a segmentation node configured to generate a ground truth segmentation mask for the endoscope tip. The ground truth segmentation mask can be provided to the model training box 406.

[0090] At model training box 406, the endoscopic end segmentation model 408 (e.g., a neural network model) can be trained using the subject image dataset 402 and the corresponding ground truth segmentation mask. For example, the weights in the neural network model can be adjusted to minimize the error during supervised learning. The subject image dataset 402 can be split into a training dataset and a validation dataset. Various metrics can be used to evaluate the segmentation task accuracy of the trained endoscopic end segmentation model 408, including the Dice similarity coefficient, which is configured to measure the similarity between the segmentation mask and the ground truth label.

[0091] It should be understood that training of the endoscope end segmentation model 408 can be performed offline and / or on dedicated hardware (e.g., a server cloud), while inference using the endoscope end segmentation model 408 can be performed online and in conjunction with the medical system 100. Furthermore, it should be understood that, in some respects, the endoscope end segmentation model 408 used in training can be modified prior to deployment. For example, any of quantization, pruning, model compression, or other optimizations can make the trained endoscope end segmentation model 408 ready for deployment.

[0092] II. Reasoning using the peephole segmentation model

[0093] During inference, at segmentation mask box 410, pipeline 400 may involve generating a segmentation mask for new but unseen image data using a trained endoscope tip segmentation model 408. The new image data may be subject image data 405 captured intraoperatively or near real-time and provided as input to the endoscope tip segmentation model 408. Subject image data 405 may be a CT image dataset including 2D slices (or 3D reconstructions based on 2D slices) depicting the endoscope tip. As output from the endoscope tip segmentation model 408, a 2D / 3D segmentation mask corresponding to the CT image can be generated.

[0094] The segmentation mask can be a map that labels each pixel / voxel as the endpiece of the endoscope or other maps. In some implementations, the mapping map can be a binary image that represents pixels / voxels corresponding to the endpiece of the endoscope with a first value (e.g., "1") and the background with a second value (e.g., "0"). The resulting segmentation mask can identify / segment the region corresponding to the endpiece of the endoscope in the subject image data 405.

[0095] A point cloud representing the end of a scope in 3D space can be generated based on a 2D / 3D segmentation mask generated at box 410. In some implementations, the point cloud can be a volumetric fusion set of 2D segmentation masks or the 3D segmentation mask itself. Figure 6 An example point cloud 602 according to one or more implementation schemes is illustrated. In some specific implementations, point cloud 602 may be a collection of all pixels / voxels labeled as the end of a peephole. Figure 6 In the example, the 3D model at the end is 604 (e.g., Figure 5B The outline 606 of the convex hull model 510 is overlaid on the top of the point cloud 602 to provide a sense of which points of the point cloud 602 correspond to which part of the end.

[0096] At Principal Component Analysis (PCA) box 412, pipeline 400 may involve performing Principal Component Analysis (PCA) to determine one or more principal axes (also called “principal components”) of the point cloud that have the maximum variance. Applying PCA to the point cloud can find the principal axis that captures the direction of the highest variance in the point cloud. Of the three possible principal axes of the point cloud in 3D space, the principal axis associated with the maximum variance can be selected. Referring to point cloud 602, performing PCA can find the principal axis 608 that provides the maximum variance in the point cloud. Principal axis 608 is an axis that extends infinitely relative to the reference point and has no sense of direction.

[0097] At centroid box 413, pipeline 400 may involve determining the centroid of the point cloud (similar to a centroid). As an example, a centroid 610 can be found for point cloud 602. Based on centroid 610 as a reference point, principal axis 608 can be divided into two possible directions as shown, each direction away from centroid 610. Either of the two possible directions may correspond to an end-effector direction (e.g., the direction component of a 3D vector). In this disclosure, the end-effector direction may be referred to as the "scope forward direction" or "scope end-effector forward direction," either of which reflects the end-effector orientation.

[0098] The pipeline 400 can estimate the end-effector pose based on the principal axis and the centroid, including the end-effector forward direction 420 and the end-effector position 419.

[0099] III. The direction of advance of the endoscope tip is determined

[0100] As described, two possible end-of-scope advance directions away from the centroid can be identified on the principal axis with the largest variance. Figure 7A An example CT image 700 depicting the endoscope 704 and the endoscope tip 706 is shown. It should be understood that the example CT image 700 is a cross-sectional image of the endoscope tip 706 in 3D representation (e.g., Figure 4 (Images in subject image data 405). Example CT image 700 shows the centroid 710 and a first direction 708a away from the endoscope tip and towards the rear of the endoscope tip, and a second direction 708b orienting the endoscope tip. Selecting the correct endoscope tip advance direction between the first direction 708a and the second direction 708b may involve sampling pixel / voxel values ​​and performing sample analysis based on the sampled values.

[0101] At sampling frame 414, pipeline 400 may involve creating two diameter-opposed sampling regions (2D or 3D) and sampling points within them. For example, Figure 7B An example sampled image 750 with two sampling regions is illustrated, which are configured as spheres 712a and 712b opposite each other on the diameter along the main axis 708. The first sphere 712a is positioned at the proximal end of the endoscope, and the second sphere 712b is positioned at the distal end of the endoscope. The spheres 712a and 712b may have the same size or different sizes.

[0102] Spheres 712a and 712b can be positioned on the main axis 708 based on the known geometry of the end of the endoscope. For example, the centers of spheres 712a and 712b can be determined by a spacing distance (denoted as d). s The distance between the two ends is approximately equal to the distance between the CAD model at the end of the endoscope (e.g., Figure 5B The end length of the endoscope in the CAD model 508 is half of the end length. Similarly, spheres 712a and 712b can be set to have a sampling radius (denoted as r) that exactly or approximately matches the radius of the cylindrical portion in the CAD model. s As an example, the interval distance can be approximately 6 millimeters, and the sampling radius can be approximately 3 millimeters.

[0103] Within the sampling area, pixel / voxel values ​​can be sampled at sampling point 714. In some specific implementations, sampling points 714 can be uniformly distributed or based on a known contour distribution. It should be understood that although spheres 712a and 712b are used as example sampling areas, any region in 2D or 3D can be used, including, for example, circles, triangles, rectangles, cubes, cones, cylinders, pyramids, ellipsoids, etc. Additionally, it will be noted that the spacing distance (d) s ), sampling radius (r) s The number of sampling points 714, the distribution contour of sampling points 714, etc. are configuration values ​​and can be adjusted based on CT images, endoscope tip, or both.

[0104] In CT images, the pixel / voxel value associated with each sampling point 714a, 714b, 714c can represent the degree of radiodensity in the subject image data 405. The Henle unit (HU) is a measurement scale used to quantify the radiodensity of tissues and materials. The Henle unit scale assigns values ​​to different tissues and materials based on their X-ray attenuation characteristics, with metallic objects and dense structures that absorb more X-rays having higher HU unit values ​​(more attenuation) compared to soft tissues and fluids (lower attenuation). Higher-density materials associated with higher HU unit values ​​appear brighter (e.g., have greater luminance) than lower-density materials associated with lower HU unit values, and lower-density materials associated with lower HU unit values ​​appear darker (e.g., have less luminance).

[0105] At sample analysis frame 416, pipeline 400 may involve analyzing the pixel / voxel values ​​sampled at sampling point 714. In example sampled image 750, all or substantially all of the sampling points 714 of the first sphere 712a are pixels / voxels depicting the endoscope. In other words, a large portion of the first sphere 712a overlaps with the endoscope. In contrast, only about half of the sampling points of the second sphere 712b are pixels / voxels depicting the endoscope, while the other half of the sampling points of the second sphere 712b are pixels / voxels that do not depict the endoscope (but rather depict objects or features surrounding the endoscope, such as tissue, bone, air, or other anatomical features). In other words, only half of the second sphere 712b overlaps with the endoscope. Therefore, the sum of pixel / voxel values ​​within the first sphere 712a (e.g., a higher sum of sampled HU values) is greater than the sum of pixel / voxel values ​​within the second sphere 712b (e.g., a lower sum of sampled HU values).

[0106] At the endoscope advance direction box 418, the correct endoscope tip advance direction can be selected based on the results of the sample analysis box 416. For example, based on the comparison of pixel / voxel values, the correct direction can be selected. Figure 7A The second direction 708b, which has a lower total, serves as the correct direction for the endoscope tip to advance.

[0107] In some implementations, only one region (e.g., either, but not both, the first sphere 712a or the second sphere 712b) can be sampled to obtain the pixel / voxel values ​​within that region and compared to a threshold level. In other words, a single vector (pointing from the centroid 710 to the sampled region) can be tested for the end-of-scope direction. For example, the threshold level can be set to a value between a first sum and a second sum, such that when the sum of the pixel / voxel values ​​within the region is greater than the threshold level, the direction opposite to the vector can be selected as the end-of-scope direction. In other words, the end-of-scope direction will be the inverse form of the vector. On the other hand, when the sum is less than the threshold level, the direction of the vector can be selected as the end-of-scope direction.

[0108] The end-effector direction defined by end-effector advance direction box 418 is the end-effector advance direction 420. In example CT image 700, the end-effector advance direction 420 is a second direction 708b. In some specific implementations, the end-effector advance direction 420 may be represented as a vector in 3D space.

[0109] IV. Determining the position of the endoscope tip

[0110] The position of the endoscope can be determined based on the centroid, the endoscope CAD model 421, and the endoscope forward direction 420.

[0111] At the end-of-scope alignment frame 415, the pipeline 400 may involve mapping / registering and / or aligning the end-of-scope CAD model 421 to the 3D reconstructed end (e.g., a point cloud or another volumetric representation). Figure 8A An example alignment process 800 is illustrated, which depicts a CAD model 421 of the endoscope tip with a first centroid 812 on a first principal axis (e.g., the Z-axis of a CAD coordinate system) and a 3D reconstructed endoscope tip 820 with a second centroid 822 on a second principal axis (e.g., the endoscope tip forward direction 420). The 3D reconstructed endoscope tip 820 may be a convex hull model reconstructed based on a segmentation mask inferred from the endoscope tip segmentation model 408.

[0112] Alignment may involve determining a transformation that aligns the first principal axis with the second principal axis and mapping the first centroid 812 to the second centroid 822. In vector space, alignment may involve rotating (denoted as θ) the end-of-scope CAD model 421 onto the 3D reconstruction end-of-scope 820 such that the CAD model 421 is rotated about a normal vector (denoted as n) onto the 3D reconstruction end-of-scope 820. In some embodiments, the transformation may additionally "roll" (not shown) the end-of-scope CAD model 421 during alignment to correspond to the rolling of the 3D reconstruction end-of-scope. Figure 8B A successful alignment of 850 is illustrated, where the CAD model is overlaid on the end of the 3D reconstruction.

[0113] At the farthest side point projection frame 417, pipeline 400 may involve projecting the farthest side point in the 3D model onto the principal axis with the maximum variance. For example, Figure 9 An example render 900 illustrates the projection of the farthest point 910 of a lap end CAD model 902 (e.g., lap end CAD model 421) onto a principal axis 904 positioned through the model's centroid 912. The lap end CAD model 902 has a known shape, and therefore the farthest point of the 3D model (relative to the centroid 912) is known. The projection of the farthest point 910 onto the principal axis 904 can represent an updated end position 914 on the principal axis 904 (such as lap end position 419).

[0114] The endoscope tip forward direction 420 can represent the tip orientation, and the endoscope tip position 419 represents the tip position. Therefore, pipeline 400 can use externally captured images to estimate the tip pose. It should be understood that the boxes depicted in pipeline 400 are exemplary, and fewer or more boxes, as well as boxes organized in different orders, may be involved in tip pose estimation. Furthermore, it should be understood that the depicted boxes can be executed within a component and / or on a separate system. For example, endoscope tip annotation box 404 and model training box 406 may be executed collectively during a training step on a cloud server, while other boxes are executed at a later time on medical system 100. Many other variations are also possible.

[0115] Endoscope pose estimation workflow

[0116] Figure 10 A flowchart illustrating an exemplary instrument end-effector pose estimation process 1000 according to one or more embodiments is provided. When followed, process 1000 can determine the end-effector advance direction and end-effector position from externally captured images.

[0117] At box 1002, process 1000 may involve accessing image data of multiple images captured by an imaging system positioned outside the subject's anatomical structures. The images may be CT images, including CBCT images. Each of the multiple images may depict an object within the anatomical structure.

[0118] At box 1004, process 1000 may involve using a machine learning model to predict at least one segmentation mask for the distal end of an object configured to segment. The object may be a hysteroscope, and the distal end may be the tip of the hysteroscope. In some embodiments, the trainable machine learning model uses a 3D model of the distal end (e.g., the tip of the hysteroscope). Figure 5B The truth segmentation mask annotated with the convex hull model (510).

[0119] At box 1006, process 1000 may involve generating a 3D representation of the distal end based on at least one segmentation mask. The segmentation mask can be used to reconstruct the 3D representation of the distal end. In some embodiments, the 3D representation may be a point cloud.

[0120] At box 1008, process 1000 may involve determining the position or orientation, or both, of the distal end based at least in part on a 3D representation of the distal end and a CAD model. The orientation of the distal end may be determined based on the corresponding centroids and principal axes of the aligned 3D representation and CAD model. In some embodiments, points along the aligned principal axes may be sampled for comparison to eliminate candidate (but incorrect) orientations, thereby identifying the correct orientation. The position may be determined by projecting the farthest point of the CAD model onto the aligned principal axes.

[0121] It should be understood that the boxes depicted in process 1000 are exemplary, and fewer or more boxes, as well as boxes organized in different orders, may be involved in end-effector pose estimation. Many variations are possible.

[0122] Figure 11 Another block diagram of an example controller 1100 for a medical system according to some specific implementations is shown. In some specific implementations, controller 1100 may be... Figure 2 An example of control circuitry 251 and / or control circuitry 211. More specifically, controller 1100 is configured to estimate the pose of an object within an anatomical structure. For example, see reference... Figure 4 The controller 110 can implement one or more stages of the end-effector pose estimation pipeline 400.

[0123] The controller 1100 includes a communication interface 1110, a processing system 1120, and a memory 1130. The communication interface 1110 is configured to communicate with one or more components of the medical system. More specifically, the communication interface 1110 includes features for communicating with one or more image sources (such as...) Figure 3 The image source interface (I / F) 1112 communicates with the CT imaging system 310 and / or the fluorescence fluoroscopy imaging system 312. In some embodiments, the image source I / F 1112 may receive image data representing a 3D model of an anatomical structure in which instruments are disposed.

[0124] The memory 1130 may include a non-transitory computer-readable medium (including one or more non-volatile memory elements, such as EPROM, EEPROM, flash memory, or hard disk drive, etc.) that stores the following software (SW) modules: a point cloud generation SW module 1132 for generating a point cloud associated with the distal end of the instrument based on image data; and a pose determination SW module 1134 for determining the pose of the distal end of the instrument based at least in part on the point cloud and the known geometry of the distal end of the instrument.

[0125] Processing system 1120 may include any suitable one or more processors capable of executing scripts or instructions of one or more software programs stored in controller 1100, such as in memory 1130. For example, processing system 1120 may execute point cloud generation SW module 1132 to generate a point cloud associated with the distal end of the instrument based on image data. Processing system 1120 may also execute pose determination SW module 1134 to determine the pose of the distal end of the instrument based at least in part on the point cloud and the known geometry of the distal end of the instrument.

[0126] Figure 12 An exemplary flowchart depicting an example pose estimation operation 1200 according to some specific implementations is shown. In some specific implementations, example operation 1200 may be provided by a controller for a medical system (such as...) Figure 11 The controller 1100) executes.

[0127] The controller receives image data (1302) representing a 3D model of an anatomical structure in which an instrument is disposed. The controller generates a point cloud (1304) associated with the distal end of the instrument based on the image data. The controller also determines the pose of the distal end of the instrument based at least in part on the point cloud and the known geometry of the distal end of the instrument (1306).

[0128] In some aspects, the controller may also segment the distal end of the instrument from a 3D model based on a machine learning model trained to infer segmentation masks from image data, wherein the point cloud is generated based on the segmentation mask. In some implementations, the machine learning model may be trained at least in part on a 3D model of the distal end of the instrument, which is generated based on known geometry. In some implementations, the 3D model may be a convex hull model.

[0129] In some aspects, determining the pose of the distal end of the instrument may include determining a principal axis that maximizes the variance of the point cloud; selecting a first sampling region on the 3D model, the first sampling region being aligned with the principal axis and located at a predetermined distance from the centroid of the point cloud in a first direction, wherein the predetermined distance and size of the first sampling region are configured based on the known geometry of the distal end of the instrument; sampling voxel values ​​of image data within the first sampling region; and determining the orientation of the distal end of the instrument based at least in part on the sampled voxel values ​​within the first sampling region.

[0130] In some embodiments, determining the orientation of the distal end of the instrument may include: determining whether the sum of voxel values ​​within a first sampling region exceeds a threshold. In other embodiments, determining the orientation of the distal end of the instrument may include: selecting a second sampling region on the 3D model, the second sampling region being aligned with the principal axis and located at a predetermined distance from the centroid of the point cloud in a second direction; sampling voxel values ​​of image data within the second sampling region; and determining whether the sum of voxel values ​​within the first sampling region is greater than the sum of voxel values ​​within the second sampling region.

[0131] In some other aspects, determining the pose of the distal end of the instrument may include: mapping a 3D model of the distal end of the instrument to a point cloud, wherein the 3D model of the distal end is generated based on known geometry; and determining the position of the distal end of the instrument based at least in part on the mapping from the 3D model of the distal end to the point cloud. In some specific implementations, determining the position of the distal end of the instrument may include: determining a principal axis that maximizes the variance of the point cloud; and projecting the point on the 3D model of the distal end of the instrument that is farthest from the centroid of the 3D model of the distal end onto the principal axis, wherein the projected point represents the position of the distal end of the instrument.

[0132] Additional Implementation Plan

[0133] Depending on the implementation, certain actions, events, or functions of any of the processes or algorithms described herein may be performed in different orders, added, combined, or completely ignored. Therefore, in some implementations, not all described actions or events are necessary for the practice of the process.

[0134] Unless otherwise specifically stated or otherwise understood in the context in which they are used, the conditional language used herein, such as “may,” “can,” “possibly,” “can,” “e.g.,” etc., refers in its ordinary sense and is generally intended to convey that certain embodiments include certain features, elements, and / or steps that are not included in other embodiments. Therefore, such conditional language is not generally intended to imply that one or more embodiments require features, elements, and / or steps in any way, or that one or more embodiments necessarily include logic for determining, with or without author input or prompting, whether such features, elements, and / or steps are included in any particular embodiment or whether they will be performed in any particular embodiment. The terms “comprising,” “including,” “having,” etc., are synonymous and used in their ordinary sense, and are used inclusively in an open-ended manner, without excluding additional elements, features, actions, operations, etc. Moreover, the term “or” is used in its inclusive sense (rather than in its exclusive sense) such that when used, for example, to connect a series of elements, the term “or” refers to one, some, or all of the elements in that series. Unless otherwise specified, combined language such as “at least one of X, Y, and Z” is understood in the context of general use to convey that an item, term, element, etc., can be X, Y, or Z. Therefore, such combined language is generally not intended to imply that a particular implementation requires the presence of at least one of X, at least one of Y, and at least one of Z.

[0135] It should be understood that in the above description of the embodiments, various features are sometimes grouped together in a single embodiment, figure, or description therein in order to simplify this disclosure and aid in understanding one or more aspects of the various inventive aspects. However, this approach of the disclosure should not be construed as reflecting an intention that any claim requires more features than those expressly recited in that claim. Furthermore, any component, feature, or step shown and / or described in the specific embodiments herein may be applied to or used with any other embodiment. Moreover, for each embodiment, no component, feature, step, or group of components, features, or steps is necessary or indispensable. Therefore, it is expected that the scope of the invention disclosed herein and claimed below is not limited to the specific embodiments described above, but should be determined solely by a fair reading of the appended claims.

[0136] It should be understood that certain ordinal terms (e.g., "first" or "second") may be provided for ease of reference and do not necessarily imply physical characteristics or ordering. Therefore, as used herein, ordinal terms used to modify elements such as structures, components, operations, etc. (e.g., "first," "second," "third," etc.) do not necessarily indicate the priority or order of that element relative to any other element, but rather serve to generally distinguish that element from another element with a similar or identical name (but used in ordinal terms). Additionally, as used herein, the indefinite article ("a") may indicate "one or more" rather than "one." Furthermore, an operation performed "based on" a certain condition or event may also be performed based on one or more other conditions or events not explicitly stated.

[0137] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the example embodiments pertain. It will be further understood that terms, such as those defined in common dictionaries, shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and shall not be interpreted in an idealized or overly formal sense, unless expressly defined herein. As used herein, the term “patient” may generally refer to a human being, an anatomical model, a simulator, a cadaver, or other living or non-living object.

[0138] For ease of description, the spatial relative terms “external,” “internal,” “upper,” “lower,” “below,” “above,” “vertical,” “horizontal,” and similar terms are used herein to describe the relationship between one element or component shown in the accompanying drawings and another. It should be understood that the spatial relative terms are intended to cover different orientations of the device in use or operation other than those depicted in the accompanying drawings. For example, if the device shown in the accompanying drawings is flipped, a device located “below” or “under” another device may be positioned “above” another device. Therefore, the illustrative term “below” can include both a lower position and an upper position. The device may also be oriented in another direction, and therefore the spatial relative terms may be interpreted differently depending on the orientation.

[0139] Unless otherwise explicitly stated, comparative and / or quantitative terms, such as “less,” “more,” “greater,” etc., are intended to encompass the concept of equality. For example, “less” may mean not only “less” in the strictest mathematical sense, but may also mean “less than or equal to.”

Claims

1. A controller for a medical system, comprising: Processing system; and The memory stores instructions that, when executed by the processing system, cause the controller to: Receive image data representing a three-dimensional (3D) model of an anatomical structure in which instruments are set; A point cloud associated with the distal end of the device is generated based on the image data; as well as The pose of the distal end of the instrument is determined at least in part based on the point cloud and the known geometry of the distal end of the instrument.

2. The controller according to claim 1, wherein, The execution of the instruction further enables the controller to: The distal end of the device is segmented from the 3D model based on a machine learning model trained to infer segmentation masks from the image data, and the point cloud is generated based on the segmentation mask.

3. The controller according to claim 2, wherein, The machine learning model is trained, at least in part, on a 3D model of the distal end of the device, which is generated based on the known geometry.

4. The controller according to claim 3, wherein, The 3D model of the distal end of the device is a convex hull model.

5. The controller according to claim 1, wherein, Determining the pose of the distal end of the device includes: Determine the principal axis that maximizes the variance of the point cloud; A first sampling region is selected on the 3D model, the first sampling region being aligned with the principal axis and located at a predetermined distance from the centroid of the point cloud in a first direction, the predetermined distance and size of the first sampling region being configured based on the known geometry of the distal end of the instrument; and The voxel values ​​of the image data within the first sampling area are sampled; and The orientation of the distal end of the device is determined at least in part based on the sampled voxel values ​​within the first sampling region.

6. The controller according to claim 5, wherein, Determining the orientation of the distal end of the device includes: Determine whether the sum of the voxel values ​​within the first sampling area exceeds a threshold.

7. The controller according to claim 6, wherein, Determining the orientation of the distal end of the device further includes: If the sum of the voxel values ​​does not exceed the threshold, then the instrument is determined to be oriented in the first direction; and If the sum of the voxel values ​​exceeds the threshold, the device is determined to be oriented in a second direction opposite to the first direction.

8. The controller according to claim 5, wherein, Determining the orientation of the distal end of the device includes: A second sampling region is selected on the 3D model, the second sampling region being aligned with the main axis and located at a predetermined distance from the centroid of the point cloud in the second direction; The voxel values ​​of the image data within the second sampling region are sampled; and Determine whether the sum of the voxel values ​​in the first sampling region is greater than the sum of the voxel values ​​in the second sampling region.

9. The controller according to claim 8, wherein, Determining the orientation of the distal end of the device further includes: If the sum of voxel values ​​in the first region is not greater than the sum of voxel values ​​in the second region, then the instrument is determined to be oriented in the first direction; and If the sum of voxel values ​​in the first region is greater than the sum of voxel values ​​in the second region, then the instrument is determined to be oriented in the second direction.

10. The controller according to claim 1, wherein, Determining the pose of the distal end of the instrument includes: Mapping the 3D model of the distal end of the instrument onto the point cloud, wherein the 3D model of the distal end is generated based on the known geometry; and The position of the distal end of the device is determined at least in part based on the mapping from the 3D model of the distal end to the point cloud.

11. The controller according to claim 10, wherein, Determining the position of the distal end of the device includes: Determine the principal axis that maximizes the variance of the point cloud; and The point on the 3D model of the distal end of the instrument that is farthest from the centroid of the 3D model of the distal end is projected onto the main axis, and the projected point represents the position of the distal end of the instrument.

12. A pose estimation method, comprising: Receive image data representing a three-dimensional (3D) model of an anatomical structure in which instruments are set; A point cloud associated with the distal end of the device is generated based on the image data; as well as The pose of the distal end of the instrument is determined at least in part based on the point cloud and the known geometry of the distal end of the instrument.

13. The method of claim 12, further comprising: The distal end of the device is segmented from the 3D model based on a machine learning model trained to infer segmentation masks from the image data, and the point cloud is generated based on the segmentation mask.

14. The method according to claim 13, wherein, The machine learning model is trained, at least in part, on a 3D model of the distal end of the device, which is generated based on the known geometry.

15. The method according to claim 14, wherein, The 3D model of the distal end of the device is a convex hull model.

16. The method according to claim 12, wherein, Determining the pose of the distal end of the device includes: Determine the principal axis that maximizes the variance of the point cloud; A first sampling region is selected on the 3D model. The first sampling region is aligned with the main axis and located at a predetermined distance from the centroid of the point cloud in a first direction. The predetermined distance and size of the first sampling region are configured based on the known geometry of the distal end of the instrument. The voxel values ​​of the image data within the first sampling area are sampled; and The orientation of the distal end of the device is determined at least in part based on the sampled voxel values ​​within the first sampling region.

17. The method according to claim 16, wherein, Determining the orientation of the distal end of the device includes: Determine whether the sum of the voxel values ​​within the first sampling area exceeds a threshold.

18. The method according to claim 16, wherein, Determining the orientation of the distal end of the device includes: A second sampling region is selected on the 3D model, the second sampling region being aligned with the main axis and located at a predetermined distance from the centroid of the point cloud in the second direction; The voxel values ​​of the image data within the second sampling region are sampled; and Determine whether the sum of the voxel values ​​in the first sampling region is greater than the sum of the voxel values ​​in the second sampling region.

19. The method according to claim 12, wherein, Determining the pose of the distal end of the device includes: Mapping the 3D model of the distal end of the instrument onto the point cloud, wherein the 3D model of the distal end is generated based on the known geometry; and The position of the distal end of the device is determined at least in part based on the mapping from the 3D model of the distal end to the point cloud.

20. The method according to claim 19, wherein, Determining the position of the distal end of the device includes: Determine the principal axis that maximizes the variance of the point cloud; and The point on the 3D model of the distal end of the instrument that is farthest from the centroid of the 3D model of the distal end is projected onto the main axis, and the projected point represents the position of the distal end of the instrument.

Images