3D reconstruction of instruments and treatment sites

Abstract

The present disclosure relates to systems, devices, and methods for reconstructing three-dimensional models of instruments and treatment sites using a trained neural network, where the trained neural network is trained using synthetic fluoroscopic images based on labeled volumetric data of anatomical structures and geometric properties of instruments to segment fluoroscopic images according to treatment sites and instruments and reconstruct three-dimensional models.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Patent Application No. 63 / 313,350, entitled “THREE-DIMENSIONAL RECONSTRUCTION OF AN INSTRUMENT AND PROCEDURE SITE,” filed February 24, 2022, the disclosure of which is incorporated herein by reference in its entirety. [Background technology]

[0002] Various medical procedures involve the use of one or more devices configured to penetrate the human anatomy to reach a treatment site. Certain operational processes can include locating the medical instrument within a patient and visualizing an area of ​​interest within the patient. To do so, many medical instruments may include sensors to track the instrument's position and may include vision capabilities, such as an embedded camera or compatible use with a visual probe. [Brief description of the drawings]

[0003] Various embodiments are illustrated in the accompanying drawings for illustrative purposes and should not be construed as limiting the scope of the present disclosure in any way. In addition, various features of different disclosed embodiments may be combined to form further embodiments that are part of the present disclosure. Throughout the drawings, reference numbers may be reused to indicate correspondence between referenced elements. [Figure 1] 1 illustrates an exemplary medical system for performing various medical procedures, according to aspects of the present disclosure. [Diagram 2] FIG. 2 illustrates components and subsystems of the control system shown in FIG. 1 in accordance with an exemplary embodiment. [Diagram 3] 1 is a flow chart illustrating a method for reconstructing a three-dimensional model of an instrument and treatment site from two-dimensional images acquired during or as part of a medical procedure, according to an exemplary embodiment. [Figure 4]FIG. 4 is a system diagram illustrating a neural network generation system 400, according to an example embodiment. [Diagram 5] 1 is a flow chart illustrating a method for generating a trained neural network that can be used to reconstruct a three-dimensional model of an instrument and a treatment site, according to an exemplary embodiment. [Figure 6] FIG. 2 is a block diagram illustrating an exemplary data architecture for fluoroscopic image processing, in accordance with an exemplary embodiment. [Figure 7] 13 is an illustration of a sequence of instrument segmentations that may be generated by a neural network in accordance with an illustrative embodiment; FIG. [Figure 8] FIG. 1 illustrates a segmentation of an instrument including sub-parts: a scope tip and an articulatable section, according to an exemplary embodiment. [Figure 9] 13 illustrates an example of segmentation of a treatment site, according to an exemplary embodiment. [Figure 10] FIG. 13 illustrates segmentation based on a region of interest, such as distance centered on a segmented instrument region, according to an exemplary embodiment. [Figure 11] 1 is a diagram illustrating a calibration object in accordance with an exemplary embodiment; [Figure 12] FIG. 1 illustrates a calibration image in accordance with an exemplary embodiment. [Figure 13] FIG. 1 illustrates a reconstructed 3D model rendering in accordance with an exemplary embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0004] The directions provided herein are for convenience only and do not necessarily affect the scope or meaning of the disclosure. Although certain exemplary embodiments are disclosed below, the subject matter extends beyond the specifically disclosed embodiments to other alternative embodiments and / or uses, as well as to modifications and equivalents thereof. Thus, the scope of claims that may arise from this specification is not limited by any of the specific embodiments described below. For example, in any method or process disclosed herein, the acts or operations of the method or process may be performed in any suitable order and are not necessarily limited to any particular disclosed order. Various operations may be described in sequence as multiple separate operations in a manner that may be helpful in understanding a particular embodiment, however, the order of description should not be construed to imply that these operations are order dependent. Furthermore, structures, systems, and / or devices described herein may be embodied as integrated or separate components. For purposes of comparing various embodiments, certain aspects and advantages of these embodiments are described. Not necessarily all such aspects or advantages are realized by any particular embodiment. Thus, for example, various embodiments may be performed in a manner that achieves or optimizes one advantage or group of advantages taught herein without necessarily achieving other aspects or advantages that may also be taught or suggested herein.

[0005] overview The present disclosure relates to systems, devices and methods for generating reconstructed three-dimensional models of treatment sites and instruments. In this specification, the term reconstruction can be understood to mean construction and vice versa. Many medical procedures rely on accurate representation of a patient's anatomy and navigation in controlling instruments within that anatomy. For example, in bronchoscopy, accurate and safe biopsy may depend on accurate alignment of a steerable bronchoscope with a biopsy site, such as a nodule or lesion. Robotic bronchoscopy can include a navigation system to facilitate navigation of the bronchoscope to the biopsy site and to provide information useful in aligning the tip of the bronchoscope with the biopsy site. The navigation system may include a three-dimensional model of the anatomy. In the case of bronchoscopy, the three-dimensional model may include data regarding the structure of the luminal network formed by the airways of the lungs. This three-dimensional model can be generated from a preoperative Computerized Tomography (CT) scan of the patient. During the procedure, the coordinate system of the three-dimensional model is aligned with the coordinate system of a position sensor (or sensors) integrated into the bronchoscope, thereby enabling the navigation system to provide an estimate of the position of the bronchoscope within the pulmonary lumen network. Examples of position sensors include robotic sensors, Inertial Measurement Units (IMUs), fiber optic shape sensors, Electromagnetic (EM) sensors, and camera sensors.

[0006] Position sensors have limitations when used to provide navigation functions. For example, the accuracy of robotic sensors may be degraded due to their miniaturized size, the accuracy of IMUs may be degraded due to accumulated errors, the accuracy of fiber optic shape sensors may be affected by environmental temperature, the accuracy of EM sensors may be degraded due to ferromagnetic materials, and the localization accuracy of camera sensors may be degraded due to poor quality images. In this regard, interventional imaging modalities such as fluoroscopy / X-ray scanning devices may be used to provide additional contextual information of the robotic bronchoscope within the patient's body.

[0007] However, fluoroscopic images are inherently two-dimensional, and it can be difficult for a physician to understand the volumetric morphology of an anatomical structure from a single or series of fluoroscopic images. The embodiments described herein can reconstruct a three-dimensional model of the instrument and treatment site from a limited set of two-dimensional images. Rendering the reconstructed three-dimensional model can provide a user interface that provides the operator with three-dimensional context, thereby allowing better insight when aligning the bronchoscope tip to the biopsy site. Additionally or alternatively, the reconstructed model can be registered to a preoperative model, which can facilitate improvements to the navigation system.

[0008] 3D Pose Estimation System FIG. 1 illustrates an example medical system 100 for performing various medical procedures in accordance with aspects of the present disclosure. The medical system 100 may be used, for example, in endoscopic procedures. Robotic medical solutions may provide relatively greater precision, greater control, and / or greater eye-hand coordination for certain instruments compared to procedures using only human hands. Although the system 100 of FIG. 1 is presented in the context of a bronchoscopy procedure, it should be understood that the principles disclosed herein may be implemented in any type of endoscopic procedure.

[0009] The medical system 100 includes a robotic system 10 (e.g., a mobile robotic cart) configured to engage and / or control a medical instrument (e.g., a bronchoscope) including a proximal handle 31 and a shaft 40 coupled to the handle 31 at its proximal portion to perform a procedure on a patient 7. It should be understood that the instrument 40 may be any type of shaft-based medical instrument, including an endoscope (such as a ureteroscope or bronchoscope), a catheter (such as a steerable or non-steerable catheter), a needle, a nephroscope, a laparoscope, or other type of medical instrument. The instrument 40 may access the internal patient anatomy through direct access (e.g., through a natural orifice) and / or through percutaneous access via a skin / tissue puncture.

[0010] The medical system 100 includes a control system 50 configured to interface with the robotic system 10 to provide information regarding a procedure and / or perform various other operations. For example, the control system 50 may include one or more displays 56 configured to present certain information to assist the physician 5 and / or other technicians or individuals. The medical system 100 may include a table 15 configured to hold a patient 7. The system 100 may further include an electromagnetic (EM) field generator, such as a robot-mounted EM field generator 80 or / and an EM field generator 85, mounted on the table 15 or other structure.

[0011] While various robotic arms 12 are shown in various positions and coupled to various tools / devices, it should be understood that such configurations are shown for convenience and illustrative purposes and that such robotic arms may have different configurations over time and / or at different points during a medical procedure. Additionally, the robotic arms 12 may be coupled to different devices / instruments than those shown in FIG. 1, and in some cases or periods, one or more of the arms may not be utilized or coupled to a medical instrument. Coupling of instruments to the robotic system 10 may be via a robotic end effector 6 associated with a distal end of each arm 12. The term "end effector" is used herein according to its broad and ordinary meaning and may refer to any type of robotic manipulator device, component, and / or assembly. The terms "robotic manipulator" and "robotic manipulator assembly" are used according to their broad and ordinary meaning and may collectively or individually refer to a robotic end effector and / or a sterile adaptor or other adaptor component coupled to an end effector. For example, a "robot manipulator" or a "robot manipulator assembly" may refer to an instrument device manipulator (IDM) that includes one or more drive outputs, whether embodied in a robotic end effector, adapter, and / or other component.

[0012] In some embodiments, the physician 5 can interact with the control system 50 and / or the robotic system 10 to cause / control the robotic system 10 to advance and navigate the medical instrument shaft 40 (e.g., scope) through the patient's anatomy to a target site and / or to perform certain operations using associated instrumentation. The control system 50 may provide information associated with the medical instrument 40 and / or other instruments of the system 100, such as real-time endoscopic images captured thereon, via the display 56 to assist the physician 5 in navigating / controlling such instruments. The control system 50 may also provide imaging / location information to the physician 5 based on a particular positioning modality, such as fluoroscopy, ultrasound, optical / camera imaging, EM field positioning, or other modalities, as described in detail herein.

[0013] Various scope / shaft-type instruments disclosed herein, such as shaft 40 of system 100, can be configured to navigate within the human anatomy, such as within a natural orifice or lumen of the human anatomy. The terms "scope" and "endoscope" are used herein according to their broad and ordinary meaning and may refer to any type of elongated (e.g., shaft-type) medical instrument having imaging, viewing, and / or capture capabilities and configured to be introduced into any type of organ, cavity, lumen, chamber, or space of the body. Scopes can include, for example, ureteroscopes (e.g., for accessing the urinary tract), laparoscopes, nephroscopes (e.g., for accessing the kidneys), bronchoscopes (e.g., for accessing the airways such as the bronchi), colonoscopes (e.g., for accessing the colon), arthroscopes (e.g., for accessing joints), cystoscopes (e.g., for accessing the bladder), colonoscopes (e.g., for accessing the colon and / or rectum), borescopes, and the like. A scope / endoscope, in some cases, may comprise at least a portion of a rigid and / or flexible tube and may be sized to be passed through an outer sheath, catheter, introducer, or other luminal device, or may be used without such a device. Endoscopes and other instruments described herein may have associated with them at their distal ends or other portions certain markers / sensors configured to be visible / detectable within a field of view / space associated with one or more positioning (e.g., imaging) systems / modalities.

[0014] The system 100 is illustrated as including an imaging device (e.g., a fluoroscopy system) 70, which includes an x-ray generator 75 and an image detector 74 (called an “image intensifier” in some contexts). Either component 74, 75 may be referred to herein as a “source,” and both may be mounted on a movable C-arm 71. A control system 50 or other system / device may be used to store and / or manipulate images generated using the imaging device 70. In some embodiments, the bed 15 is radiolucent, such that radiation from the generator 75 can pass through the bed 15 and a target area of ​​the patient's anatomy, and the patient 7 is positioned between the ends of the C-arm 71. The structure / arm 71 of the fluoroscopy system 70 may be rotatable or fixed. The imaging device 70 may be implemented to allow live images to be viewed to facilitate image-guided surgery. The structure / arm 71 may be selectively movable to allow various images of the patient 7 and / or the surgical field to be taken by the fluoroscopy panel source 74.

[0015] In the exemplary bronchoscopy configuration shown in FIG. 1, the field generator 67 is mounted to a bed. In other exemplary embodiments, the field generator 67 may be attached to a robotic arm. Because the electric field generated by the electric field generator 67 may be distorted by the presence of metal or other conductive components therein, it may be desirable to position the electric field generator 67 such that other components of the system do not substantially interfere with the electric field. For example, it may be desirable to position the electric field generator 67 at least 8 inches or more away from a support arm 71 associated with a fluoroscopy system.

[0016] System 100 (as well as other systems disclosed herein) can include an optical imaging source (not shown), such as a camera device (e.g., stereoscopic camera assembly, depth-sensing camera assembly (e.g., RGB / RGBD)). The optical imaging source may be configured / used to view a field of view within a surgical environment and identify specific markers disposed within the field of view. For example, in some embodiments, the imaging source can emit infrared (IR) or other frequency electromagnetic radiation and / or detect reflections of such radiation to identify markers including surfaces that reflect such radiation. Such optical deflections can indicate the location and / or orientation of the marker associated with a particular optical modality. System 100 can have specific markers / fiducials that can be detectable / locatable in one or more reference / coordinate frames / spaces associated with respective localization modalities.

[0017] 1, the image detector 74 may include one or more external tracking sensors 78. The external tracking sensors 78 may include position sensors, optical tracking, depth-sensing cameras, or any combination thereof, as described above, that can be used to determine the pose of the imaging device 70.

[0018] FIG. 2 is a diagram illustrating components and subsystems of the control system 50 shown in FIG. 1, according to an exemplary embodiment. As described above, the control system 50 can be configured to provide various functions to assist in the performance of a medical procedure. For example, the control system 50 can communicate with the robotic system 10 via a wireless or wired connection (e.g., to control the robotic system 10). In some embodiments, the control system 50 can communicate with the robotic system 10 to receive position data from the robotic system regarding the position of the distal end of the scope 40 or other instrumentation. Such positioning data may be derived using one or more position sensors (e.g., electromagnetic sensors, shape-sensing fibers, accelerometers, gyroscopes, satellite-based positioning sensors (e.g., Global Positioning System (GPS)), radio frequency transceivers, etc.) associated with the respective instrumentation and / or based at least in part on robotic system data (e.g., arm position / orientation data, known parameters or dimensions of various system components, etc.) and vision-based algorithms. In some embodiments, the control system 50 can communicate with an EM field generator to control the generation of an EM field in an area around the patient 7 and / or around the tracked instrumentation.

[0019] As mentioned above, the system 100 may include certain control circuits configured to perform certain functions described herein, including the control circuitry 251 of the control system 50. That is, the control circuitry of the system 100 may be part of the robotic system 10, the control system 50, or some combination thereof. Thus, all references to control circuits herein may refer to circuitry embodied in the robotic system, the control system, or any other component of a medical system, such as the medical system 100 shown in FIG. 1, respectively. The term "control circuitry" is used herein according to its broad and ordinary meaning and may refer to any collection of processors, processing circuits, processing modules / units, chips, dies (e.g., semiconductor dies including one or more active and / or passive devices and / or connectivity circuits), microprocessors, microcontrollers, digital signal processors, microcomputers, central processing units, field programmable gate arrays, programmable logic devices, state machines (e.g., hardware state machines), logic circuits, analog circuits, digital circuits, and / or any devices that manipulate signals (analog and / or digital) based on hard-coding of circuits and / or operational instructions. The control circuitry referred to herein may further include one or more circuit boards (e.g., printed circuit boards), conductive traces and vias, and / or mounting pads, connectors, and / or components. The control circuitry referred to herein may further include one or more storage devices, which may be embodied in a single memory device, multiple memory devices, and / or embedded circuitry of a device. Such data storage devices may include read-only memory, random access memory, volatile memory, non-volatile memory, static memory, dynamic memory, flash memory, cache memory, data storage registers, and / or any device that stores digital information.It should be noted that in embodiments in which the control circuitry comprises hardware and / or software state machines, analog circuits, digital circuits, and / or logic circuits, the data storage devices / registers that store any associated operational instructions may be embedded within or external to the circuitry that comprises the state machines, analog circuits, digital circuits, and / or logic circuits.

[0020] The control circuitry 251 may comprise a computer-readable medium that stores and / or is configured to store hard-coded and / or operational instructions corresponding to at least some of the steps and / or functions illustrated in one or more of the figures and / or described herein. Such a computer-readable medium may, in some cases, be included in an article of manufacture. The control circuitry 251 may be maintained / located entirely locally or may be at least partially remotely located (e.g., indirectly communicatively coupled via a local area network and / or wide area network). Any of the control circuits 251 may be configured to perform any aspect(s) of the various processes disclosed herein.

[0021] 2, the control system 50 can include various I / O components 258 configured to assist the physician 5 or others in performing a medical procedure. For example, the input / output (I / O) components 258 can be configured to allow user input to control / navigate the scope 40 and / or basket system within the patient 7. In some embodiments, for example, the physician 5 can provide input to the control system 50 and / or the robotic system 10, and in response to such input, can send control signals to the robotic system 10 to operate the scope 40 and / or other robotically controlled instrumentation.

[0022] The control system 50 and / or the robotic system 10 may include specific user controls (e.g., controls 55), which may comprise one or more buttons, keys, joysticks, handheld controllers (e.g., video game type controllers), computer mice, track pads, track balls, control pads, and / or any type of user input (and / or output) device or device interface, such as sensors (e.g., motion sensors or cameras) that capture hand and finger gestures, touch screens, and / or interfaces / connectors therefor. Such user controls are communicatively and / or physically coupled to respective control circuits. The control system may include a structural tower 51, as well as one or more wheels 58 that support the tower 51. The control system 50 may further include specific communication interfaces 254 and / or power interfaces 259.

[0023] The control circuitry 251 may include a data store 260 that stores various types of data, such as intraoperative images 220, trained network 222, sensor data 224, calibration data 226, and preoperative 3D model 228. The intraoperative images 220 may be data representing images acquired during a procedure, for example, via a fluoroscopy scanner, cone beam, or C-arm scanner. The intraoperative images may be two-dimensional images showing 2D positioning of the instruments and treatment site. As discussed in more detail later in this disclosure, the intraoperative images 220 may be used as input to generate a 3D representation of these positionings to provide more insight into the proper 3D alignment of the instruments and treatment site.

[0024] The trained network 222 may include data and logic configured to identify one or both of the instruments or treatment sites shown in the intraoperative images. In some embodiments, the trained network 222 may include one or more trained networks that segment only the instruments, and then another set of one or more trained networks that segment only the treatment sites. In other embodiments, the trained network 222 may include a network configured to identify both the instrument (or instruments) and the treatment sites. As discussed in more detail below, the trained networks may be generated by a network training system communicatively or electronically coupled to the control system 50.

[0025] The sensor data 224 may include raw data collected from and / or processed by input devices (e.g., control system 50, optical sensors, EM sensors, IDM) to generate estimated state information and output navigation data for the instrument. By way of example and not limitation, the sensor data 224 may include image data, position sensor data, and robotic data. Image data may include one or more image frames captured by an imaging device at the instrument tip, as well as information such as a frame rate or timestamp that allows for the determination of the time elapsed between pairs of frames. Robotic data includes data related to the physical movement of the medical instrument or a portion of the medical instrument (e.g., the instrument tip or sheath) within the tubular network. Exemplary robotic data include command data instructing the instrument tip to reach and / or change its orientation (e.g., with a particular pitch, roll, yaw, insertion, and retraction of one or both of the leader and sheath), insertion data describing the insertion motion of a portion of the medical instrument (e.g., the instrument tip or sheath), IDM data, and mechanical data describing the mechanical movement of an elongated member of the medical instrument, e.g., the motion of one or more pull wires, tendons, or shafts of the endoscope that drive the actual movement of the medial instrument within the tubular network. Position sensor data may include data collected by the instrument's position sensors (e.g., EM sensors, shape sensing fibers, etc.).

[0026] The calibration data 226 may include data representing intrinsic parameters of the imaging device such as principal point, focal length, distortion coefficients, etc. The calibration data 226 may be obtained by an imaging device calibration procedure.

[0027] The pre-operative three-dimensional model data 228 may be a computer-generated 3D model representing an anatomical space, according to one embodiment. The pre-operative three-dimensional model data 228 may be generated using a centerline obtained by processing a pre-operatively generated CT scan. In some embodiments, the computer software may be capable of mapping a navigation path within the tubular network to access a treatment site within the model represented by the pre-operative three-dimensional model data.

[0028] Various modules of the control circuitry 251 may process the data stored in the data store 221. For example, the control circuitry may include a 3D model renderer 240, a navigation module 242, a tool segmenter 244, a site segmenter 246, and a calibration module 248. The tool segmenter 244 may process the intraoperative image data using one or more trained networks (e.g., trained network 222) to generate segmented data corresponding to instruments shown in the intraoperative image data. Similar to the tool segmenter 244, the site segmenter 246 may similarly process the intraoperative image data using one or more trained networks (e.g., trained network 222) to generate segmented data corresponding to treatment sites shown in the intraoperative image data. Although the tool segmenter 244 and the site segmenter 246 are shown in FIG. 2 as separate modules, it should be understood that other embodiments may have a single module that performs the functions of both the tool segmenter 244 and the site segmenter 246. In general, the segmented data generated by the segmenters 244, 246 may be fed to the input of a 3D model renderer, described below. However, in some embodiments, the segmented data may be provided to an operator of the system 100 and rendered in a manner useful to the operator. As described below, the segmented stream may be provided to the operator as the operator adjusts the placement of the instrument in real time.

[0029] The 3D model renderer 240 may be a control circuit configured to process the segmented intraoperative image data and reconstruct a 3D rendering of the instrument relative to the treatment site.

[0030] The navigation module 242 processes various data (e.g., sensor data 224) and provides localization data of the instrument tip as a function of time, the localization data indicating an estimated position and orientation information of the instrument tip within the anatomical structure. In some embodiments, the navigation module 242 registers a coordinate frame corresponding to the instrument's position sensors to the coordinate frame of the pre-operative 3D model 228.

[0031] 3D model reconstruction method and operation Details of the operation of an exemplary model reconstruction system will now be described. The methods and operations disclosed herein will be described with reference to the model reconstruction system 100 shown in Figure 1 and the modules and other components shown in Figure 2. However, it should be understood that the methods and operations may be performed by any of the components discussed herein, alone or in combination.

[0032] The model reconstruction system can generate a representation of a three-dimensional anatomical structure based on a relatively limited number of two-dimensional images acquired during or as part of a medical procedure. FIG. 3 is a flow chart illustrating a method 300 for reconstructing a 3D model of an instrument and a treatment site from two-dimensional images acquired during or as part of a medical procedure, according to an exemplary embodiment. As FIG. 3 illustrates, the method 300 may begin at block 310, where the system 100 acquires a two-dimensional image (e.g., a fluoroscopic image) of a patient's anatomy acquired by an imaging device 70. In some embodiments, the domain shift is performed when the imaging device 70 is configured to replicate the desired characteristics. As an example, some embodiments may configure the imaging device 70 to adjust the principal point, focal length, distortion coefficient, or any other parameter of the imaging device.

[0033] In block 320, the system retrieves one or more neural networks previously trained with images generated based on one or more computed tomography scans. As used herein, a system or device may "retrieve" data in any number of mechanisms, such as a push or pull model or access through local storage. For example, a neural network service (described below) may send one or more neural networks to the control system 50 based on a determinable event (e.g., determining whether a neural network has been updated or a new one is available) or based on a schedule for periodically updating the neural networks of the control system. In another example, the control system 50 may send a request for a neural network to the neural network service, and in response to the request, the neural network service may send the requested neural network to the control system 50. Still further, some embodiments of the control system may generate their own local copies of the neural networks and retrieve the neural networks in block 320 via retrieving them from a local storage device, such as the data store 260.

[0034] In block 330, the system uses one or more neural networks to identify segmentations corresponding to the instrument in the two-dimensional images (e.g., fluoroscopic images) acquired in block 310. Examples of segmentations of two-dimensional images corresponding to the instrument are shown with reference to FIGS. 7 and 8. FIG. 7 illustrates a series of instrument segmentations 700 that may be generated by a neural network, according to an exemplary embodiment. Each of the fluoroscopic images 712, 714, 716 is obtained from an imaging device 70 oriented at the same anatomical location of the patient but at a different angle. By way of example and not limitation, image 712 may be taken in a 15 degree left anterior oblique view. Image 714 may be taken in a 0 degree anterior-posterior view. Image 716 may be taken in a 15 degree right anterior oblique. Each of the images 712, 714, 716 includes a respective segmentation 720a, 720b, 720 of the instrument. In some embodiments, the segmentation includes sub-segments that represent detectable portions of the instrument. 8 is a diagram illustrating a segmentation of an instrument, according to an example embodiment, including sub-parts scope tip 820 and articulatable section 822. Additionally, in some embodiments, directional relationships can be determined based on the relationships between the sub-parts.

[0035] 3, in block 340, the system 100 may use one or more neural networks to identify a segmentation in the fluoroscopic image that corresponds to the treatment site (e.g., a tumor or a legion). Figure 9 shows an example of a treatment site segmentation, according to an exemplary embodiment. For example, Figure 9 is a segmentation of a fluoroscopic image in which a tumor nodule has been segmented from the remainder of the image.

[0036] Referring back to FIG. 3, it should be understood that the order and timing of performing blocks 330 and 340 is flexible. For example, blocks 330 and 340 can be performed in parallel or sequentially. In some embodiments, block 340 can be performed depending on the results of block 330. For example, the instrument segmentation in block 330 may be used to generate a Region-Of-Interest (ROI), which can then be used to facilitate finer treatment site segmentation in block 340. That is, the system segments a determinable region around the instrument or an identifiable portion of the instrument, such as the instrument tip. FIG. 10 illustrates segmentation based on a region of interest 1020, such as a distance centered on the segmented instrument region, according to an exemplary embodiment. An exemplary distance includes 3 centimeters, but can be any appropriate distance based on the treatment situation.

[0037] Referring back to FIG. 3, at block 350, the system may reconstruct a three-dimensional model of the instrument and treatment site using the segmentation in the two-dimensional image corresponding to the instrument and the segmentation in the two-dimensional image corresponding to the treatment site. More than one two-dimensional image may be used for reconstruction. While it is contemplated that two two-dimensional images provide sufficient accuracy, many embodiments may choose to increase the number of two-dimensional images, for example, three or more. For example, one embodiment may reconstruct using three two-dimensional images, one from a 15 degree left anterior oblique view, a second image from a 0 degree anterior-posterior view, and a third image from a 15 degree right anterior oblique view.

[0038] The reconstruction can be performed using triangulation of the different 2D images and the intraoperative imaging device pose and the imaging device intrinsic parameters. The intrinsic parameters of the imaging device (principal point, focal length, and distortion coefficients) can be obtained via a camera calibration procedure. The intraoperative imaging device pose can be obtained in several ways. If a motorized scanner is used, the pose at which the fluoroscopic images are taken can be obtained from the provided output of the imaging device. If the pose information is not provided by the system, an external tracking sensor 78 (optical tracking, inertial measurement unit, RGB / RGBD camera) in FIG. 1 can be placed on the scanner to obtain the pose information. In some embodiments, the system may instruct the operator to acquire images at predefined angles, and the system can assume that the operator followed the instructions.

[0039] Neural Network Training As mentioned above, referring to block 320 of FIG. 3, the system 100 may obtain a neural network trained by images from one or more CT scans. Also as mentioned above, the neural network may be accessed via a neural network service. The concepts including neural network generation and neural network services will now be described in more detail. FIG. 4 is a system diagram illustrating a neural network generation system 400, according to an exemplary embodiment. The neural network generation system 400 may include network communication between the control system 50 of FIG. 1 and the neural network training system 410 via a network 420. The network 420 may include any suitable communication network that allows two or more computer systems to communicate with each other, which may include wireless and / or wired networks. Exemplary networks include one or more personal area networks (PANs), local area networks (LANs), wide area networks (WANs), Internet area networks (IANs), cellular networks, the Internet, and the like.

[0040] The neural network training system 410 may be a computer system configured to generate a neural network 412 usable to generate a reconstructed three-dimensional model of instruments and a treatment site from intraoperative two-dimensional images (e.g., fluoroscopic images). The neural network training system 410 shown in FIG. 4 communicates with the control system 50 via a network 420, however, it should be understood that the neural network training system 410 may be on-site with respect to the control system and may operate according to pre-operative software associated with the control system. The pre-operative software of the control system 50 may be software usable to generate a pre-operative three-dimensional model of a patient's anatomy, plan a path to a treatment site, etc.

[0041] The neural network training system 410 may be coupled to a two-dimensional image database 430. The two-dimensional image database 430 may include annotated images of anatomical structures such as lungs and kidneys. The CT scan may include the known shape and location of the tumor. In one example, the shape may be known based on manual annotations included by an experienced human observer or based on automatic annotations from a computer-based vision algorithm. In some cases, instead of or in addition to a patient population, the two-dimensional image database 430 may include CT scans of patients for whom a procedure is planned.

[0042] The neural network training system 410 may also be coupled to an instrument model database 440. The instrument model database 440 includes data characterizing known geometric properties of instruments, which may be stored, for example, in a computer-aided design file.

[0043] As shown, the neural network training system 410 and the control system 50 may exchange domain data. The domain data 414 may be data characterizing the calibration-based operation of the imaging device or the patient. The neural network training system 410 may use the domain data 414 to specialize the data being used to train the neural network that is sent to the control system 50.

[0044] 5 is a flow chart illustrating a method 500 for generating a trained neural network usable to reconstruct a three-dimensional model of an instrument and a treatment site, according to an example embodiment. The method 500 may begin at block 510, where a neural network training system 410 acquires volumetric data from one or more CT scans. In some embodiments, the neural network training system 410 may select the one or more CT scans based on domain data 414, which characterizes a patient based on age, race, sex, health status, etc. In other embodiments, the one or more CT scans include CT scans acquired from a patient as part of a pre-operative planning procedure.

[0045] At block 520, the neural network training system 410 obtains the geometric characteristics of the instrument. As mentioned above, the geometric characteristics of the instrument may include geometric data derived from a CAD file.

[0046] In block 530, the neural network training system 410 generates a synthetic fluoroscopic image using the volumetric data and geometric properties. In one embodiment, as part of block 530, the neural network training system 410 utilizes patient-specific CT scan data to generate a Digitally Reconstructed Radiology (DRR) image. The input for DRR data generation includes the known 3D shape and location of the treatment site (e.g., tumor / nodule) and known instrument geometric properties (available in a CAD file). Based on the sampling of the viewing angle and distance in the 3D space of the CT scan coordinate frame, the DRR data generation can be performed using backprojection / ray tracing techniques or deep learning based techniques. The generated DRR image shares similar image characteristics as the actual fluoroscopic image. To further generate a DRR image that shares similar image characteristics as the actual fluoroscopic image, the settings and example images generated by the imaging device may be provided as part of the domain transfer. The neural network training system 410 may generate the DRR image using these settings and example images.

[0047] In block 540, the neural network training system 410 trains the neural network using the synthetic fluoroscopic images. Once the neural network is trained, it is configured to segment the fluoroscopic images according to the treatment site or the instrument. In the above example where the neural network training system 410 generates DRR images from the CT scan in block 530, these DRR images can be used as synthetic fluoroscopic images to train the neural network for instrument segmentation and treatment site segmentation. The architecture of the segmentation network can be based on a convolutional neural network (e.g., UNet and ResNet), a transformer, or a graph neural network.

[0048] The neural network training system 410 then makes the trained neural network or networks available to the control system 50. The neural network training system 410 may make the neural networks available by providing access via a network 420 or by providing transfer via a computer-readable medium, where the neural network training system 410 and the control system 50 are co-located.

[0049] Example of the entire framework FIG. 6 is a block diagram illustrating an exemplary data architecture 600 for fluoroscopic image processing, according to an exemplary embodiment. The functional blocks of the exemplary data architecture 600 are labeled according to the corresponding blocks from the flowcharts described above. As FIG. 6 illustrates, there is domain data that is transferred or exchanged as part of the synthetic fluoroscopic image and the intraoperative fluoroscopic image. This is so that the synthetic fluoroscopic image can be generated by the neural network training system to accurately reflect the intraoperative fluoroscopic image generated by the medical system performing the medical procedure. For example, the domain data may include the principal point, focal length, and distortion coefficients of the imaging device 70 of FIG. 1. The generated DRR may then reflect these parameters to more closely align the image that the imaging device 70 would likely generate.

[0050] Similarly, if the imaging device has the ability to configure itself, the domain data may include data usable by the medical system to configure the imaging device. Examples of domain data that may be used by the control system include imaging angle, image contrast, depth, etc.

[0051] It should also be appreciated that the example data architecture 600 shown in FIG. 6 also illustrates that the segmenters 330' and 340' may be used in a correlated manner. That is, the instrument segmenter 330' may first segment the instrument from the intraoperative image, and then the system may identify a region of interest based on the instrument segmentation. Based on the region of interest, the treatment site segmenter 340' may generate a refined segmentation of the treatment site based on regions within the region of interest.

[0052] Navigation Integration As mentioned above, with reference to FIG. 2, the control system 50 may include a navigation system that locates the instruments to the patient via the 3D pre-operative model. After the control system generates the reconstructed 3D model of the instruments and treatment site, the control system may register the reconstructed 3D model of the instruments and treatment site to the CT / patient reference frame. Once registered, the segmented 3D scope or 3D nodule information can be used as another input to the navigation module to improve the accuracy of the navigation output, which is also in the CT / patient reference frame.

[0053] The information can be used in multiple ways, for example, the control system 50 may use the 3D reconstructed scope pose information as another input to the "Fusion" framework, combining it with other outputs from EM-based, vision-based, and robotics-based algorithms to improve the accuracy of the navigation, "Fusion" framework output.

[0054] As another example, the control system 50 may use the 3D reconstructed nodes to correct intraoperative node position estimates and update node positions, which can help compensate for errors caused by CT-to-body displacement and anatomical deformations induced during surgery.

[0055] As another example, the control system 50 may use the 3D reconstructed scope and node positions to calculate the relative distance between the scope tip and the node, and use this information to correct distance measurements to targets displayed to the user.

[0056] As another example, the control system 50 may use the 3D reconstructed shape of the scope to input shape information into the navigation framework. The shape information can be registered to the skeleton / path to provide more stable navigation information relative to the scope tip position alone.

[0057] As another example, the control system 50 may use geometry information from the scope to detect and model intraoperative anatomical deformations. In some embodiments, the control system can adaptively update the 3D map / lung model based on the scope geometry information.

[0058] Segmentation Stream In some situations, once the operator has the instrument near the treatment site, the operator may choose to take a fluoroscopic image to verify the instrument's pose relative to the treatment site. While much of this disclosure describes embodiments that generate a reconstructed 3D model of the instrument and treatment site, these and other embodiments may render intermediate results to assist the operator in making fine adjustments to the instrument. For example, in some embodiments, the control system may render the intraoperative fluoroscopic image to the operator along with data identifying different segmentations within the intraoperative fluoroscopic image. Augmenting the intraoperative fluoroscopic image with segmentation data may provide an enhanced view of the instrument and treatment site that may be useful to the operator in visualizing the instrument's pose, even if the intraoperative image is only 2D. This is particularly useful when the operator places the imaging device in a mode that provides fluoroscopic images in rapid succession or a video-like appearance, referred to herein as an "intraoperative image stream." With an intraoperative image stream, the processing time to reconstruct the 3D model of the instrument and treatment site may be longer than the rate at which intraoperative images are captured. Thus, if a stream of 2D intraoperative images is provided, the control system may render a segmented view of the stream of intraoperative images, the segmented view including visual indicators of segmented instruments and segmented treatment sites. As used herein, a view of the stream of intraoperative images augmented with segmentation data may be referred to herein as a "segmentation stream."

[0059] This 2D visualization of the treatment site can be shown on fluoroscopic images taken from different angles based on the pose information of the imaging device at the time the 2D intraoperative images are taken. The control system may obtain the pose information (or some aspect of the pose, such as orientation or position) manually (e.g., date entered by an operator of the system via a user interface provided by the control system), through a communication interface between the control system and the imaging device, or through an external tracking sensor. The control system may then use the camera parameters and pose information to backproject a 3D model of the treatment site onto the 2D intraoperative images.

[0060] Imaging device calibration Fluoroscopic / X-ray images acquired by a C-arm scanner can be used for the reconstruction of a 3D anatomical scene. The reconstruction module of the control system 50 (see FIG. 2) may use the intrinsic and extrinsic parameters of the C-arm scanner to achieve a relatively accurate 3D reconstruction. As mentioned above, the C-arm scanner includes a source generator (that emits an X-ray source) and a detector that identifies the X-rays to create an image. The calibration module can treat this source-detector imaging setup as a pinhole camera model, and thus the intrinsic parameters are the principal point, focal length, and distortion coefficients. If the C-arm scanner has a flat panel detector, there will be no distortion in the acquired fluoroscopic / X-ray images, and thus the distortion coefficients can be ignored in the calibration process.

[0061] To calculate the intrinsic parameters of a fluoroscopy / X-ray scanner, a calibration object for the calibration module can be designed to facilitate this. FIG. 11 illustrates a calibration object 1100 according to an exemplary embodiment. The calibration object 1100 can be a planar object with small balls attached thereon. The material used to fabricate the calibration object 1100 may have low opacity under fluoroscopy imaging so that the metal balls are clearly visible to the imaging device. The rectangular geometric pattern of ball placement on the calibration object 1100 can be generated arbitrarily prior to tool machining, and this known geometric pattern is then used in the calibration process.

[0062] i. Intrinsic calibration Next, intrinsic calibration will be described. To collect data, an operator performing the calibration process may place a calibration object on the patient bed and adjust the height of the fluoroscopic scanner so that the calibration object is approximately centered between the source generator and the detector. The scanner is then rotated at various predetermined angles and fluoroscopic images are taken on the calibration object at each angle. An exemplary image 1200 of these data is provided in FIG.

[0063] For data processing, for any collected image, the calibration module detects the metal ball positions on the image of the calibration object, which can be done, for example, by a blob detection algorithm. After obtaining the ball positions on the image of the calibration object, the calibration module matches these detected balls with their correspondence in the known geometric pattern of the calibration object. This matching process can have the following steps: 1: Calculate pairwise distances between ball positions, loop over each ball position and do the following: 1a: For each ball position, collect the top X (say 8) nearest neighbor positions. 2: Find the dominant orientation on the image. 2a: For each ball position, create a vector to each of its eight nearest neighbors. 2b: Calculate the orientation of the vector, which then outputs a histogram of orientations (where the orientations range from 0 to 180 degrees, which means that 225 is assigned as 45 degrees). 2c: From the orientation histogram, identify two orientations that correspond to the horizontal and vertical directions on a known geometric pattern. 3: Create a graph by checking the ball positions and their nearest neighbors. The cost of a connection between two vertices in the graph can be defined as a combination of their position distance and orientation. 4: Find the maximal connected component of the graph and match this maximal connected component in a known geometric pattern. After graph matching, a positional correspondence between the detected ball and the pattern is determined.

[0064] The calibration module performs the above correspondence search between all collected images and known geometric patterns, and then uses these obtained correspondences to obtain the intrinsic parameters including the principal point, focal length, and distortion coefficients (optional) via a camera calibration algorithm.

[0065] i. Extrinsic Calibration For extrinsic calibration, most fluoroscopy / X-ray scanners do not provide 6 Degree-of-Freedom (DoF) attitude information (orientation and translation) during imaging. To obtain this information, an external tracking sensor can be used. This external tracking sensor can be either an IMU device or an RGB-D camera (or a combination of the two). This external tracking sensor generates 6DoF attitude data on its own coordinate frame. To transform the attitude data into the scanner imaging coordinate frame, a hand-eye calibration can be performed.

[0066] To collect data for this calibration process, an external tracking sensor is attached to the scanner, preferably near the detector. The system then performs fluoroscopy / x-ray imaging on the calibration object at various angles / translations where the object is at least partially visible in the fluoroscopy image. At each pose when an image is taken, the calibration module records the pose output from the external tracking sensor. With these paired image and pose data sets, the calibration module can then run a hand-eye calibration algorithm and calculate a sensor-to-scanner transformation. This transformation can then be used intraoperatively to transform readings from the sensor into the scanner imaging coordinate frame, thus providing the pose information required for 3D scene reconstruction.

[0067] Exemplary 3D reconstructions The control system may cause the reconstructed 3D model to be rendered on a display device, as described above with reference to block 360 of Figure 3. Figure 13 illustrates a reconstructed 3D model rendering 1300, according to an example embodiment. The reconstructed 3D model rendering 1300 may include data indicative of the shape and orientation of the instrument 1302 relative to the treatment site 1304 when rendered by a display device. Methods and systems for generating the reconstructed 3D model are described in more detail above.

[0068] Although not shown, in some embodiments, when a reconstructed 3D model is aligned to a virtual model of an anatomical structure (e.g., lungs, kidneys, gastrointestinal system, etc.), features of the virtual model of the anatomical structure may be depicted in conjunction with (e.g., overlaid on) the reconstructed 3D model.

[0069] Although not shown, in some embodiments, the reconstructed 3D model rendering 1300 may include user interface features that facilitate alignment of the instrument with the treatment site. For example, in one embodiment, the reconstructed 3D model rendering 1300 may include graphical elements such as a line (patterned or not) extending axially from the tip of the instrument 1302. In these embodiments, the operator may have the robotic system adjust the pose of the instrument until the line extending from the instrument 1302 shown in the rendering 1300 intersects with the treatment site 1304. In some embodiments, the rendering 1300 may be updated by the system upon detection of a particular event, such as alignment of the instrument 1302 with the treatment site 1304. One such update may be to change the color of the treatment site or graphical elements extending from the instrument 1302 depending on the alignment of the instrument with the treatment site. For example, the treatment site 1304 may be updated to be rendered in one color when there is no alignment and rendered in another color when there is alignment. In some embodiments, the color may represent the strength of the alignment, so that the operator can distinguish between alignments that may result in sampling the edge of the nodule and alignments that may result in sampling the center of the nodule.

[0070] Mounting system and terminology Implementations disclosed herein provide systems, methods, and devices for reconstructing three-dimensional models of anatomical structures using two-dimensional images. Various implementations described herein provide improved visualization of anatomical structures during medical procedures using medical robots.

[0071] System 100 can include various other components. For example, system 100 can include one or more control electronics / circuitry, power sources, pneumatics, light sources, actuators (e.g., motors for moving a robotic arm), memory, and / or a communication interface (e.g., for communicating with another device). In some embodiments, the memory can store computer-executable instructions that, when executed by the control circuitry, cause the control circuitry to perform any of the operations discussed herein. For example, the memory can store computer-executable instructions that, when executed by the control circuitry, cause the control circuitry to receive input and / or control signals related to the operation of the robotic arm and, in response, control the robotic arm to position it in a particular arrangement.

[0072] The various components of system 100 may be electrically and / or communicatively coupled using certain connection circuits / devices / features, which may or may not be part of the control circuitry. For example, the connection feature(s) may include one or more printed circuit boards configured to facilitate mounting and / or interconnection of at least some of the various components / circuitry of system 100. In some embodiments, two or more of the control circuitry, data storage / memory, communication interface, power supply unit(s), and / or input / output (I / O) component(s) may be electrically and / or communicatively coupled to one another.

[0073] The term "memory" is used herein according to its broad and ordinary meaning and may refer to any suitable or desired type of computer-readable medium, including, for example, one or more volatile, non-volatile, removable, and / or non-removable data storage devices implemented using any suitable or desired technology, layout, and / or data structures / protocols that contain any suitable or desired computer-readable instructions, data structures, program modules, or other types of data.

[0074] Computer-readable media that may be implemented according to embodiments of the present disclosure may include, but are not limited to, phase-change memory, static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage device, magnetic cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, or any other non-transitory medium that may be used to store information for access by a computing device. As used in certain contexts herein, computer-readable media may generally not include communication media such as modulated data signals and carrier waves. Thus, computer-readable media should generally be understood to refer to non-transitory media.

[0075] Described herein are systems, devices, and methods for generating reconstructed three-dimensional models of treatment sites and instruments. Some implementations of the present disclosure relate to methods that include acquiring volumetric data from one or more computed tomography (CT) scans labeled according to portions of anatomy, acquiring geometric properties of the instruments, generating synthetic fluoroscopic images based on the volumetric data and the geometric properties, and training one or more neural networks using the synthetic fluoroscopic images. The neural networks can be configured to segment the fluoroscopic images according to the treatment site or the instruments.

[0076] In some embodiments, the one or more neural networks can include a first neural network configured to segment the fluoroscopic image according to the treatment site and a second neural network configured to segment the fluoroscopic image according to the instrument, hi some embodiments, the one or more neural networks include a single neural network configured to segment the fluoroscopic image according to both the treatment site and the instrument.

[0077] In some embodiments, the method may further include acquiring domain data corresponding to a procedure performed on the patient by the medical system. In some embodiments, the domain data may include at least one of a principal point, a focal length, or a distortion coefficient. In some embodiments, generating the synthetic fluoroscopic image may be further based on the domain data.

[0078] In some embodiments, generating the composite fluoroscopic image can be based on superimposing a representation of the instrument onto the composite fluoroscopic image based on the geometric characteristics and the preoperative trajectory. In some embodiments, the CT scan can lack a representation of the instrument. In some embodiments, the treatment site can be a pulmonary nodule. In some embodiments, generating the composite fluoroscopic image can include generating a first composite fluoroscopic image focused on a portion of the anatomical structure at a first angle and a second composite fluoroscopic image focused on a portion of the anatomical structure at a second angle different from the first angle. In some embodiments, generating the composite fluoroscopic image can further include generating a third composite fluoroscopic image focused on a portion of the anatomical structure at a third angle different from the first angle and the second angle.

[0079] Some implementations of the present disclosure relate to a system for training one or more neural networks usable to segment intraoperative fluoroscopic images, the system comprising a control circuit and a computer-readable medium. The computer-readable medium can have instructions that, when executed, cause the control circuit to obtain at least one of volumetric data and geometric properties of instruments from one or more computed tomography (CT) scans labeled according to portions of anatomy, generate a composite fluoroscopic image based on the volumetric data and at least one of the geometric properties, and train the one or more neural networks using the composite fluoroscopic image. The neural network can be configured to segment the intraoperative fluoroscopic image according to a treatment site or an instrument.

[0080] Certain implementations of the present disclosure relate to a method for reconstructing a three-dimensional model of an instrument and a treatment site within an anatomical structure, the method may include acquiring fluoroscopic images of a patient's anatomical structure, acquiring one or more neural networks, identifying a segmentation within the fluoroscopic images corresponding to the instrument based on the one or more neural networks, identifying a segmentation within the fluoroscopic images corresponding to the treatment site based on the one or more neural networks, reconstructing a three-dimensional model of the instrument and the treatment site based on the segmentation within the fluoroscopic images corresponding to the instrument and the segmentation within the fluoroscopic images corresponding to the treatment site, and causing the reconstructed three-dimensional model to be rendered on a display device.

[0081] In some embodiments, the method may further include determining a region of interest based on a segmentation in the fluoroscopic image corresponding to the instrument. Identifying a segmentation in the fluoroscopic image corresponding to the treatment site may be based on the region of interest. In some embodiments, identifying a segmentation in the fluoroscopic image corresponding to the instrument may be performed in parallel with identifying a segmentation in the fluoroscopic image corresponding to the treatment site. In some embodiments, the reconstruction of the three-dimensional model of the instrument and treatment site may be further based on calibration data derived from an imaging device that generated the fluoroscopic image of the patient's anatomy.

[0082] In some embodiments, the method may further include causing a segmentation in the fluoroscopic image corresponding to the instrument and a segmentation in the fluoroscopic image corresponding to the treatment site to be rendered on a display device.

[0083] In some embodiments, a segmentation in the fluoroscopic image corresponding to the treatment site and a segmentation in the fluoroscopic image corresponding to the instrument can be rendered on a display device before the reconstructed three-dimensional model is rendered on the display device. In some embodiments, the segmentation in the fluoroscopic image corresponding to the instrument can include a first sub-segmentation and a second sub-segmentation, the first sub-segmentation and the second sub-segmentation corresponding to different components of the instrument.

[0084] In some embodiments, the treatment site can correspond to a biopsy site. In some embodiments, the fluoroscopic images of the patient's anatomy can include a first fluoroscopic image focused on the anatomy at a first angle and a second fluoroscopic image focused on the anatomy at a second angle different from the first angle.

[0085] In some embodiments, reconstructing the three-dimensional model of the instrument and treatment site can include triangulating an identified segment in the first fluoroscopic image and an identified segment in the second fluoroscopic image.

[0086] In some embodiments, the method may further include acquiring at least one of the first angle or the second angle via a communication interface of an imaging device that generated the first fluoroscopic image and the second fluoroscopic image. In some embodiments, the method may further include acquiring at least one of the first angle or the second angle via an operator user interface. In some embodiments, the method may further include acquiring at least one of the first angle or the second angle via an external tracking sensor.

[0087] Some implementations of the present disclosure relate to a system for reconstructing a three-dimensional model of an instrument and a treatment site within an anatomical structure. The system can include a control circuit and a computer-readable medium. The computer-readable medium can have instructions that, when executed, cause the control circuit to acquire fluoroscopic images of a patient's anatomy, acquire one or more neural networks, identify a segmentation within the fluoroscopic images corresponding to an instrument based on the one or more neural networks, identify a segmentation within the fluoroscopic images corresponding to the treatment site based on the one or more neural networks, reconstruct a three-dimensional model of the instrument and the treatment site based on the segmentation within the fluoroscopic images, and render the reconstructed three-dimensional model at a display device.

[0088] For purposes of summarizing the disclosure, certain aspects, advantages, and novel features have been described. It is to be understood that not necessarily all such advantages may be achieved in accordance with any particular embodiment. Thus, the disclosed embodiments may be performed in a manner that achieves or optimizes one advantage or group of advantages taught herein without necessarily achieving other advantages that may be taught or suggested herein.

[0089] Further embodiments Depending on the embodiment, certain acts, events, or functions of any of the algorithms or processes described herein may be performed in a different order, added, merged, or omitted entirely, and thus, in a particular embodiment, not all of the described acts or events are necessary to the execution of a process.

[0090] In particular, conditional language used herein, such as "can," "could," "might," "may," "eg," and the like, unless specifically stated otherwise or understood otherwise within the context in which it is used, is intended to have its ordinary meaning and is generally intended to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not. Thus, such conditional language is not intended to imply that features, elements, and / or steps are generally required in any way for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether those features, elements, and / or steps are included or performed in any particular embodiment, with or without author input or prompting. Terms such as "comprising," "including," "having," and the like, are synonymous and used in their ordinary sense and are used inclusively in a non-limiting manner and do not exclude additional elements, features, acts, operations, etc. Also, when the term "or" is used, for example, to connect a list of elements, the term "or" is used in its inclusive sense (and not its exclusive sense) to mean one, some, or all of the listed elements. Unless specifically stated otherwise, connective language such as the phrase "at least one of X, Y, and Z" is understood in the context as it is commonly used to convey that an item, term, element, etc. can be either X, Y, or Z. Thus, such connective language is not generally intended to imply that a particular embodiment requires that at least one of X, at least one of Y, and at least one of Z, respectively, be present.

[0091] In the above description of the embodiments, it should be understood that various features are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the method of the disclosure should not be interpreted as reflecting an intention that any claim requires more features than are expressly recited in that claim. Moreover, any component, feature, or step illustrated and / or described in a particular embodiment herein may be applied to or used in conjunction with any other embodiment. Moreover, no component, feature, step, or group of components, features, or steps is necessary or essential for each embodiment. Thus, it is intended that the scope of the present disclosure should not be limited by the particular embodiments described above, but should be determined solely by a fair reading of the following claims.

[0092] It should be understood that certain ordinal terms (e.g., "first" or "second") may be provided for ease of reference and do not necessarily imply physical characteristics or ordering. Thus, as used herein, ordinal terms (e.g., "first," "second," "third," etc.) used to modify an element, such as a structure, component, operation, etc., do not necessarily indicate a priority or order of the element with respect to any other elements, but rather may generally distinguish the element from another element having a similar or identical name (apart from the use of the ordinal term). In addition, as used herein, the indefinite articles ("a" and "an") may indicate "one or more" rather than "one." Furthermore, an operation performed "based on" a condition or event may also be performed based on one or more other conditions or events not expressly recited.

[0093] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the example embodiments belong. It is further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and should not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0094] Spatially relative terms such as "outer", "inner", "upper", "lower", "below", "upper", "vertical", "horizontal", and similar terms may be used herein for ease of description to describe the relationship between one element or component and another element or component as illustrated in the figures. It should be understood that the spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if the device shown in the figures is inverted, a device positioned "below" or "under" another device may be placed "above" the other device. Thus, the illustrative term "lower" may include both lower and upper positions. The device may also be oriented in other directions, and thus the spatially relative terms may be interpreted differently depending on the orientation.

[0095] Unless otherwise specified, comparative and / or quantitative terms such as "less," "more," "greater than," and the like are intended to encompass the notion of equality. For example, "less" can mean not only "less than" in the strict mathematical sense, but also "less than or equal to."

[0096] [Embodiment] (1) acquiring volumetric data from one or more computed tomography (CT) scans labeled according to portions of anatomy; Obtaining a geometric characteristic of the instrument; generating a synthetic fluoroscopic image based on the volumetric data and the geometric characteristics; training one or more neural networks using the synthetic fluoroscopic images, the neural networks configured to segment the fluoroscopic images according to a treatment site or according to the instrument; The method includes: (2) The method of embodiment 1, wherein the one or more neural networks include a first neural network configured to segment the fluoroscopic image according to the treatment site and a second neural network configured to segment the fluoroscopic image according to the instrument. (3) The method of any one of the preceding claims, wherein the one or more neural networks include a single neural network configured to segment the fluoroscopic image according to both the treatment site and the instrument. (4) A method according to any one of embodiments 1 to 3, further comprising acquiring domain data corresponding to a procedure performed on the patient by the medical system. (5) The method of claim 4, wherein the domain data includes at least one of a principal point, a focal length, or a distortion coefficient.

[0097] (6) The method of claim 4, wherein generating the synthetic fluoroscopic image is further based on the domain data. (7) A method according to any of claims 1 to 3, wherein generating the synthetic fluoroscopy image is based on superimposing a representation of the instrument onto the synthetic fluoroscopy image based on the geometric characteristics and preoperative path. (8) The method of any one of claims 1 to 3, wherein the CT scan lacks a representation of the instrument. (9) The method according to any one of embodiments 1 to 3, wherein the treatment site is a pulmonary nodule. (10) A method according to any one of embodiments 1 to 3, wherein generating the composite fluoroscopic image includes generating a first composite fluoroscopic image focused on a portion of the anatomical structure at a first angle and a second composite fluoroscopic image focused on the portion of the anatomical structure at a second angle different from the first angle.

[0098] (11) The method of embodiment 10, wherein generating the composite fluoroscopic image further comprises generating a third composite fluoroscopic image focused on the portion of the anatomical structure at a third angle different from the first angle and the second angle. (12) A system for training one or more neural networks usable to segment intraoperative fluoroscopic images, comprising: A control circuit; and a computer-readable medium having instructions that, when executed, cause the control circuitry to: acquiring at least one of volumetric data from one or more computed tomography (CT) scans labeled according to portions of anatomy and geometric characteristics of the instrument; generating a synthetic fluoroscopic image based on at least one of the volumetric data and the geometric characteristics; the synthetic fluoroscopic images are used to train the one or more neural networks, the neural networks configured to segment the intraoperative fluoroscopic images according to a treatment site or according to the instrument. (13) A method for reconstructing a three-dimensional model of an instrument and a treatment site within an anatomical structure, comprising: obtaining a fluoroscopic image of a patient's anatomy; Obtaining one or more neural networks; identifying a segmentation in the fluoroscopic image corresponding to the instrument based on the one or more neural networks; and identifying a segmentation within the fluoroscopic image corresponding to the treatment site based on the one or more neural networks; reconstructing the three-dimensional model of the instrument and the treatment site based on the segmentation in the fluoroscopic image corresponding to the instrument and the segmentation in the fluoroscopic image corresponding to the treatment site; rendering the reconstructed three-dimensional model on a display device; and A method comprising: (14) The method of claim 13, further comprising determining a region of interest based on the segmentation in the fluoroscopic image corresponding to the instrument, wherein the identification of the segmentation in the fluoroscopic image corresponding to the treatment site is based on the region of interest. (15) The method of any one of claims 13 to 14, wherein the identification of the segmentation in the fluoroscopy image corresponding to the instrument is performed in parallel with the identification of the segmentation in the fluoroscopy image corresponding to the treatment site.

[0099] (16) A method according to any of embodiments 13 to 15, wherein the reconstruction of the three-dimensional model of the instrument and the treatment site is further based on calibration data derived from the imaging device that generated the fluoroscopic image of the patient's anatomical structure. (17) A method according to any one of embodiments 13 to 15, further comprising rendering on the display device the segmentation in the fluoroscopy image corresponding to the instrument and the segmentation in the fluoroscopy image corresponding to the treatment site. (18) The method of embodiment 17, wherein the segmentation in the fluoroscopic image corresponding to the treatment site and the segmentation in the fluoroscopic image corresponding to the instrument are rendered on the display device before the reconstructed three-dimensional model is rendered on the display device. (19) A method according to any of embodiments 13 to 15, wherein the segmentation in the fluoroscopic image corresponding to the instrument includes a first sub-segmentation and a second sub-segmentation, the first sub-segmentation and the second sub-segmentation corresponding to different components of the instrument. (20) The method according to any one of embodiments 13 to 15, wherein the treatment site corresponds to a biopsy site.

[0100] (21) A method according to any of embodiments 13 to 15, wherein the fluoroscopic images of the patient's anatomical structure include a first fluoroscopic image focused on the anatomical structure at a first angle and a second fluoroscopic image focused on the anatomical structure at a second angle different from the first angle. (22) The method of embodiment 21, wherein reconstructing the three-dimensional model of the instrument and the treatment site includes triangulating a segment identified in the first fluoroscopic image and a segment identified in the second fluoroscopic image. (23) The method of embodiment 21, further comprising acquiring at least one of the first angle or the second angle via a communication interface of an imaging device that generated the first fluoroscopic image and the second fluoroscopic image. (24) The method of claim 21, further comprising obtaining at least one of the first angle or the second angle via an operator user interface. (25) The method of embodiment 21, further comprising obtaining at least one of the first angle or the second angle via an external tracking sensor.

[0101] (26) A system for reconstructing a three-dimensional model of an instrument and a treatment site within an anatomical structure, comprising: A control circuit; and a computer-readable medium having instructions that, when executed, cause the control circuitry to: obtaining a fluoroscopic image of the patient's anatomy; Obtain one or more neural networks, identifying a segmentation within the fluoroscopic image corresponding to the instrument based on the one or more neural networks; identifying a segmentation within the fluoroscopic image corresponding to the treatment site based on the one or more neural networks; reconstructing the three-dimensional model of the instrument and the treatment site based on the segmentation in the fluoroscopic image; The system causes the reconstructed three-dimensional model to be rendered on a display device.

Claims

1. A method for reconstructing a three-dimensional (3D) model from two-dimensional (2D) images, comprising: obtaining a fluoroscopic image of an anatomical structure; identifying a first segmentation in the fluoroscopic image associated with an instrument based on one or more neural networks; identifying a second segmentation in the fluoroscopic image associated with a treatment site based on the one or more neural networks; reconstructing the 3D model of the instrument and the treatment site based at least in part on the identified first segmentation and the identified second segmentation; Rendering the reconstructed 3D model on a display device; and A method comprising:

2. The method of claim 1, further comprising determining a region of interest associated with the fluoroscopic image based on the identified first segmentation, wherein the second segmentation is further identified based on the region of interest.

3. The method of claim 1, wherein the first segmentation is identified in parallel with the second segmentation.

4. The method of claim 1, wherein the 3D model is further reconstructed based on calibration data associated with the imaging device used to capture the fluoroscopic image.

5. The method of claim 1, further comprising rendering the first segmentation and the second segmentation on the display device.

6. The method described in claim 5, wherein the first segmentation and the second segmentation are rendered on the display device before rendering the reconstructed 3D model.

7. The method of claim 1, wherein the first segmentation includes a first sub-segmentation and a second sub-segmentation associated with different components of the instrument.

8. The method described in claim 1, wherein the fluoroscopic images include a first fluoroscopic image focused on the anatomical structure at a first angle and a second fluoroscopic image focused on the anatomical structure at a second angle different from the first angle.

9. The method described in claim 8, wherein reconstructing the 3D model includes triangulating the first segmentation and the second segmentation identified in the first fluoroscopy image and the first segmentation and the second segmentation identified in the second fluoroscopy image.

10. A system for reconstructing a three-dimensional (3D) model from a two-dimensional (2D) image, comprising: a control circuit; a computer-readable medium having instructions that, when executed, cause the control circuitry to: obtaining a fluoroscopic image of the anatomical structure; identifying a first segmentation in the fluoroscopic image associated with an instrument based on one or more neural networks; identifying a second segmentation within the fluoroscopic image associated with a treatment site based on the one or more neural networks; reconstructing the 3D model of the instrument and the treatment site based at least in part on the identified first and second segmentations; The system causes the reconstructed 3D model to be rendered on a display device.

Images