Using fluoroscopy projection images to calculate inertia moments of an object

Abstract

A system comprising: an imaging device configured to rotate axially about an axis of rotation and capture a plurality of images associated with a sinogram at a plurality of angles along the axis of rotation, respectively; one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the system to: detect an object in each of the plurality of images; determine a moment of inertia associated with the object based on the plurality of images; simulate one or more images of the sinogram based at least in part on the moment of inertia; generate a reconstruction of the object based on the sinogram; and determine one or more characteristics of the object based on the reconstruction.

Description

[0001] Cross-references to related applications This application claims priority and benefit under 35 USC § 119(e) to U.S. Provisional Patent Application No. 63 / 609,452, filed December 13, 2023, and U.S. Non-Provisional Patent Application No. 18 / 966,094, filed December 2, 2024, the entire contents of which are incorporated herein by reference. Background Technology

[0002] This disclosure relates to an object recognition and object semantic segmentation system. The system can generate a complete sine map from multiple fluorescence perspective images containing an object projected at a selected angle, and reconstruct a cross-section of the object from the complete sine map. The generation of the complete sine map may involve determining the moment of inertia and principal axes from the multiple fluorescence perspective images, and using them to simulate one or more fluorescence perspective images projected at other, unselected angles. When reconstructing the cross-section of the object, various geometric properties can be derived from the reconstructed cross-section. These geometric properties can be used for automatic segmentation of objects in fluorescence perspective images, and particularly for providing initial pose estimations of objects in fluorescence perspective images. Summary of the Invention

[0003] The present invention is presented in a simplified form to introduce some concepts, which will be further described in the detailed embodiments below. This summary is not intended to identify key or essential features of the subject matter protected by the claims, nor is it intended to limit the scope of the subject matter protected by the claims.

[0004] One innovative aspect of the subject matter of this disclosure can be implemented in a system including an imaging apparatus, one or more processors, and a memory. The imaging apparatus is configured to rotate axially about a rotation axis and capture multiple images associated with a sine wave at multiple angles along the rotation axis. The memory stores instructions that, when executed by the one or more processors, cause the system to: detect an object in each of the multiple images; determine the moment of inertia associated with the object based on the multiple images; simulate one or more images of the sine wave based at least in part on the moment of inertia associated with the object; generate a reconstruction of the object based on the sine wave; and determine one or more properties of the object based on the reconstruction.

[0005] Another innovative aspect of the subject matter of this disclosure can be implemented in a computer-based method. The method includes the steps of: receiving a plurality of images associated with a sine wave, wherein the plurality of images are captured by an imaging device at a plurality of angles along a rotation axis of the imaging device; detecting an object in each of the plurality of images; determining a moment of inertia associated with the object based on the plurality of images; simulating one or more images of the sine wave based at least in part on the moment of inertia associated with the object; generating a reconstruction of the object based on the sine wave; and determining one or more properties of the object based on the reconstruction.

[0006] Another innovative aspect of the subject matter of this disclosure can be implemented in a controller for a medical system, the controller including a memory and one or more processors. The memory stores instructions that, when executed by the one or more processors, cause the controller to: receive a plurality of images associated with a sine wave, wherein the plurality of images are captured by an imaging device at a plurality of angles along a rotation axis of the imaging device; detect an object in each of the plurality of images; determine, based on the plurality of images, a moment of inertia associated with the object; simulate one or more images of the sine wave based at least in part on the moment of inertia associated with the object; generate a reconstruction of the object based on the sine wave; and determine one or more properties of the object based on the reconstruction. Attached Figure Description

[0007] Various embodiments are depicted in the accompanying drawings for illustrative purposes and should in no way be construed as limiting the scope of the invention. Furthermore, various features of different disclosed embodiments may be combined to form additional embodiments that are part of this disclosure. Throughout the drawings, reference numerals may be repeated to indicate correspondences between reference elements.

[0008] Figure 1 Examples are illustrated of embodiments of a robotic medical system deployed for diagnostic and / or therapeutic ureteroscopy, according to one or more embodiments.

[0009] Figure 2 An example is illustrated of a robotic system deployed for diagnostic and / or therapeutic bronchoscopy according to one or more embodiments.

[0010] Figure 3 An example of a platform-based robotic system according to one or more implementation schemes is shown.

[0011] Figure 4 Examples are given of implementation schemes that can be implemented according to one or more embodiments. Figures 1 to 3 A medical system component implemented in any medical system within a medical system.

[0012] Figure 5An axially rotatable fluorescent perspective system for an image configured as a projection object, according to one or more embodiments, is illustrated.

[0013] Figure 6-1 , Figure 6-2 , Figure 6-3 and Figure 6-4 An example is illustrated of axial rotation of a fluorescence fluoroscopic imaging apparatus according to one or more embodiments and the corresponding captured image.

[0014] Figure 7-1 , Figure 7-2 and Figure 7-3 A conventional method for generating a sine graph by changing the projection angle is illustrated according to one or more implementation schemes.

[0015] Figure 8 An example is illustrated of a process for simulating a complete sine curve using selected fluorescence perspective images, according to one or more embodiments.

[0016] Figure 9-1 and Figure 9-2 An exemplary simulated sine curve and a reconstructed cross section of an object from the sine curve are illustrated according to one or more embodiments.

[0017] Figure 10 The presentation of derived geometric properties of fluorescent perspective images according to one or more embodiments is illustrated.

[0018] Figure 11 A flowchart illustrating the process of reconstructing an object from a fluorescence perspective image according to one or more implementation schemes is provided.

[0019] Figure 12 A block diagram of an exemplary controller for a medical system, according to some specific implementations, is shown.

[0020] Figure 13 An exemplary flowchart is shown, according to some specific implementations, depicting exemplary operations for determining one or more properties of an object. Detailed Implementation

[0021] The headings provided herein are for convenience only and do not necessarily affect the scope or meaning of the claimed invention. Although specific preferred embodiments and examples are disclosed below, the subject matter of the invention extends beyond the specifically disclosed embodiments to other alternative embodiments and / or uses, as well as modifications and equivalents thereof. Therefore, the scope of the claims that may appear herein is not limited to any particular embodiment of the specific embodiments described below. For example, in any method or process disclosed herein, the actions or operations of the method or process may be performed in any suitable sequence and are not necessarily limited to any particular disclosed sequence. Various operations may then be described as multiple discrete operations in a manner that may aid in understanding certain embodiments; however, the order of description should not be construed as implying that these operations depend on the order. Furthermore, the structures, systems, and / or apparatuses described herein may be embodied as integrated components or separate components. Certain aspects and advantages of these embodiments are described for the purpose of comparing the various embodiments. Not all such aspects or advantages are necessarily achieved by any particular embodiment. Therefore, for example, various embodiments may be performed by implementing or optimizing one or a set of advantages taught herein without necessarily achieving other aspects or advantages that may also be taught or proposed herein.

[0022] Although specific spatial relative terms such as “external,” “internal,” “upper,” “lower,” “below,” “above,” “vertical,” “horizontal,” “top,” “bottom,” “outer,” and similar terms are used herein to describe the spatial relationship of one device / element or anatomical structure to another device / element or anatomical structure, it should be understood that these terms are used herein for descriptive convenience to describe the positional relationship between elements / structures, such as with respect to the orientation shown in the accompanying drawings. It should be understood that spatial relative terms are intended to cover different orientations of elements / structures in use or operation other than those depicted in the accompanying drawings. For example, an element / structure described as “above” another element / structure may indicate a position below or beside such other element / structures relative to the subject patient or an alternative orientation of the element / structure. It should be understood that spatial relative terms, including those listed above, can be understood relative to the corresponding illustrated orientation in the reference drawings.

[0023] To facilitate the use of devices, components, systems, features, and / or modules having similar features in one or more aspects, specific reference numerals are repeated in different figures within the group of figures disclosed herein. However, the repeated use of common reference numerals in the figures with respect to any embodiment of the embodiments disclosed herein does not necessarily indicate that such features, devices, components, or modules are identical or similar. Rather, those skilled in the art will be informed by the context that the use of common reference numerals may imply the degree of similarity between the referenced subjects. The use of specific reference numerals in the context of the description of a particular figure may be understood to refer to the devices, components, aspects, features, modules, or systems identified in that particular figure, and not necessarily to any devices, components, aspects, features, modules, or systems identified by the same reference numerals in another figure. Furthermore, aspects of individual figures identified by common reference numerals may be interpreted as sharing characteristics or being completely independent of each other. In some contexts, features associated with individual figures identified by common reference numerals are unrelated and / or similar in at least some aspects.

[0024] This disclosure provides systems, apparatus, and methods for mathematically reconstructing objects in projected images, which can be used to derive various geometric properties of objects within the image and can be used as initialization for object semantic segmentation. In particular, the systems, apparatus, and methods according to one or more aspects of this disclosure can reduce the number of projected images required to generate a complete sine wave. Additionally, one or more aspects can be used to identify objects in projected images and estimate their initial poses. It is conceivable that objects may include, but are not limited to, endoscopic eyes and nodules.

[0025] With respect to the medical devices described in this disclosure, the term "device" is used in its broad and general sense and can refer to any type of tool, apparatus, component, system, subsystem, device, part, etc. In some contexts herein, the term "device" is used substantially interchangeably with the term "device".

[0026] Robotic surgical systems can be used to facilitate instrument navigation into areas within a patient's body. In some implementations, the robotic system can be configured to provide an interface that allows an operator to navigate robotically controlled instruments by guiding their movement in multiple degrees of freedom. For example, an operator can guide the axial translation (i.e., insertion and / or retraction), joint angles, and / or roll angles (i.e., joint angle directions) of endoscopes, access sheaths, guidewires, working instruments (e.g., needles, baskets, lithotripters, etc.). Navigation within organs, branch vascular orifices, or other relatively open three-dimensional spaces can be challenging because it requires understanding the three-dimensional relationship of the navigation / tracking instrument relative to the anatomical target and / or determining in which plane the instrument will bend. This task can be difficult, partly because navigating an instrument from a distant patient entry point through the patient's lumen to the desired site of the procedure requires manipulating the instrument without a direct line of sight. Positioning / tracking systems can be used to aid in locating the desired site of the procedure and to visualize the navigation of the instrument to that site. Positioning / tracking systems allow users to visualize the patient's internal anatomy and the location and / or orientation of detectable markers of the instrument within the patient's anatomy.

[0027] Positioning systems can include imaging systems / modalities such as positron emission tomography (PET), X-ray computed tomography (CT), X-ray fluoroscopy, magnetic resonance imaging (MRI), camera-based optical systems, and ultrasound or other acoustic imaging systems. Positioning systems can further include electromagnetic (EM) tracking systems (e.g., using an electromagnetic field generator as described in detail herein), fiber optic tracking systems, and robot tracking / positioning based on robot data (e.g., robot actuators, torque, attitude data). Some imaging systems / modalities are not suitable for continuous real-time tracking of instruments, such as PET, CT, and MRI, which typically generate and combine many cross-sectional images of an object to produce a computer-processed image; this image capture process can be relatively slow, and motion within the image field during image capture can produce image artifacts, making such systems unsuitable for real-time tracking of instruments moving within the body. Furthermore, some imaging systems / modalities (such as X-ray, CT, and fluoroscopy) emit potentially harmful ionizing radiation, thus potentially requiring limitations on their usage time.

[0028] Electromagnetic (EM) tracking systems and fiber optic tracking systems can provide real-time instrument tracking. EM tracking typically functions by detecting / determining the position / orientation of an EM sensing coil (i.e., an EM marker / sensor) in a fluctuating magnetic field. The fluctuating magnetic field induces a current in the coil based on its position and orientation within the magnetic field. Therefore, the position and orientation of the coil can be determined by measuring the current in the coil. In some cases, a single EM sensor / marker can sense its position and orientation in three-dimensional space with five degrees of freedom. That is, an EM sensor can provide data indicating orientation in every direction except about the axis of symmetry of the coil (i.e., the roll angle). Two EM sensors / markers held in a fixed relative position and orientation on the instrument or other marking device can be used to sense all six degrees of freedom of the instrument. In navigation systems employing EM tracking, images of the anatomical space can be acquired, where system control circuitry is configured to determine registration between a reference frame of the EM sensor / marker associated with the tracked instrument and a reference frame of the imaging system / modality used for imaging the anatomical space to depict the motion of the tracked instrument within the imaged anatomical space.

[0029] Although certain aspects of this disclosure are described in detail in the context of bronchoscopy and ureteroscopy procedures, it should be understood that such context is provided for convenience and clarity, and that the device positioning concepts disclosed herein are applicable to any suitable medical procedure.

[0030] Regarding ureteroscopy procedures, a surgeon inserts an endoscope (e.g., a ureteroscope) through the urethra to remove urinary stones from the bladder and ureters, such as for the purpose of removing kidney stones. In some procedures, the physician may use percutaneous nephrolithotomy (“PCNL”) techniques, which involve inserting a nephroscope through the skin (i.e., percutaneously) and intervening in the tissue to provide access to the treatment site for breaking up and / or removing the stones. Relatively large kidney stones can be broken up into relatively smaller fragments to facilitate their removal using certain instruments, such as laser lithotripters. Depending on the procedure, a basket device / system may be used to capture relatively small stone fragments and extract them from the patient's treatment site. Any instruments associated with such ureteroscopic procedures can be robotically controlled and / or positionally tracked using markers / sensors associated with the instrument, as described in detail herein.

[0031] healthcare system Figure 1Example medical systems 100 for performing various medical procedures are illustrated according to various aspects of this disclosure. Medical system 100 can be used, for example, in endoscopic procedures. Compared to fully manual procedures, robotic medical solutions offer relatively higher precision, superior control, and / or superior hand-eye coordination relative to a particular instrument. Although Figure 1 System 100 is presented in the context of ureteroscopy procedures, but it should be understood that the principles disclosed herein can be implemented in any type of endoscopic procedure.

[0032] Medical system 100 includes a robotic system 10 (e.g., a mobile robotic cart) configured to engage and / or control a medical device (e.g., a ureteroscope) to perform procedures on patient 7. The medical device includes a proximal handle 31 and an axis 40 coupled to the handle 31 in its proximal portion. It should be understood that the device 40 can be any type of axis-based medical device, including endoscopes (such as ureteroscopes or bronchoscopes), catheters (such as manipulable or non-manipulable catheters), needles, nephroscopes, laparoscopes, or other types of medical devices. The device 40 can access the patient's internal anatomy through direct entry (e.g., through natural orifices) and / or percutaneous entry via skin / tissue puncture.

[0033] The medical system 100 includes a control system 50 configured to interact with the robotic system 10, providing information about procedures and / or performing various other operations. For example, the control system 50 may include one or more displays 56 configured to present information to assist physician 5 and / or other technicians or individuals. The medical system 100 may include a table 15 configured to hold patient 7. The system 100 may further include an electromagnetic (EM) field generator, such as an EM field generator 80 or EM field generator 85 mounted on the table 15 or other structure for robotic installation.

[0034] Although various robotic arms 12 are shown in various positions and coupled to various tools / devices, it should be understood that such configurations are shown for convenience and illustrative purposes, and the medical system 100 may include any number of robotic arms 12, which may be configured differently over time and / or at different points during medical procedures. Furthermore, the robotic arms 12 may be coupled to... Figure 1The devices / instruments shown are different devices / instruments, and in some cases or for some time periods, one or more of these arms may not be used or coupled to the medical device. The instrument can be coupled to the robotic system 10 via robotic end effectors 6a-6c associated with the distal end of the respective arm 12. The term "end effector" is used herein in its broad and general sense and can refer to any type of robotic manipulator device, component, and / or assembly. The terms "robotic manipulator" and "robotic manipulator assembly" are used in their broad and general sense and can refer to a robotic end effector and / or a sterile adapter or other adapter components (commonly or separately) coupled to the end effector. For example, "robotic manipulator" or "robotic manipulator assembly" can refer to an instrument device manipulator (IDM) including one or more drive outputs, whether embodied in a robotic end effector, adapter, and / or other component.

[0035] In some implementations, physician 5 may interact with control system 50 and / or robotic system 10 to cause / control robotic system 10 to advance and navigate medical device axis 40 (e.g., endoscope) through patient anatomy to target sites and / or perform certain procedures using associated instruments. Control system 50 may provide information associated with medical device 40 and / or other instruments of system 100, such as real-time endoscopic images captured by the medical device, via display 56 to assist physician 5 in navigating / controlling such instrumentation. Control system 50 may provide physician 5 with imaging / positioning information based on certain positioning modalities, such as fluoroscopy, ultrasound, optical / camera imaging, EM field localization, or other modalities, as described in detail herein.

[0036] The various endoscopes / axial instruments disclosed herein (such as axis 40 of system 100) can be configured to navigate within human anatomical structures, such as within natural openings or cavities of human anatomical structures. The terms “endoscope” and “scope” are used herein according to their broad and general meaning and can refer to any type of elongated (e.g., axial) medical device having image generation, viewing, and / or capture capabilities and configured to be introduced into any type of organ, cavity, lumen, chamber, or space of the body. Endoscopes may include, for example, ureteroscopes (e.g., for access to the urinary tract), laparoscopes, nephroscopes (e.g., for access to the kidneys), bronchoscopes (e.g., for access to airways, such as bronchi), colonoscopes (e.g., for access to the colon), arthroscopes (e.g., for access to joints), cystoscopes (e.g., for access to the bladder), colonoscopes (e.g., for access to the colon and / or rectum), tubular endoscopes, etc. In some cases, endoscopes may include at least partially rigid and / or flexible tubes, and may be sized to pass through an outer sheath, catheter, guide, or other lumen-type device, or may be used without such a device. The endoscopes and other instruments described herein may have certain markers / sensors associated with their distal ends or other portions, which are configured to be visible / detectable in a field / space associated with one or more positioning (e.g., imaging) systems / modalities.

[0037] System 100 is shown to include a fluoroscopy system 70, which includes an X-ray generator 75 and an image detector 74 (referred to in some contexts as an "image intensifier"; either component 74 or 75 may be referred to herein as a "source" or "emitter"), both of which may be mounted on a movable C-arm 71. In some embodiments, the fluoroscopy system 70 and any part thereof may be referred to as an imaging device. A control system 50 or other system / device may be used to store and / or manipulate images generated using the fluoroscopy system 70. In some embodiments, the bed 15 is X-ray permeable, such that X-rays from the generator 75 can pass through the bed 15 and a target area of ​​the patient's anatomy, with the patient 7 positioned between the ends of the C-arm 71. The structure / arm 71 of the fluoroscopy system 70 may be rotatable or fixed. The fluoroscopy system 70 may be implemented to allow viewing of real-time images to facilitate image-guided surgery. The structure / arm 71 may be selectively movable to allow the fluoroscopy panel source 74 to capture various images of the patient 7 and / or the operating room.

[0038] exist Figure 1In the exemplary urological configuration shown, robotic arm 12c is shown as holding field generator 80. Since the electric field generated by field generator 80 can be distorted due to the presence of metal or other conductive components, it is desirable to position arm 12c in a manner that ensures other components of the system do not substantially interfere with the electric field. For example, it may be desirable to position field generator 80 at least 8" or more away from the support arm 71 associated with the fluoroscopy system. In some embodiments, system 100 includes an EM field generator 85 mounted to stage 15 or other structure (e.g., a stand-alone structure).

[0039] Figure 2 An exemplified example is a cart-based robotic system 101 arranged for diagnostic and / or therapeutic bronchoscopy, according to one or more embodiments. During bronchoscopy, the arm 12 of the robotic system 10 may be configured to drive a medical instrument axis 40, such as a manipulable endoscope (which may be a procedure-specific bronchoscope for bronchoscopy), through a natural orifice access point (e.g., the mouth of the patient 7 positioned on table 15 in this example) to deliver diagnostic and / or therapeutic instruments. As shown, the robotic system 10 (e.g., a cart) may be positioned close to the patient's upper torso to provide access to the access point. This can also be utilized when performing gastrointestinal (GI) procedures with an endoscope. Figure 2 The layout within.

[0040] as Figure 1 Similar to system 100, the instrument / endoscopy 40 can access the target anatomical structure through a access sheath. For surgical bronchoscopy, after insertion, the endoscope 40 can be guided downwards through the patient's trachea and lungs using precise commands from the robotic system 10 until the target surgical site is reached. For example, the endoscope 40 can be guided to deliver a biopsy needle to a target, such as, for example, a lesion or nodule within the patient's lungs.

[0041] Figure 3 A table-based robotic system 102 according to one or more embodiments of the present disclosure is illustrated. System 102 includes robotic components integrated with a table / platform 115, thereby allowing for a reduction in the amount of capital equipment in the operating room compared to some cart-based robotic systems. In some cases, a table-integrated robotic system like system 102 can allow for better patient access. Much like cart-based systems 101, 102, the instrument manipulator assembly associated with the robotic arm 112 of system 102 typically includes instruments and / or instrument feeders designed to manipulate elongated medical instruments / axis, such as catheters 40, etc.

[0042] As shown, the robot-enabled stage system 104 may include a column 144 coupled to one or more brackets 141 (e.g., a ring-shaped movable structure), from which one or more robotic arms 112 may extend. The brackets 141 may translate along a vertical column interface extending at least a portion of the length of the column 144 to provide different vantage points from which the robotic arms 112 may be positioned. In some embodiments, the brackets 141 may rotate about the column 144 to allow the robotic arms 112 to access multiple sides of the stage 104. Rotation and / or translation of the brackets 141 may allow the system 102 to align medical devices such as endoscopes and catheters to different access points on the patient.

[0043] refer to Figures 1 to 3 and Figure 4 , Figure 4 It shows Figures 1 to 3 In exemplary embodiments of any of the subsystems, control system 50 can be configured to provide various functions to assist in the execution of medical procedures. Control system 50 can communicate with robotic system 10 via a wireless or wired connection (e.g., to control robotic system 10). In some embodiments, control system 50 can communicate with robotic system 10 to receive location data from the robotic system relating to the position of the distal end of endoscope 40 or other instruments. This location data can be derived using one or more markers associated with the respective instrument (e.g., electromagnetic sensors, radiopaque markers, etc.) and / or at least partially based on robotic system data (e.g., arm position / posture data, known parameters or dimensions of various system components, etc.). In some embodiments, control system 50 can communicate with EM field generators 80 / 85 to control the generation of EM fields in the area surrounding patient 7 and / or the area surrounding the tracked instrument.

[0044] Figure 4 Further shown Figures 1 to 3 An exemplary embodiment of the robotic system 10 is described above. The robotic system 10 may include one or more robotic arms 12, each of which may include multiple arm segments 23 coupled to joints 24, providing multiple degrees of motion / freedom. When the robotic system 10 is correctly positioned, the endoscope 40 may be inserted into the patient 7 robotically using the robotic arms 12, manually by a physician 5, or in combination thereof. One of the arms 112 may have an associated instrument coupling / manipulator 31 configured to facilitate the advance and manipulation of the endoscope 40.

[0045] The robotic system 10 can be physically and / or communicatively coupled to any component of the medical system, such as the control system 50, the stage 15, the EM field generator 80 / 85, the endoscope 40, the fluoroscopy system 70, and / or any type of percutaneous access instrument (e.g., needles, catheters, nephroscopes, etc.). For example, the robotic system 10 may include a communication interface 214 for communicating with the communication interface 254 of the control system 50. The robotic system 10 can be configured to receive control signals from the control system 50 to perform certain operations, such as positioning one or more robotic arms 12, manipulating the endoscope 40, etc. In response, the robotic system 10 can use certain control circuitry 211 to control actuators 217 and / or other components of the robotic system 10 to perform operations. For example, the control circuitry 211 can control various motors / actuators associated with various joints of the robotic arms 12 and / or arm supports 17. In some embodiments, the robotic system 10 and / or control system 50 are configured to receive images and / or image data from the endoscope 40, which represent portions of the patient's internal anatomy and / or access sheath or other device components.

[0046] The robot system 10 typically includes an elongated support structure 14 (also referred to as a "pillar"), a robot system base 25, and a console 13 at the top of the pillar 14. The pillar 14 may include supports for one or more robotic arms 12 (in... Figure 1 One or more arm supports 17 (also referred to as “carriers”) are deployed (three are shown in the diagram). Arm supports 17 can be configured to translate vertically along a column 14. In some embodiments, arm supports 17 are connected to the column 14 via slots 20 positioned on opposite sides of the column 14 to guide the vertical translation of the arm supports 17. Slots 20 include vertical translation interfaces to position and hold the arm supports 17 relative to the robot system base 25 at various vertical heights. The base 25 balances the weight of the column 14, arm supports 17, and arm 12 on the floor.

[0047] The robotic arm 12 typically includes a robotic arm base 21 and end effectors 6a-6c separated by a series of linked arm segments 23 connected by a series of joints 24, each joint including one or more independent actuators 217. Each actuator may include an independently controllable motor. Each independently controllable joint 24 provides or represents an independent degree of freedom available to the robotic arm. In some embodiments, each arm in the arm 12 has seven joints and thus provides seven degrees of freedom, including “redundant” degrees of freedom. Redundant degrees of freedom allow the robotic arm 12 to position its corresponding end effectors 6a-6c in specific locations, orientations, and trajectories in space using different linkage positions and joint angles. This allows the system to locate and guide medical devices from desired points in space, while allowing physicians to move arm joints to clinically advantageous positions away from the patient to generate greater access while avoiding arm collisions. A console 13 positioned at the upper end of the column 14 provides both a user interface for receiving user input and a display screen 56 (or a dual-purpose device, such as a touchscreen) for providing preoperative and intraoperative data to the physician / user. The robot cart 10 may further include a handle 27 and one or more wheels 28.

[0048] Each end effector 6a-6c of the robotic arm 12 may include or be configured to couple an instrument device manipulator (IDM; e.g., endoscope handle 31) thereto, which may, in some cases, be attached using a sterile adapter component. The combination of the end effectors 6a-6c and the associated IDM, along with any intervention mechanism or coupling element (e.g., a sterile adapter), may be referred to as a manipulator assembly. The IDM may provide a power and control interface. For example, the interface may include connectors for transmitting pneumatic pressure, power, electrical signals, and / or optical signals from the robotic arm 12 to the IDM. The IDM may be configured to manipulate medical instruments (e.g., surgical instruments / tools) such as the endoscope 40 using techniques including, for example, direct drive, harmonic drive, gear drive, belt and pulley, magnetic drive, etc. The robotic system 10 may also include one or more power interfaces 219.

[0049] As mentioned above, system 100 may include certain control circuitry configured to perform certain functions described herein, including control circuitry 211 of robot system 10 and control circuitry 251 of control system 50. That is, the control circuitry of systems 100, 101, and 102 may be part of robot system 10, control system 50, or some combination thereof. Therefore, any reference herein to control circuitry may refer to systems embodied in robot systems, control systems, or medical systems (such as those described in…). Figures 1 to 3The term "control circuitry" is used herein in its broad and general sense and may refer to any combination of the following: processor, processing circuitry, processing module / unit, chip, die (e.g., a semiconductor die including one or more active and / or passive devices and / or connectivity circuitry), microprocessor, microcontroller, digital signal processor, microcomputer, central processing unit, field-programmable gate array, programmable logic device, state machine (e.g., hardware state machine), logic circuitry, analog circuitry, digital circuitry, and / or any means of manipulating signals based on hard-coded circuitry and / or operating instructions. The control circuitry mentioned herein may further include one or more circuit substrates (e.g., printed circuit boards), conductive traces and vias and / or mounting pads, connectors, and / or components. The control circuitry mentioned herein may further include one or more memory devices, which may be embodied in the embedded circuitry of a single memory device, multiple memory devices, and / or devices. Such data storage devices may include read-only memory, random access memory, volatile memory, non-volatile memory, static memory, dynamic memory, flash memory, cache, data storage registers, and / or any device for storing digital information. It should be noted that in embodiments where the control circuitry includes hardware and / or software state machines, analog circuits, digital circuits, and / or logic circuits, the data storage device / register storing any associated operational instructions may be embedded within or outside the circuitry including the state machine, the analog circuitry, the digital circuitry, and / or the logic circuitry.

[0050] Control circuits 211, 251 may include a computer-readable medium that stores and / or is configured to store hard-coded and / or operational instructions corresponding to at least some of the steps and / or functions shown in one or more of the figures shown in this document and / or described herein. In some cases, such a computer-readable medium may be included in an article of manufacture. Control circuits 211 / 251 may be maintained / set up entirely locally or may be located at least partially remotely (e.g., communicatively coupled indirectly via a local area network and / or a wide area network). Either control circuit 211, 251 may be configured to perform any aspect of the various processes disclosed herein.

[0051] Further reference Figure 4The robotic system 10 may provide one or more input / output (I / O) components 218, such as a user interface and / or display (or dual-purpose device, such as a touchscreen) for receiving user input, to provide preoperative and / or intraoperative data to the physician / user. The control system 50 may also include various I / O components 258 configured to assist the physician 5 or others in performing medical procedures. For example, I / O components 258 may be configured to allow user input to control / navigate the endoscope 40 and / or basket system within the patient 7. In some embodiments, for example, the physician 5 may provide input to the control system 50 and / or the robotic system 10, wherein, in response to such input, control signals may be sent to the robotic system 10 to manipulate the endoscope 40 and / or other robot-controlled instruments.

[0052] The control system 50 and / or robot system 10 may include certain user controls (e.g., control 55), which may include any type of user input (and / or output) device or device interface, such as one or more buttons, keys, joysticks, handheld controllers (e.g., video game type controllers), computer mice, touchpads, trackballs, control pads, and / or sensors for capturing gestures and finger postures (e.g., motion sensors or cameras), touchscreens, and / or interfaces / connectors for them. Such user controls are communicatively and / or physically coupled to corresponding control circuitry. The control system may include a structural tower 51 and one or more wheels 58 supporting the tower 51. The control system 50 may further include certain communication interfaces 254 and / or power interfaces 259.

[0053] In some embodiments, the endoscope assembly 30 includes a handle or base 31 coupled to the endoscope shaft 40 (referred to herein as an "endoscope" or "scope" in certain contexts). For example, the endoscope 40 may include an elongated shaft that includes one or more lamps 49 and one or more cameras or other imaging devices 48. The scope 40 may further include one or more working channels 44 that extend the length of the scope 40.

[0054] The endoscope assembly 30 may further include one or more positioning markers and / or sensors 63, which may be configured to generate signals indicating the position of the markers / sensors 63 within an electromagnetic field. Such markers 63 may include, for example, one or more conductive coils (or other embodiments of antennas), which may be positioned relative to each other in a known fixed orientation to allow determination of multiple degrees of freedom regarding position determination. The markers 63 may be configured to generate sensor position data and / or transmit sensor position data to another device and / or generate detectable distortions or features in the electromagnetic field. The sensor / marker position data may indicate the position and / or orientation of the medical device 40 (e.g., its distal end 42), and / or may be used to determine / infer the position / or orientation of the medical device.

[0055] The endoscope 40 may be capable of articulation, such as relative to at least its distal portion 42, allowing it to be steered within the human anatomy. In some embodiments, the endoscope 40 is configured to articulate with, for example, six degrees of freedom, including XYZ coordinate movement, as well as pitch, yaw, and roll. A position sensor of the endoscope 40 (e.g., an electromagnetic sensor, in some implementation) may similarly have similar degrees of freedom relative to position information generated / provided by that position sensor.

[0056] In a robotic implementation, the robotic arm of the robotic system may be configured to / can be configured to manipulate the endoscope 40. For example, an instrument manipulator (e.g., the endoscope handle) may be coupled to an end effector of the robotic arm and the endoscope 40 may be manipulated using an elongated moving member. This elongated moving member may include one or more drawwires (e.g., pull wires or push wires), cables, fibers, and / or flexible shafts. For example, the robotic end effector may be configured to actuate multiple drawwires (not shown) coupled to the endoscope 40 to deflect the end 42 of the endoscope 40.

[0057] In various embodiments, the anatomical space in which the endoscope 40 or other instrument can be located (i.e., the location of the endoscope / instrument is determined / estimated) is a three-dimensional portion of the patient's vascular system, tracheobronchial airway, urethra, gastrointestinal tract, or any organ or space accessed via such lumens. Various localization / imaging modalities can be implemented to provide images / representations of the anatomical space. Suitable imaging subsystems include, for example, X-ray, fluoroscopy, CT, PET, PET-CT, CT angiography, cone-beam CT, 3DRA, single-photon emission computed tomography (SPECT), MRI, optical coherence tomography (OCT), and ultrasound. One or both of pre-procedure and in-procedure images can be acquired. In some embodiments, a C-arm fluoroscopy is used to acquire pre-procedure and / or in-procedure images. Specific localization and imaging systems / modalities are described in conjunction with some embodiments; it should be understood that such descriptions may refer to any type of localization system / modality.

[0058] Capturing multiple images Figure 5 An axially rotatable fluorescence fluoroscopy system 70, configured to project an image of an object according to one or more embodiments, is illustrated. In this disclosure, "projecting" a fluorescence fluoroscopy image can be used synonymously for "capturing" a fluorescence fluoroscopy image when a fluorescence fluoroscopy imaging device projects a fluorescence fluoroscopy field onto a region of interest to capture an object within that region of interest. The fluorescence fluoroscopy system 70 may include a mounting arm (such as a C-arm 71) or other structures that hold / support the fluorescence fluoroscopy imaging device. The fluorescence fluoroscopy imaging device may include a detector / receiver 74 or a source / emitter 75. While a C-arm 71 is illustrated for the fluorescence fluoroscopy system 70, it should be understood that other support structures and imaging devices are contemplated.

[0059] In the fluorescence fluoroscopy system 70, a source / emitter 75 and a detector / receiver 74 can be positioned at opposite ends of a C-arm 71 to maintain a fixed position and orientation. The source / emitter 75 can generate a fluorescence fluoroscopic field (e.g., X-rays) 601, and the detector / receiver 74 can receive this field to record the intensity of the fluorescence fluoroscopic field passing through an object 78 positioned on a bed 15. More densely packed areas of the object 78 attenuate the fluorescence fluoroscopic field more and produce lower intensity on the detector / receiver 74, and vice versa. As shown, the C-arm 71 can rotate axially about a rotation center along one or more axes. For example, the illustrated C-arm 71 can rotate along the "X" axis (e.g., to provide a horizontally rotated image), along the "Y" axis (e.g., to provide a vertically rotated image), or along both. The object 78 (which can be any semantically separable object) can be positioned at or near the rotation center of the C-arm 71. Some exemplary semantically separable objects include, but are not limited to, endoscopes, medical tools (e.g., biopsy needles, baskets, forceps, etc.) and anatomical features.

[0060] Rotating the C-arm 71 axially to capture multiple projected images at various angles can provide a more complete two-dimensional or three-dimensional understanding of the various geometric properties of the object 78 than a single image can provide. However, capturing multiple projected images can be time-consuming and resource-intensive. Therefore, an improved technique is desired that can efficiently and reliably derive various geometric properties using a minimum number of projected images.

[0061] Figure 6-1 , Figure 6-2 , Figure 6-3 and Figure 6-4 An example is illustrated of axial rotation of a fluorescence fluoroscopic imaging apparatus according to one or more embodiments and the corresponding captured image. Figure 6-1An example of a fluorescence imaging system 70 is attached to a C-arm 71, including an emitter / source 75 or a detector / receiver 74. As shown, the device arm 71 can rotate along the "Y" axis to provide a neutral configuration 70b (0°) and offset at a negative angle (-). θ The negative configuration 70c indicated by ) and the offset by a positive angle (+ θ The positive configuration 70a is indicated by ) . It should be understood that the axial rotation of the imaging device can be along different axes (such as Figure 5 (The "X" axis shown) or multiple axes are used. Positive configuration 70a and negative configuration 70c can be defined relative to neutral configuration 70b along any arbitrary axis of rotation. Although the axis of rotation can be arbitrary, a positive angular offset + θ and negative angle offset - θ They can be selected to have the same or substantially the same size (e.g., |+ θ = |- θ This allows them to be positioned symmetrically relative to the neutral configuration 70b. In some implementations, the neutral configuration may be synonymously referred to as "reference configuration," "reference angle," "reference point," "zero angle," etc.

[0062] In some cases, conventional angles often used for C-arm or environmental settings can be chosen to serve as neutral configuration 70b. For example, a 0° configuration or a 90° configuration (referred to as an "anterior-posterior" configuration or a "lateral" configuration) can be such conventional angles used as neutral configuration 70b. Conventional angles may be preferred clinically due to their ease of identification and reference.

[0063] Angular offset between positive configuration 70a and neutral configuration 70b or between negative configuration 70c and neutral configuration 70b θ These can be any angular intervals, as long as their dimensions are substantially the same. For example, when the positive angular offset is + θ When selected as +5°, +10°, +15°, +30°, +45°, the corresponding negative angle offset is - θ These angles can be -5°, -10°, -15°, -30°, -45°, etc. These two angles will be positioned along the same axis of rotation defined by the positive angle and symmetrical with respect to the neutral configuration 70b. While any angle can be used, for computational efficiency, it is generally preferred that the angles differ sufficiently (>15°) to show some variability in the projected cross-section, but not so much that the similarity is too low (<30°).

[0064] Figure 6-2 , Figure 6-3 and Figure 6-4Examples of fluorescence fluoroscopic images are shown, each depicting a cross-section of bone projected at an angle corresponding to configurations 70a-c. From neutral configuration 70b, the C-arm 71 can rotate about its axis by a first angular offset. θ To reach the negative configuration 70a, and similarly, the C-arm 71 can rotate around the axis by a second angular offset + θ To reach the positive configuration 70c. Additional images can be projected from the negative configuration 70a and the positive configuration 70c to provide at least three fluorescent perspective images of the object 78 from multiple angles. That is, Figure 6-2 It is in the case of negative angular offset - θ The first fluorescence permeation image captured at the negative configuration 70a, Figure 6-3 The second fluorescence perspectral image was captured at a neutral configuration 70b with a zero (0°) angle, and Figure 6-4 It is in the case of having a positive angle + θ The third fluorescence perspectral image captured at 70°C in the positive configuration.

[0065] It should be noted that, although Figure 6-2 , Figure 6-3 and Figure 6-4 Bone is shown as the captured object of interest, but object 78 can be any semantically identifiable object. For example, object 78 can be any part of endoscope 30 or any anatomical feature including lesions or nodules. In terms of being composed of a denser material relative to its surroundings, object 78 can appear with distinguishable brightness, such that its cross-section is identifiable on the fluorescence fluoroscopic image, as... Figure 6-2 , Figure 6-3 and Figure 6-4 As shown.

[0066] In some implementations, objects appearing in a fluorescence perspectral image can be segmented to include some identified objects while excluding other objects or features. For example, Figure 6-2 , Figure 6-3 and Figure 6-4 The fluorescence fluoroscopic image includes bone but excludes other objects and background. In some implementations, segmentation can be based on one or more filters, such as density filters or thickness filters. Various segmentation algorithms or models can be utilized regarding filtering, including AI-driven algorithms / models. In some cases, segmentation can involve breaking down objects captured in the fluorescence fluoroscopic image into underlying materials, such as the metal of a hysteroscope or the tissue of nodules. Many variations are conceivable.

[0067] Sine curve generation and projection image simulation A fluorescent light source (such as a source / emitter 75) emits a beam that passes through the object at various angles. Detectors (such as receivers 74) measure the intensity of the fluorescent light field after they pass through the object. The data collected at each angle is represented in a sine plot, which is a two-dimensional array where one axis represents the angle of the beam and the other axis represents the position of the detector / receiver 74.

[0068] Figure 7-1 , Figure 7-2 and Figure 7-3 A conventional method for generating a sine graph by changing the projection angle is illustrated according to one or more implementation schemes. Figure 7-1 , Figure 7-2 , Figure 7-3 Each of the examples illustrates a region of interest 710a-c, which receives a corresponding imaging field 716a-c at a certain angle to generate a corresponding projected image 718a-c.

[0069] Projected images 718a-c can reflect first objects 712a-c and second objects 714a-c, each with different physical properties that reflect the brightness of objects 712 and 714 depicted by the projected images 718a-c. Projected images 718a-c can be attached as slices to another projected image at adjacent angles to gradually fill the sine curve 720a-c. Note that the region of interest 710 is a two-dimensional region, and the projected image 718 is a one-dimensional line. However, the general ideas related to the generation of sine curves described herein can be extended to three-dimensional space of interest and two-dimensional projected images.

[0070] Traditionally, an imaging field 716 is applied at fixed angular intervals (e.g., 0.5° interval, 1° interval, 2° interval, etc.) to project images 718 at various angles from 0° to 180°, and each projected image in the projected images 718 is combined to form a complete sine curve.

[0071] If the goal is to reconstruct a cross-section of an object, the complete sine wave can be filtered and back-projected to generate the reconstruction. Reconstruction is equivalent to generating a cross-section of a region of interest 710 for a given angle from the complete sine wave 720. A common method for reconstructing a cross-section from a sine wave is Filtered Backprojection (FBP). This process involves applying a transform, inverse transform, or other transform to the sine wave, applying filters in the frequency domain to enhance certain features or suppress artifacts, and then back-projecting the filtered data to obtain a two-dimensional image. Besides FBP, other reconstruction algorithms exist, such as iterative methods. Iterative reconstruction algorithms iteratively refine an initial estimate of the image to fit the measured sine wave data. Examples include the Maximum Likelihood Expectation Maximization (MLEM) algorithm and Algebraic Reconstruction Techniques (ART).

[0072] In some implementations, it is conceivable that a first cross-section of the first object 712 can be reconstructed separately from a second cross-section of the second object 714, and the second cross-section of the second object can also be reconstructed independently for a given angle. Furthermore, the first and second cross-sections can be merged, superimposed, or otherwise combined to provide a third cross-section depicting both the first object 712 and the second object 714. Compared to conventional methods, the techniques proposed herein effectively eliminate the need to project objects at every angle. That is, unlike conventional methods for generating sine waves, these techniques can rely on a selected number (e.g., three or more) of projected images and simulate a complete sine wave for all angles from these projected images.

[0073] Figure 8 An example of process 800, according to one or more embodiments, for simulating a complete sine curve using selected fluorescence perspective images. The selected fluorescence perspective images may be fluorescence perspective images projected at a known angle of symmetrical rotation, such as... Figure 6-2 , Figure 6-3 and Figure 6-4 The projected image. Process 800 can be implemented, at least in part, by the control circuitry of any system component disclosed herein, such as a robot cart / system and / or a control tower / system.

[0074] At box 802, process 800 may involve acquiring or otherwise accessing at least three projected images. In some embodiments, the projected images may be... Figure 6-2 , Figure 6-3 and Figure 6-4 The fluorescent perforated images are separated by a fixed-angle rotation, for example, a 30° rotation along the axis of rotation, symmetrically oriented in the positive and negative directions of the axis of rotation. In some embodiments, an object can be identified, segmented, and included in the adjusted image for subsequent use in process 800, while other objects and features can be subtracted, as shown in the fluorescent perforated images.

[0075] At box 804, the centroid of the fluorescence fluoroscopic image can be determined. In each fluorescence fluoroscopic image, the centroid of the fluorescence fluoroscopic image ( It can be calculated based on the following formula: in ∆p It is the pixel spacing along the contour (e.g., the axis in a projected image, such as the axis along the linear projected images 718a-c of Figure 7). m i It is the value of the i-th pixel in the contour. x i It is the position of the i-th pixel. xc It is the locus of the centroid in an orthogonal plane, and These are the known material properties of the object being projected. For example, Known for use in nodular tissue or endoscopic materials.

[0076] At box 806, after calculating the centroid of the projected image, the area quadratic moment of the plane orthogonal to the given projected image can be determined. I x In some cases, calculations can be based on the following formula. I x : This disclosure relies on at least three images projected from three-dimensional space (3D) onto a two-dimensional plane (2D). A major limitation of using a single 2D projection is that the cross-section of the object (such as the cross-section of a hysteroscope) may not be rotationally symmetric, and therefore, the moment of inertia ( I x The angle of projection around the object's axis (e.g., the endoscope axis) may vary. The operational concept of mathematical reconstruction is based on the fact that the object's principal moment of inertia (…) I max , I min ( ) can be completely determined using only three angularly shifted projections. That is, any two orthogonal projections ( I x , I y ) can generate a polar moment of inertia ( I p The constant of ) is equal to the principal moment of inertia ( I max , I min The sum of ). In other words, the constant polar moment of inertia ( I p ) equals two orthogonal moments of inertia ( I x , I y The sum of any of the principal moments of inertia ( I max , I min The sum of ), and this relationship can be summarized by the following formula: At box 808, process 800 may involve determining orthogonal to I x The moment of inertia, orthogonal moment of inertia is called I yAs described with respect to Figure 6, in an environment with a C-arm structure, it is typically possible to obtain two additional projected images around the C-arm axis with the same rotation angle ±θ on either side of the original projected image (e.g., symmetrically positioned). This disclosure utilizes the two additional projected images in conjunction with the rotation axis theorem to determine... I y Specifically, the rotation axis theorem specifies that the moment of inertia at angle θ can be given by the following equation: in I xy It is an inertial product, and I y It is orthogonal to I x The moment of inertia. Calculated separately for θ+ and θ- using equation (4). I θ+ and I θ- ,right I θ+ and I θz Summation, and finally, solution. I y ,get: Therefore, it is possible to base our understanding on two additional projected images and the known information from equation (2). I x And equation (4) to determine I y .

[0077] At box 810, process 800 may involve determining the inertial product ( I xy ).use I θ+ or I θ- Solve using equation (4) using any one of the options. I xy ,get: Note that the two orthogonal projections ( I x , I y The angle is known. θ ) and at that angle ( θ Moment of inertia under ) I θ The known value is . Therefore, by substituting the known value into equation (6), it is possible to find . I xy .

[0078] At box 812, process 800 may involve determining the principal moment of inertia ( I max , I min In three-dimensional space, for any arbitrary shape with a assumed uniform mass distribution, there always exist three perpendicular axes passing through any point (usually the center of mass for convenience) where the moment of inertia is maximized or minimized. These axes are called principal axes. The moments of inertia about these principal axes are called principal moments of inertia, which are here... I max and I min Principal moment of inertia I max and I min It can be based on the previously determined moment of inertia ( I x , I y and I θ The following determinations are made: At box 814, process 800 may involve determining I max The principal angle of the reference coordinate system ( φ When assuming I x and I y When referring to the reference coordinate system, φ It can be calculated in the following ways: At box 816, process 800 may involve generating a sine curve. In some implementations, the principal moment of inertia (… I max , I max ) and inertial product ( I xy This can be used to generate sine waves. Specifically, I max , I max and I xyThis can be used to calculate the area quadratic moment for all angles that are not measured (e.g., unrelated to the projected image) until a complete set of projections from 0° to 180° can be generated at fixed intervals (e.g., 1° intervals). This complete set can be all projection angles from 0° to 180° for a specific object (e.g., only endoscopes, only nodules, etc.). φ The sine curve.

[0079] The complete data will consist of the object thickness distribution at each angle (such as the thickness distribution of a trocar with uniform density). That is, the thickness distribution can be generated based on the assumption that the object has a uniform density profile. The characteristics of the object thickness distribution can lie in the total area ( A Typically, all projections for the same imaging device are fixed, and the centroid position is ( c φ ) and angle φ The area of ​​the second moment ( I φ From the mathematical calculations above, everything about each projection is known except for the actual distribution. The subsequent process will begin by generating or simulating the known projections. A , I φ and c φ The corresponding distribution of all missing projections.

[0080] First, the generated endoscope thickness profile is fitted to a function with a sufficient number of vertices, such that the fitting metric provides a large number of fits, such as at least R0. 2 ≥99% least squares fit or other sufficient fit values, while also taking the original A , I φ and c φ Maintaining an acceptable error is crucial. Functions can include polynomial functions, spline functions, etc. It is conceivable that Non-Uniform Rational Basis Splines (NURBS) is a model that uses basis splines (B-splines) for this purpose, which can be used as the fitting function, but various other functions are envisioned.

[0081] The generation of the distribution can begin with a projected image at the neutral configuration, which is at an angle. φ An image with known (e.g., measured) properties at 0°. From φ Starting with a fitted distribution at 0°, one can target angles... φ The simulated projected image at =1° iteratively adjusts the vertex of the fitting function while maintaining the original... A and c φ ,until I φThe conditions associated with the threshold are met, such as rising above the threshold, falling below the threshold, or reaching the threshold. This can be achieved by utilizing... φ Similar iterative adjustments are made starting with a fitted distribution of 1° to simulate... φ The projected image is at 2°, and so on. A similar process can be repeated for each missing projected image at other angles and also towards the negative angles until the sine graph is complete.

[0082] Therefore, compared with traditional methods, the technique disclosed herein can utilize equations (1)-(9) to simulate other projected images and complete the sine curve using three projected images. For example, Figure 9-1 The process 800 is illustrated by applying these three projected images to... Figure 6-2 , Figure 6-3 and Figure 6-4 A complete sine wave 900 is generated from the fluorescent perspective image of the bone.

[0083] Sine curve and reconstructed cross section of the object Figure 9-1 and Figure 9-2 An exemplary simulated sine curve 900 and a reconstructed cross section 950 of an object from the sine curve are illustrated according to one or more embodiments. Figure 9-1 It shows the use of Figure 8 The process 800 generates a complete sine plot of the object. As previously mentioned, the sine plot is a representation of fluorescence attenuation measurements taken around the object at various angles. Figure 9-2 The reconstructed cross-section of object 952 is shown. Typically, object reconstruction involves a sine plot from which cross-sectional slices of the object can be obtained. In the context of a fluorescence perspective sine plot, reconstruction refers to the process of transforming the collected raw data (in the form of a sine plot) into a meaningful image (such as a cross-section of the object at a given angle).

[0084] It should be noted that the exemplary sine wave 900 and the reconstructed cross section 950 of object 952 are bone-based; however, as previously stated, object 952 can be any semantically separable object, including but not limited to, a endoscopic endoscope, its tip, or any part of the endoscope, nodular tissue, or any other anatomical feature. Reconstruction 950 shows object 952 at a zero angle, which confirms the actual projected image of reconstruction 950 and object 952 at zero angle (e.g., Figure 6-3 The neutral configuration (70b) is a perfect match.

[0085] In some implementations, reconstructing 950 may involve filtering and backprojecting onto the analog sine curve 900. For example, Figure 9-2An example is shown: a cross-section of bone reconstructed from a sinusoid 900 to a reconstruction 950. Similar filtering and backprojection can be applied to reconstruct other objects from the same sinusoid or another similarly generated sinusoid. Therefore, cross-sections can be repeatedly reconstructed with various objects until the entire cross-section is reconstructed. Specifically, it is conceivable that the endoscope (including the distal end) and the tubercle can be reconstructed from three projected images.

[0086] The reconstructed cross-sectional image can include a mathematical reconstruction of the underlying object / material in the image (e.g., endoscope and nodule), and the image can be used to derive various geometric properties of the objects and materials within the image. These derived properties can then be used to initialize semantic segmentation of the objects and to indicate more accurate object identification to users viewing both the projected and simulated images. For example, derived geometric properties (including principal moments of inertia, area moments, the centroid of the object (which may be close to the object's centroid if the object is homogeneous), axes of rotation, or other properties) can be displayed to the user in association with the image and aid in the identification of objects and their properties. As another example, some derived properties (such as the centroid of the endoscope tip) can help estimate the pose of the endoscope tip in the image.

[0087] Object semantic segmentation using sine graphs Figure 10 A presentation 1000 illustrating derived geometric properties of a fluorescence fluoroscopic image according to one or more embodiments is illustrated. In some embodiments, the presentation 1000 may be provided on one or more displays 56 for viewing by a physician 5. In some other embodiments, the presentation 1000 may include a reconstructed cross-section (e.g., a simulated projection image) simulated in a manner consistent with this disclosure. In some other embodiments, the presentation 1000 may include an actual projection image associated with a known angle.

[0088] A 1000-dimensional reconstruction may include at least one reconstructed cross-section. This reconstructed cross-section can be displayed on a computer monitor or printed for analysis. In some implementations, modern medical imaging systems can also provide detailed 3D reconstructions.

[0089] The reconstructed object can be the endoscope 1002 or any part thereof (e.g., endoscope tip, sheath tip, biopsy needle, basket, forceps, etc.) and various anatomical features 1004 (e.g., nodules, lesions, tissues, etc.). Additionally, presentation 1000 may include an indicator pointing to the reconstructed object and presenting the indicator in association with the object. For example, the indicator may be a location indicator pointing to the centroid or location regarding other physical properties of the object. As another example, the indicator may point to one or more principal axes and provide valuable information to navigation software about the object's orientation, such as endoscope orientation. In some embodiments, accurate reconstruction of the object can facilitate the segmentation and detection of one or more objects.

[0090] Reconstruction and location estimation initialization Figure 11 A flowchart illustrating a process 1100 for reconstructing an object from a fluorescence fluoroscopic image according to one or more embodiments is provided. For example, process 1100 may be implemented by combining the reconstruction of the object (such as a endoscopic endoscope and a nodule) with the initialization of a later pose estimation of the object. Process 1100 may be implemented, at least in part, by control circuitry of any system component disclosed herein, such as a robot cart / system and / or a control tower / system.

[0091] At box 1102, process 1100 may involve accessing at least three projected images. As described above, the projected images may be axially rotated images generated by a fluorescence imaging device attached to the C-arm with a known symmetrical angle offset. For example, the projected images may be... Figure 6-2 , Figure 6-3 , Figure 6-4 The cross-section shown.

[0092] At box 1104, process 1100 may involve using three projected images to determine at least one principal moment of inertia. In some specific implementations, the object of interest may be left out, while other objects or features may be excluded from the three projected images. Specifically, the principal moment of inertia may be determined from the three projected images based on the rotation axis theorem. I max , I min ).

[0093] At box 1106, process 1100 may involve generating a sine graph. Here, for angles other than the three angles associated with the three projected images, the projected images for the remaining angles can be simulated based on at least one principal moment of inertia. In some embodiments, the projected images can be simulated at fixed intervals from 0° to 180° to complete the sine graph, such that the complete sine graph can consist of the thickness distribution of the object at each angle. In some embodiments, simulating the projected image for each adjacent angle may involve iteratively adjusting the number of vertices of a function fitted to the projected image of the object projected at the previous angle. The projected image at the previous angle may be based on the original projected image or a simulated projected image.

[0094] At box 1108, process 1100 may involve reconstructing an object seen from a specific angle based on the generated sine wave. In some implementations, the reconstruction may involve post-processing steps. As an example of a post-processing step, the reconstruction may involve an inverse transform. The actual reconstruction process can be quite complex, depending on the various post-processing algorithms and steps employed, the specific imaging techniques used, and the equipment employed. The choice of reconstruction method can affect image quality, spatial resolution, and the ability to distinguish different structures within the imaged object.

[0095] At box 1110, procedure 1100 may involve deriving geometric properties from the reconstructed object. Some example derivable geometric properties include the object's centroid, moment of inertia, area quadratic moment, etc. In some implementations, the object can be annotated in the original or simulated projected image based on the derived geometric properties.

[0096] Optionally, process 1100 may involve determining the three-dimensional position of an object, such as the distal end of a endoscopic endoscope or the centroid of a nodule. The three-dimensional position of the object can later be used as initialization for a pose estimation algorithm. When determining such a three-dimensional position of the object, process 1100 may involve performing some additional boxes.

[0097] At box 1112, process 1100 may involve performing triangulation. Triangulation is a common method in some implementations for projecting a point from a two-dimensional plane into three-dimensional space using capture point parameters. In fluoroscopic applications, capture point parameters can be determined based on the C-arm position or the fluoroscopic imaging device attached to the C-arm. Therefore, such parameters determined during installation and / or calibration can be used to perform triangulation.

[0098] At box 1114, process 1100 may involve initializing the pose estimation based on triangulation. For example, the three-dimensional position of an object may be determined based on triangulation, and this three-dimensional position may be used as the initial estimated pose of the object. The initial estimated pose may be presented to the user on a display in association with a projected image.

[0099] Figure 12 A block diagram of an exemplary controller 1200 for a medical system according to some specific embodiments is shown. In some specific embodiments, controller 1200 may be... Figure 4 An example of either control circuit 211 and / or control circuit 251.

[0100] The controller 1200 includes a communication interface 1210, a processing system 1220, and a memory 1230. The communication interface 1210 is configured to communicate with one or more components of the medical system. For example, the communication interface 1210 may include communication with one or more image sources (such as...) Figures 1 to 3 and Figure 5The image source interface 1212 communicates with the fluorescence fluoroscopic imaging system 70. In some embodiments, the image source interface 1212 can receive multiple images associated with a sine wave, wherein the multiple images are captured by the imaging device at multiple angles along the rotation axis of the imaging device.

[0101] Memory 1230 includes a non-transitory computer-readable medium (including one or more non-volatile memory elements, such as EPROM, EEPROM, flash memory, or hard disk drive, etc.) storing the following software (SW) modules: object detection SW module 1231 for detecting objects in each of a plurality of images; moment of inertia SW module 1232 for determining the moment of inertia associated with the object based on the plurality of images; sine wave simulation SW module 1233 for simulating one or more images of a sine wave based at least in part on the moment of inertia associated with the object; object reconstruction SW module 1234 for generating a reconstruction of the object based on the sine wave; and object analysis SW module 1235 for determining one or more characteristics of the object based on the reconstruction. Each software module 1231-1235 includes instructions that, when executed by processing system 1220, cause controller 1200 to perform a corresponding function.

[0102] Processing system 1220 may include any suitable one or more processors capable of executing scripts or instructions of one or more software programs stored in controller 1200 (such as in memory 1230). For example, processing system 1220 may execute object detection module 1231 to detect objects in each of a plurality of images. Processing system 1220 may also execute moment of inertia module 1232 to determine the moments of inertia associated with objects based on the plurality of images. Processing system 1220 may execute sine wave simulation module 1233 to simulate one or more images of a sine wave, at least in part, based on the moments of inertia associated with objects. Processing system 1220 may further execute object reconstruction module 1234 to generate a reconstruction of the object based on the sine wave. Furthermore, processing system 1220 may execute object analysis module 1235 to determine one or more characteristics of the object based on the reconstruction.

[0103] Figure 13 An exemplary flowchart illustrating exemplary operation 1300 for determining one or more characteristics of an object, according to some specific implementations, is shown. In some specific implementations, exemplary operation 1300 may be provided by a controller for a medical system (such as...). Figure 12 Controller 1200 or Figure 4 The control circuit (either of 211 and / or 251) is used to execute this.

[0104] The controller receives multiple images associated with the sine wave, wherein the multiple images are captured by the imaging device at multiple angles along the rotation axis of the imaging device (1302). In some embodiments, the imaging device may include a fluorescence fluoroscopy system. The controller detects an object in each of the multiple images (1304). In some embodiments, the object may include an instrument or anatomical feature configured to be driven through an intraluminal network. The controller further determines the moment of inertia associated with the object based on the multiple images (1306).

[0105] The controller simulates one or more images of a sine wave based at least in part on the moment of inertia associated with the object (1308). The controller generates a reconstruction of the object based on the sine wave (1310). In some implementations, the reconstruction may include a cross-section of the object. The controller further determines one or more properties of the object based on the reconstruction (1312). In some implementations, the controller may further present a position indicator pointing to the object on a display.

[0106] In some embodiments, the multiple images may include a first image captured at a first angle among a plurality of angles, a second image captured at a second angle among a plurality of angles, and a third image captured at a third angle among a plurality of angles, wherein the angular rotation degree separating the first and second angles is the same as the angular rotation degree between the second and third angles. In some embodiments, adjacent angles among the multiple angles may be separated by a fixed interval.

[0107] In some aspects, a sine graph can include the thickness distribution of an object at multiple angles. In some implementations, the controller can further determine a fitting function with multiple vertices that fits a known thickness profile to the thickness distribution of the object. In some implementations, simulation of each of one or more images of the sine graph can include iteratively adjusting the multiple vertices of the fitting function until the area quadratic moment at the angle associated with the simulated image satisfies a threshold condition.

[0108] In some implementations, the controller may further perform semantic segmentation operations based at least in part on one or more characteristics of the object, which are determined based on reconstruction. In some implementations, the controller may further determine an axis associated with the moment of inertia; determine a quadratic area moment associated with the axis, which is orthogonal to the axis associated with the moment of inertia; and determine the principal angle associated with the object based on the moment of inertia and the quadratic area moment.

[0109] Additional Implementation Plan Depending on the implementation, certain actions, events, or functions of any of the processes or algorithms described herein may be performed in different orders, added, combined, or completely ignored. Therefore, in some implementations, not all described actions or events are necessary for the practice of the process.

[0110] Unless otherwise specifically stated or otherwise understood in the context in which they are used, the conditional language used herein, such as “may,” “can,” “possibly,” “can,” “e.g.,” etc., refers in its ordinary sense and is generally intended to convey that certain embodiments include certain features, elements, and / or steps that are not included in other embodiments. Therefore, such conditional language is not generally intended to imply that one or more embodiments require features, elements, and / or steps in any way, or that one or more embodiments necessarily include logic for determining, with or without author input or prompting, whether such features, elements, and / or steps are included in any particular embodiment or whether they will be performed in any particular embodiment. The terms “comprising,” “including,” “having,” etc., are synonymous and used in their ordinary sense, and are used inclusively in an open-ended manner, without excluding additional elements, features, actions, operations, etc. Moreover, the term “or” is used in its inclusive sense (rather than in its exclusive sense) such that when used, for example, to connect a series of elements, the term “or” refers to one, some, or all of the elements in that series. Unless otherwise specified, combined language such as “at least one of X, Y, and Z” is understood in the context of general use to convey that an item, term, element, etc., can be X, Y, or Z. Therefore, such combined language is generally not intended to imply that a particular implementation requires the presence of at least one of X, at least one of Y, and at least one of Z.

[0111] It should be understood that in the above description of the embodiments, various features are sometimes grouped together in a single embodiment, figure, or description therein in order to simplify this disclosure and aid in understanding one or more aspects of the invention. However, this approach of the disclosure should not be construed as reflecting an intention that any claim requires more features than those expressly recited in that claim. Furthermore, any component, feature, or step shown and / or described in the specific embodiments herein may be applied to or used with any other embodiment. Moreover, for each embodiment, no component, feature, step, or group of components, features, or steps is necessary or indispensable. Therefore, it is expected that the scope of the invention disclosed herein and claimed below is not limited to the specific embodiments described above, but should be determined solely by a fair reading of the appended claims.

[0112] It should be understood that certain ordinal terms (e.g., "first" or "second") may be provided for ease of reference and do not necessarily imply physical characteristics or ordering. Therefore, as used herein, ordinal terms used to modify elements such as structures, components, operations, etc. (e.g., "first," "second," "third," etc.) do not necessarily indicate the priority or order of that element relative to any other element, but rather serve to generally distinguish that element from another element with a similar or identical name (but used in ordinal terms). Additionally, as used herein, the indefinite article ("a") may indicate "one or more" rather than "one." Furthermore, an operation performed "based on" a certain condition or event may also be performed based on one or more other conditions or events not explicitly stated.

[0113] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which the example embodiments pertain. It will be further understood that terms, such as those defined in common dictionaries, should be interpreted as having meanings consistent with their meanings in the context of the relevant field and not as idealized or overly formal, unless expressly defined herein.

[0114] For ease of description, the spatial relative terms “external,” “internal,” “upper,” “lower,” “below,” “above,” “vertical,” “horizontal,” and similar terms are used herein to describe the relationship between one element or component shown in the accompanying drawings and another. It should be understood that the spatial relative terms are intended to cover different orientations of the device in use or operation other than those depicted in the accompanying drawings. For example, if the device shown in the accompanying drawings is flipped, a device located “below” or “under” another device may be positioned “above” another device. Therefore, the illustrative term “below” can include both a lower position and an upper position. The device may also be oriented in another direction, and therefore the spatial relative terms may be interpreted differently depending on the orientation.

[0115] Unless otherwise explicitly stated, comparative and / or quantitative terms, such as “less,” “more,” “greater,” etc., are intended to encompass the concept of equality. For example, “less” may mean not only “less” in the strictest mathematical sense, but may also mean “less than or equal to.”

Claims

1. A system comprising: An imaging device configured to rotate axially about a rotation axis and capture multiple images associated with a sine wave at multiple angles along the rotation axis; One or more processors; and The memory stores instructions that, when executed by the one or more processors, cause the system to: Detect objects in each of the plurality of images; The moment of inertia associated with the object is determined based on the multiple images; One or more images of the sine curve are simulated, at least in part, based on the moment of inertia associated with the object; The reconstruction of the object is generated based on the sine wave; as well as One or more characteristics of the object are determined based on the reconstruction.

2. The system according to claim 1, wherein, The execution of the instructions further enables the system to: A location indicator pointing to the object is displayed on the monitor.

3. The system according to claim 1, wherein, The object includes instruments or anatomical features configured to be driven through an internal network of lumens.

4. The system according to claim 1, wherein, The imaging device includes a fluorescence imaging system.

5. The system according to claim 1, wherein, The plurality of images includes a first image captured at a first angle among the plurality of angles, a second image captured at a second angle among the plurality of angles, and a third image captured at a third angle among the plurality of angles, wherein the angular rotation degree separating the first angle and the second angle is the same as the angular rotation degree between the second angle and the third angle.

6. The system according to claim 1, wherein, Adjacent angles among the plurality of angles are separated by a fixed interval.

7. The system according to claim 1, wherein, The sine curve includes the thickness distribution of the object at the plurality of angles.

8. The system according to claim 7, wherein, The execution of the instructions further enables the system to: A fitting function with multiple vertices is determined, which fits a known thickness profile to the thickness distribution of the object.

9. The system according to claim 8, wherein, The simulation of each of the one or more images of the sine graph includes: The plurality of vertices of the fitting function are iteratively adjusted until the area quadratic moment at the angle associated with the simulated image satisfies a threshold condition.

10. The system according to claim 1, wherein, The reconstruction includes a cross-section of the object.

11. The system according to claim 1, wherein, The execution of the instructions further enables the system to: The semantic segmentation operation is performed at least in part based on one or more characteristics of the object, which are determined based on the reconstruction.

12. The system according to claim 1, wherein, The execution of the instructions further enables the system to: Determine the axis associated with the moment of inertia; Determine the area quadratic moment associated with the axis, which is orthogonal to the axis associated with the moment of inertia; and The principal angle associated with the object is determined based on the moment of inertia and the second moment of area.

13. A computer-implemented method, comprising: Receive multiple images associated with a sine wave, the multiple images being captured by the imaging device at multiple angles along the rotation axis of the imaging device; Detect objects in each of the plurality of images; The moment of inertia associated with the object is determined based on the multiple images; One or more images of the sine curve are simulated, at least in part, based on the moment of inertia associated with the object; The reconstruction of the object is generated based on the sine wave; as well as One or more characteristics of the object are determined based on the reconstruction.

14. The method of claim 13, the method further comprising displaying a location indicator pointing to the object on a display.

15. The method according to claim 13, wherein, The plurality of images includes a first image captured at a first angle among the plurality of angles, a second image captured at a second angle among the plurality of angles, and a third image captured at a third angle among the plurality of angles, wherein the angular rotation degree separating the first angle and the second angle is the same as the angular rotation degree between the second angle and the third angle.

16. The method according to claim 13, wherein, Adjacent angles among the plurality of angles are separated by a fixed interval.

17. The method according to claim 13, wherein, The sine curve includes the thickness distribution of the object at the plurality of angles, and the method further includes determining a fitting function having a plurality of vertices, the fitting function fitting a known thickness profile to the thickness distribution of the object.

18. The method according to claim 17, wherein, The simulation of each of the one or more images of the sine graph includes: iteratively adjusting the plurality of vertices of the fitting function until the area quadratic moment at the angle associated with the simulated image satisfies a threshold condition.

19. The method according to claim 13, wherein, The object includes at least one of a medical device or a nodule.

20. A controller for a medical system, the controller comprising: One or more processors; and The memory stores instructions that, when executed by the one or more processors, cause the controller to: Receive multiple images associated with a sine wave, the multiple images being captured by the imaging device at multiple angles along the rotation axis of the imaging device; Detect objects in each of the plurality of images; The moment of inertia associated with the object is determined based on the multiple images; One or more images of the sine curve are simulated, at least in part, based on the moment of inertia associated with the object; The reconstruction of the object is generated based on the sine wave; as well as One or more characteristics of the object are determined based on the reconstruction.

Images