Gesture-based three-dimensional structure reconstruction system and method

Abstract

The invention discloses a three-dimensional structure reconstruction system and a related method. In some examples, a three-dimensional structure reconstruction system may include at least one processor configured to receive a plurality of temporally ordered X-ray images of an object, where the plurality of X-ray images are taken in a plurality of poses relative to the object, wherein at least one of the plurality of X-ray images depicts at least one object having a known shape and / or position; determining an initial estimate of at least one pose of at least one of the plurality of X-ray images; and refining the at least one pose based on at least one comparison with the plurality of X-ray images. In some examples, a method may include receiving temporally ordered images taken in some gestures; determining an initial estimate of the at least one pose; and refining the pose based on the comparison with at least one of the plurality of X-ray images.

Description

[0001] Cross-referenced applications

[0002] This application claims priority to and the benefit of U.S. Provisional Application No. 63 / 327,119, filed on April 4, 2022, entitled “Pose-Based Three-DimensionalStructure Reconstruction Systems and Methods,” which is incorporated herein by reference in its entirety. Technical Field

[0003] Disclosed examples relate to three-dimensional structure reconstruction systems and methods. Background Art

[0004] C-arm machines are commonly used to take X-rays of a patient on a table. Manual C-arm machines allow an operator to manually rotate the C-arm around the patient to acquire images at various positions and orientations relative to the subject. Summary of the invention

[0005] In one example, a three-dimensional structure reconstruction system may include at least one processor configured to: receive a plurality of time-ordered X-ray images of an object, wherein the plurality of X-ray images are taken at a plurality of poses relative to the object, wherein at least one of the plurality of X-ray images depicts at least one object having a known shape and / or position; determine an initial estimate of at least one pose of at least one of the plurality of X-ray images; and refine at least one pose based on at least one comparison with the plurality of X-ray images.

[0006] In another example, at least one non-transitory computer-readable medium may have instructions thereon that, when executed by at least one processor, perform a method for three-dimensional structure reconstruction, the method comprising: receiving a plurality of time-ordered X-ray images of an object, wherein the plurality of X-ray images are taken at a plurality of poses relative to the object, wherein at least one of the plurality of X-ray images depicts at least one object having a known shape and / or position; determining an initial estimate of at least one pose of at least one of the plurality of X-ray images; and refining at least one pose based on at least one comparison with the plurality of X-ray images.

[0007] In yet another example, a method for three-dimensional structure reconstruction may include: receiving a plurality of time-ordered X-ray images of an object, wherein the plurality of X-ray images are taken at a plurality of poses relative to the object, wherein at least one of the plurality of X-ray images depicts at least one object having a known shape and / or position; determining an initial estimate of at least one pose of at least one of the plurality of X-ray images; and refining at least one pose based on at least one comparison with the plurality of X-ray images.

[0008] It should be understood that the above concepts and the additional concepts discussed below can be arranged in any suitable combination, as the present disclosure is not limited in this regard. In addition, other advantages and novel features of the present disclosure will become apparent from the following detailed description of various non-limiting examples when considered in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The accompanying drawings are not intended to be drawn to scale. In the accompanying drawings, each identical or nearly identical component shown in various figures may be represented by a like numeral. For clarity, not every component may be labeled in every drawing. In the accompanying drawings:

[0010] Figure 1A An illustrative C-arm imaging system is depicted in accordance with an embodiment of the present disclosure.

[0011] Figure 1B An illustrative imaging system is depicted operating with a subject in place in accordance with an embodiment of the present disclosure.

[0012] Figure 2 is a block diagram showing a schematic representation of an imaging system having a flexible medical system including a shape sensor disposed within a field of view of the imaging system according to an embodiment of the present disclosure.

[0013] Figure 3 An illustrative flexible elongated device visualized in certain contexts is depicted in accordance with an embodiment of the present disclosure.

[0014] Figure 4A is a first part of a flowchart illustrating a method for reconstructing a three-dimensional structure according to an embodiment of the present disclosure.

[0015] Figure 4B yes Figure 4A The second part of the flowchart is shown.

[0016] Figure 5 is a flowchart illustrating a method for reconstructing a three-dimensional structure according to an embodiment of the present disclosure.

[0017] Figure 6 An illustrative bundling adjustment is depicted in accordance with an embodiment of the present disclosure.

[0018] Figure 7 An illustrative C-arm imaging system including an object for calibrating distortion parameters is depicted in accordance with an embodiment of the present disclosure.

[0019] Figure 8 is an illustration of an object for calibrating distortion parameters according to the present disclosure. DETAILED DESCRIPTION

[0020] In certain applications, a user (such as a medical practitioner) views the two-dimensional views output from a C-arm system and makes educated guesses about the shape and orientation of objects and anatomical structures in the different views. However, the two-dimensional fluoroscopic images (which are typically produced by these systems) have no depth information due to the overlap of different parts of the three-dimensional structures within the image. As a result, it is difficult or impossible to discern the three-dimensional relative pose of the medical tool relative to the target tissue (such as a lesion) in the subject, and users of manual C-arm imaging systems are typically left to guess. This is because, unlike CT scanners and automated C-arm machines, conventional manual two-dimensional C-arm systems do not measure or track the reference frame and pose of images relative to each other. Therefore, typical image reconstruction and segmentation methods cannot be used for image streams captured of a subject using a conventional manual C-arm.

[0021] Thus, it may be desirable to improve positioning, in particular the relative orientation of a tip portion of a medical tool (e.g., a catheter, biopsy needle, or other desired tool) to a target tissue (e.g., a lesion on an organ of a subject) during operation. Furthermore, it may be desirable to allow three-dimensional reconstruction and / or segmentation of objects located in the field of view of the imaging system even if the poses of the different images relative to each other are initially unknown.

[0022] In some embodiments, the system may receive an image stream of a series of sequential two-dimensional images (such as X-ray fluoroscopic images) captured from different sequentially arranged postures of an imaging device (e.g., a detector in a C-arm system or other type of imaging system) taken along the motion path of the imaging device relative to the subject during image capture. As used herein, the posture of an image refers to the position and orientation of the imaging device that captures the image. Each image is captured from a perspective defined by the posture of the imaging device at the time of capture. These images can be used to reconstruct three-dimensional structures within the body of the imaged subject, including, for example, a part of the subject's body and associated instruments that interact with the subject's body. However, in order to use the image stream, it is desirable to know the posture of the imaging device associated with each image captured by the imaging device. In view of this limitation associated with typical imaging devices, having an object with a known shape and / or position located in the field of view of the imaging device can be used to determine the posture associated with different images of the image stream. Specifically, one or more estimated postures of the image stream can be refined using at least one comparison with multiple images of the image stream. In some embodiments, the comparison can include generating one or more two-dimensional projection images of at least a portion of the object projected using an initial posture estimate. The resulting projection image can be compared to a corresponding image of the image stream to provide a refined pose estimate of at least one pose, as described in further detail below. The resulting pose can optionally be used in conjunction with the captured image for any desired application (including use with any suitable reconstruction algorithm).

[0023] Any suitable type of shape sensor capable of sensing the shape of at least a portion of an instrument disposed within the field of view of an imaging system may be used with the various embodiments disclosed herein. Suitable types of shape sensors may include, but are not limited to, fiber optic shape sensors, encoders / displacement sensors of articulatable instruments, electromagnetic sensors positioned at known locations along the instrument, position sensors, combinations of the above sensors, and / or any other suitable sensor configured to sense the shape, orientation, and / or position of one or more portions of an instrument.

[0024] The received image used in the various embodiments described herein can have any suitable resolution. For example, the image received can have a resolution of at least 256 pixels by 256 pixels. In some embodiments, the image received can have a resolution of at least 512 pixels by 512 pixels. In some embodiments, the image received can have a resolution of at most 976 pixels by 976 pixels or 1200 pixels by 1200 pixels. For example, the image received can have a resolution between or equal to 256 pixels by 256 pixels and 976 pixels by 976 pixels. Although specific resolutions are mentioned above, any suitable resolution can be used for the image described herein.

[0025] The reconstructed structure can have any suitable resolution. For example, the reconstructed structure can have a voxel resolution of at least 256 voxels by 256 voxels by 256 voxels. In some embodiments, the reconstructed structure can have a voxel resolution of at least 512 voxels by 512 voxels by 512 voxels. In some embodiments, the reconstructed structure can have a resolution of at most 976 voxels by 976 voxels by 976 voxels. For example, the reconstructed structure can have a resolution between or equal to 256 voxels by 256 voxels by 56 voxels and 976 voxels by 976 voxels by 976 voxels. Although specific resolutions for the reconstructed structure are mentioned above, any suitable resolution can be used.

[0026] In various embodiments disclosed herein, the C-arm 110 can be configured to rotate through any suitable range of angles. For example, a typical C-arm can be configured to rotate around an object (e.g., a subject on an imaging table) up to an angle between or equal to 140 and 270 degrees. As further described below, in some embodiments, scanning can be performed throughout such a rotation range of the C-arm. Alternatively, scanning can be performed within a subset of the rotation range of the system that is less than the total rotation range of the system. For example, for a system capable of operating within a rotation range greater than 0 to 90 degrees, scanning can be performed between 0 and 90 degrees. Although specific rotation ranges are mentioned above, the systems and methods disclosed herein can be used with any suitable rotation range. As the rotation range increases, the quality of the reconstruction can be improved, and the techniques described herein allow as many rotations as desired by the operator.

[0027] Some embodiments may be widely used and adapted for simple and common inputs from manually operated C-arm machines. Some embodiments may even operate without additional hardware. For example, some embodiments may be installed as part of a scanner's firmware or software, or used independently by transferring images to any device hosting the algorithm. Thus, the disclosed embodiments may provide an inexpensive alternative to automated 3D C-arms, which are less common and significantly more expensive than manual 2D C-arm machines. In some embodiments, no additional sensors or fiducial markers are required for any of these processes.

[0028] Additionally, a manually rotated C-arm has a varying trajectory between runs, so techniques that use only a rotation sensor to determine the position of the C-arm on the trajectory may not be sufficient to provide acceptable reconstruction. Some embodiments herein can correct for these varying trajectories, and can detect larger amounts of jitter induced in manual motion (e.g., by accurately detecting the pose of the source and detector of the C-arm on a per-image basis).

[0029] It may be desirable to improve the accuracy of the initial estimate of the pose of the image stream used in the disclosed method and system. In such an embodiment, a pose sensor configured to sense parameters related to the estimated pose of the system during imaging may be used. In some cases, this may correspond to one or more additional pose sensors added to an existing imaging system. For example, suitable pose sensors may include, but are not limited to, inertial measurement units (IMUs), accelerometers, gyroscopes, magnetometers, encoders, gravimeters, cameras pointing to the surrounding environment (SLAM), optical trackers (e.g., cameras pointing to a C-arm), combinations of the above, and / or any other suitable type of pose sensor capable of sensing parameters related to the pose of the system relative to an object during imaging. Such sensors can use data related to the pose of the image to improve the estimate, and the use of such additional hardware (particularly inexpensive additional sensors) can still greatly limit costs relative to traditional automated three-dimensional C-arms.

[0030] Although specific dimensions and ranges of various components and aspects of the systems and methods disclosed herein are described above and elsewhere in this disclosure, dimensions greater and less than those noted herein may be used.

[0031] Embodiments herein can be used with imaging and positioning of any medical device, including robotic-assisted endoscopes, catheters, and rigid arm systems used in medical procedures. For example, the disclosed methods and systems can be used to provide updated posture information that can enable 3D reconstruction and / or segmentation of objects (e.g., lungs and medical devices during medical procedures) using inexpensive medical imaging devices (e.g., 2D fluoroscopic C-arms) with or without additional sensors. The disclosed techniques are not limited to use with only these specific applications. For example, while the disclosed methods are primarily described for use with a C-arm system that is used to take X-ray images at different postures relative to a subject, the disclosed methods can be used with any X-ray imaging system that takes X-ray images at different postures relative to an object imaged by the system.

[0032] The disclosed methods and systems can provide many benefits relative to automated C-arms capable of 3D reconstruction and manual C-arms that are generally not capable of 3D reconstruction. For example, the disclosed methods and systems can be used to achieve 3D reconstruction and / or segmentation of target tissue (e.g., a target with the lungs or other parts of the body of a subject) based on the standard output of relatively inexpensive and commonly used medical imaging equipment (e.g., a traditional manual two-dimensional C-arm system). In some embodiments, the disclosed imaging system and method can also be used without affecting the workflow of the medical procedure relative to the current system. Therefore, in some embodiments, the disclosed methods and systems can be used as an alternative to more expensive 3D C-arm imaging systems in the workflow of current 2D C-arms. In addition, the ability of the disclosed methods and systems to consider differences in the amount of rotation applied by the user during manual imaging can also provide a flexible imaging system, although a larger rotation range may be associated with improved reconstruction quality. Using a benchmark (e.g., an object of known shape) to improve the pose estimation of the image stream can also provide a robust and accurate method for considering differences in the manual rotation trajectory of the system during separate manual scans. Although several potential benefits are described above, there may be other benefits in the system that are different from the above benefits.

[0033] The received images and / or output of the disclosed processes may correspond to any desired format. However, in some embodiments, the received and / or output images may be in Digital Imaging and Communications in Medicine (DICOM) format. This format may be viewed (e.g., like a CT scan), may be widely compatible with other systems and software, and may be easily saved to a storage device and viewed at a later time.

[0034] As used herein, the term "orientation" refers to the position of an element or a portion of an element in three-dimensional space (e.g., three translational degrees of freedom along Cartesian x, y, and z coordinates). As used herein, the term "orientation" refers to the rotational arrangement of an element or a portion of an element (three rotational degrees of freedom (e.g., roll, pitch, and yaw), axis-angles, rotation matrices, quaternion representations, etc.). As used herein, the term "pose" refers to the multi-degree-of-freedom (DOF) spatial position and orientation of a coordinate system of interest (e.g., attached to a rigid body). In general, a pose includes pose variables for each DOF in the pose. For example, a complete 6-DOF pose will include 6 pose variables corresponding to 3 DOF of orientation (e.g., x, y, and z) and 3 DOF of orientation (e.g., roll, pitch, and yaw).

[0035] Turning to the drawings, specific non-limiting examples are described in more detail. It should be understood that the various systems, components, features, and methods described with respect to these examples may be used alone and / or in any desired combination, as the present disclosure is not limited to the specific examples described herein.

[0036] Figure 1A An illustrative two-dimensional C-arm imaging system 100 according to an embodiment of the present disclosure is depicted. The imaging system 100 can be configured to image any desired object. In an example where the imaging system is a medical imaging system, the object to be imaged can correspond to tissue of a subject, and in some cases, can correspond to a medical system that interacts with the target tissue. The tissue can correspond to a site within a natural cavity of a subject and / or an intervention site. The imaging system 100 includes a manual C-arm 110 operably coupled to a source 114, a detector 116, and a controller 120. In some embodiments, the source 114 can be configured to emit X-rays toward the detector 116, and the detector 116 can be configured to detect an X-ray image of an object disposed between the source 114 and the detector 116. In some embodiments, the controller 120 can be operably coupled to the detector 116 so that it receives an image stream from the detector 116. The C-arm 110 can also be rotatably coupled to a base 118, which is configured to support the entire C-arm imaging system. In some embodiments, the imaging system 100 includes a manual handle 112 attached to the C-arm 110, which an operator can use to control the pose of the C-arm 110 and the source 114 and detector 116 as they rotate relative to a base 118 and an object disposed between the source 114 and detector 116. Although the embodiments disclosed herein are primarily directed to a manually controlled C-arm, in some embodiments, the pose of the C-arm 110 can be controlled programmatically or by a user via a user input device.

[0037] In some embodiments, the imaging system 100 may include a posture sensor 160. In some cases, the posture sensor 160 may be an additional posture sensor attached to the manual C-arm 110 or other suitable portion of the imaging system, such that the posture sensor 160 may sense one or more parameters related to the posture of the source 114 and the detector 116 relative to the object being imaged within the field of view of the system. The posture sensor 160 may be attached to the C-arm 110 of the imaging system 100. In other examples, the posture sensor 160 may be attached to the detector 116 and / or the source 114. In some embodiments, the attachment may use adhesives, hooks and loops, screws, bolts, or any other suitable attachment mechanism. In some embodiments, the orientation of the rotation axis of the posture sensor 160 may be aligned with the C-arm rotation axis, which may improve the accuracy of the sensor's measurements. In some embodiments, communication between the posture sensor 160 and the controller 120 or other computer may be performed via Wi-Fi, Bluetooth, wired, near field communication, or any other suitable communication method.

[0038] Figure 1B An illustrative imaging system 100 is depicted operating with a subject 150 in place in accordance with an embodiment of the present disclosure. Figure 1B A manual C-arm imaging system 100 is shown having a C-arm 110 , source 114 , detector 116 , and manual handle 112 similar to those described above. In some embodiments, the imaging system 100 includes a display 130 . Figure 1B Also shown is an illustrative operator 140 operating the manual handle 112 and an illustrative subject 150 being scanned by the imaging system 100. The source 114 and the detector 116 can be rotated around the subject 150 in pairs. As described above, the C-arm and the associated detector 116 and source 114 are rotatable so that they can rotate through a plurality of different postures relative to the subject 150 or other objects disposed between the source 114 and the detector 116. Thus, when the operator 140 manually rotates the C-arm 110 between an initial posture and a final posture, the source 114 and the detector 116 can be used to obtain a sequential stream of X-ray images of the subject 150 or other objects in a plurality of postures relative to the subject. As described above, this can correspond to rotations between any desired postures (including rotations over the entire rotation range of the C-arm 110 or a portion of the rotation range of the C-arm 110). In some embodiments, the imaging system 100 may include a posture sensor 160 as described above.

[0039] In some embodiments, a pose estimation and / or 3D structure reconstruction system as described herein may be part of a controller 120 of an imaging system. Alternatively or additionally, a pose estimation and / or 3D structure reconstruction system may be part of a separate computer (e.g., a desktop computer, a portable computer, and / or a remote or local server). In some embodiments, a pose estimation and / or 3D structure reconstruction system may include at least one processor (e.g., controller 120).

[0040] Figure 2 1 is a block diagram illustrating the relationship of one embodiment of an imaging system similar to that described above. In the depicted embodiment, the imaging system includes a source 114 and a detector 116. In some embodiments, a shape sensor 190 may be configured to detect the shape or position of at least one or more portions of an object 1010 disposed in the field of view of the imaging system. In some embodiments, the object is a medical system or device, such as a catheter, an endoscope, a laparoscope, or any other object that a shape sensor can characterize. The system may also include a posture sensor 160 connected to a suitable moving portion of the imaging system as described above. In the depicted embodiment, various components such as the shape sensor 190, the posture sensor 160, and the detector 116 may be operably coupled to a control system 120 such that signals from these various components may be output to the control system 120 for use in various embodiments disclosed herein. In some embodiments, the control system 120 may include at least one processor 122 and at least one memory 124. In some embodiments, the memory may be a non-transitory computer-readable memory 124 including computer-executable instructions thereon that, when executed by at least one processor, may perform any of the methods disclosed herein.

[0041] Figure 3How a shape sensor may be used to determine the position or posture of one or more portions of an instrument disposed within a field of view of an imaging system is shown. In the depicted embodiment, an illustrative flexible elongated device 1010 (e.g., a catheter) including a shape sensor (not depicted) is visualized in a captured image 1000 of a human anatomy. The corresponding shape of the flexible elongated device relative to the reference frame of the imaging system is shown in a corresponding three-dimensional diagram 1100, wherein the position and posture of each intermediate portion of the flexible elongated device may correspond to the integrated posture of the intermediate portion of the flexible elongated device. As further described below, a two-dimensional projection of the measured three-dimensional shape, or the position of a portion of the imaged three-dimensional object may be associated with the corresponding positions of these features in the captured two-dimensional image. For example, a known position of a distal portion of the flexible elongated device in the reference frame of the imaging system may be associated with the position of the distal portion of the flexible elongated device in the captured image to determine the posture of the image, as further described below. Although a flexible elongated device is described with respect to the accompanying drawings, any suitable type of object and corresponding sensor capable of measuring the position or posture of the object within the reference frame of the imaging system may be used.

[0042] Figure 4A-4B Depicted is a flow chart illustrating a method 2000 for estimating a pose associated with a captured image stream from an imaging system and reconstructing a three-dimensional structure based on the estimated pose and the image stream. Figure 4A is the first part of the flowchart, and Figure 4B is from Figure 4A The indicator A shown continues the second part of the flowchart. In some embodiments, the described process, system and control system can be used to implement the depicted method. The method 2000 is shown as a set of phases, blocks, steps, operations or processes. Not all of the enumerated operations shown can be performed in all embodiments of the method 2000. Additionally, Figure 4A-4B Some additional operations not explicitly shown in the method 2000 may be included before, after, between, or as part of the enumerated stages. Some embodiments of the method 2000 include instructions corresponding to the processes of the method 2000 stored in a memory. These instructions may be executed by a processor, such as a processor of a controller or control system.

[0043] In some embodiments, a pose sensor as described above may optionally be used to determine an initial estimate of the pose of the C-arm's source 114 and detector 116. In some embodiments, other types of devices may provide this information in lieu of sensors, including optical tracking sensors, cameras, encoders, Hall sensors, distance sensors, or any other suitable device or system.

[0044] Some embodiments of method 2000 may begin with a phase of capturing data, which may include phases 2010, 2110, 2210, and 2310 as described below. In some embodiments, it may be desirable to know when to begin data capture for phases 2010, 2110, 2210, and 2310. In one embodiment, a physical button may be pressed by an operator to indicate that a scan will be performed using a C-arm imaging system. Alternatively, a device and / or software application may detect the start of a scan from a video capture or based on data from a posture sensor (e.g., exceeding a threshold change in posture). In some embodiments, the data may be passed through a smoothing filter for such detection, which may reduce false positives. In some embodiments, other sensors and / or algorithms may be used to detect the start of motion. Alternatively or additionally, detection may be performed by detecting an X-ray beam-on signal from the C-arm imaging system 100, which may include a radiation signal. In some embodiments, when a scan begins, the signal may trigger the start of a sensor and video recording. Regardless of how the process is initiated, after triggering the start of data capture, the operator can manually rotate the C-arm to complete the scan.

[0045] In addition to starting data capture, it may also be desirable to determine when the scan is complete and when data capture should be terminated. For example, in one embodiment, various sensor data may continue to be recorded until a suitable input is received indicating an end to rotating the C-arm and / or imaging with the C-arm. For example, user input, termination of image capture, turning off the X-ray source and / or a change in posture of the imaging system being below a threshold for a predetermined period of time and / or any other suitable input may be used to determine when to terminate data capture.

[0046] Various types of feedback can be provided to the user during the scan. For example, in some embodiments, the system can monitor the streaming video images to see if any image is overexposed, and the display settings can be adjusted to include all structures as much as possible. In other embodiments, the system can check for user errors. For example, if the user is rotating the C-arm but has not stepped on the imaging or fluoroscopic pedal, no image is real-time, or if the user is stepping on the pedal but has not rotated the C-arm, the image will not change. The system can provide an output indicating such events to the user. In another embodiment, if the sensor detects any irregular pattern (e.g., too fast, too slow, not enough angle, etc.), the information may potentially be output from the system to the operator to notify the operator that they should adjust their speed or add more rotations, or make any other suitable adjustments. Of course, other suitable types of feedback can be provided to the operator, because the present disclosure is not limited to this approach.

[0047] In stage 2010, calibration of the posture sensor can be performed, which can be done in one go. In some embodiments, this one-time calibration can be used to improve reconstruction quality. In some embodiments, the calibration can include placing a phantom with easily identifiable markers of known shapes within the field of view of the imaging system. In some embodiments, a scan can then be performed, recording data from the optional posture sensor and video data from the imaging system. In some embodiments, the calibration is used to determine the approximate trajectory of the source and detector of the imaging system during the scan.

[0048] In stage 2110, an image of the object can be captured. For example, the image can be an X-ray fluoroscopic image captured by a C-arm imaging system as described above. In addition, the object can be a human subject or a part of a subject (e.g., an organ of the subject). As described above, in some embodiments, the received images are taken in different postures relative to the object (e.g., taken from different positions and orientations through which an operator moves a manual C-arm). In some embodiments, different postures can correspond to different orientations (e.g., angles) of the system. However, embodiments in which different postures are characterized as changes in orientation and orientation can also be considered. In either case, the captured sequential image stream can be output to a corresponding control system or other computing system (including a processor configured to perform the methods disclosed herein).

[0049] In stage 2210, data from a gesture sensor may optionally be captured. As previously described, this sensed gesture data may be used to determine an initial estimate of the gesture of an individual image in the captured image stream. For example, temporal data associated with different sensed gestures and captured images may be used to associate the estimated gestures with the corresponding captured images to provide an initial estimated gesture.

[0050] In stage 2310, data from a shape sensor may be captured. For example, data from a flexible elongated device or other medical device with shape sensing may be captured. As previously described, the shape sensor data may provide information related to the shape of one or more portions of the medical device within the reference frame of the imaging system. In one such embodiment, the pose of the distal portion of the catheter or other medical device may be known within the reference frame of the imaging system.

[0051] After capturing the initial data, the method 2000 may then proceed to a stage for preprocessing the captured data, which may include Figure 4A 2020, 2120, 2220 and 2320. In stage 2020, the calibration data from stage 2010 may be loaded by the processor.

[0052] In stage 2120, the image captured in stage 2110 (e.g., an X-ray fluoroscopic image) may be subjected to any suitable preprocessing including, but not limited to, contrast adjustment, image correction, filtering, cropping, and / or any other suitable type of image preprocessing.

[0053] In stage 2220, the optional gesture sensor data captured in stage 2210 may be preprocessed. The gesture sensor data may correspond to sensor input related to the gesture of the imaging system during image capture. Suitable types of preprocessing for the gesture sensor data may include, but are not limited to, signal averaging, filtering, smoothing, and / or any other suitable type of preprocessing that may be desired for the sensed gesture data.

[0054] In stage 2320, the shape sensor data captured in stage 2310 may be preprocessed. Suitable types of preprocessing for the shape sensor data may include, but are not limited to, integration of the sensor data to determine one or more positions of one or more parts of the object within a reference frame of the imaging system. Other suitable types of preprocessing of the shape sensor data may also be used.

[0055] Method 2000 may then proceed to stage 2040 for data alignment of preprocessed data. In stage 2040, the received image data may be aligned with the sensor data from the gesture sensor. For example, the image data preprocessed in stage 2120 may be aligned with the sensor data preprocessed in stage 2220.

[0056] Method 2000 may then proceed to a phase of pose estimation, which may include phase 2050, see Figure 4A and Figure 4B In stage 2050, the aligned sensor data may be mapped to the calibrated pose. For example, if calibration was performed, the sensor data aligned in stage 2040 may be mapped to the calibration data loaded in stage 2020 to determine a more accurate pose estimate for different images of the image stream.

[0057] In some embodiments, the sensed pose given by the optional pose sensor can be used to determine the initial pose estimates associated with the different images in 2050. For example, if the rotation of the C-arm is only around one axis, the rotation information from the pose sensor can be mapped to a single angle. For example, the rotation can be mapped to an axis-angle representation, and the axis can be constrained. In such an embodiment, the angle can then represent how much the C-arm has rotated around the axis.

[0058] As another example, a frame at the origin may be rotated by a three-dimensional rotation and then translated out by the C-arm radius. In such an embodiment, the points at these frame positions may then be constrained to a three-dimensional sphere, and the angle formed by each pair of adjacent points may represent how much the C-arm has been rotated.

[0059] In yet another embodiment of stage 2050, the initial estimated poses of the corresponding images of the image stream may correspond to angular orientations that are uniformly distributed along the expected trajectory within the rotation range of the imaging system. For example, a system with a 180 degree rotation range may be estimated to have an image stream that includes images uniformly distributed within the rotation range of the system extending from 0 degrees to 180 degrees (e.g., for 100 frames taken within 180 degrees, each frame may be estimated to be 1.8 degrees from its adjacent frames). In some embodiments, larger or smaller rotation ranges may be used, depending on the system being used. Additionally, instances where the initial estimate is a random number, all zeros, or any other suitable initial estimate may also be used.

[0060] Due to variability in manual rotation of the C-arm, sensor errors and / or other sources of error, the initial estimate of posture using a posture sensor or based on any other initial estimation technique may be inaccurate. Therefore, after estimating the posture of the images of the image stream, the method can proceed to stage 2060, where the posture of the image can be further refined to improve its accuracy. As previously described, the use of information related to the shape, position and / or posture of a portion of the imaged object can be used to improve the estimated posture using a separate posture correction for each frame that can account for differences in manual rotation trajectories. For example, the refinement of the posture can use at least a portion of an object with a known shape and / or position in the reference frame of the imaging device and within the field of view of the images of the image stream. Therefore, the refined posture of each image of the image stream may be more robust and accurate.

[0061] In one possible example of stage 2060, a portion of an object with a known shape and / or position may correspond to a medical device (e.g., a distal portion of a catheter or other device disclosed herein) present within the field of view of the imaging system. Using a suitable shape sensor associated with the medical device, the shape of one or more portions of the medical device relative to the reference frame of the imaging system may be determined. The initial pose estimate and the known shape and / or position of at least a portion of the medical device may be used to determine the error in the estimated pose. Specifically, the measured shape and / or position of one or more portions of the object may be projected into a two-dimensional image using the estimated pose associated with each image of the image stream. The position of one or more portions of the object in the projected image may be compared with the position of one or more portions of the object in the original image to determine the error. These determined errors may then be used to determine an updated estimate of the pose of each image of the image stream. For example, in some embodiments, a refined and more accurate pose may be found by minimizing the error between the estimated position of one or more portions of the medical device and the position where it is actually seen in the received image under various poses. Suitable methods for determining a pose may include, but are not limited to: alignment between features identified in the image and a projected image of an object having a known shape and / or position; bundle adjustment; a trained statistical model configured to identify a suitable pose based at least in part on the position and / or shape of at least a portion of the object; and / or any other suitable type of method for determining a suitable pose based on a reprojected image and an original corresponding image of the image stream.

[0062] In various embodiments disclosed herein, the estimated and refined posture information may include external posture parameters, such as the position and / or orientation of the captured image. In some embodiments, intrinsic parameters such as the dimensions associated with the source and detector used for imaging may be known and input into a suitable algorithm. However, embodiments are also contemplated in which one or more of these intrinsic parameters are included in the estimated posture and refined posture determined by the processes and systems disclosed herein, as the present disclosure is not limited in this manner.

[0063] After determining the pose, method 2000 may store the pose and image stream for subsequent use, or may proceed to reconstruction in stage 2070. In stage 2070, the three-dimensional structure may be reconstructed using the refined pose estimate and image stream from stage 2060. Since the pose is now known, any suitable reconstruction method used in the art may be used, including but not limited to: filtered back projection (FBP), simultaneous iterative reconstruction technique (SIRT), simultaneous algebraic reconstruction technique (SART), iterative reconstruction technique (ART), conjugate gradient least squares (CGLS), FDK, ADMM total variation, ADMM wavelet, ordered subset expectation maximization (OSEM), statistical image reconstruction (SIR), coordinate ascent algorithm, expectation maximization algorithm (EM), and / or any other reconstruction technique.

[0064] After reconstructing the three-dimensional object, method 2000 may optionally proceed to displaying information in stage 2080. Information related to the reconstructed three-dimensional object may be displayed to an operator of the system in stage 2080. This may include displaying a three-dimensional rendering, a segmentation, and / or any other suitable type of information related to the reconstructed three-dimensional object.

[0065] In some embodiments, as described herein, at least some portions of stages 2060 and 2070 may be repeated as needed. Although not shown in method 2000, it may be checked whether the reconstruction has been completed (e.g., convergence has been reached where the difference between the received image and the projected image is within a threshold). If the reconstruction has not been completed, method 2000 may return to at least some portion of stage 2060. Alternatively, if the reconstruction has been completed, method 2000 may end or repeat as needed.

[0066] In the above-described embodiments, the posture has been refined from an initial estimate of the posture of the received image. However, in some embodiments, it may be desirable to proceed directly from stage 2050, where an estimated posture is determined based on one or more sensed posture parameters from an additional posture sensor and an optional calibration of the reconstruction at stage 2070. In such embodiments, the initial estimated posture from the posture sensor data may be deemed sufficiently accurate to allow reconstruction based on time-coordinated image and posture information. Therefore, it should be understood that the present disclosure may be implemented with or without further refinement of posture information associated with the captured image stream.

[0067] Figure 5 2 is a flow chart showing a method 200 for reconstructing a three-dimensional structure according to an embodiment of the present disclosure. In some embodiments, the method described may be implemented using the above-described processes, systems, and controllers. Figure 5200 is shown as a set of phases, blocks, steps, operations or processes. Not all of the enumerated operations shown may be performed in all embodiments of method 200. Additionally, Figure 5 Some additional operations not explicitly shown in the method 200 may be included before, after, between, or as part of the enumerated stages. The operations may also be performed in an order different from the order shown. Some embodiments of the method 200 include instructions corresponding to the processes of the method 200 stored in a memory. These instructions may be executed by a processor, such as a processor of a controller or control system.

[0068] Some embodiments of method 200 may begin at stage 210, where time-ordered images of an object captured by an imaging device in different poses may be received. In some embodiments, the object may be a human patient or subject and / or an organ of a patient or subject. In some embodiments, the image of the object may be an X-ray image. In some embodiments, the image of the object may be captured from different perspectives, each perspective corresponding to a posture of the imaging device relative to the object. In some embodiments, the plurality of images may be a series of sequential images captured in a plurality of sequential postures, the postures being positioned along a motion path of a detector of the imaging system relative to an object within the field of view of the detector. In some embodiments, at least one object having a known shape and / or position relative to a reference frame of the imaging system may be present in the field of view of the image. For example, a medical device comprising a shape sensor may be present within the field of view of the imaging system during imaging of the subject.

[0069] The method 200 may then optionally proceed to stage 220, where the shape and / or position of at least a portion of the object may be determined. For example, the processor may receive a sensed or otherwise determined position of at least a portion of the object relative to a reference system of the imaging system. In other embodiments, the processor may receive a sensed shape of at least one object relative to a reference system of the imaging system.

[0070] The method 200 may then proceed to stage 230 where an initial estimate of the pose of each image of the image stream may be determined.For example, the processor may determine an initial estimate of at least one pose of at least one received image.

[0071] In some embodiments, stage 230 may optionally include stage 232, where an initial estimate of the pose may be determined using sensor data as described above. For example, the processor may determine the initial pose estimate based at least in part on data from a pose sensor, such as an inertial measurement unit, an accelerometer, a gyroscope, a magnetometer, or any other suitable pose sensor attached to the imaging system during capture of the received image.

[0072] Method 200 may then optionally proceed to stage 240, where a two-dimensional image may be projected based on the initially estimated pose and shape and / or position information associated with the object as described above. For example, in some embodiments, position and / or shape information determined relative to one or more parts of an object (e.g., a medical device) may be projected into a two-dimensional image using pose estimation. Figure 4A and Figure 4B An example of this process is described further.

[0073] Method 200 may then proceed to stage 250, in which the gesture may be refined based on at least one comparison between the projected image and the corresponding image of the image stream. In some embodiments, stage 250 may include stage 252, wherein the comparison includes comparing at least one projected two-dimensional image of the shape and / or position of one or more parts of the object with at least one received image. For example, the comparison may include alignment of data from at least one projected two-dimensional image with the received image. As an alternative or in addition to stage 252, stage 250 may include stage 254, wherein the processor may use bundling adjustments to refine at least one gesture based on at least one object having a known shape and / or position. In some embodiments, stage 254 may include stage 256, wherein the processor may use the position of one or more parts of the object when refining at least one gesture using bundling adjustments.

[0074] Figure 6 An example of bundling adjustment is shown. Figure 6 middle, P is an estimated point in the real world, p' is where the point would be seen if the camera were in a particular position relative to the orientation of the source at O ​​used to create the signal detected by the camera, and p is where the point was actually observed. In some embodiments, the bundle adjustment includes a nonlinear optimization that attempts to correct the point and the camera (e.g., X-ray imaging orientation) by minimizing a reprojection error that can be used to refine the pose estimate. However, as previously described, other optimization techniques may also be used.

[0075] In stage 260, in some embodiments, at least some portions of stage 250 may be repeated as needed, as described above. For example, if the pose has converged to within a desired threshold accuracy, method 200 may proceed to stage 270 and reconstruct using the refined pose. Alternatively, if the refined pose still requires further refinement, method 200 may return to at least some portion of stage 250 until the pose exhibits the desired accuracy.

[0076] In stage 270, the image may be reconstructed to provide a three-dimensional reconstruction of the image stream including the object. This reconstruction may be performed using the refined gestures and images of the image stream as described above. For example, the processor may reconstruct the three-dimensional structure using the refined gestures of the image stream and the received images. The gestures, the reconstructed three-dimensional structure (including the reconstructed object), the image stream, and / or any other suitable information may be stored in a memory for future recall and use, displayed to an operator as described above, or used for any other suitable application.

[0077] C-arm systems (particularly those systems that include image intensifier detectors) may experience spatial distortion. In some examples, spherical distortion (e.g., radial, pincushion, and / or barrel distortion) may be caused by optical lenses and may be relatively consistent with changes in C-arm posture. In some examples, S-shaped distortion (e.g., S-distortion) may be caused by an external magnetic field, and the distortion field may vary with the C-arm posture. Without compensation, these forms of distortion may result in inaccurate posture determination and distort the three-dimensional reconstruction generated by the C-arm. Determining a set of distortion parameters for a C-arm system or a specific configuration of a C-arm system may allow compensation for distortion. In some examples, an object with a known shape (e.g., object 1010) in the field of view of a C-arm detector may be used to optimize both posture and distortion parameters, but it may be difficult to optimize both parameters using a common object. Using different objects with known shapes and / or configurations may allow the optimization to be separated and may result in more accurate determination of posture and distortion parameters. Figure 7 A C-arm imaging system 300 is shown that includes an object 302 that can be used to calibrate distortion parameters. System 300 can be similar to system 100, but there are differences as described. In this example, object 302 can be attached to C-arm 110. Object 302 can have known features, such as a known shape and / or position, such as a known fiducial pattern, and can be used to calibrate the distortion parameters of the C-arm. Object 302 can be fixed or coupled to C-arm 110 so that the object rotates with the C-arm and remains in the same orientation in the C-arm field of view for all generated images. In some examples, object 302 can be attached to detector 116 and rotate with detector 116.

[0078] Figure 8is a top view of object 302. Object 302 may include a platform 352 and a set of fiducials 354. One or more attachment devices 356 (e.g., clamps, clips, threaded connectors, or other mechanical fixtures) may be configured to removably or permanently couple object 302 to detector 116. The fiducials 354 have a fixed position and orientation relative to detector 116. As detector 116 rotates, the position of fiducials 354 in the generated image remains unchanged, while the position and orientation of object 1010 in the field of view of detector 116 changes. The fiducials 354 may be used to determine distortion parameters, and object 1010 may be used independently to determine posture parameters. The distortion parameters may be used to correct distortion and generate an undistorted image or three-dimensional reconstruction. In some examples, platform 352 may be formed of a metal material, and fiducials 354 may be a set of holes through platform 352. The metal material may obscure the field of view (except for the location of the fiducials). In some examples, platform 352 can be formed of a radiolucent material such as plastic, and fiducial 354 can be formed of a radiopaque material such as metal spheres. These metal spheres can have an invariant position and orientation in the generated images. In some examples, fiducial 354 can have a different shape than object 1010 and can therefore be distinguished based on the shape in the generated images. In other examples, fiducial 354 can have the same shape as object 1010, but can be distinguished from object 1010 in successive images because fiducial 354 is positionally invariant in the generated images.

[0079] One or more elements in the embodiments of the present disclosure can be implemented in software to execute on a processor of a computer system including a control system disclosed herein. When implemented in software, the elements of the embodiments of the present disclosure are basically code segments that perform necessary tasks. A program or code segment can be stored in a processor-readable storage medium or device, which may have been downloaded by a computer data signal embodied in a carrier wave on a transmission medium or a communication link. A processor-readable storage device may include any medium that can store information, including optical media, semiconductor media, and magnetic media. Processor-readable storage device examples include electronic circuits, semiconductor devices, semiconductor memory devices, read-only memories (ROMs), flash memories, erasable programmable read-only memories (EPROMs), floppy disks, CD-ROMs, optical disks, hard disks, or other storage devices. Code segments can be downloaded via a computer network (such as the Internet, intranets, etc.).

[0080] Please note that the processes and displays presented may not be inherently related to any particular computer or other device. The structures of these different systems appear as elements in the claims. In addition, the embodiments of the present disclosure are described without reference to any particular programming language. It should be understood that various programming languages ​​can be used to implement the teachings of the present disclosure as described herein.

[0081] Although the present teaching has been described in conjunction with various examples, it is not intended to limit the present teaching to these examples. On the contrary, as will be appreciated by those skilled in the art, the present teaching includes various alternatives, modifications and equivalents. Therefore, the above description and accompanying drawings are only intended as examples.

Claims

1. A three-dimensional structure reconstruction system, comprising: At least one processor configured to: receiving a time-ordered plurality of x-ray images of an object, wherein the plurality of x-ray images are taken at a plurality of poses relative to the object, wherein at least one of the plurality of x-ray images depicts at least one object having a known shape and / or position; determining an initial estimate of at least one pose of at least one of the plurality of x-ray images; as well as The at least one gesture is refined based on at least one comparison with the plurality of X-ray images.

2. The system of any one of the preceding claims, wherein the at least one processor is further configured to reconstruct a three-dimensional structure using the refined at least one pose and the plurality of X-ray images.

3. A system according to any of the preceding claims, wherein the at least one processor is further configured to determine the initial estimate of the at least one posture based at least in part on data from at least one of an inertial measurement unit, an accelerometer, a gyroscope, or a magnetometer attached to the imaging system during capture of the multiple X-ray images.

4. A system according to any one of claims 1-2, wherein the at least one processor is further configured to determine the initial estimate of the at least one posture using a uniform distribution of the multiple postures within a range of motion of an imaging system used to capture the multiple X-ray images.

5. The system of any of the preceding claims, wherein the at least one comparison comprises comparing at least one projected two-dimensional image of the shape and / or position of at least a portion of the at least one object with at least one of the plurality of X-ray images.

6. The system of claim 5, wherein the at least one comparison comprises an alignment of data from the at least one projected two-dimensional image with the plurality of x-ray images.

7. The system of any one of claims 1-4, wherein the at least one processor is further configured to use bundle adjustments to refine the at least one gesture based at least in part on the object having a known shape and / or position.

8. The system of claim 7, wherein the at least one processor is further configured to receive a position of at least one feature of the at least one object, and the at least one processor is further configured to use the position when refining the at least one gesture using binding adjustments.

9. The system of any of the preceding claims, wherein the at least one object comprises a medical device.

10. The system of any of the preceding claims, wherein the at least one processor is further configured to receive a sensed shape of the at least one object.

11. A system according to any of the preceding claims, wherein the at least one object having a known shape and / or position comprises a first object having a known shape and / or position and a second object having a known shape and / or position, and wherein the processor is further configured to determine distortion parameters based on the second object having a known shape and / or position.

12. The system of claim 11, wherein the second object is movable relative to the first object.

13. A method for three-dimensional structure reconstruction, the method comprising: receiving a time-ordered plurality of x-ray images of an object, wherein the plurality of x-ray images are taken at a plurality of poses relative to the object, wherein at least one of the plurality of x-ray images depicts at least one object having a known shape and / or position; determining an initial estimate of at least one pose of at least one of the plurality of x-ray images; as well as The at least one gesture is refined based on at least one comparison with the plurality of X-ray images.

14. The method of claim 13, further comprising reconstructing a three-dimensional structure using the refined at least one pose and the plurality of X-ray images.

15. The method of any one of claims 13-14, further comprising determining the initial estimate of the at least one posture based at least in part on data from at least one of an inertial measurement unit, accelerometer, gyroscope, or magnetometer attached to the imaging system during capture of the plurality of X-ray images.

16. The method of any one of claims 13-14, wherein the at least one processor is further configured to determine the initial estimate of the at least one posture using a uniform distribution of the multiple postures within a range of motion of an imaging system used to capture the multiple X-ray images.

17. The method according to any one of claims 13-16, wherein the at least one comparison comprises comparing at least one projected two-dimensional image of the shape and / or position of the object with at least one of the plurality of X-ray images.

18. The method of claim 17, wherein the at least one comparison comprises alignment of data from the at least one projected two-dimensional image with the plurality of x-ray images.

19. The method of any one of claims 13-16, wherein the method further comprises using bundle adjustment to refine the at least one gesture based on the at least one object having a known shape.

20. The method of claim 19, further comprising receiving a position of at least one feature of the at least one object and using the position in refining the at least one gesture using a binding adjustment.

21. The method of any one of claims 13-20, wherein the at least one object comprises a medical device.

22. The method of any one of claims 13-21, further comprising receiving a sensed shape of the at least one object.

23. At least one non-transitory computer readable medium having instructions thereon which, when executed by at least one processor, perform any one of the methods of claims 13-22.

24. The method of claim 13, wherein the at least one object having a known shape and / or position comprises a first object having a known shape and / or position and a second object having a known shape and / or position, and the method further comprises determining distortion parameters based on the second object having a known shape and / or position.

25. The method of claim 24, wherein the second object is movable relative to the first object.

Images