Three-dimensional structure reconstruction system and method

Abstract

Three-dimensional structure reconstruction systems and related methods are disclosed herein. In some examples, a three-dimensional structure reconstruction system may include at least one processor configured to: receive a plurality of X-ray images of an object, where the plurality of X-ray images are taken in a plurality of poses relative to the object; determining a loss function based on the estimated three-dimensional structure and the plurality of X-ray images of the object; and determining a reconstructed three-dimensional structure by minimizing the loss function. In some examples, at least one non-transitory computer readable medium may have instructions thereon that, when executed by at least one processor, perform a method for three-dimensional structure reconstruction. In some examples, a method may include receiving an X-ray image of an object; determining a loss function based on the estimated three-dimensional structure and the X-ray image; and determining a reconstructed three-dimensional structure by minimizing the loss function.

Description

[0001] Cross-referenced applications

[0002] This application claims priority to and the benefit of U.S. Provisional Application No. 63 / 327,133, filed on April 4, 2022, entitled “Three-Dimensional Structure Reconstruction Systems and Methods,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] The disclosed examples relate to three-dimensional structure reconstruction systems and methods. Background Art

[0004] C-arm machines are commonly used to take X-rays of a patient on a table. Manual C-arm machines allow an operator to manually rotate the C-arm around the patient to acquire images at various positions and orientations relative to the subject. Summary of the invention

[0005] In one example, a three-dimensional structure reconstruction system may include at least one processor configured to: receive multiple X-ray images of an object, wherein the multiple X-ray images are taken at multiple postures relative to the object; determine a loss function based on an estimated three-dimensional structure and the multiple X-ray images of the object; and determine a reconstructed three-dimensional structure by minimizing the loss function.

[0006] In another example, at least one non-transitory computer-readable medium may have instructions thereon that, when executed by at least one processor, perform a method for three-dimensional structure reconstruction, the method comprising: receiving multiple X-ray images of an object, wherein the multiple X-ray images are taken at multiple postures relative to the object; determining a loss function based on the estimated three-dimensional structure and the multiple X-ray images of the object; and determining a reconstructed three-dimensional structure by minimizing the loss function.

[0007] In another example, a method for reconstructing a three-dimensional structure may include: receiving multiple X-ray images of an object, wherein the multiple X-ray images are taken at multiple postures relative to the object; determining a loss function based on the estimated three-dimensional structure and the multiple X-ray images of the object; and determining a reconstructed three-dimensional structure by minimizing the loss function.

[0008] It should be understood that the above concepts and other concepts discussed below can be arranged in any suitable combination, as the present disclosure is not limited in this regard. In addition, other advantages and novel features of the present disclosure will become apparent from the following detailed description of various non-limiting examples in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The drawings are not intended to be drawn to scale. In the drawings, each identical or nearly identical component shown in various figures may be represented by the same numeral. For clarity, not every component may be labeled in every figure. In the drawings:

[0010] Figure 1A is a schematic diagram of an exemplary C-arm imaging system according to an embodiment of the present disclosure.

[0011] Figure 1B is a schematic diagram of an exemplary imaging system operating with a subject in place according to an embodiment of the present disclosure.

[0012] Figure 2 is a flow chart illustrating a method for reconstructing a three-dimensional structure according to an embodiment of the present disclosure.

[0013] Figure 3 is a series of images showing the comparison of the generated two-dimensional projections with the original captured X-ray images, and the comparison between their gradients.

[0014] Figure 4 is a series of images illustrating the increase in detail shown in a generated two-dimensional projection according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0015] In certain applications, users (such as medical personnel) view the two-dimensional views output from a C-arm system and make skilled guesses about the shape and position of objects and anatomical structures in different views. However, the two-dimensional fluorescence images (which are typically produced by these systems) have no depth information due to the overlap of different parts of the three-dimensional structures within the image. As a result, it is difficult or impossible to determine the three-dimensional relative pose of the surgical tool relative to the target (such as a lesion) on the subject, so it often needs to be guessed by the user using the manual C-arm imaging system. This is because conventional manual two-dimensional C-arm systems cannot measure or track the reference frame and pose of images relative to each other like CT scanners and automatic C-arm machines. As a result, typical image reconstruction methods cannot be used for image streams of subjects captured with conventional manual C-arms.

[0016] Thus, it may be desirable to improve the positioning, in particular, of an end effector of a surgical tool (e.g., a biopsy needle or other desired end effector) relative to a target (e.g., a lesion on an organ of a subject). Furthermore, it may be desirable to allow a three-dimensional reconstruction of an object located in the field of view of an imaging system even if the poses of different images relative to each other are unknown. As a result, it may be possible to provide a usable three-dimensional representation of a target (e.g., the lungs or other portion of a subject's body) based on the standard output of a relatively inexpensive medical imaging device (e.g., a conventional manual two-dimensional C-arm system).

[0017] In view of this, by adopting the specific reconstruction techniques described in some embodiments of this article, various improvements can be made to the positioning and three-dimensional reconstruction based on the image flow, thereby improving surgical operations. For example, the inventors have recognized and understood that using some embodiments can achieve better positioning than conventional methods.

[0018] In some embodiments, a system can receive a series of continuous two-dimensional images (such as X-ray fluorescence images) captured from different continuous positions and orientations relative to a subject. These images can be used to reconstruct the three-dimensional structure being imaged, such as a portion of the subject's body and associated instruments that interact with the subject's body. In addition, the system can also (in some embodiments) recover projection parameters associated with these received two-dimensional images so that simulated two-dimensional images generated as part of the reconstruction process match the received images. In some embodiments, none of these processes require additional position sensors or fiducial markers.

[0019] In a specific embodiment, multiple X-ray images of an object are received. This can be achieved by using real-time capture of images, transmission or downloading of received images, or any other appropriate method for obtaining images. These images can correspond to images of an object or multiple objects within the field of view of the X-ray imaging device, and these images are taken in multiple different postures relative to the object. In some embodiments, these images can be taken in continuous postures relative to the object. Comparison between the estimated three-dimensional structure and the multiple X-ray images can be used to determine information about different postures associated with the image, which can allow reconstruction of the three-dimensional structure. For example, in some embodiments, a loss function based on the estimated three-dimensional structure and the multiple X-ray images can be used to reconstruct the three-dimensional structure corresponding to the object.

[0020] The received image used in the various embodiments described herein can have any suitable resolution. For example, the received image can have a resolution of at least 256 pixels by 256 pixels. In certain embodiments, the received image can have a resolution of at least 512 pixels by 512 pixels. In certain embodiments, the received image can have a resolution of at most 2048 pixels by 2048 pixels. For example, the received image can have a resolution between 256 pixels by 256 pixels and 2048 pixels by 2048 pixels or equal to these pixels. Although specific resolution is noted above, any suitable resolution can be used for the image described herein.

[0021] The reconstructed structure can have any suitable resolution. For example, the reconstructed structure can have a voxel resolution of at least 16 voxels by 16 voxels by 16 voxels. In some embodiments, the reconstructed structure can have a voxel resolution of at least 512 voxels by 512 voxels by 512 voxels. In some embodiments, the reconstructed structure can have a resolution of at most 1024 voxels by 1024 voxels by 1024 voxels. For example, the reconstructed structure can have a resolution between or equal to 16 voxels by 16 voxels by 16 voxels and 1024 voxels by 1024 voxels by 1024 voxels. 512 pixels by 512 pixels by 512 pixels. Although the specific resolution of the structure used for reconstruction is noted above, any suitable resolution can be used. In addition, as described below, a coarse-to-fine analysis process can be used to implement the increased resolution of the structure used for reconstruction.

[0022] In various embodiments disclosed herein, the C-arm 110 can be configured to rotate through any suitable angular range. For example, a typical C-arm can be configured to rotate around an object (e.g., a subject on an imaging table) to an angle between 180 and 270 degrees or equal to these degrees. As further described below, in some embodiments, scanning can be performed throughout this rotation range of the C-arm. Alternatively, scanning can be performed on a subset of the rotation range of the system that is less than the total rotation range of the system. For example, for a system that can operate within a rotation range greater than 0 to 90 degrees, scanning can be performed between 0 and 90 degrees. Although specific rotation ranges are noted above, the systems and methods disclosed herein can be used with any suitable rotation range.

[0023] Some embodiments may be widely used and may be adapted for simple and commonly used inputs from manually operated C-arm machines. Some embodiments may even operate without additional imaging hardware. For example, some embodiments may be installed as part of a scanner's firmware or software, or used independently by passing the image to a device separate from the C-arm machine. Thus, the disclosed embodiments may provide an inexpensive alternative to automated 3D C-arms, which are uncommon and much more expensive than manual 2D C-arm machines.

[0024] Although specific dimensions and ranges for various components and aspects of the systems and methods disclosed herein are described above and elsewhere herein, it should be understood that dimensions larger than those indicated herein and dimensions smaller than those indicated herein may be used.

[0025] Embodiments herein can be used for imaging and positioning of any medical device, including robot-assisted endoscopes, catheters, and rigid arm systems. In some instances, the technology disclosed herein can be used for manually operated systems, robot-assisted surgical systems, remotely operated robotic surgical systems, and / or other desired applications. The disclosed technology is not limited to being used only for these specific applications. For example, although the disclosed method is primarily described as being used with a C-arm system for taking X-ray images in different postures relative to a subject, the disclosed method can be used with any X-ray imaging system that takes X-ray images in different postures relative to an object imaged by the system.

[0026] The received images and / or output of the disclosed processes may correspond to any desired format. However, in some embodiments, the received and / or output images may be in Digital Imaging and Communications in Medicine (DICOM) format or some other standard format. Such a format may be browsed (e.g., like a CT scan), may be widely compatible with other systems and software, and may be conveniently saved to a storage device and viewed at a later time.

[0027] As used herein, the term "position" refers to the positioning of an element or a portion of an element in three-dimensional space (e.g., three translational degrees of freedom along Cartesian x-, y-, and z-coordinates). As used herein, the term "orientation" refers to the rotational arrangement of an element or a portion of an element (three rotational degrees of freedom - e.g., roll, pitch, and yaw, angular axes, rotation matrices, quaternion representations, etc.). As used herein, the term "pose" refers to the multi-degree-of-freedom (DOF) spatial position and orientation of a coordinate system of interest (e.g., attached to a rigid body). In general, a pose includes pose variables for each DOF in the pose. For example, a complete 6-DOF pose will include 6 pose variables corresponding to 3 position DOFs (e.g., x, y, and z) and 3 orientation DOFs (e.g., roll, pitch, and yaw).

[0028] Turning to the drawings, specific non-limiting examples will be described in further detail. The various systems, components, features, and methods described in connection with these examples may be used alone and / or in any desired combination and are not limited to the specific examples described herein.

[0029] Figure 1Ais a schematic diagram of an exemplary two-dimensional C-arm imaging system 100 according to an embodiment of the present disclosure. The imaging system 100 can be configured to image any desired object. In an example where the imaging system is a medical imaging system, the object to be imaged can correspond to tissue of a subject, such as a site within a natural cavity of the subject and / or a surgical site. The imaging system 100 includes a manual C-arm 110 operably coupled to a source 114, a detector 116, and a controller 120. In some embodiments, the source 114 can be configured to emit X-rays toward the detector, and the detector can be configured to detect an X-ray image of an object disposed between the source 114 and the detector 116. In some embodiments, the controller 120 can be operably coupled to the detector 116 so that it receives an image stream from the detector 116. The C-arm 110 can also be rotatably coupled to a base 118, which is configured to support the entire C-arm imaging system. In some embodiments, the imaging system 100 includes a manual handle 112 attached to the C-arm 110, which an operator can use to control the posture of the C-arm 110 and the source 114 and detector 116 as the C-arm 110 and the source 114 and detector 116 are rotated relative to the base 118 and an object disposed between the source 114 and the detector 116. Although the disclosed embodiments are primarily directed to a manually operated C-arm, in some embodiments, the posture of the C-arm 110 can be controlled by a program or by a user through a user input device.

[0030] Figure 1B is a schematic diagram of an exemplary imaging system operating with a subject in place according to an embodiment of the present disclosure. Figure 1B A manual C-arm imaging system 100 is shown having a C-arm 110 , source 114 , detector 116 , and manual handle 112 similar to those described above. In some embodiments, the imaging system 100 includes a display 130 . Figure 1B Also shown is an exemplary operator 140 operating the manual handle 112 and an exemplary subject 150 scanned by the imaging system 100. The source 114 and the detector 116 are rotatable around the subject as a pair. As described above, the C-arm 110 and the associated detector 116 and source 114 are all rotatable so that they can move through a variety of different postures relative to the subject 150 or other objects disposed between the source 114 and the detector 116. Therefore, when the C-arm 110 is manually rotated by the operator 140 between the initial posture and the final posture, the source 114 and the detector 116 can be used to obtain a continuous stream of X-ray images of the subject 150 or other objects in a plurality of postures relative to the subject 150. As described above, this can correspond to rotations between any desired postures, including rotations within the entire rotation range of the C-arm 110 or within a partial rotation range of the C-arm 110.

[0031] In some embodiments, the three-dimensional structure reconstruction system described herein may be part of the controller 120 of the imaging system 100. Alternatively or additionally, the three-dimensional structure reconstruction system may be part of a stand-alone computer (such as a desktop computer, a portable computer, and / or a remote or local server). In some embodiments, the three-dimensional structure reconstruction system may include at least one processor, such as the controller 120. In some embodiments, the processor may be configured to receive an image of an object. For example, the image may be an X-ray fluorescence image obtained from a C-arm imaging system as described above. In addition, the object may be a human subject or an organ of the subject. In some embodiments, the received image of the object may be taken from different perspectives by an imaging device (e.g., a detector 116). For example, these images may be taken at different postures of the imaging device relative to the object (such as from different positions and orientations). These images taken at different positions and orientations of the imaging device may be obtained via movement of the C-arm 110 attached to the source 114 and the detector 116 (e.g., may be controlled by an operator via a manual handle 112 or otherwise).

[0032] In some embodiments, an initial estimate of the three-dimensional structure can be made. In some embodiments, an initial estimate of projection parameters associated with different poses of the received images (e.g., captured using different orientations and positions of the imaging device in some embodiments) can also be made. Projection parameters are values ​​that define the viewing angle of the detector and / or source relative to the object, used to provide one or more captured images. For example, each image can capture the object from a viewing angle defined by one or more projection parameters (such as angular position, orientation angle, position, etc.). In some embodiments, these initial estimates are not necessarily accurate or even close to the actual values. On the contrary, in some embodiments, very rough "guesses" are acceptable. In some embodiments, the initial estimate of the three-dimensional structure can be all zeros or random numbers in the voxels of the reconstructed structure. For example, the estimated three-dimensional structure may initially include voxels with random intensity or zero intensity.

[0033] In some embodiments, the initially estimated projection parameters may correspond to angular positions uniformly distributed along a circular trajectory, such as from 0 to 180 degrees (e.g., for 100 frames captured within a 180 degree range, each frame may be estimated to differ from its adjacent frames by 1.8 degrees). In some embodiments, a larger or smaller rotation range may be used, which may be varied based on appropriate constraints.

[0034] In some embodiments, a better initial estimate of the projection parameters, such as from an inertial measurement unit (IMU) sensor or a benchmark-based calibration, can be used to fine-tune the reconstruction process. In such embodiments, the minimization of the loss function described below can be accelerated by such fine-tuning. However, in some embodiments, the reconstructed three-dimensional structure is determined without relying on information derived from position sensors or benchmark markers.

[0035] In some embodiments, the estimated three-dimensional structure can be projected into a two-dimensional image using the estimated or determined projection parameters. For example, the three-dimensional structure initially estimated above can be projected into these two-dimensional images using the projection parameters. In some embodiments, the projection operation can be distinguishable for both the three-dimensional structure and the projection parameters. In some embodiments, the projection parameters can include intrinsic parameters related to the imaging system (e.g., the separation distance and orientation of the source and the detector relative to each other), and parameters related to the interaction between the imaging system and the imaged object, for example, including the position and orientation of the imaging system relative to the viewing angle or viewpoint at which the image has been or will be projected.

[0036] In some embodiments, the received image may be an 8-bit image, the image contrast may vary, and / or some areas of the image may be overexposed or underexposed. In this case, information may be lost, and during the projection of the estimated 3D structure into a 2D image, this may be mitigated by modeling this process as a linear mapping using value clipping.

[0037] In some embodiments, the processor may determine the projection parameters and the reconstructed three-dimensional structure. In some embodiments, the projection parameters and the reconstructed three-dimensional structure may be determined together in time. For example, in addition to reconstructing the three-dimensional structure, the projection parameters actually used when capturing the received image may also be reconstructed, and in some embodiments the projection parameters may be reconstructed in the same process and at the same time as the three-dimensional structure is reconstructed. For example, in some embodiments, the processor may determine a loss function based on the estimated three-dimensional structure and based on the received image of the object. For example, the loss function may be determined by comparing the projected two-dimensional image with the received image. In some embodiments, the loss function may be a way of expressing and quantifying the difference between the projected two-dimensional image and the received image.

[0038] In some embodiments, the comparison includes comparing a gradient of at least one of the two-dimensional images with a gradient of at least one of the received images. In some embodiments, the gradient may be obtained by shifting at least a portion of the original image.

[0039] In some embodiments, the processor may determine the reconstructed three-dimensional structure by minimizing a loss function. For example, the processor may pass the gradient of the loss back to the three-dimensional structure and the projection parameters and update them. In some embodiments, minimizing the loss function may make the loss or difference as low as possible. In some embodiments, a derivative of the loss function, such as a first-order derivative, may be used. Therefore, some embodiments make the projected image more similar to the received image through each iteration of the disclosed reconstruction process. In some embodiments, these iterations may continue to include the following steps: 1) generating a projected two-dimensional image using the updated reconstructed three-dimensional structure and an estimate of the projection parameters; 2) comparing the projected two-dimensional image with the captured image (e.g., using a loss function); and 3) updating the estimate of the reconstructed three-dimensional structure and projection parameters based on this comparison (e.g., by minimizing the loss function) until the estimated projection parameters and the reconstructed three-dimensional structure converge, at which point the projected image and the received image may have a threshold degree of difference at most.

[0040] In some embodiments, an Adam optimizer may be used for such minimization or optimization. Alternatively or additionally, Adadelta, Adagrad, AdamW, SparseAdam (a "lazy" Adam algorithm suitable for sparse tensors), Adamax (an Adam variant based on infinite norms), averaged stochastic gradient descent, L-BFGS, NAdam, RAdam, RMSprop, resilient backpropagation, stochastic gradient descent, DENSE_QR, DENSE_NORMAL_CHOLESKY, SPARSE_NORMAL_CHOLESKY, CGNR, DENSE_SCHUR, SPARSE_SCHUR, ITERATIVE_SCHUR, JACOBI, SCHUR_JACOBI, Levenberg-Marquardt, STEEPEST_DESCENT, NONLINEAR_CONJUGATE_GRADIENT, and / or BFGS may be used.

[0041] To help avoid getting stuck in suboptimal solutions during the above process, it may be desirable to implement a coarse-to-fine optimization using the disclosed reconstruction method. This can help improve the overall accuracy of the reconstructed structure. For example, in some embodiments, the coarse-to-fine optimization can prevent the reconstruction process from getting stuck in a local solution instead of a global solution or minimum. Several potential methods for implementing this coarse-to-fine optimization method are described in detail below.

[0042] In some embodiments, minimizing the loss function may include using a coarse-to-fine optimization. For example, the coarse-to-fine optimization may include an initial lower resolution (e.g., 50 or less voxels) and a final resolution greater than the initial resolution (e.g., 200 or more voxels). The reconstruction method disclosed herein may be used first to determine the reconstructed structure for a coarser resolution. The coarse reconstructed structure and associated projection parameters may then be used as the initial input to the next iterative process with increasing resolution until the desired final resolution is obtained.

[0043] In another embodiment, a total variation technique can be used to implement coarse-to-fine optimization. For example, one way to calculate the total variation on a three-dimensional array is as follows: calculate the distance of each voxel to its neighboring voxels and add all of these distances; add the total variation to a weight coefficient and add it to the loss; at the beginning of training, set the weight coefficient of the total variation to be large so that the volume is forced to be smooth at the beginning; then gradually reduce the weight coefficient so that the volume can capture more details and therefore lead to more accurate pose and volume estimates.

[0044] In some embodiments, coarse-to-fine optimization can include using a three-dimensional array and modeling the three-dimensional structure as an implicit neural representation. For example, a neural network can take a three-dimensional coordinate as input and output a value representing the intensity of the volume at that coordinate. In some embodiments, the trainable parameters can be parameters in the neural network instead of the three-dimensional array. In some embodiments, if the volume is modeled as an implicit neural representation, coarse-to-fine optimization can be achieved by changing the position encoding in the neural network.

[0045] In some embodiments using coarse-to-fine optimization, attention to detail may be avoided at the beginning of the optimization. For example, downsampling and upsampling may be used in some embodiments. In one such embodiment, initially, the three-dimensional structure, the received image, and the projection step may be downsampled based on the original resolution of the received image (e.g., a 32x factor may be used for downsampling). In some embodiments, the three-dimensional structure, the received image, and the projection step may be upsampled (e.g., at a 2x factor) for each iteration to gradually introduce detailed information. The inventors have recognized and appreciated that this may help the optimization and avoid falling into a suboptimal solution.

[0046] In the above embodiments, the posture of the received image is unknown, and in other embodiments, the processor implementing the method disclosed herein may receive projection parameters (e.g., posture and distance parameters) associated with multiple images. Then, the system may implement the reconstruction method disclosed herein. For example, projection parameters and / or data related to the posture of the image may be received (rather than generated) from an IMU, an accelerometer, a gyroscope, a magnetometer, an encoder, or other sensors configured to measure the posture of the C-arm during imaging. In some embodiments, if the projection parameters have been received, there is no need to optimize them. For example, in some embodiments, since the projection parameters have been received, only the three-dimensional structure can be reconstructed. In some embodiments, the initial estimate may correspond to the three-dimensional structure. For example, as described above, the initial estimate may be all zeros or random numbers, but the projection parameters may be at least approximately known from the measurement results. In some embodiments, the three-dimensional structure may be projected into a two-dimensional image based on the received projection parameters, similar to the above-mentioned projection.

[0047] Figure 2 is a flow chart illustrating a method 200 for reconstructing a three-dimensional structure according to an embodiment of the present disclosure. In some embodiments, the depicted method may be implemented using the above-described processes, systems, and controllers. Figure 2 The method 200 is illustrated as a set of phases, blocks, steps, operations, or processes. Not all illustrated, listed operations may be performed in all embodiments of the method 200. Figure 2 Some additional operations not explicitly shown in the figure may also be included before, after, in between, or as part of the enumerated stages. Some embodiments of method 200 include instructions corresponding to the process of method 200 stored in a memory. These instructions can be executed by a processor (such as a processor of a controller or control system).

[0048] Some embodiments of method 200 may begin at stage 210, in which images of an object captured from different postures may be received. For example, images may be taken at different postures of an imaging device (e.g., a detector of an imaging system), such as from different positions and / or orientations of the imaging device. In some embodiments, the object may be a human patient or subject and / or an organ of a patient or subject. In some embodiments, the image of the object may be an X-ray image. In some embodiments, the image of the object may be taken at multiple postures relative to the object. In some embodiments, the multiple images may be a series of continuous images taken from multiple continuous viewpoints (corresponding to continuous postures of the imaging device), which are positioned along the path of motion of the detector of the imaging system relative to the object within the field of view of the detector.

[0049] At stage 230, a loss function may be determined based on the estimated three-dimensional structure and the captured image of the object. In some embodiments, stage 230 may optionally include stage 232, in which the estimated three-dimensional structure may be projected into a two-dimensional image using projection parameters associated with the received image. In some embodiments, stage 232 may include stage 233, in which the projection parameters may be determined by a processor. Alternatively or additionally, stage 232 may include stage 234, in which the projection parameters may be received (e.g., from user input or measurements from appropriate sensors).

[0050] For example, the estimated three-dimensional structure may initially include voxels with random or zero intensities, or some other values ​​corresponding to an initial guess. The projection parameters used to generate the projected two-dimensional image may include initial projection parameters, such as projection parameters corresponding to angular positions distributed (e.g., uniformly) along a semicircular or other appropriately shaped trajectory (e.g., each angular position corresponds to a viewing angle), or some other values ​​corresponding to the initial guess. If the actual projection parameters used to capture the image are known, these projection parameters may be used as initial projection parameters. The initial three-dimensional structure and initial projection parameters are used to generate a projected two-dimensional image for each viewing angle.

[0051] In some embodiments, stage 232 may optionally include stage 235, in which the projected two-dimensional image may be compared with the captured image of the object to determine a loss function. In some embodiments, stage 235 may optionally include stage 236, in which the comparison to determine the loss function may be a comparison between the gradient of the projected two-dimensional image and the gradient of the image of the object.

[0052] At stage 250, the reconstructed three-dimensional structure can be determined using a loss function using any method disclosed herein. For example, the determination can be made by minimizing the loss function. In some embodiments, stage 250 may optionally include stage 252, in which a coarse-to-fine optimization can be used. In some embodiments, stage 250 may optionally include stage 254, in which method 200 (including the determination of the three-dimensional structure) can be performed without a position sensor or a reference. Here, a loss function can also be used to determine the reconstructed projection parameters, such as when the projection parameters are not measured values, but are iteratively derived via minimizing the loss function to correspond to the viewing angle of the captured image. In each iteration, the three-dimensional structure and / or projection parameters can be reconstructed to include a value that minimizes the difference (defined by the loss function) between the projected two-dimensional image and the captured image.

[0053] In some embodiments, as described above, at least some portions of stages 230 and 250 may be repeated as needed. For example, method 200 may then proceed to stage 270, in which it may be checked whether convergence has been reached, at which the difference or other loss function between the captured image and the projected image is within a threshold. If convergence has not occurred, method 200 may return to at least some portion of stage 230 for the next iteration. For example, in stages 230 and 250 in the next iteration, the reconstructed three-dimensional structure (used as the estimated three-dimensional structure in the next iteration), the reconstructed projection parameters (used as the determined projection parameters in the next iteration), and the captured image may be used to further refine the three-dimensional structure and / or projection parameters. In this iteration, the three-dimensional structure and / or projection parameters have more accurately reconstructed values ​​than their initial values. Additional iterations will further improve the accuracy of the reconstructed three-dimensional structure and / or projection parameters. Alternatively, if convergence has occurred, method 200 may end or repeat as needed.

[0054] Example: Reconstruction of 3D structures

[0055] Figure 3 A series of images are shown showing a comparison of a two-dimensional projection 310 generated according to an embodiment of the present disclosure and a received image 320, as well as a comparison between their gradients (330 and 340, respectively). Similar to the process described above, the projection based on the initial estimate of the three-dimensional structure and the estimate of the projection parameters (e.g., orientation in this example) is refined through multiple iterations of minimizing the loss function associated with the illustrated gradients until the loss function converges (e.g., the gradient difference between the projected image and the received image is less than a predetermined threshold). This provides a reconstructed structure that closely matches the actual imaged object, as shown by the matching generated projected images and the corresponding received images and the illustrated gradients in the final image set.

[0056] Example: Coarse-to-fine optimization

[0057] Figure 4 is a series of images showing the increase in detail obtained during the coarse-to-fine optimization shown in two-dimensional projections generated at three different orientations (410, 420, and 430). Figure 3The optimization shown and described elsewhere in this paper is used to determine the structure and projection parameters of initial rough reconstruction, wherein the received image is modified to have an initial low resolution of 8 pixels by 8 pixels. Then the resulting rough estimated structure and projection parameters of the reconstruction are used as initial estimation to input into the subsequent iteration of the optimization process of the increased resolution of the structure with image and reconstruction. This iterative process is continued for the resolution of 16 pixels by 16 pixels, 32 pixels by 32 pixels, 64 pixels by 64 pixels and 128 pixels by 128 pixels. However, the process can be carried out with any desired resolution set (including a resolution greater than or less than the above-mentioned resolution).

[0058] One or more elements in the embodiments of the present disclosure may be implemented with software to be executed on a processor (such as controller 120) of a computer system. When implemented with software, the elements of the embodiments of the present disclosure are essentially code segments that perform necessary tasks. The program or code segment may be stored in a processor-readable storage medium or device, which may have been downloaded in the form of a computer data signal embodied in a carrier wave via a transmission medium or a communication link. The processor-readable storage device may include any medium that can store information, including optical media, semiconductor media, and magnetic media. Examples of processor-readable storage devices include electronic circuits, semiconductor devices, semiconductor memory devices, read-only memories (ROMs), flash memories, erasable programmable read-only memories (EPROMs), floppy disks, CD-ROMs, optical disks, hard disks, or other storage devices. The code segment may be downloaded via a computer network (such as the Internet, an intranet, etc.).

[0059] It should be noted that the processes and displays presented may not be inherently related to any particular computer or other device. The structures required for various such systems will appear as elements in the claims. In addition, the embodiments of the present disclosure are not described with reference to any particular programming language. It should be understood that various programming languages ​​can be used to implement the teachings of the disclosure described herein.

[0060] Although the present teaching has been described in conjunction with various examples, it is not intended that the present teaching is limited to such examples. On the contrary, as will be appreciated by those skilled in the art, the present teaching encompasses various alternatives, modifications and equivalents. Therefore, the above description and accompanying drawings are only examples.

Claims

1. A three-dimensional structure reconstruction system, comprising: At least one processor configured to: receiving a plurality of X-ray images of an object, wherein the plurality of X-ray images of the object are captured from different poses; The loss function is determined based on the following terms: The estimated three-dimensional structure, and the plurality of X-ray images of the object; as well as The reconstructed three-dimensional structure is determined by minimizing the loss function. 2 . The system of claim 1 , wherein the at least one processor is further configured to project the estimated three-dimensional structure into a projected two-dimensional image using projection parameters.

3. The system of claim 2, wherein the at least one processor is further configured to determine projection parameters of the reconstruction by minimizing the loss function.

4. The system according to any one of claims 2-3, wherein the projection parameters and the reconstructed three-dimensional structure are determined together in time.

5. The system of any one of claims 2-4, wherein the projection parameters correspond to the different postures from which the plurality of X-ray images were captured.

6. The system of any one of claims 2-5, wherein the at least one processor is configured to determine the projection parameters without using information derived from position sensors or fiducial markers.

7. The system of any one of claims 2-6, wherein the loss function defines a difference between the projected two-dimensional image and the plurality of X-ray images. 8 . The system of claim 7 , wherein determining the loss function comprises comparing a gradient of at least one of the two-dimensional images to a gradient of at least one of the plurality of X-ray images.

9. The system of any one of claims 1-8, wherein minimizing the loss function comprises using coarse-to-fine optimization.

10. The system of claim 9, wherein an initial three-dimensional structure comprises a resolution of 50 voxels or less, and a subsequent three-dimensional structure generated using the coarse-to-fine optimization comprises a resolution of 200 voxels or more.

11. The system of any one of claims 1-10, wherein the estimated three-dimensional structure initially comprises voxels having random or zero intensity.

12. The system of any one of claims 1-11, wherein the at least one processor is further configured to project the estimated three-dimensional structure into the projected two-dimensional image using projection parameters, and the projection parameters initially include angular positions distributed along a trajectory.

13. At least one non-transitory computer readable medium having instructions thereon which, when executed by at least one processor, perform a method for three-dimensional structure reconstruction, the method comprising: receiving a plurality of X-ray images of an object, wherein the plurality of X-ray images of the object are captured from different poses; The loss function is determined based on the following terms: The estimated three-dimensional structure, and the plurality of X-ray images of the object; as well as The reconstructed three-dimensional structure is determined by minimizing the loss function.

14. The at least one non-transitory computer-readable medium of claim 13, wherein the method further comprises projecting the estimated three-dimensional structure into a projected two-dimensional image using projection parameters.

15. The at least one non-transitory computer-readable medium of claim 14, wherein the method further comprises determining projection parameters of the reconstruction by minimizing the loss function.

16. At least one non-transitory computer-readable medium according to any one of claims 14-15, wherein the projection parameters and the reconstructed three-dimensional structure are determined together in time.

17. The at least one non-transitory computer-readable medium of any one of claims 14-16, wherein the method further comprises receiving the projection parameters.

18. The at least one non-transitory computer-readable medium of any one of claims 13-17, wherein the loss function defines a difference between the projected two-dimensional image and the plurality of X-ray images.

19. A method for three-dimensional structure reconstruction, the method comprising: receiving a plurality of X-ray images of an object, wherein the plurality of X-ray images of the object are captured from different poses; The loss function is determined based on the following terms: The estimated three-dimensional structure, and the plurality of X-ray images of the object; as well as The reconstructed three-dimensional structure is determined by minimizing the loss function.

20. The method of claim 19, further comprising projecting the estimated three-dimensional structure into a projected two-dimensional image using projection parameters.

21. The method of claim 20, further comprising determining projection parameters of the reconstruction by minimizing the loss function.

22. The method according to any one of claims 20-21, wherein the projection parameters and the reconstructed three-dimensional structure are determined together in time.

23. The method of any one of claims 20-22, further comprising receiving the projection parameters.

24. The method of any one of claims 19-23, wherein the loss function defines a difference between the projected two-dimensional image and the plurality of X-ray images.

Images