Multi-scale image reconstruction of 3D objects

Abstract

A technique for reconstructing an image of a three-dimensional object. In one example, a projection image dataset can be obtained from an imaging data detector, and a reduced image dataset and an image residual dataset can be generated. A first scale image reconstruction of the three-dimensional object can be generated using a reconstruction technique and the reduced image dataset, and a second scale image reconstruction of the three-dimensional object can be generated using an iterative reconstruction technique and the image residual dataset. A multi-scale image reconstruction of the three-dimensional object can be generated using the reconstruction technique and the first scale image reconstruction and the second scale image reconstruction.

Description

[0001] Related Applications

[0002] This application is related to U.S. Provisional Application No. 62 / 731,652 filed on September 14, 2018, which is incorporated herein by reference. Technical Field Background Art

[0003] Tomography is imaging performed by using any kind of penetrating waves through cross sections or slices. Tomographic reconstruction can be a mathematical procedure for reconstructing an image of an object. For example, X-ray computed tomography can produce images from multiple projection radiographs. Tomographic reconstruction is a multidimensional inverse problem involving the challenge of producing an estimate of a particular object from a finite number of projections. Summary of the invention

[0004] One aspect of the present technology is to provide low-latency tomographic reconstruction through multi-scale image decomposition of projection images and multi-scale image reconstruction of three-dimensional objects. In one aspect, low latency refers to the latency associated with transmitting projection image data to computing resources for tomographic reconstruction. The present technology reduces the latency by decomposing at least a portion of the projection image into a reduced image set that is smaller in size than the projection image, and the reduced image set can be transmitted to a remote data center (e.g., "cloud") over a network for processing. Decomposing the projection image into a reduced image set allows the projection image data to be transmitted faster over the network. In addition, projection image data determined to be more relevant can be transmitted first through a communication bottleneck to allow the projection image data to be used to generate tomographic reconstruction, and then, over time, as less relevant data is received, the tomographic reconstruction can be refined. On the other hand, low latency refers to the latency associated with generating tomographic reconstruction. The present technology can use computing resources of a remote data center to process projection image data in parallel, which can reduce the latency associated with generating tomographic reconstruction. On the other hand, low latency refers to the latency associated with displaying tomographic reconstruction to a display device. In one instance, the present technology reduces this latency using a multi-resolution, multi-scale reconstruction process that first generates a downscaled image reconstruction to be displayed to a display device and incrementally improves the quality of the image reconstruction by generating additional higher resolution and / or depth image reconstructions to be displayed to the display device.

[0005] Thus, the present technology can provide low latency tomographic reconstruction by parallelizing a reconstruction method that uses a series of two-dimensional images to create a three-dimensional image, wherein the reconstruction process is divided into multiple reconstructions using multiple progressive inputs to generate a tomographic reconstruction, thereby benefiting interventional imaging systems that require fast tomographic reconstruction. For example, when used with an interventional system such as CT (computed tomography) or CBCT (cone beam computed tomography), the present technology can reduce reconstruction latency, thereby allowing 3D (three-dimensional) imaging modalities to be used as image guidance systems during procedures (e.g., 0.1s and above), providing faster 3D imaging. Many surgical solutions that suffer from latency issues (e.g., closed-loop systems (such as robotic systems), automatic injection of contrast agents, automatic injection of cements in kyphosis / vertebroplasty, or remote surgery and remote imaging in challenging environments (such as rural areas, imaging in space or underwater environments, etc.)) can be improved.

[0006] As a specific example, the present technology can be used to improve the performance of tomographic systems, such as those described in U.S. Patent No. 10,070,828, U.S. Application Publication No. US-2017-0200271-A1, and International Application Publication No. WO 2019 / 060843 (collectively referred to as "nView systems"), by reducing latency and enabling virtual fluoroscopy (i.e., computer-generated fluoroscopy-like projections from tomographic images), which are incorporated by reference. A challenge with systems that provide real-time cone beam tomosynthesis systems such as nView for standard fluoroscopy is that x-rays are projected at an angle (i.e., not perpendicular to the x-ray detector), which can result in images that may be disorienting to the user. One option that can be used to address this challenge is to generate virtual projections based on tomographic reconstructions. However, in the past, the extended latency associated with acquiring projection data, reconstructing tomographic images, and then generating virtual projections to the user (i.e., greater than 20-30 seconds, and sometimes longer) has made this option less preferred. The present technology can be used to reduce the latency associated with acquiring projection data and reconstructing tomographic images to render virtual projections using the reconstructed images for display to a user, making tomographic reconstruction a more desirable option for near real-time fluoroscopy. For example, reconstruction resolution, field of view, and / or image depth can be increased while providing faster tomographic reconstruction. The present technology can also be used with a SaaS (software as a service) business model to provide low-latency tomographic reconstruction and / or virtual fluoroscopy (e.g., a real-time imaging mode based on fast tomography, where the images presented to the user include computer-generated projections through volume reconstruction).

[0007] Thus, the more important features of the invention have been outlined rather broadly so that the following detailed description may be better understood and the contribution of the invention to the prior art may be better appreciated. Other features of the invention will become more apparent from the following detailed description of the invention in conjunction with the accompanying drawings and claims, or may be learned by practice of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a flow chart illustrating an example method for tomographic reconstruction of a three-dimensional object using multi-scale decomposition and multi-scale reconstruction.

[0009] Figure 2 is a flow chart illustrating an example method for tomographic reconstruction of a three-dimensional object using multi-scale decomposition and multi-scale reconstruction using parallel processing.

[0010] Figure 3 is a flow chart illustrating an example method for performing tomographic reconstruction of a three-dimensional object using batch data for multi-scale decomposition and multi-scale reconstruction.

[0011] Figure 4 is a flow chart illustrating an example method for generating a multi-scale reconstruction of a three-dimensional object using parallel computing in a remote data center to process layers of the multi-scale reconstruction.

[0012] Figure 5 is a block diagram illustrating an example imaging system configured to reconstruct a multi-scale image of a three-dimensional object using an iterative reconstruction technique.

[0013] Figure 6 is a block diagram illustrating an example system including an imaging system in network communication with computing resources in a remote data center.

[0014] Figure 7 is a flow chart illustrating an example method for reconstructing a three-dimensional object using multi-scale decomposition and multi-scale reconstruction.

[0015] Figure 8 is a block diagram illustrating an example of a computing device that can be used to perform a method for tomographic reconstruction of a three-dimensional object using multi-scale decomposition and multi-scale reconstruction.

[0016] Fig. 9A is a high-resolution final reconstruction of the spine model using an example reconstruction method consistent with the present disclosure.

[0017] Fig. 9B yes Fig. 9A First scale reconstruction of the spine model in .

[0018] Fig. 9C yes Fig. 9A Final reconstruction of the second scale of the spine model.

[0019] These figures are provided to illustrate various aspects of the invention and are not intended to be limiting in scope with respect to size, material, configuration, arrangement, or scale, unless otherwise limited by the claims. DETAILED DESCRIPTION

[0020] Although these exemplary embodiments are described in sufficient detail to enable those skilled in the art to practice the present invention, it should be understood that other embodiments may be implemented and various changes may be made to the present invention without departing from the spirit and scope of the present invention. Therefore, the following more detailed description of the embodiments of the present invention is not intended to limit the scope of the present invention as required, but is presented only for the purpose of illustration, without limiting the features and characteristics of the present invention, setting forth the best operating mode of the present invention and being sufficient to enable those skilled in the art to practice the present invention. Therefore, the scope of the present invention is limited only by the appended claims.

[0021] definition

[0022] In describing and claiming the present invention, the following terminology will be used.

[0023] Unless the context clearly dictates otherwise, the singular forms "a," "an," and "the" include plural referents. Thus, for example, reference to "constraint" includes reference to one or more such values ​​and reference to "binning" refers to one or more such steps.

[0024] As used herein, for convenience, multiple items, structural elements, constituent elements and / or materials may be presented in a common list. However, these lists should be interpreted as if each member of the list is individually identified as a single and unique member. Therefore, in the absence of contrary indications, individual members in such a list should not be interpreted as de facto equivalents of any other members in the same list solely based on their presentation in a common group.

[0025] As used herein, the term "at least one of" is intended to be synonymous with "one or more of." For example, "at least one of A, B, and C" expressly includes only A, only B, only C, and combinations of each of them.

[0026] Any steps recited in any method or process claim may be performed in any order and are not limited to the order presented in the claims. Means-plus-function or step-plus-function limitations shall be used only if all of the following conditions apply to a particular claim limitation: a) "means for" or "step for" are clearly recited; and b) the corresponding function is clearly recited. The structure, material, or act that supports the means-plus-function is clearly recited in the description herein. Therefore, the scope of the present invention should be determined solely by the appended claims and their legal equivalents, rather than by the description and examples given herein.

[0027] This technology

[0028] Techniques for providing low-latency tomographic reconstruction of a three-dimensional object in an imaging system are described. In one example, projection images may be captured using imaging techniques including, but not limited to: medical imaging, computed tomography (CT), tomosynthesis (including real-time cone beam tomosynthesis), diagnostic imaging, interventional and surgical imaging to enable virtual fluoroscopy (e.g., a near real-time imaging mode based on rapid tomography, where the image presented to the user includes projections generated by a computer through volume reconstruction), other x-ray based imaging, magnetic resonance imaging (MRI), elastic imaging, ultrasound, ultrasound transmission, etc. Regardless of the image data source, the projection images may be included in the projection image data set to allow the use of iterative reconstruction techniques to generate reconstructed image data of the three-dimensional object.

[0029] As part of reducing the time delay associated with generating reconstructed image data of a three-dimensional object, a projection image data set can be decomposed into a reduced image data set and a residual image data set. The reduced image data set can have a smaller amount of data than the amount of data of the projection image data set, and the residual image data set can indicate an image difference between the projection image data set and the reduced image data set. In one example, a data compression technique can be used to generate the reduced image data set, and the difference between the reduced image data set and the projection image data set can be calculated to form the residual image data set. In other aspects, the reduced image data set can be considered as a primary layer data set, and the residual image data set can be considered as a secondary layer data set. As described in more detail later, the secondary layer data set can be further decomposed into multiple additional layers.

[0030] After decomposing the projection image data set into a reduced image data set and a remaining image data set, the data set (i.e., the reduced image data set and the remaining image data set) can be transmitted to a computing resource configured to reconstruct the image data of a three-dimensional object using the data set. In particular, the above-mentioned projection image data set can be a portion of the entire image data set or the acquired image. Therefore, the process described herein can be applied to a subset of image data. For example, the decomposed projection image data set can be an entire stack of 2D images (e.g., a complete sinogram, a 3D data set), a single 2D projection, a limited image fragment (i.e., a smaller fragment of a larger image), etc. A relevant subset of the image data can be identified, and the subset of the image data can be included in the reduced image data set to generate a relevant portion (e.g., an anatomical structure) of a first scale image reconstruction, which is a higher quality representation than other portions of the first scale image reconstruction. Similarly, the reduced image data set can be a reduced resolution image data set, a reduced depth image data set, a multi-resolution image, and / or a multi-depth image. As used herein, "resolution" can refer to the number of multiple pixels or voxels in an image, and / or the depth (i.e., chroma or grayscale) of an image. A reduced multi-resolution image dataset may include portions of the image having a higher resolution, while other portions have a reduced resolution. As an example, a central portion of the image may be maintained at a fully uncompressed resolution, while outer portions of the image are a reduced resolution. A similar approach may be applied to depth (i.e., chrominance or grayscale) by allowing portions of the image dataset to remain at full depth, while other portions are a reduced depth (e.g., 8 bits instead of 24 bits, or 2 bits instead of 24 bits, etc.). When acquiring multi-color or multi-dimensional images, such as in a multi-energy x-ray data acquisition utilizing a photon counting detector or dual energy exposure, the reduced dataset may be a monochrome image (represented in grayscale), and the remaining dataset may contain one or more color channels (e.g., to encode the multi-energy components of the image).

[0031] As one example, the data set may be transmitted to a remote data center (e.g., the "cloud") that hosts computing resources configured to use the data set to reconstruct image data of the three-dimensional object. As another example, computing resources included in a local imaging system (e.g., an imaging system for capturing projection images) may be used to reconstruct image data of the three-dimensional object using the data set. By decomposing the projection image data set into a reduced image data set and a remaining image data set, the size of the projection image data set can be reduced to allow for reduced latency associated with transmitting the reduced image data set and the remaining image data set to the computing resources for reconstructing image data of the three-dimensional object.

[0032] An iterative reconstruction technique may be used to generate reconstructed image data of the three-dimensional object. In one example, the reconstruction technique may include using the reduced image data set to generate a first scale image reconstruction of the three-dimensional object (e.g., a first image volume representing the three-dimensional object), and using the remaining image data set to generate a second scale image reconstruction of the three-dimensional object (e.g., a second image volume representing the three-dimensional object). In some cases, the reconstruction technique may be an iterative reconstruction.

[0033] In an alternative, a residual image data set can be obtained based on a lower resolution and / or lower depth description of a first scale image reconstruction of a three-dimensional object. When the reconstruction is low resolution and low depth, a description of the reconstruction is obtained, and then the residual image data set can be identified by projection and comparison with the initial image. Therefore, the size of the first data set is not always reduced. For example, a first layer with a low resolution projection image can be used to describe the low resolution reconstruction, and then a full resolution image is taken and compared with the projection of the lower resolution reconstruction to create a residual image, which is then reconstructed into a second scale reconstruction. An advantage of this alternative method is that a simpler (and therefore faster in some cases) reconstruction method can be used, because any modeling or mathematical errors produced in the first scale reconstruction can be recovered by the second scale reconstruction. One such simpler reconstruction method is the filtered back projection method, which is faster but less accurate than other iterative methods.

[0034] When the first scale image reconstruction and the second scale image reconstruction are generated, the multiscale reconstruction of the three-dimensional object (e.g., a third image volume representing the three-dimensional object) can be generated using the first scale image reconstruction and the second scale image reconstruction. In one example, the delay associated with displaying the three-dimensional object to the display device can be reduced by directly providing the first scale image reconstruction for display to the display device after the first scale image reconstruction has been generated. In doing so, the image of the three-dimensional object can be provided to the user before the higher quality image of the three-dimensional object is generated. Subsequently, after the second scale image reconstruction is generated, the second scale image reconstruction can be provided to the display device for display, thereby providing the user with an image of the three-dimensional object of higher quality than the image of the first scale image reconstruction. When the multiscale reconstruction of the three-dimensional object becomes available, the multiscale reconstruction can be provided for display to the display device, thereby providing an image of the three-dimensional object of higher quality than the image of the second scale image reconstruction. As used herein, "higher quality" can refer to higher image resolution and / or higher depth (chroma or grayscale). The delay can typically be less than about twenty seconds, and the delay is most often less than about five seconds depending on the image resolution, available computing resources, and image content. In some low-resolution situations, the delay can be less than one second. In any case, in the context of the present invention, the terms "real time" and "near real time" refer to less than 5 seconds. These same principles can also be applied to individual projections rather than a complete three-dimensional image. In either case, a first reduced image is reconstructed, and then the remainder of the image can be reconstructed independently of the first reduced image. Although the reconstructed image is typically displayed to a display device for human viewing, the reconstructed image can also be used as input to a computer vision algorithm. For example, if the image is used by a robotic system, the computer vision algorithm can process the layers without reassembly.

[0035] To further describe the present technology, examples are now provided with reference to the accompanying drawings. Figure 1 1 is a flow chart illustrating an example method 100 for tomographic reconstruction of a three-dimensional object using multi-scale decomposition and multi-scale reconstruction. As in block 102, projection images may be acquired from one or more imaging data detectors configured to detect x-ray radiation from one or more radiation sources. The projection images may be one-dimensional images (e.g., generated using a CT scanner) or two-dimensional images (e.g., generated using a CT scanner or a CBCT scanner), wherein the projection images may include a complete set of available images or a subset of available images. In some examples, pre-processing of the projection images, such as denoising, scale change, etc., may be performed before acquiring the projection images for tomographic reconstruction of the three-dimensional object.

[0036] As in block 104, the multi-scale decomposition process may be used to generate a reduced image (shown as “I 1 ”) and generates a remaining image (shown as “I2 ”). Note that step 104 is optional. When omitted, the first layer image reconstruction 106 can be performed on the complete image data set, and as shown in step 108, the V 1 The forward projection of I 2 However, a reduced resolution image may be generated having a lower resolution than the resolution of the projection image (shown as "I" in box 104) acquired from one or more imaging data detectors. A variety of lossy data compression techniques may be used to generate the reduced resolution image. For simplicity, we refer to reduced resolution as any lossy data compression of the original image, with reducing the resolution being the most common compression technique.

[0037] In one example, the projected image can be reduced to a smaller size by encoding each scale of the projected image with a smaller depth to produce a lower depth image with fewer bits than the projected image. For example, a 16-bit or floating point value projected image can be scaled to 8 bits to produce a reduced resolution image. As an example, combining image merging with 8-bit encoding to divide the resolution by 2 on each image axis can reduce the size of a 16-bit projected image to a reduced resolution image that is one-eighth the size of the projected image. Illustratively, when the projected image is decomposed into an 8-bit depth reduced resolution image, a GPU (graphics processing unit) can advantageously use 8-bit fixed-point operations to further accelerate the reconstruction of three-dimensional objects using reduced resolution images.

[0038] As another example, the projected image can be reduced by generating a blurred version of the projected image before the images are merged. In another example, a most significant bit technique (e.g., a bit tree) can be used to identify a portion of the projected image to be included in the reduced image, and the remainder of the projected image can be left for later use. In some examples, a combination of the above techniques can be used to generate a reduced image. Illustratively, given the compression properties of the reduced image (e.g., due to depth, the reduced image can be half the size of the projected image, and by halving the resolution on each axis of the projected image, the reduced image can be one-quarter the size of the projected image, resulting in one-eighth the total size of the projected image), the reduced resolution image can be transmitted in a portion of the time required to transmit the projected image. As will be appreciated, the present technology is not limited to the above-mentioned data compression techniques. The present technology can use any type of data compression to generate a reduced image.

[0039] The residual image generated in box 104 can be generated to indicate the image resolution difference between the projection image and the reduced image obtained from one or more imaging data detectors. In one example, the residual image can be obtained by determining the resolution difference between the projection image and the reduced image. In another example, the first scale image reconstruction (described below) can be forward projected (FPJ) to produce a residual image. The size of the residual image can be approximately smaller than the size of the full resolution image. In one example, in order to further reduce the delay associated with transmitting and processing larger images, the residual image can be further decomposed by compressing the first residual image set using a lossy compression technique to generate a second residual image set used in reconstructing the three-dimensional object and discarding the first residual image set. Lossy compression techniques may include, but are not limited to, JPEG, JPEG2000, H.264, etc. Lossless compression may include, but are not limited to, ZIP, GZIP, sparse image coding, etc.

[0040] As in block 106, the downscaled image may be used to generate a first scale image reconstruction of the three-dimensional object (shown as “V 1 ”). Due to the reduced resolution, reconstruction of image data for a three-dimensional object can be performed in a shorter amount of time than reconstructing the three-dimensional object using a higher resolution projection image. The first scale image reconstruction can be performed using an iterative forward / back projection process or a non-iterative reconstruction process (e.g., filtered back projection). For example, an image volume X including the reduced image can be forward projected (FPJ) to produce a two-dimensional (2D) set of projections. The difference between the reduced image and the forward projection of the image reconstruction can be back projected (BPJ) into 3D space to obtain an updated volume ε. The iterative process can be repeated until an exit criterion is detected. In one example, the exit criterion can be a mathematical norm of the corresponding residual image, or a metric derived from the corresponding residual image can be used to define the exit criterion. In one example, the FPJ and BPJ operators can be matrix operations. For example, the FPJ and BPJ can be transposed matrices that can be calculated once and then used in each iteration step. The matrices can be large to store in computer memory, and therefore, it may be advantageous to use computing resources provided by a computing service provider and a fixed point implementation.

[0041] The iterative forward / backward projection process can be used to operate on batches of images (e.g., reduced images) (such as ordered subsets) to reconstruct an image of a three-dimensional object, which can reduce the delay associated with waiting for a complete set of images to begin reconstructing an image of the three-dimensional object. For example, if the reconstruction process uses a batch of one image (or a variable batch size, but starts with one image), the delay associated with starting the reconstruction is the time to transfer the first image from the image capture device (e.g., an imaging detector) to the computing resource used to reconstruct the image of the three-dimensional object. A similar subset approach can be taken on a "per image portion" basis, down to a "per ray basis" or a "per pixel basis". In these examples, a subset of data used to reconstruct a three-dimensional object can be started when a portion of an image is received and down to a single image pixel. Batch processing can be advantageously used to match specific memory and computing power. For example, at one extreme, each single ray or pixel can be processed with virtually no memory and no computer parallelization capabilities, and can be suitable for very simple and fast processors (e.g., quantum computers). At the other extreme, large batches can be better suited to multi-GPU cloud environments with large amounts of available memory and the ability to parallelize computing. In practice, a scheme where batches are variable and increase in size over time is advantageous because it minimizes latency due to data availability and initial transmission, while minimizing computation latency later as more data becomes available and can be exploited by parallelization.

[0042] In one example, machine learning can be used to generate or modify a first scale image reconstruction of a three-dimensional object. For example, after obtaining an updated volume ε, a deep learning regularization matrix such as a neural network (NN) can produce an updated solution. The deep learning regularization matrix can operate in a registration framework to enable the NN to add features and remove artifacts. In some examples, density constraints can be applied when generating a first scale image reconstruction. Density constraints may include, but are not limited to, empty space, known objects and medical devices, anatomical features, object boundaries, etc. In some examples, a regularization matrix can be used to introduce object features and density constraints into the first scale image reconstruction. For example, the regularization matrix in tomographic reconstruction can be a smoothness constraint (e.g., total variation) and a density constraint (e.g., positivity constraint). In addition, blurring, denoising, or total variation can be used as a regularization matrix. The second layer may require a very different regularization matrix, usually symmetric and non-fuzzy properties. Each step of the iterative reconstruction process can improve the quality of the first scale image reconstruction, and the regularization matrix can be used to limit the solution space by incorporating prior knowledge about the three-dimensional object. For example, a positivity constraint or a smoothness constraint during first scale image reconstruction may be used to enforce the knowledge that the projection image of the imaged three-dimensional object contains only positive values ​​or that the projection image of the three-dimensional object has a specified degree of smoothness over homogeneous regions, respectively.

[0043] In some examples, the first scale image reconstruction of the three-dimensional object can be displayed to a display device (e.g., a monitor, a touch screen, etc.) during reconstruction of the three-dimensional object and / or after reconstruction of the three-dimensional object. For example, the first scale image reconstruction can be provided for display on a display device to allow a user to view the first scale image reconstruction of the three-dimensional object prior to reconstruction of a multi-scale reconstruction (e.g., a high-resolution and / or deep reconstruction) of the three-dimensional object. Displaying the first scale image reconstruction reduces the latency associated with generating a higher quality reconstructed image of the three-dimensional object and displaying the three-dimensional object to the display device once the image is available.

[0044] As in block 108, an updated multi-scale decomposition may be performed, where the updated multi-scale decomposition may be performed by applying a forward projection (FPJ) operator to a first scale image reconstruction (shown as “V 1 ”) and determines the resolution / depth difference between the result and the projected image (shown as “I” in box 108) to update the remaining image (shown as “I” in box 108). 2 ”). In one example, referring to blocks 104 and 108, m I 2 The image can be angled with n I 1 The images may be n images with the same angle, or a different number of images at any angle. If the angles do not match, sampling between the n images or the available images I can be performed.

[0045] As in block 110, the remaining image (shown as “I 2 ”) to generate a second scale image reconstruction of the three-dimensional object (shown as “V 2 ”). The second scale image reconstruction can be the same as the first scale image reconstruction or have a higher resolution / depth than the first scale image reconstruction. The second scale image reconstruction can be performed using the iterative forward / back projection process previously described in conjunction with block 106. In one example, the second scale image reconstruction can have a higher quality than the first scale image reconstruction by having a smaller region of interest. For example, the second scale image reconstruction can be a higher resolution and / or depth image of the region of interest within the area contained in the first scale image reconstruction. As an example, one or more regions of interest contained within the first scale image reconstruction can be identified, and the second scale image reconstruction can be generated to provide a higher resolution and / or depth image of the one or more regions of interest. In one example, the region of interest can be dynamically selected (e.g., based on a view of the first scale image reconstruction selected by a user, or based on an identified feature of a three-dimensional object), or the region of interest can be manually selected (e.g., by manually selecting a region of the first scale image reconstruction of the three-dimensional object displayed to the touch screen device via the touch screen device).

[0046] In another example, the second scale image reconstruction may be of the same resolution and / or depth as the first scale image reconstruction, and the second scale image reconstruction may be shifted in position to create a volume interlaced with the first scale image reconstruction. This avoids encoding and processing zero values ​​at overlapping nodes of the image. In some instances, the second scale image reconstruction of the three-dimensional object may be generated based on a grid that is selected to not include any nodes in common with the first scale image reconstruction in order to improve the computational efficiency associated with generating the second scale image reconstruction. For example, a staggered grid with a half-voxel offset of the same resolution may be used to generate the second scale image reconstruction. In some examples, after the second scale image reconstruction of the three-dimensional object has been generated, the second scale image reconstruction may be provided directly for display on a display device to allow a user to view the second scale image reconstruction of the three-dimensional object before the reconstruction of the multi-scale reconstruction of the three-dimensional object.

[0047] As in block 112, a first scale image reconstruction (shown as “V 1 ”) and a second scale image reconstruction (shown as “V 2 ”) to generate a multiscale image reconstruction of the three-dimensional object (shown as “V” in box 112). Multiple layers (e.g., resolutions, regions of interest, and / or image depths included in the first scale image reconstruction and the second scale image reconstruction) can be sequentially obtained to be included in the multiscale image reconstruction, and the layers can be combined to generate the multiscale image reconstruction of the three-dimensional object. In some examples, the multiscale image reconstruction can take into account the region of interest, which in some cases can involve resampling using GPU textures. The multiscale image reconstruction of the three-dimensional object can have a higher quality than the first scale image reconstruction and the second scale image reconstruction. For example, the multiscale reconstruction can have a higher image resolution and / or a higher depth than the first scale image reconstruction and the second scale image reconstruction.

[0048] As in box 114, a multiscale image reconstruction of a three-dimensional object (shown as "V" in box 114) can be displayed to a display device. The visualization of the three-dimensional object can include slices, three-dimensional rendering, projection, etc. of volume data. Visualization can also include post-processing. Such processing can include denoising, interpolation, resampling, nonlinear operations (such as gamma correction and dimensional expansion via color mapping, to name a few examples). The display of the image reconstruction (e.g., first scale image reconstruction, second scale image reconstruction, or multiscale image reconstruction) can be provided to a local display, or to a remote display, and in the case where the processing is performed in a remote data center, the image can be rendered at the remote data center and the rendering of the image can be provided for display to a local display device. This remote-to-local display transmission can be achieved, for example, via video compression such as H.264. Compression techniques such as JPEG2000 decompose images into layers. It can be advantageous to match the multiscale reconstruction layers with the compression technology input by reducing the encoding time and thus reducing the latency. Similarly, when the images are paired with a computer vision algorithm as an image consumer (e.g., for robotic surgery), it may be advantageous to match the multi-scale reconstruction layers to the computer vision algorithm input (e.g., the residual layers typically have differential content that can be directly exploited by the algorithm to identify edges of anatomical structures, such as Fig. 9A , 9B and 9C). Fig. 9A is the high-resolution final reconstruction of the spine model. Fig. 9B is the first scale reconstruction of the spine model. Note that Fig. 9A compared to, Fig. 9B is blurry. Fig. 9C It is the final reconstruction of the second scale.

[0049] Figure 2 2 is a flow chart illustrating an example method 200 for tomographic reconstruction of a three-dimensional object using multi-scale decomposition and multi-scale reconstruction using parallel processing. As in block 202, a projection image dataset including a plurality of projection images may be acquired to allow reconstructed image data of a three-dimensional object to be generated using an iterative reconstruction technique. The projection images may include projection data generated by an imaging detector in response to detecting x-ray radiation from a radiation source. As in combination Figure 1 As described in detail, as in box 204, a multi-scale decomposition of the projection image dataset can be performed to generate a reduced image for each of the multiple projection images included in the projection image dataset, and a residual image is generated for the reduced image, which indicates the image difference between the multiple projection images and the reduced image.

[0050] As in box 206 and box 208, after decomposing at least a portion of the projection image data set into a reduced image and a residual image data set, a first scale image reconstruction and a second scale image reconstruction can be performed. The first scale image reconstruction and the second scale image reconstruction of the three-dimensional object can be performed in parallel. In one example, a remote data center ("cloud") including computing resources for generating the first scale image reconstruction and the second scale image reconstruction in parallel can be used. The computing resources can utilize a GPU cluster or a multi-GPU environment, and the GPU cluster or the multi-GPU environment can be used to generate the first scale image reconstruction and the second scale image reconstruction. Illustratively, local computing resources can be used to decompose the projection image data set into a reduced image and a residual image, after which the reduced image and the residual image can be transmitted to the remote data center for parallel reconstruction of the first scale image and the second scale image. In one example, a local computer can reconstruct the first layer, while the second layer can be processed in the cloud. In another example, computing resources can be used to generate the first scale image reconstruction and the second scale image reconstruction in parallel. One disadvantage associated with the parallel generation of the first scale image reconstruction and the second scale image reconstruction is that due to not performing an updated multi-scale decomposition (such as Figure 1 108), may not be able to resolve reconstruction incompleteness of the first scale image reconstruction, where the incompleteness can be resolved when generating the second scale image reconstruction (e.g. Figure 1 110). To compensate for this disadvantage, additional reconstruction iterations may be performed after the multi-scale image reconstruction to resolve eventual differences that may occur. In contrast, one advantage of this approach is that there is limited need to synchronize data between different computational units, making the computation more efficient. For example, if the same scale reconstruction is split into different computational units, the iteratively changing reconstruction solution must be distributed over the different computational units and merged with the solutions of other computational units, for example via averaging.

[0051] As in block 210, as previously described, a multiscale image reconstruction of a three-dimensional object may be generated using the first scale image reconstruction and the second scale image reconstruction. In the example where the first scale image reconstruction and the second scale image reconstruction are generated at a remote data center, the first scale image reconstruction and the second scale image reconstruction may be transmitted back to a local computing resource for generating a multiscale image reconstruction of the three-dimensional object, and the multiscale image reconstruction may be provided for display on a display device, as in block 212. Alternatively, a multiscale image reconstruction of a three-dimensional object may be generated at a remote data center, and the multiscale image reconstruction may be transmitted to a local computing resource for display to a display device (as in block 212), or the multiscale image reconstruction may be rendered at a remote data center and a rendering of the three-dimensional object may be provided for display to a local display device.

[0052] Figure 3is a flow chart illustrating an example method 300 for performing tomographic reconstruction of a multi-scale decomposition and multi-scale reconstruction of a three-dimensional object using batch data. The method 300 may include acquiring projection images of a three-dimensional object in batches to allow processing to begin when the first projection image is available for processing and to allow parallel processing of the projection images, thereby reducing latency caused by a slow acquisition system in providing projection images.

[0053] As in block 302a, a first projection image may be acquired, and then after acquiring the first projection image, a multi-scale decomposition process may be used to generate a reduced image and a residual image from the projection image, as in block 304a. The reduced image may be used to start a first scale reconstruction of a three-dimensional object, as in block 306a.

[0054] When the first scale reconstruction shown in box 306a is being performed, additional projection images become available, and as in box 302b, the additional projection images can be acquired. As in box 304b, a multi-scale decomposition process can be performed on the additional projection images, and as in box 306b, the first scale reconstruction of the three-dimensional object can be continued using the reduced image generated based on the additional projection images. As in box 302n, the most recent projection image can be acquired, and as in box 304n, the multi-scale decomposition process can be performed on the most recent projection image. Thereafter, as in box 306n, the first scale reconstruction of the three-dimensional object can be completed using the reduced image generated based on the most recent projection image. Examples of reconstruction methods that can be used for the first scale reconstruction of the three-dimensional object include, but are not limited to, ordered subset reconstruction methods, stochastic gradient descent methods, Nesterov methods, or other momentum-based methods, as well as other methods that can utilize batches of image data using subsets of image data at each optimization step.

[0055] As shown in block 302b, the second input (shown as “I i ”) may be the volume of the projected image, which may be equivalent to solving based on the changed data, rather than transplanting the previous solution as the initial best guess for the next step. Similarly, the “V 0 ” can be a volume of zero, or a volume of the most likely solution, the expected density of the imaged material, or a registration volume, such as an expected image derived from a database of scans or a machine learning process.

[0056] After generating the first scale image reconstruction of the three-dimensional object, as in block 308, an updated multi-scale decomposition may be performed, wherein, as in blocks 304a-n, a residual image generated from the projection image may be updated by applying a forward projection (FPJ) operator to the first scale image reconstruction and determining a resolution / depth difference between the result and the projection image. Thereafter, as in block 310, the updated residual image may be used to generate a second scale image reconstruction, and as in block 312, a multi-scale image reconstruction of the three-dimensional object may be generated using the first scale reconstruction and the second scale reconstruction. Finally, as in block 314, the multi-scale image reconstruction of the three-dimensional object may be displayed to a display device.

[0057] Figure 4 4 is a flow chart showing an example method 400 for processing multi-scale reconstruction of layers of a multi-scale reconstruction of a three-dimensional object using parallel computing in a remote data center. As shown, projection images of the three-dimensional object can be acquired in batches. After receiving the projection images, the multi-scale decomposition process described previously can be performed locally to reduce the time required to transmit the resulting reduced image and residual image to the computing resources located in the remote data center. The computing resources can be configured to use the reduced image and the residual image to start the first scale reconstruction and the second scale reconstruction of the three-dimensional object.

[0058] When additional projection images are acquired, reduced images and residual images may be generated based on the projection images, and the reduced images and residual images may be transmitted to a remote data center to allow the reduced images and residual images to be used to generate first scale reconstructions and second scale reconstructions of the three-dimensional object.

[0059] like Figure 4 As shown, the reduced image may be stored to data memory 402 and the residual image may be stored to data memory 404. As part of generating a first scale image reconstruction of the three-dimensional object, a subset of the available reduced images may be selected from data memory 402 and the subset of the reduced images may be used to generate the first scale reconstruction. A subset of the available residual images may be selected from data memory 404 and the subset of the residual images may be used to generate a second scale reconstruction of the three-dimensional object. Figure 1 The described iterative reconstruction technique can be used to generate a first scale image reconstruction and a second scale image reconstruction, and an exit criterion (e.g., a mathematical norm of the reduced image / residual image or a metric derived from the reduced image / residual image) can be used to determine when to exit the iterative reconstruction process. The resulting first scale reconstruction and second scale reconstruction can then be used to generate a multi-scale image reconstruction, which can be displayed to a display device as described above.

[0060] Figure 5is a block diagram showing an example imaging system 502 configured to reconstruct a multi-scale image of a three-dimensional object using an iterative reconstruction technique. As shown, the imaging system 502 may include an imaging modality 510 configured to generate a projection image dataset of a three-dimensional object, a computing device 512, and a display device 514, wherein the imaging modality 510 is configured to generate a projection image dataset of a three-dimensional object, the computing device 512 is used to host various modules associated with generating and displaying image reconstruction of the three-dimensional object, and the display device 514 is used to display the image reconstruction of the three-dimensional object.

[0061] The components of the imaging system 502 may be included in a workstation, or the components of the imaging system 502 may be located separately and may be configured to communicate with each other over a network (e.g., a local area network (LAN), a wide area network (WLAN), a short-range network protocol, or a cellular network such as 4G or 5G, etc.). Illustratively, the imaging system 502 may be a CT scanner or a CBCT imaging system. The imaging modality 510 may be any imaging device that combines imaging techniques such as computed tomography, radiography, fluoroscopy, and x-ray tomosynthesis, although other techniques such as elastic imaging, tactile imaging, thermal imaging, and / or medical photography and nuclear medicine functional imaging techniques (such as positron emission tomography (PET) and single photon emission computed tomography (SPECT)) may also be used. In one example, the imaging modality 510 may be a computed tomography (CT) scanner or a tomosynthesis system. As will be appreciated, imaging modalities not specifically described herein are also within the scope of the present disclosure. For example, imaging systems such as those described in U.S. Patent No. 10,070,828 and U.S. Application Publication No. 2017-0200271-A1 and PCT Application Publication No. WO 2019 / 060843 (all of which are incorporated herein by reference) are particularly effective systems for image reconstruction.

[0062] The computing device 512 may include a processor-based system and may include any such device capable of receiving projection image data from the imaging modality 510 and outputting the reconstructed projection image data to the image display module 508 and hosting the image decomposition module and the iterative reconstruction module 506. When executed on the computing device 512, the image decomposition module 504 decomposes the projection image data set into a set of image data sets having a smaller amount of data than the projection image data set. In particular, the image decomposition module 504 generates a reduced image data set and an image residual data set. The image decomposition module 504 may be configured to apply a lossy data compression technique to the projection image data set to generate a reduced image data set having a smaller amount of data than the resolution of the projection image data set. In addition, the image decomposition module 504 may be configured to generate an image residual data set by calculating the image resolution difference between the projection image data set and the reduced image data set. The reduced image data set and the image residual data set generated by the image decomposition module 504 may be provided to the iterative reconstruction module 506.

[0063] When the iterative reconstruction module 506 is executed on the computing device 512, the iterative reconstruction module 506 reconstructs an image of the three-dimensional object using an iterative reconstruction technique applied to the reduced image dataset and the image residual dataset received from the image decomposition module 504. The iterative reconstruction module 506 generates a first scale image reconstruction of the three-dimensional object using the reduced image dataset and generates a second scale image reconstruction of the three-dimensional object using the image residual dataset. When generating the first scale image reconstruction and the second scale image reconstruction, the iterative reconstruction module 506 generates a multi-scale image reconstruction of the three-dimensional object using the first scale image reconstruction and the second scale image reconstruction.

[0064] As described above, the iterative reconstruction module 506 generates image reconstructions (i.e., first scale image reconstructions, second scale image reconstructions, and multiscale image reconstructions) using an iterative reconstruction technique. In one example, the iterative reconstruction technique includes the following steps: (i) forward projecting a ground truth image volume to produce a two-dimensional projection image set, (ii) determining the difference between the projection image data set and the two-dimensional projection image set, (iii) generating an update volume by back-projecting the difference into a three-dimensional space, and (iv) incorporating the update volume into the reconstruction of the image of the three-dimensional object. In one example, a regularization matrix can be used to introduce object features and constraints (e.g., density, boundaries, curves, etc.) into the image of the three-dimensional object being reconstructed. In at least one example, a machine learning model can be used as a regularization matrix. For example, after one or more iterations of the reconstruction process, the output of the machine learning model can be provided as an input to the next iteration of the reconstruction process. In some examples, multiple machine learning models can be used as regularization matrices at different stages of the iterative reconstruction technique.

[0065] The image reconstructions (i.e., first scale image reconstructions, second scale image reconstructions, and multi-scale image reconstructions) generated by the iterative reconstruction module 506 may be provided to an image display module 508, which is configured to output the image reconstructions to a display device 514, which includes a monitor, a mobile device, or other type of display for presenting the reconstructed image to a user (e.g., a medical professional). In one example, the image reconstructions may be provided directly to the image display module 508 after being generated to reduce the latency associated with generating a higher quality image reconstruction. As an example, the iterative reconstruction module 506 may provide the first scale image reconstruction to the image display module 508 for display on the display device 514 when the first scale image reconstruction is ready. Subsequently, the iterative reconstruction module 506 may provide the second scale image reconstruction to the image display module 508, and thereafter provide the multi-scale image reconstruction for display on the display device 514, so as to increase the resolution, field of view, and / or image depth of the image reconstruction of the three-dimensional object. The visualization of the three-dimensional object provided by the image display module 508 may include three-dimensional rendering, projection, slicing of volume data, and the like.

[0066] Figure 6 It shows that including Figure 5 6 is a block diagram of an example system 600 of an imaging system 604 that is in network communication with computing resources in a remote data center 602 (e.g., a "cloud" computing environment). In this example, the imaging system 604 may include an imaging modality 608 configured to generate a projection image dataset of a three-dimensional object, a computing device 616 for hosting various modules associated with generating and displaying an image reconstruction of the three-dimensional object, and a display device 614 for displaying the image reconstruction of the three-dimensional object.

[0067] The computing device 616 may host the image decomposition module 610, the multi-scale reconstruction module 620, and the image display module 612. When the image decomposition module 610 is executed on the computing device 616, the image decomposition module 610 decomposes the projection image data set received from the imaging modality 608 into a reduced image data set and an image residual data set, and sends the data sets to the remote data center 602 for parallel processing using entities of the iterative reconstruction module 606, thereby generating a first scale image reconstruction and a second scale image reconstruction as described above.

[0068] The multi-scale reconstruction module 620 receives the first scale image reconstruction and the second scale image reconstruction sent from the remote data center 602, and provides the first scale image reconstruction and the second scale image reconstruction to the multi-scale reconstruction module 620. Thereafter, when the multi-scale reconstruction module 620 is executed on the computing device 616, the multi-scale reconstruction module 620 uses the above Figure 5 The described first scale image reconstruction and second scale image reconstruction and iterative reconstruction techniques are used to generate a multi-scale image reconstruction of the three-dimensional object, and the multi-scale image reconstruction of the three-dimensional object is provided to the image display module 612 for display on the display device 614.

[0069] The remote data center may include computing resources for entities that host iterative reconstruction modules 606. The computing resources may include servers and / or virtual machines executed on servers. Image data for reconstruction of three-dimensional objects may be sent between the remote data center 602 and the imaging system 604 via a network 618. The network 618 may include any useful computing network, including an intranet, the Internet, a local area network, a wide area network, a wireless data network, or any other such network or a combination thereof. The components used for such a network 618 may depend at least in part on the type of selected network and / or environment. Communication on the network may be achieved by wired or wireless connections and a combination thereof.

[0070] When the reconstruction module 606 hosted on the remote data center 602 is executed, the reconstruction module 606 can reconstruct an image of the three-dimensional object using a reconstruction technique applied to the reduced image dataset and the image residual dataset received from the image decomposition module 610. The reconstruction module 606 generates a first scale image reconstruction of the three-dimensional object using the reduced image dataset, and generates a second scale image reconstruction of the three-dimensional object using the image residual dataset. The generation of the reconstructed image can be performed in parallel using multiple instances of the reconstruction module 606. After generating the reconstructed image (e.g., the first scale image reconstruction or the second scale image reconstruction), the iterative reconstruction module 606 sends the first scale image reconstruction and the second scale image reconstruction to the multi-scale reconstruction module 620 located in the imaging system 604 to allow the multi-scale reconstruction module 620 to generate a multi-scale image reconstruction of the three-dimensional object for display on the display device 614.

[0071] Figure 7 7 is a flow chart illustrating an example method 700 for reconstructing a three-dimensional object using multi-scale decomposition and multi-scale reconstruction. As in block 710, a projection image dataset may be received, wherein the projection image dataset may be generated by at least one imaging detector in response to detecting x-ray radiation from at least one radiation source to allow for the generation of reconstructed image data of the three-dimensional object using an iterative reconstruction technique.

[0072] In one example, a lossy data compression technique is used to generate the reduced image data set.In some examples, the projection image data set may include ordered subsets of the projection image data to allow reconstruction of the three-dimensional object to begin prior to receiving the complete projection image set.

[0073] As in block 720, a reduced image dataset may be generated from the projection image dataset, wherein the reduced image dataset has a smaller amount of data than the amount of data of the projection image dataset received from the at least one imaging data detector. As in block 730, an image residual dataset may be generated to indicate an image resolution difference between the projection image dataset and the reduced image dataset.

[0074] As in box 740, a reconstruction technique can be performed to generate reconstructed image data of the three-dimensional object, including generating at least: (i) a first scale image reconstruction of the three-dimensional object displayed to the display device generated using the reduced image data set, (ii) a second scale image reconstruction of the three-dimensional object displayed to the display device generated using the image residual data set, and (iii) a multi-scale image reconstruction of the three-dimensional object displayed to the display device generated using the first scale image reconstruction and the second scale image reconstruction.

[0075] In one example, an iterative reconstruction technique includes: (ii) forward projecting a ground truth image volume to produce a two-dimensional projection image set, (iii) determining a difference between the projection image data set and the two-dimensional projection image set, (iii) generating an update volume by back-projecting the difference into three-dimensional space, and (iv) incorporating the update volume into a reconstruction of an image of a three-dimensional object.

[0076] In one example, after the first scale image reconstruction of the three-dimensional object is generated, the first scale image reconstruction may be directly displayed to the display device, and after the second scale image reconstruction of the three-dimensional object is generated, the second scale image reconstruction may be directly displayed to the display device.

[0077] In one example, the first scale image reconstruction and the second scale image reconstruction can be generated in parallel. In some examples, generating the first scale image reconstruction and the second scale image reconstruction of the three-dimensional object includes applying constraints (e.g., boundaries, curves, empty spaces, known objects and medical devices, anatomical features, etc.) to the reconstruction.

[0078] In one example, generating the second scale image reconstruction includes increasing the resolution of the region of interest within the second scale image reconstruction of the three-dimensional object. In some examples, the second scale image reconstruction of the three-dimensional object can have a higher resolution than the resolution of the first scale image reconstruction. In other examples, the second scale image reconstruction of the three-dimensional object can be the same resolution as the first scale image reconstruction, and the second scale image reconstruction can be shifted in position to create a volume that is interlaced with the first scale image reconstruction.

[0079] Figure 8 A computing device 810 is shown on which a service module of the present technology may be executed. A computing device 810 is shown on which a high-level example of the technology may be executed. The computing device 810 may include one or more processors 812 in communication with a memory device 820. The computing device 810 may include a local communication interface 818 for components in the computing device. For example, the local communication interface 818 may be a local data bus and / or any related address or control bus that may be desired.

[0080] The memory device 820 may contain a module 824 and data for the module 824 to provide various services, and the module 824 may be executed by the (one or more) processors 812. In one aspect, the memory device 820 may include an image decomposition module, a reconstruction module, a multi-scale reconstruction module, an image display module, and other modules. A data memory 822 may also be located in the memory device 820 for storing data related to the module 824 and other applications and operating systems that may be executed by the (one or more) processors 812.

[0081] Other applications may also be stored in the memory device 820 and may be executed by the processor(s) 812. The components or modules discussed in this specification may be implemented in software using a high-level programming language that is compiled, interpreted, or executed using a mixture of methods.

[0082] The computing device may also have access to I / O (input / output) devices 814 that may be used by the computing device. An example of an I / O device is a display screen 830 that may be used to display output from the computing device 810. Networking devices 816 and similar communication devices may be included in the computing device. Networking devices 816 may be wired or wireless networking devices that connect to the Internet, a LAN, a WAN, or other computing network.

[0083] The components or modules shown as being stored in the memory device 820 can be executed by (one or more) processors 812. The term "executable" can represent a program file in a form that can be executed by the processor 812. For example, a program in a high-level language can be compiled into machine code in a format that can be loaded into a random access portion of the memory device 820 and executed by the processor 812, or a source code can be loaded by another executable program and interpreted to generate instructions to be executed by the processor in a random access portion of the memory. The executable program can be stored in any part or component of the memory device 820. For example, the memory device 820 can be a random access memory (RAM), a read-only memory (ROM), a flash memory, a solid-state drive, a memory card, a hard drive, an optical disk, a floppy disk, a tape, or any other memory component.

[0084] Processor 812 may represent multiple processors, including but not limited to a central processing unit (CPU), a graphics processing unit (GPU), an FPGA, a quantum computer, or a cluster of the above, and memory device 820 may represent multiple memory units operating in parallel with the processing circuit. This can provide parallel processing channels for processing and data in the system. Local communication interface 818 can be used as a network to facilitate communication between any of the multiple processors and multiple memories. Local communication interface 818 can use additional systems designed to coordinate communication, such as load balancing, batch data transfer, and similar systems.

[0085] Although the flowchart presented for this technology can imply a specific execution order, the execution order may be different from the illustrated one. For example, the order of two or more frames can be rearranged relative to the order shown. In addition, two or more frames shown in succession can be executed in parallel or partially in parallel. In some configurations, one or more frames shown in the flowchart may be omitted or skipped. In order to enhance practicality, counting, performance, measurement, troubleshooting or for similar reasons, any number of counters, state variables, warning signals or messages may be added to the logic flow.

[0086] Some functional units described in this specification have been labeled as modules to more specifically emphasize their implementation independence. For example, a module can be implemented as a hardware circuit that includes a custom VLSI circuit or gate array, a finished semiconductor (such as a logic chip, transistor, or other discrete component). A module can also be implemented in a programmable hardware device (e.g., a field programmable gate array, programmable array logic, a programmable logic device, etc.).

[0087] The module can also be implemented in software so that it can be executed by various types of processors (such as CPU or GPU, hybrid environment and cluster). The identification module of executable code can, for example, include one or more computer instruction blocks, and one or more computer instruction blocks can be organized as objects, programs or functions. However, the executable file of the identification module does not need to be physically located together, but can include different instructions stored in different locations, which include the module and realize the specified purpose of the module when logically combined together.

[0088] In fact, the module of executable code can be a single instruction, or many instructions, and can even be distributed on several different code segments, between different programs and across several memory devices. Similarly, operational data can be identified and shown in the module herein, and can be embodied in any suitable form and organized in any suitable type of data structure. Operational data can be collected as a single data set, or can be distributed in different locations, including distribution on different storage devices. Modules can be passive or active, including a subject that can be operated to perform a desired function.

[0089] The techniques described herein may also be stored on computer-readable storage media, which includes volatile and nonvolatile, removable and non-removable media implemented using any technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer-readable storage media include, but are not limited to, non-transitory machine-readable storage media (such as RAM, ROM, EEPROM, flash memory or other memory technology), CD-ROM, digital versatile disk (DVD) or other optical storage device, magnetic cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, or any other computer storage medium that can be used to store the desired information and described techniques.

[0090] The devices described herein may also include communication connections or networking devices and networking connections that allow the device to communicate with other devices. A communication connection is an example of a communication medium. Communication media are typically embodied as computer-readable instructions, data structures, program modules, and other data (such as carrier waves or other transmission mechanisms) in a modulated data signal, and include any information transfer medium. A "modulated data signal" refers to a signal that sets or changes one or more of its characteristics in a manner that encodes information in the signal. By way of example and not limitation, communication media include wired media (such as a wired network or direct line connection) and wireless media (such as sound, radio frequency, infrared, and other wireless media). The term computer-readable medium used herein includes communication media.

[0091] Reference is made to the examples shown in the accompanying drawings, and specific language is used herein to describe these examples. However, it should be understood that this is not intended to limit the scope of the present technology. Changes and further modifications to the features shown herein and additional applications of the examples shown herein will be considered within the scope of this specification.

[0092] In addition, the described features, structures or characteristics can be combined in one or more examples in any suitable manner. In the previous description, many specific details (such as examples of various configurations) are provided to provide a thorough understanding of the examples of the described technology. However, it will be appreciated that the present technology can be practiced without one or more specific details, or using other methods, components, devices, etc. In other examples, well-known structures or operations are not shown or described in detail to avoid obscuring various aspects of the present technology.

[0093] Although the subject matter has been described in language specific to structural features and / or operations, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features and operations described above. Instead, the specific features and actions described above are disclosed as example forms of implementing the claims. Many modifications and alternative arrangements may be devised without departing from the spirit and scope of the technology.

Claims

1. An imaging system, include: at least one memory device comprising instructions that, when executed by at least one processor, cause the imaging system to: receiving a projection image data set generated by at least one imaging data detector in response to detecting image data to allow generation of reconstructed image data of the three-dimensional object using a reconstruction technique; generating a reduced image dataset from the projection image dataset, wherein the reduced image dataset has a smaller amount of data than an amount of data of the projection image dataset received from the at least one imaging data detector; generating an image residual data set to indicate an image difference between the projected image data set and the reduced image data set; and Performing the reconstruction technique to generate reconstructed image data of the three-dimensional object includes at least: (i) generating a first scale image reconstruction of the three-dimensional object using the reduced image data set, and (ii) generating at least a second scale image reconstruction of the three-dimensional object using the image residual data set; After generating the first scale image reconstruction, sending the first scale image reconstruction of the three-dimensional object to a client for direct display to a display device; and After generating the second scale image reconstruction, the second scale image reconstruction of the three-dimensional object is sent to the client for direct display to the display device.

2. The imaging system according to claim 1, in, The reconstruction technique further includes: (iii) using the first scale image reconstruction and the second scale image reconstruction to generate a multi-scale image reconstruction of the three-dimensional object, wherein the multi-scale image reconstruction of the three-dimensional object is used for display on a display device.

3. The imaging system according to claim 1, in, Both the first scale image reconstruction and the second scale image reconstruction are provided as input to a computer vision algorithm or to a robot.

4. The imaging system according to claim 1, in, The reduced data set is encoded using a data compression technique.

5. The system according to claim 1, in, The remaining data set is encoded using a data compression technique.

6. The imaging system according to claim 1, in, The first scale image reconstruction and the second scale image reconstruction are generated in parallel.

7. The imaging system according to claim 1, in, Generating a first scale image reconstruction of the three-dimensional object comprises: forward projecting the ground truth image volume to produce a two-dimensional projection image set; determining a difference between the projection image data set and the two-dimensional projection image set; generating an update volume by backprojecting the difference into three-dimensional space; and The update volume is incorporated into a reconstruction of an image of the three-dimensional object.

8. The imaging system according to claim 1, in, Generating a first scale image reconstruction of the three-dimensional object includes applying a density constraint to the first scale image reconstruction.

9. The imaging system according to claim 1, in, Generating the second scale image reconstruction includes increasing the resolution of a region of interest within the second scale image reconstruction of the three-dimensional object.

10. The imaging system according to claim 1, in, The second scale image reconstruction of the three-dimensional object has a higher resolution than the resolution of the first scale image reconstruction.

11. The system according to claim 1, in, The second scale image reconstruction of the three-dimensional object has a higher depth than the depth of the first scale image reconstruction.

12. The imaging system according to claim 1, in, The second scale image reconstruction of the three-dimensional object has the same resolution as the first scale image reconstruction, and the second scale image reconstruction is shifted in position to create a volume that is interleaved with the first scale image reconstruction.

13. The imaging system according to claim 1, in, The first scale image reconstruction of the three-dimensional object and the second scale image reconstruction of the three-dimensional object are both displayed to a display device.

14. The imaging system according to claim 1, in, The memory device also includes instructions that, when executed by the at least one processor, cause the system to send the reduced image data set and the image remainder data set to a remote data center having computing resources configured to perform the reconstruction technique to generate reconstructed image data of the three-dimensional object.

15. The imaging system according to claim 1, in, The reconstruction technique is an iterative reconstruction technique or a filtered back projection technique.

16. The imaging system according to claim 1, in, The projection image data set is generated by x-rays or by magnetic resonance.

17. A computer-implemented method, include: receiving an image dataset from a client at a data center having computing resources to perform a reconstruction technique to generate reconstructed image data of a three-dimensional object, wherein the image dataset comprises (i) a reduced image dataset having a smaller amount of data than a resolution of a projection image dataset obtained from at least one imaging data detector configured to receive x-ray radiation from at least one radiation source, and (ii) an image residual dataset indicating an image resolution difference between the projection image dataset and the reduced image dataset; Using the reconstruction technique, initiating a first scale image reconstruction of the three-dimensional object using the reduced image dataset, a second scale image reconstruction of the three-dimensional object using the image residual dataset, and a multi-scale image reconstruction of the three-dimensional object using the first scale image reconstruction and the second scale image reconstruction; After generating the first scale image reconstruction, sending the first scale image reconstruction of the three-dimensional object to the client for direct display to a display device; After generating the second scale image reconstruction, sending the second scale image reconstruction of the three-dimensional object to the client for direct display to the display device; and A multiscale image reconstruction of the three-dimensional object is caused to be displayed to a display device, wherein the multiscale image reconstruction provides a higher resolution view of the three-dimensional object than the resolution views of the first scale image reconstruction and the second scale image reconstruction.

18. The method according to claim 17, in, The first scale image reconstruction and the second scale image reconstruction of the three-dimensional object are performed in parallel using computing resources of the data center.

19. The method according to claim 17, in, Causing the multiscale image reconstruction of the three-dimensional object to be displayed on the display device further includes: sending the multiscale image reconstruction to the client, the client being configured to display the multiscale image reconstruction on the display device.

20. The method according to claim 17, in, Causing the multiscale image reconstruction of the three-dimensional object to be displayed on the display device further includes sending a rendering of the multiscale image reconstruction to the client for display on the display device.

21. The method according to claim 17, in, The projection image data set includes an ordered subset of projection image data generated by the at least one imaging data detector to allow reconstruction of the three-dimensional object to begin prior to receiving a complete projection image set.

22. An imaging system, include: at least one memory device comprising instructions that, when executed by at least one processor, cause the imaging system to: receiving a projection image data set generated by at least one imaging data detector in response to detecting imaging data to allow generation of reconstructed image data of a three-dimensional object using a reconstruction technique; performing a reconstruction technique to generate the reconstructed image data of the three-dimensional object, including generating at least (i) a first scale image reconstruction of the three-dimensional object, generating an image residual data set by forward projecting a first scale reconstruction of the three-dimensional object and comparing the first scale reconstruction of the three-dimensional object to the imaging data; wherein the first scale image reconstruction is generated using a reduced image dataset, wherein the reduced image dataset has a smaller amount of data than an amount of data of the projection image dataset received from the at least one imaging data detector, wherein the reconstruction technique further comprises (ii) a second scale image reconstruction using an image residual dataset, wherein the image residual dataset is generated as an image difference between the projection image dataset and the forward projection of the first scale reconstruction of the three-dimensional object; performing a reconstruction technique to generate a reconstruction of the image residual data set to generate a second scale image reconstruction; After generating the first scale image reconstruction, sending the first scale image reconstruction of the three-dimensional object to a client for direct display to a display device; After generating the second scale image reconstruction, the second scale image reconstruction of the three-dimensional object is sent to the client for direct display to the display device.

23. The system according to claim 22, in, The data sets are combined to generate a second scale image reconstruction of the three-dimensional object having a higher resolution than the first scale reconstruction, and the second scale image reconstruction of the three-dimensional object is displayed on a display device.

24. The system according to claim 22, in, Both the first scale image reconstruction and the second scale image reconstruction are provided as input to a computer vision algorithm.

25. The system according to claim 22, in, The reconstructed image data is a subset of the data of the region of interest having a higher resolution and a higher image depth.

Images