Systems and methods for near real-time and unsupervised coordinate projection networks for computed tomography image reconstruction

Abstract

A computer-implemented method for constructing a three-dimensional image is described. A computing system may obtain a plurality of sparse view sinograms of an object. A computing system may encode a machine learning model to represent a continuous density field of an object to map spatial coordinates of voxels in the continuous density field to intensity values. The machine learning model may include an encoding module configured to map voxels of the continuous density field to a higher dimensional space by a multi-resolution hash encoding technique, and a neural radiation module configured to output intensity values according to spatial coordinates of the voxels of the continuous density field. A computing system can train a machine learning model using a plurality of sparse view sinograms. A computing system may determine spatial coordinates of voxels of a three-dimensional image of an object to be constructed. The computing system may input spatial coordinates of voxels of the three-dimensional image into a trained machine learning model. The computing system may construct a three-dimensional image based on a dense view sinogram generated by the trained machine learning model.

Description

[0001] Citation: This application claims the benefit of PCT International Application No. PCT / CN2022 / 124249, entitled “System and Method for Near Real-Time and Unsupervised Coordinate Projection Network for Computed Tomography Images Reconstruction”, filed on October 10, 2022. The contents of PCT International Application No. PCT / CN2022 / 124249 are incorporated herein by reference. Technical Field

[0002] The present invention relates generally to image rendering and more particularly to systems and methods for performing machine learning based image rendering. Background Art

[0003] X-ray computed tomography (CT) has been widely used in many fields such as clinical diagnosis, non-destructive industrial testing, and safety inspection. In recent years, CT has become one of the most important medical imaging technologies for diagnosing and detecting viral infections (such as COVID-19 or other new coronaviruses). However, the high level of radiation exposure caused by CT may increase the lifelong risk of cancer, especially for pregnant women and children. Therefore, reducing the number of radiation views (e.g., sparse view CT) can significantly reduce the radiation dose received by patients and shorten the scanning time. However, under such an approach, the insufficient projection data in sparse view CT (i.e., sparse view sinusoids) will produce severe streak artifacts when applying various analytical reconstruction techniques, such as filtered back projection (FBP), which seriously affects the image quality.

[0004] To address these streak artifact issues, many optimization-based iterative techniques (e.g., gradient-based methods, low-rank-based methods, total variation-based methods, etc.) have been developed. Since these optimization-based iterative techniques make overly idealistic assumptions, their performance in artifact removal is limited. Recently, convolutional neural network (CNN)-based machine learning models have shown potential in improving the quality of sparse view computed tomography (CT) images, due to the nonlinear characteristics of CNNs in learning the transformation between low-quality input images and artifact-free high-quality output images. Thanks to the well-designed network architecture, these CNN-based machine learning models can produce satisfactory artifact removal effects. However, these CNN-based machine learning models also have disadvantages. For example, training a CNN requires a large training dataset containing many low-quality and high-quality image pairs. In addition, due to the limitation of excessive radiation exposure to patients, it may be very difficult to construct a training dataset containing low-quality and high-quality image pairs. In addition, the model generalization ability of CNN mainly depends on the data distribution in the training dataset. In addition, for real clinical applications, supervised CNN-based machine learning models need to be fine-tuned according to the scanner model and the scanning parameters of the scanner model. Therefore, the performance of CNN-based machine learning models is limited when reconstructing images that are not included in the training data (for example, image patterns of rare diseases). In other words, if the training dataset used to train such machine learning models does not include images of specific objects, the quality of images reconstructed by CNN-based machine learning models is often poor. Summary of the invention

[0005] The present invention describes a computer-implemented method. A computing system may obtain a plurality of sparse view sinograms of an object. The computing system may encode a machine learning model to represent a continuous density field of the object to map the spatial coordinates of voxels in the continuous density field to intensity values. The machine learning model may include an encoding module configured to map voxels of the continuous density field to a higher dimensional space through a multi-resolution hash coding technique, and a neural radiation module configured to output intensity values ​​according to the spatial coordinates of the voxels of the continuous density field. The computing system is capable of training the machine learning model using the plurality of sparse view sinograms. The computing system may determine the spatial coordinates of voxels of a three-dimensional image of the object to be constructed. The computing system may input the spatial coordinates of the voxels of the three-dimensional image into the trained machine learning model. The computing system may construct the three-dimensional image based on a dense view sinogram generated by the trained machine learning model.

[0006] In some embodiments, the projection of a ray into the continuous density field may be estimated based on a Radon transform.A point sample may be performed on each of the projections along the ray to obtain the spatial coordinates of the voxel.

[0007] In some embodiments, the projection of the ray may be optimized based on minimizing a loss on a loss function.

[0008] In some embodiments, the projection of the rays is at least one of a parallel beam, a fan beam, or a cone beam.

[0009] In some embodiments, the trained machine learning model generates the dense visual angle sinusoidal map based on the intensity values ​​corresponding to the spatial coordinates of the voxels of the three-dimensional image of the object to be constructed output by the neural radiation module.

[0010] In some embodiments, the three-dimensional image of the object may be constructed based on applying filtered backprojection to the dense view sinusoidal map.

[0011] In some embodiments, the machine learning model can be trained using the plurality of sparse view sinograms without requiring additional external data, wherein the external data includes a plurality of high-quality and low-quality image pairs of the object.

[0012] In some embodiments, the spatial coordinates of the voxels of the continuous density field may be obtained. The surrounding voxels on a particular resolution layer may be determined based on the spatial coordinates of the voxels. Indexes may be assigned to the corner points of the surrounding voxels. A feature vector may be obtained from one or more hash tables. The feature vector may be interpolated based on the relative positions of the voxels. An encoded feature vector may be generated by concatenating the results of the interpolation at each resolution layer. The encoded feature vector has a higher dimension than the spatial coordinates of the voxels.

[0013] In some embodiments, the index may be assigned to the corner points of the surrounding voxels based on hashed integer coordinates of the surrounding voxels.

[0014] In some embodiments, the neural radiation module may be based on a neural network.

[0015] In some embodiments, the neural network comprises a fully connected neural network comprising eight neural layers and having skip connections at the fourth neural layer.

[0016] In some embodiments, each of the first seven neural layers is associated with an activation layer.

[0017] In some embodiments, the continuous density field is a neural radiation field of the subject.

[0018] The present invention describes a computer-implemented method for constructing a three-dimensional image. A computing system may obtain a plurality of sparse view sinograms of an object. The computing system may encode a neural radiation field of the object into a machine learning model. The neural radiation field of the object may be encoded using at least a multi-resolution hash coding technique. The computing system determines the spatial coordinates of voxels of the three-dimensional image of the object to be constructed. The computing system generates a dense view sinogram of the object based on the spatial coordinates of the voxels through the machine learning model. The computing system constructs the three-dimensional image of the object by applying filtered back projection to the dense view sinogram.

[0019] In some embodiments, the spatial coordinates of the voxels of the neural radiation field may be mapped to a higher dimensional imaging space based on the multi-resolution hash coding technique.

[0020] In some embodiments, the projection of the ray into the continuous density field may be estimated based on a Radon transform.Point sampling may be performed along each of the projections of the ray to obtain the spatial coordinates of the voxel.

[0021] In some embodiments, the dense viewing angle sinusoidal map of the object may be generated based on intensity values ​​corresponding to the spatial coordinates of the voxels of the three-dimensional image of the object to be constructed output by the neural radiation field.

[0022] In some embodiments, the neural radiation field can be encoded using the plurality of sparse view sinusoids and no additional external data is required. The external data includes a plurality of high-quality and low-quality image pairs of the object.

[0023] In some embodiments, the spatial coordinates of the voxels of the continuous density field can be obtained. The surrounding voxels on a specific resolution layer can be determined based on the spatial coordinates of the voxels. Indexes can be assigned to the corner points of the surrounding voxels. A feature vector can be obtained from one or more hash tables. The feature vector can be interpolated according to the relative positions of the voxels. An encoded feature vector can be generated by connecting the results of the interpolation at each resolution layer. The encoded feature vector has a higher dimension than the spatial coordinates of the voxels.

[0024] In some embodiments, the neural radiation field may be based on a neural network.

[0025] The present invention describes a system. The system may include an encoding circuit and a neural radiation circuit communicatively coupled to the encoding circuit. The encoding circuit may be configured to receive a plurality of sparse view sinograms of an object, and map the spatial coordinates of voxels in a continuous density field of the object to a higher dimensional space by a multi-resolution hash coding technique to generate an encoding feature vector. The neural radiation circuit may be configured to train a neural network to encode an implicit neural representation of the object, receive the encoding feature vector from the encoding circuit, and generate an intensity value corresponding to the spatial coordinate of the voxel based on the encoding feature vector using the trained neural network.

[0026] In some embodiments, the encoding circuit may include a plurality of multiply-accumulate (MAC) circuits. Each MAC circuit may be configured to perform a matrix-vector multiplication on the spatial coordinates of the voxel in parallel with one or more of the plurality of MAC circuits.

[0027] In some embodiments, the encoding circuit can also be configured to determine surrounding voxels at a specific resolution layer based on a set of the spatial coordinates of one of the voxels; assign indices to corner points of the surrounding voxels; and obtain feature vectors from one or more hash tables. The plurality of MAC circuits are configured to interpolate the feature vectors according to the relative positions of the voxels.

[0028] In some embodiments, the encoding circuit may further include a plurality of coordinate rotation digital computers (CORDIC) circuits, each of which is configured to perform a sine or cosine function on the results generated in parallel from the matrix-vector multiplication to generate the encoded feature vector.

[0029] In some embodiments, the encoding circuit is configured to perform Fourier eigenmapping.

[0030] In some embodiments, the neural radiation circuit includes a plurality of processing element (PE) arrays configured to operate in parallel. Each of the plurality of PE arrays includes a plurality of PE circuits. Each of the plurality of PE circuits from the same PE array in the plurality of PE arrays is configured to perform multiplications between shared elements from the encoded feature vector and different weights of the trained neural network in parallel.

[0031] In some embodiments, the neural radiation circuit further includes two weight buffers configured to provide weights of the trained neural network to the plurality of PE arrays in a ping-pong manner.

[0032] In some embodiments, the neural radiation circuit may further include a partial sum circuit configured to receive a result from a multiplication performed by each of the plurality of PE circuits and generate a partial sum vector corresponding to the multiplication.

[0033] In some embodiments, the neural radiation circuit may further include a ReLU and quantization circuit configured to perform a ReLU activation based on the partial sum vector, and a bias buffer configured to store a bias for the ReLU activation.

[0034] In some embodiments, the encoding circuit also includes a coordinator generation circuit, which is configured to estimate the projection of the ray into the continuous density field based on the Radon transform; and perform point sampling along each of the projections of the ray to obtain the spatial coordinates of the voxel.

[0035] These and other features of the apparatus, system, method and non-transitory computer readable medium disclosed herein, as well as methods of operation and functions of related elements of the structure and the combination and manufacturing economy of parts will become more apparent when considering the following description and the appended claims with reference to the accompanying drawings, all of which form a part of this specification, wherein like reference numerals represent corresponding parts in the various figures. It should be expressly understood, however, that the drawings are for illustration and description purposes only and are not intended as a definition of the limitations of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Certain features of various embodiments of the invention are particularly set forth in the appended claims. A better understanding of the features and advantages of the invention will be obtained by reference to the following detailed description of illustrative embodiments that utilize the principles of the invention and the accompanying drawings, in which:

[0037] Figure 1 A computing environment including a machine learning model configured to generate a dense perspective sinusoidal graph according to various embodiments disclosed herein is shown.

[0038] Figure 2A A machine learning model configured to generate a dense perspective sinusoidal graph according to various embodiments disclosed herein is shown.

[0039] Figure 2B The flowchart of the multi-resolution hash coding module for mapping a low-dimensional image into a high-dimensional space according to various embodiments disclosed in the present invention is shown.

[0040] Figure 3A Reconstructed CT images using various image reconstruction techniques (including using SCOPE disclosed herein) according to various embodiments disclosed herein are shown.

[0041] Figure 3B The three most common types of projections used in CT imaging according to various embodiments of the present disclosure are shown.

[0042] Figure 3C A method of reconstructing a CT image using SCOPE according to various embodiments disclosed herein is shown.

[0043] Figure 4 A computing component according to various embodiments disclosed herein is shown, the computing component comprising one or more hardware processors and a machine-readable storage medium storing a set of machine-readable / machine-executable instructions that, when executed, cause the hardware processor to perform a method for constructing a three-dimensional image.

[0044] Figure 5 is a block diagram of a computer system upon which any of the various embodiments described herein may be implemented.

[0045] Figure 6 FIG. 4 is an architecture diagram of an accelerator according to various embodiments disclosed in the present invention.

[0046] Figure 7 Schematic diagram of an encoding engine according to various embodiments disclosed in the present invention.

[0047] Figure 8 The diagram is an architecture diagram of a network computing engine according to various embodiments disclosed in the present invention.

[0048] Fig. 9 FIG. 4 is a diagram of an algorithm executed by an accelerator according to various embodiments disclosed in the present invention.

[0049] Fig.10 A graph is generated for coordinates according to various embodiments disclosed herein.

[0050] The accompanying drawings depict various embodiments of the disclosed technology for illustrative purposes only, wherein the accompanying drawings use the same reference numerals to identify the same elements. Those skilled in the art will readily appreciate from the following discussion that alternative embodiments of the structures and methods shown in the accompanying drawings may be adopted without departing from the principles of the disclosed technology described in the present invention. DETAILED DESCRIPTION

[0051] The present invention describes a solution to the above-mentioned problems. Inspired by implicit neural representations (also known as coordinate-based neural networks), a machine learning model is proposed to generate high-quality CT images. In various embodiments, the machine learning model may be referred to as a single image self-supervised coordinate projection network, or SCOPE for short. SCOPE can reconstruct artifact-free, high-quality CT images from only sparse view sinusoids (i.e., x-ray projections) without using any additional data during training, such as low-quality and high-quality image pairs. Specifically, in SCOPE, an imaging object can be represented as an implicit neural representation whose independent variables correspond to two-dimensional (2D) or three-dimensional (3D) spatial coordinates of the imaging plane, depending on the type of projection beam used. By projecting through a differentiable Radon transform, the implicit neural representation of the imaging object can be directly encoded into a neural network based on a sparse view sinusoid. That is, the neural network can be trained based on a sparse view sinusoid. After the neural network is encoded (i.e., trained), the neural network can generate a dense view sinusoid from the encoded implicit neural representation through a Radon transform. Thereafter, high-quality CT images can be reconstructed by applying filtered back projection (FBP) to the dense view sinusoidal graphs. However, since SCOPE is image-specific (i.e., in order to generate a high-quality CT scan image from each image object, a unique neural network needs to be trained to encode the implicit representation of the image object), SCOPE's reconstruction efficiency is generally low. To this end, multi-resolution hash coding can be implemented to accelerate neural network training. In this way, the training time of the neural network is about 1-5 minutes, which enables SCOPE to reconstruct high-quality CT images in near real time in practice.

[0052] Figure 1 FIG. 1 shows a computing environment 100 including a machine learning model 104 configured to generate a dense perspective sinusoidal graph according to various embodiments disclosed herein. Figure 1 As shown, in some embodiments, the computing environment 100 may include a neural radiation field 102. The neural radiation field 102 may be associated with an object, such as an organ or a biological cavity. The neural radiation field 102 may be encoded into a machine learning model 104. For example, in some embodiments, the machine learning model 104 may be trained using a training data set containing multiple sparse view images (e.g., sinusoids) depicting an object to encode the neural radiation field 102. In some embodiments, the machine learning model 104 may be referred to as a single image self-supervised coordinate projection network, or SCOPE for short. By projecting an x-ray beam to an object from multiple view angles on a Z axis parallel to a source generating the x-ray beam, measurement data associated with multiple sparse view sinusoids may be acquired. The projection of the x-ray beam may be represented as follows:

[0053]

[0054] Where R(θ,ρ,z) is the measurement data, f(x,y,z) is a plurality of sparse view sinograms acquired in the spatial domain, θ∈[0,π) is the projection view of the x-ray beam, ρ is the position of the x-ray beam in each projection, and δ() is the Dirac delta function. After a circle of x-ray beams (i.e., a radiation cone) is projected from the source through the object, a CT image depicting the object can be reconstructed using filtered back projection (FBP) based on the acquired plurality of sparse view sinograms.

[0055] In order to reconstruct an artifact-free CT image from multiple sparse view sinograms in a self-supervised manner, the objects depicted in the multiple sparse view sinograms can be modeled as implicit neural representations fΘ(x) (where Θ is a parameter of the implicit neural representation). In some embodiments, the implicit neural representation can be encoded (i.e., trained) into the machine learning model 104. The implicit neural representation fΘ(x) maps the spatial coordinates x=(x, y, z) in the imaging plane to the corresponding voxel intensity I. Due to the differentiable line integral projection of Equation 1, the implicit neural representation fΘ can be finely approximated from multiple sparse view sinograms. After the implicit neural representation fΘ is encoded (i.e., trained), a dense view sinogram can be generated from the implicit neural representation fΘ. Then, an artifact-free CT image can be obtained by simply applying filtered back projection (FBP) to the dense view sinogram.

[0056] In some embodiments, the machine learning model 104 can be optimized. For example, consider the sparse perspective sinusoidal graph R s (θ,ρ)∈R M×N×N , where M is the number of x-ray projection views and N×N is the number of x-ray beams in each projection. Along each of the N x-ray beams, N positions x are sampled, as Figure 1 As shown. Afterwards, the implicit neural representation fΘ takes the coordinate x of each sampling point as input and outputs the corresponding voxel intensity I = fΘ(x) at each sampling point. Applying the transformation of Equation 1, the projection along each x-ray beam can be estimated. In some embodiments, the transformation of Equation 1 can be a Radon transform. In this case, each estimated projection from the x-ray beam L{θ,ρ,z} It can be expressed as follows:

[0057]

[0058] Here, since Equation 2 is differentiable, we can use the mini-batch gradient descent technique to optimize the implicit neural representation fΘ(x) to minimize the loss function L as follows:

[0059]

[0060] where m and n×n are the number of sampled views during each training iteration and the number of x-ray beams per view, respectively.

[0061] Figure 2A A machine learning model 200 configured to generate a dense perspective sinusoidal graph according to various embodiments disclosed herein is shown. In some embodiments, Figure 1 The machine learning model 104 can be implemented as a machine learning model 200. Figure 2A As shown, the machine learning model 200 may include a multi-resolution hash coding module 202 coupled to a neural network 204. The multi-resolution hash coding module 202 may be configured to perform multi-resolution hash coding, and the neural network 204 may be configured to encode the implicit neural representation. In some embodiments, the neural network 204 may be a fully connected neural network. In other embodiments, the neural network 204 may be a multi-layer perceptron (MLP). There are many possible variations and assumptions. For example, Figure 2A As shown, the machine learning model 200 can take the three-dimensional voxel coordinates x = (x, y, z) as input and output the corresponding voxel intensity I corresponding to the three-dimensional voxel coordinates.

[0062] In some embodiments, the multi-resolution hash coding module 202 can be configured to map the three-dimensional coordinates of the voxels to a high-dimensional space. Mapping the three-dimensional coordinates of the voxels to a high-dimensional space can facilitate the neural network 204 to learn high-frequency images, thereby significantly accelerating the optimization of the machine learning model 200. In some embodiments, the multi-resolution hash coding module 202 can map the three-dimensional coordinates of the voxels to a high-dimensional space based on a hash coding technique. For example, referring to Figure 2B . Figure 2B The flowchart 250 of the multi-resolution hash coding module 202 mapping the three-dimensional coordinates of the voxel to the high-dimensional space according to various embodiments disclosed in the present invention is shown. Figure 2B As shown, for a given input voxel coordinate The surrounding voxels of the input voxel coordinates at L resolution levels can be identified. By hashing the integer coordinates, indices can be assigned to the corners of these surrounding voxels. For all the resulting corner indexes, the corresponding F-dimensional feature vectors can be looked up from the hash table. These F-dimensional feature vectors are then linearly interpolated based on the relative position of the input voxel coordinates within their respective l-th voxels. The results at each level are concatenated to generate an encoding vector Using this hash encoding technique, the encoded vector can have higher dimensionality than the input voxel coordinates.

[0063] In some embodiments, the parameters of the multi-resolution hash coding module 202 may be configured by a user. For example, in some embodiments, the parameters of the multi-resolution hash coding module 202 may be set according to the values ​​shown in Table 1.

[0064] Hyperparameters value L 16 T <![CDATA[[2 14 ,2 24 ]]]> F 2

[0065] Table 1

[0066] Wherein L, T and F are parameters used by the multi-resolution hash coding module 202 to encode the low-dimensional image, and the values ​​associated with L, T and F are the values ​​used by the multi-resolution hash coding module 202 when encoding the low-dimensional image into a higher-dimensional space. In some embodiments, the parameters of the multi-resolution hash coding module 202 may be hyperparameters.

[0067] In some embodiments, the neural network 204 can be configured to encode an implicit neural representation of an object. The neural network 204 can be trained based on a training data set containing a plurality of sparse view sinograms (e.g., low-dimensional images) depicting an object to encode the implicit neural representation. Once trained, the neural network 204 can output a dense view sinogram of the object based on the three-dimensional voxel coordinates. Review Figure 2A In some embodiments, neural network 204 may include multiple neural layers. For example, in some embodiments, neural network 204 may include eight neural layers. In some cases, the eight neural layers may be eight fully connected neural layers. In some embodiments, neural network 204 may include multiple activation layers. Multiple activation layers may be associated with multiple neural layers. For example, in one embodiment, except for the last fully connected neural layer, all other neural layers may be followed by an activation layer, such as ReLU activation. In some embodiments, dense connections 206 may be used between the input of neural network 204 and the fourth neural layer of neural network 204 for activation. In some embodiments, the first and fifth neural layers of neural network 204 may have 10 and 554 neurons, respectively, while other neural layers may have 512 neurons. There are many possible variations and ideas here.

[0068] Typically, 720 view projections are considered fully sampled measurements, while 90 view projections are considered sparse view observations. Therefore, images reconstructed using 720 view projections by the machine learning model 200 can be used as the true value (GT), while images reconstructed using 240, 180, 120, and 90 view projections can be used as input images corresponding to four different factors or multiples (3x, 4x, 6x, and 8x) of GT.

[0069] Table 2 shows a quantitative comparison of various image reconstruction techniques including the SCOPE technique disclosed in the present invention when applied to an image dataset (i.e., the AAPM dataset), where the best performance values ​​are shown in bold. Note that in Table 2, DD-Net and TF-Net are machine learning models based on supervised convolutional neural networks (CNNs). Comparing the results with three baseline metrics (i.e., peak signal-to-noise ratio PSNR, structural similarity index SSIM, and normalized root mean square error NRMSE), SCOPE performs best in most cases. For example, please refer to Figure 3A . Figure 3A According to various embodiments of the present disclosure, a CT image 300 reconstructed using various image reconstruction techniques, including SCOPE disclosed herein, is shown. Figure 3A A scenario is shown in which 90 projection views are used when reconstructing a CT image 300 . Figure 3A A qualitative comparison of various image reconstruction techniques is shown on two representative test samples from the AAPM dataset. Figure 3A As can be seen from the above, SCOPE generally produces the best results in terms of image contrast and details. For example, the image reconstructed by Filtered Back Projection (FBP) contains severe streak artifacts. For example, the image reconstructed by Algebraic Reconstruction Technique (ART) produces extremely blurry results. For example, the images reconstructed by DD-Net and TF U-Net only recover limited details. In addition, Figure 3A As can be seen in , image reconstruction using SCOPE can achieve similar reconstruction performance as using full sampling (i.e., true values) of 720 projection views.

[0070] Figure 3A Results from other deep learning methods, such as TF U-Net and DD-Net, which require large-scale training datasets consisting of high-resolution and low-resolution image pairs, are also included. Unlike other techniques, SCOPE's training is completely self-supervised and requires only a single image set - no image pairs are required. Table 2 below compares the quantitative evaluation results of each technique.

[0071]

[0072] Table 2

[0073] In general, image reconstruction using SCOPE can be divided into three steps: (i) dense projection views (i.e., a total of 720 projection views) are sampled along the Z axis, and N×N x-ray beams are generated from each dense projection view; (ii) N coordinates along each x-ray beam are input into SCOPE to output the intensity I=fΘ(x) at the N coordinates, and the projections of all x-ray beams are calculated by Equation 2 to generate a dense view sinusoidal map; (iii) then, filtered back projection (FBP) is applied to the dense view sinusoidal map to reconstruct an artifact-free 3D CT image.

[0074] Figure 3B The three most common projection types used in CT imaging according to various embodiments disclosed herein are shown. Figure 3B As shown in Figure 1, the three most common projection types are (1) parallel X-ray beam, (2) fan-shaped X-ray beam, and (3) cone-shaped X-ray beam. Despite the differences between the three projections, the projection values ​​of the corresponding X-ray beam in each sinusoidal diagram are line integrals, as described in Equation 1. Therefore, SCOPE can be applied to any of the three projection strategies without any modification.

[0075] Figure 3C Methods 340 and 342 for reconstructing a CT image using SCOPE according to various embodiments disclosed herein are shown. Method 340 shows a scenario where a CT image is reconstructed based on multiple sparse view sinograms, while method 342 shows a scenario where a CT image is reconstructed based on multiple limited view sinograms. In method 340, the interval between adjacent projection view angles is increased to reduce the total number of projection view angles. In method 342, the range of projection view angles is set to [a, b] ∈ [0, π), where a>0 and b<π. Similarly, SCOPE works under both conditions without any modification.

[0076] Figure 4 A computing component 400 according to various embodiments disclosed herein is shown, which includes one or more hardware processors 402 and a machine-readable storage medium 404 storing a set of machine-readable / machine-executable instructions that, when executed, cause the hardware processor 402 to perform a method for constructing a three-dimensional image. The computing component 400 may be, for example, Figure 5 The computing system 500 of FIG. The hardware processor 402 may include, for example, Figure 5 The machine-readable storage medium 404 may include a processor 504 in the embodiment of the present invention or any other processing unit described herein. Figure 5 The main memory 506, the read only memory (ROM) 508, the storage device 510 and / or any other suitable machine-readable storage media described herein.

[0077] At block 406 , processor 402 may obtain a plurality of sparse view sinograms of the object.

[0078] At box 408, the processor 402 may encode the machine learning model to represent the continuous density field of the object, thereby mapping the spatial coordinates of the voxels of the continuous density field to the intensity value. In some embodiments, the machine learning model may include an encoding module configured to map the voxels of the continuous density field to a higher dimensional space through a multi-resolution hash coding technique. In some embodiments, the machine learning model may also include a neural radiation module configured to output the intensity value according to the spatial coordinates of the voxels of the continuous density field. In some embodiments, the multi-resolution hash coding technique may include: obtaining the spatial coordinates of the voxels of the continuous density field; determining the surrounding voxels on a specific resolution layer based on the spatial coordinates of the voxels; assigning indexes to the corners of the surrounding voxels; obtaining feature vectors from one or more hash tables; interpolating the feature vectors according to the relative positions of the voxels; and generating an encoded feature vector by connecting the results of the interpolation on each resolution layer, wherein the encoded feature vector has a higher dimension than the spatial coordinates of the voxels. In some embodiments, the corners of the surrounding voxels may be assigned indices based on the hash integer coordinates of the surrounding voxels. In some embodiments, the neural radiation module may be based on a neural network. In some embodiments, the neural network may include a fully connected neural network including eight neural layers with skip connections at the fourth neural layer. Each of the first seven neural layers can be associated with an activation layer. In some embodiments, the continuous density field can be a neural radiation field of an object. At box 410, the processor 402 can use multiple sparse perspective sinusoids to train a machine learning model. In some embodiments, the trained machine learning model can generate a dense perspective sinusoid based on the intensity values ​​corresponding to the spatial coordinates of the voxels of the three-dimensional image of the object to be constructed output by the neural radiation module. In some embodiments, the machine learning model can be trained using multiple sparse perspective sinusoids and no additional external data is required. The external data can include multiple high-quality and low-quality image pairs of the object.

[0079] At block 412, the processor 402 may determine the spatial coordinates of the voxels of the three-dimensional image of the object to be constructed. In some embodiments, the spatial coordinates of the voxels may be obtained by estimating the projection of the ray into the continuous density field based on the Radon transform and performing point sampling along each of the projections of the ray. In some embodiments, the projection of the ray may be optimized based on minimizing the loss on the loss function. In some embodiments, the projection of the ray may be at least one of a parallel ray beam, a fan ray beam, or a cone ray beam.

[0080] At box 414, the processor 402 can input the spatial coordinates of the voxels of the three-dimensional image into the trained machine learning model.

[0081] At block 416, processor 402 may construct a three-dimensional image based on the dense view sinusoidal map generated by the trained machine learning model. In some embodiments, a three-dimensional image of an object may be constructed based on applying filtered back projection to the dense view sinusoidal map.

[0082] The techniques described herein are implemented, for example, by one or more special-purpose computing devices. The special-purpose computing devices may be hardwired to perform the techniques, or may include circuits or digital electronic devices, such as one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs), that are continuously programmed to perform the techniques, or may include one or more hardware processors programmed to perform the techniques according to program instructions in firmware, memory, other storage devices, or a combination.

[0083] Figure 5 5 is a block diagram of a computer system 500 on which any of the various embodiments described herein may be implemented. The computer system 500 includes a bus 502 or other communication mechanism for communicating information, and one or more hardware processors 504 coupled to the bus 502 for processing information. Descriptions of devices performing tasks are intended to indicate that one or more hardware processors 504 perform the tasks.

[0084] The computer system 500 also includes a main memory 506, such as a random access memory (RAM), cache, and / or other dynamic storage device, coupled to the bus 502 for storing information and instructions to be executed by the processor 504. The main memory 506 may also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by the processor 504. When these instructions are stored in a storage medium accessible to the processor 504, they cause the computer system 500 to become a customized, special-purpose machine for performing the operations specified in the instructions.

[0085] The computer system 500 also includes a read-only memory (ROM) 508 or other static storage device coupled to the bus 502 for storing static information and instructions for the processor 504. A storage device 510, such as a magnetic disk, an optical disk, or a USB flash drive (flash drive), is also provided and coupled to the bus 502 for storing information and instructions.

[0086] The computer system 500 may be coupled to an output device 512, such as a cathode ray tube (CRT) or LCD display (or touch screen), via the bus 502 for displaying information to a computer user. An input device 514, including alphanumeric and other keys, is coupled to the bus 502 for communicating information and command selections to the processor 504. Another type of user input device is a cursor control 516. The computer system 500 also includes a communication interface 518 coupled to the bus 502.

[0087] Figure 6 FIG. 1 is an architecture diagram of an accelerator according to various embodiments disclosed in the present invention. Figure 6 As shown, the accelerator 600 includes two parts, namely, the encoding engine 700 and the network computing engine 800. The encoding engine 700 is a dedicated module that is configured to implement the mapping from the input coordinates to the encoded high-dimensional features. The network computing engine 800 can be configured to generate the final voxel intensity through network evaluation. In some embodiments, the accelerator 600 can implement Figure 2A The machine learning model 200 shown in .

[0088] In some embodiments, the accelerator 600 also includes a control engine configured as a top controller to control the execution of the algorithm according to external control signals. For example, the control engine can accept and decode two control signals, namely parameter loading and configuration setting & reconstruction start. Two first-in-first-out (FIFO) modules connect the accelerator 600 to the external memory, where FIFO IN is used to load the feature matrix and network parameters into the accelerator 600, and FIFO OUT is responsible for writing out the generated voxel intensities.

[0089] Figure 7 Schematic diagram of the encoding engine according to various embodiments disclosed in the present invention. In some embodiments, Figure 7 The encoding engine 700 shown can implement Figure 2A The multi-resolution hash coding module 202 is shown. Figure 7 The mapping operation performed by the encoding engine 700 shown in FIG. 7 may include two parts: (i) the voxel coordinates x(x, y, z) and the matrix B (eg, the eigenvectors, such as Figure 2B The encoding engine 700 may be configured to perform a mapping of input coordinates (e.g., spatial coordinates of voxels) using a multiply-accumulate (MAC) array and a coordinate rotation digital computer (CORDIC) array, such as Figure 7As shown. The MAC array containing 12 MAC blocks performs matrix-vector multiplication between the voxel coordinates x(x, y, z) and the matrix B(3) in a pipelined manner. The four CORDIC modules can be configured to perform sine and cosine function operations on the four parallel results from the MAC array and output the encoded features to the input feature map memory, i.e., IMEM, in the network computing module. The coordinate generator can be configured to receive the reconstruction configuration from the top control engine and generate the coordinates (x i ,y i ,z i ), and send them to the MAC array. The matrix B (e.g., size 3×128) can be stored in a matrix memory (e.g., 768B).

[0090] For example, Figure 7 As shown, in each cycle, the coordinate generator outputs a coordinate (x i ,y i ,z i ), and the matrix memory outputs 4×3 feature parameters to the 12MACs in 3 cycles. Every four MACs on the same row of the MAC array read data from the same axis. For example, the top four MACs read x i Along the vertical direction, three parameters of the same frequency (corresponding to a column in the feature matrix) are written into three MACs respectively to be multiplied with the three axes. i and z i The registers are used to pipeline the inner products of the voxel and frequency parameters. After the MAC array, the four CORDIC blocks operate on the four inner products from the bottom four MACs to generate encoded features. These features are then written to the IMEM of the network computation engine.

[0091] In some embodiments, Figure 7 The encoding engine 700 shown in can be configured to perform Fourier feature mapping operations. For example, the Fourier feature mapping operation may include two parts: (i) matrix-vector multiplication between voxel coordinates x(x, y, z) and the Fourier feature matrix B, and (ii) performing trigonometric operations on the resulting inner product. The encoding engine 700 can be configured to perform Fourier feature mapping on the input coordinates using a MAC array and a CORDIC array, such as Figure 7As shown. The MAC array containing 12 MAC blocks performs matrix-vector multiplication between voxel coordinates x(x, y, z) and the Fourier feature matrix B(3) in a pipelined manner. The four CORDIC modules can be configured to perform sine and cosine function operations on four parallel results from the MAC array and output the encoded features to the input feature map memory, i.e., IMEM, in the network computing module. The coordinate generator in the FFM can be configured to receive the reconstruction configuration from the top control engine and generate the coordinates (x i ,y i ,z i ), and send them to the MAC array. The Fourier feature matrix B (e.g., size 3×128) can be stored in a matrix memory (e.g., 768B).

[0092] For example, Figure 7 As shown, in each cycle, the coordinate generator outputs a coordinate (x i ,y i ,z i ), and the Fourier feature memory outputs 4×3 feature parameters to 12 MACs in 3 cycles. Every four MACs on the same row of the MAC array read data on the same axis. For example, the top four MACs read x i Along the vertical direction, three parameters of the same frequency (corresponding to a column in the Fourier characteristic matrix) are written into three MACs respectively to be multiplied with the three axes. i and z i The registers are used to pipeline the inner products of the voxel and frequency parameters. After the MAC array, the four CORDIC blocks operate on the four inner products from the bottom four MACs to generate encoded features. These features are then written to the IMEM of the network computation engine.

[0093] Return to reference Figure 6 , the network computing engine 800 includes multiple parallel processing element (PE) arrays (e.g., 64 PE arrays), multiple partial and accumulation (PSA) blocks (e.g., 64 PSA blocks), ReLU & quantization units, and multiple memory blocks (e.g., four memory blocks). For example, a weight memory block (e.g., 1.5MB in size) can be used to store all weights of the network during the execution of the algorithm. In addition, there can be three activation memories, for example, one input memory (IMEM) for saving the input activations of the first layer, and two inter-layer activation memories (MEMA and MEMB) for buffering the activations of internal layers. Note that MEMA and MEMB are designed to work in an interchangeable manner, that is, if MEMA is used as the activation memory of the current layer, then it will be the input memory of the next layer, and vice versa.

[0094] Figure 8 FIG. 8 is an architecture diagram of a network computing engine according to various embodiments disclosed in the present invention. In some embodiments, the network computing engine 800 may implement Figure 2A The neural network 204 shown in FIG. Figure 8 As shown in (a), for each PE array, the same input neuron x can be shared by multiple PEs (e.g., 64 PEs) to be multiplied with multiple weights (e.g., 64 weights) from the same column of the submatrix, thereby generating multiple products (e.g., 64 products). For example, during calculation, 64×64 weights stay in the PE array for Bi=128 cycles. In each cycle, a new input vector with 64 neurons is loaded into the 64 PE arrays to be multiplied with these weights to obtain a different partial sum (Psum) vector. In addition, there are two weight buffers in each PE that work in a ping-pong manner, as shown in FIG. Figure 8 (b) One of them is used to save the next part of the weight matrix, while the other is used for the calculation of the current product.

[0095] In some embodiments, Figure 8 As shown, the products in the PE array can be transferred to the PSA block for accumulation to generate activations. There can be two types of Psum accumulation, such as addition tree accumulation and block accumulation. In each cycle, 64 PE arrays can generate 64 product vectors, each of which has 64 products, such as Figure 8 (c). When the products at the same horizontal position (i.e., the same row) are summed, a partial sum vector corresponding to the matrix-vector multiplication between the sub-weight matrix and the input vector can be generated. For example, the first column of the partial sum matrix P1 can be generated. The addition tree can then sum 64 products (e.g., in three stages). Each stage can include an adder with four input data. The block accumulation accumulates the Psums from the same position of the Psum matrix to generate the final result. Each block accumulation can be responsible for calculating one row of the output matrix O1. Note that the block accumulation block contains a 128×32-bit memory (PMEM) for storing 128 Psums corresponding to 128 input vectors. In addition, there can be a bias buffer to store the bias for the current output activation. When the addition tree accumulation generates a new Psum, it can be added to the bias and saved in PMEM. After 128 cycles, the Psum from the addition tree accumulation will be added to the Psum read from PMEM instead of the bias.

[0096] like Figure 8 As shown, Bi = 128 inputs can be combined into a batch to compute the entire network. Accordingly, the encoding engine (e.g., Figure 7The encoding engine 700 shown in can generate the required 128×256 features. Since there are only 12 MACs in the encoding engine, it is necessary to traverse the frequency Ffreq and the input coordinates to generate these encoded features. For example, there may be two implementations: (i) keeping the input coordinates in the MAC and traversing the frequency, or (ii) traversing the input while maintaining the frequency. For example, if the input coordinates are traversed, the X axis can be looped first, and then during the ResolutionX loop, the values ​​of yi and zi remain unchanged, where ResolutionX is the number of pixels on the X axis. This is because these voxels maintain the same y and z coordinate values. Therefore, the frequency in the MAC array can be maintained, and only x values ​​are generated for the ResolutionX cycle. Since the frequency in the MAC array is maintained when traversing the coordinates, and the y and z coordinate values ​​are also maintained in the ResolutionX cycles, the product between {y,z} and the corresponding frequency may not change. Therefore, the dynamic power consumption of y and z multiplication can be saved.

[0097] Fig. 9 FIG. 1 is an algorithm diagram executed by an accelerator according to various embodiments disclosed in the present invention. It should be understood that Fig. 9 The algorithm 900 shown in FIG. 9 is pseudocode and can be represented by Figure 6 The accelerator 600 shown in FIG. Fig. 9 As shown, the input of algorithm 900 includes voxel coordinates x(x, y, z) (e.g., similar to Figure 6 and Figure 7 voxel coordinates) and matrix B (e.g., Figure 6 and Figure 7 The output of the algorithm includes voxel intensities.

[0098] In some embodiments, the input of algorithm 900 includes the 3D spatial coordinates of all voxels. To reconstruct three scanned images with 256×256 pixels, we need to input 256×256×3=1966608 sets of coordinates into the system. For medical image reconstruction applications, the coordinates of three planes of interest can be determined by corresponding intersection points, because these planes are perpendicular to one of the three coordinate axes.

[0099] Fig.10 A coordinate generation diagram is provided for various embodiments disclosed according to the present invention. Fig.10 As shown, the sagittal scan is perpendicular to the Y axis, so that the voxels on this plane have the same y coordinate value. Therefore, given the boundaries of the 3D volume of the model and the image resolution to be rendered, we can input the intersection coordinates (x0, y0, z0) and calculate all other remaining coordinates in the accelerator. For example, Fig.10The coordinates of the voxels in the sagittal scan plane can be generated as follows:

[0100] x i =x 左 +cnt1×△x

[0101] y i =y o

[0102] z i =z 底 +cnt2×△z (4)

[0103] where x 左 and z 底 are the x value of the leftmost voxel and the z value of the bottommost voxel, respectively, Δx and Δz are the resolutions on the X and Z axes, respectively, and cnt1 and cnt2 are the number of pixels on the X and Z axes, respectively. Therefore, instead of storing all coordinate sets for the three scans, an on-chip coordinate generator can be built to calculate all coordinate sets on-chip. In this way, the input can be simplified to the boundary, the resolution of the reconstructed image, and the intersection coordinates, which significantly reduces the amount of data transmission.

[0104] Unless the context requires otherwise, throughout this specification and claims, the word "comprise" and variations thereof, such as "comprises" and "comprising", should be understood to have an open and inclusive meaning, i.e., "including, but not limited to". Recitation of numerical ranges throughout the specification is intended to be used as a shorthand notation for individually referring to each separate value falling within the range, including the values ​​defining the range, and each separate value is incorporated into the specification as individually recited herein. In addition, the singular forms "a", "an", and "the" include plural referents unless the context clearly dictates otherwise. The phrases "at least one of", "at least one selected from the group of", or "at least one selected from the group consisting of", etc. should be interpreted as disjuncts (e.g., not to be interpreted as at least one of A and at least one of B).

[0105] References throughout this specification to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the phrases "in one embodiment" or "in an embodiment" appearing in various places throughout this specification do not necessarily all refer to the same embodiment, but may refer to the same embodiment in some cases. Furthermore, in one or more embodiments, the particular features, structures, or characteristics may be combined in any suitable manner.

[0106] A component implemented as another component may be interpreted as a component that operates in the same or similar manner as the other component and / or includes the same or similar features, characteristics, and parameters as the other component.

Claims

1. A computer-implemented method comprising: A computing system obtains a plurality of sparse perspective sinusoidal graphs of the object; The computing system encodes a machine learning model to represent a continuous density field of the object to map spatial coordinates of voxels in the continuous density field to intensity values, wherein the machine learning model comprises: An encoding module configured to map voxels of the continuous density field to a higher dimensional space by a multi-resolution hash encoding technique; and a neural radiation module configured to output intensity values ​​according to spatial coordinates of the voxels of the continuous density field; Training the machine learning model using a plurality of sparse view sinusoidal graphs by the computing system; Determining, by the computing system, the spatial coordinates of voxels of the three-dimensional image of the object to be constructed; inputting, by the computing system, the spatial coordinates of the voxels of the three-dimensional image into the trained machine learning model, and The three-dimensional image is constructed by the computing system based on a dense view sinusoidal map generated by the trained machine learning model.

2. The computer-implemented method of claim 1, wherein: Determining the spatial coordinates of the voxels of the three-dimensional image of the object to be constructed comprises: estimating a projection of a ray into the continuous density field based on a Radon transform; and Point sampling is performed along each of the projections of the ray to obtain the spatial coordinates of the voxel.

3. The computer-implemented method of claim 2, wherein: The projection of the ray is optimized based on minimizing a loss on a loss function.

4. The computer-implemented method of claim 2, wherein: The projection of the rays is at least one of a parallel ray beam, a fan ray beam, or a cone ray beam.

5. The computer-implemented method of claim 1, wherein: The trained machine learning model generates the dense view sinusoidal map based on the intensity values ​​corresponding to the spatial coordinates of the voxels of the three-dimensional image of the object to be constructed output by the neural radiation module.

6. The computer-implemented method of claim 1, wherein: The three-dimensional image of the object is constructed based on applying filtered backprojection to the dense view sinusoidal map.

7. The computer-implemented method of claim 1, wherein: The machine learning model is trained using the plurality of sparse view sinusoids without requiring additional external data, wherein the external data comprises a plurality of high-quality and low-quality image pairs of the object.

8. The computer-implemented method of claim 1, wherein: The multi-resolution hash coding technique includes: obtaining spatial coordinates of voxels of the three-dimensional image of the object; determining surrounding voxels at a specific resolution layer based on the spatial coordinates of the voxel; assigning indices to corner points of the surrounding voxels; Obtaining feature vectors from one or more hash tables; interpolating the feature vector according to the relative positions of the voxels; and An encoded feature vector is generated by concatenating the results of the interpolation at each resolution level, wherein the encoded feature vector has a higher dimension than the spatial coordinates of the voxel.

9. The computer-implemented method of claim 8, wherein: The corner points of the surrounding voxels are assigned the indices based on hashed integer coordinates of the surrounding voxels.

10. The computer-implemented method of claim 1, wherein: The neural radiation module is based on a neural network.

11. The computer-implemented method of claim 10, wherein: The neural network comprises a fully connected neural network including eight neural layers and having a skip connection at the fourth neural layer.

12. The computer-implemented method of claim 11, wherein: Each of the first seven neural layers is associated with an activation layer.

13. The computer-implemented method of claim 1, wherein: The continuous density field is the subject's neural radiation field.

14. A computer-implemented method for constructing a three-dimensional image, comprising: A computing system obtains a plurality of sparse perspective sinusoidal graphs of the object; encoding, by the computing system, the neural radiation field of the subject into a machine learning model, wherein the neural radiation field of the subject is encoded using at least a multi-resolution hash coding technique; Determining, by the computing system, the spatial coordinates of voxels of the three-dimensional image of the object to be constructed; generating, by the computing system, a dense view sinusoidal map of the object based on the spatial coordinates of the voxels by the machine learning model; as well as The three-dimensional image of the object is constructed by the computing system by applying filtered backprojection to the dense view sinusoidal map.

15. The computer-implemented method of claim 14, wherein encoding the neural radiation field of the subject into the machine learning model comprises: Based on the multi-resolution hash coding technique, the spatial coordinates of the voxels of the neural radiation field are mapped to a higher dimensional imaging space.

16. The computer-implemented method of claim 14, wherein: Determining the spatial coordinates of the voxels of the three-dimensional image of the object to be constructed comprises: estimating a projection of a ray into the continuous density field based on a Radon transform; and Point sampling is performed along each of the projections of the ray to obtain the spatial coordinates of the voxel.

17. The computer-implemented method of claim 14, wherein: The dense viewing angle sinusoid of the object is generated based on intensity values ​​corresponding to the spatial coordinates of the voxels of the three-dimensional image of the object to be constructed output by the neural radiation field.

18. The computer-implemented method of claim 14, wherein: The neural radiation field is encoded using the plurality of sparse view sinusoids without requiring additional external data, wherein the external data comprises a plurality of high-quality and low-quality image pairs of the object.

19. The computer-implemented method of claim 14, wherein: The multi-resolution hash coding technique includes: obtaining spatial coordinates of voxels of the three-dimensional image of the object; Based on the spatial coordinates of the voxel, determining surrounding voxels at a specific resolution layer; assigning indices to corner points of the surrounding voxels; Obtaining feature vectors from one or more hash tables; interpolating the feature vector according to the relative positions of the voxels; and An encoded feature vector is generated by concatenating the results of the interpolation at each resolution level, wherein the encoded feature vector has a higher dimension than the spatial coordinates of the voxel.

20. The computer-implemented method of claim 14, wherein the neural radiation field is based on a neural network.

21. A system comprising: The encoding circuit is configured as follows: receiving multiple sparse view sinusoids of the object, and Mapping the spatial coordinates of voxels in the continuous density field of the object to a higher dimensional space by a multi-resolution hash coding technique to generate a coding feature vector; a neural radiation circuit, which is communicatively coupled to the encoding circuit, the neural radiation circuit being configured to: training a neural network to encode an implicit neural representation of the object; receiving the encoded feature vector from the encoding circuit; and Intensity values ​​corresponding to the spatial coordinates of the voxels are generated based on the encoded feature vector using the trained neural network.

22. The system of claim 21, wherein: The encoding circuit includes a plurality of multiply-accumulate (MAC) circuits, and Each MAC circuit is configured to perform a matrix-vector multiplication on the spatial coordinates of the voxel in parallel with one or more of the plurality of MAC circuits.

23. The system of claim 22, wherein: The encoding circuit is further configured to: determining surrounding voxels at a particular resolution layer based on a set of the spatial coordinates of one of the voxels; assigning indices to corner points of the surrounding voxels; as well as Obtaining feature vectors from one or more hash tables; and The plurality of MAC circuits are configured to interpolate the feature vectors according to relative positions of the voxels.

24. The system of claim 22, wherein: The encoding circuit also includes a plurality of coordinate rotation digital computer (CORDIC) circuits, and Each of the plurality of CORDIC circuits is configured to perform a sine or cosine function on results generated in parallel from matrix-vector multiplications to generate the encoded feature vector.

25. The system of claim 21, wherein: The encoding circuit is configured to perform Fourier eigenmapping.

26. The system of claim 21, wherein: The neural radiation circuit includes a plurality of processing element (PE) arrays configured to operate in parallel, Each of the plurality of PE arrays includes a plurality of PE circuits, and Each of the plurality of PE circuits from a same one of the plurality of PE arrays is configured to perform, in parallel, multiplications between shared elements from the encoded feature vector and different weights of the trained neural network.

27. The system of claim 26, wherein the neural radiation circuit further comprises two weight buffers configured to provide weights of the trained neural network to the plurality of PE arrays in a ping-pong manner.

28. The system of claim 26, wherein the neural radiation circuit further comprises: A partial sum circuit is configured to receive a result from a multiplication performed by each of the plurality of PE circuits and generate a partial sum vector corresponding to the multiplication.

29. The system of claim 28, wherein the neural radiation circuit further comprises: a ReLU and quantization circuit configured to perform a ReLU activation based on the partial sum vector; as well as A bias buffer configured to store biases for ReLU activations.

30. The system of claim 21, wherein the encoding circuit further comprises a coordinator generation circuit, wherein the coordinator generation circuit is configured to: estimating a projection of a ray into the continuous density field based on a Radon transform; and Point sampling is performed along each of the projections of the ray to obtain the spatial coordinates of the voxel.

Images