Sparse-view three-dimensional tomographic reconstruction method and system based on diffusion model prior and neural representation optimization

Abstract

The present application relates to a sparse view three-dimensional tomographic reconstruction method and system based on diffusion model prior and neural representation optimization, belonging to the field of computer tomography and visual technology. The method comprises: inputting an axial slice containing artifacts into a SliceFixer model for processing to obtain a latent variable; inputting multi-modal conditional data, performing slice repair through the SliceFixer model, and outputting a repaired slice; periodically calling the SliceFixer model for slice repair to obtain a pseudo reference body; calculating a three-dimensional structure perception loss of the pseudo reference body and a reconstructed body and injecting it into the optimization of the neural representation model in a weighted form, and realizing three-dimensional tomographic reconstruction based on the optimized neural representation model. The SliceFixer model is used to repair the axial slice with artifacts in combination with multi-modal conditional data, which can repair the missing details and improve the calculation efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer tomography and computer vision, in particular to a sparse view three-dimensional tomographic reconstruction method and system based on diffusion model prior and neural representation optimization. BACKGROUND

[0002] X-ray computed tomography (CT) is an important imaging technique for non-invasive examination of internal structures. A CT scanner acquires projections from multiple views, recording the attenuation information of X-rays penetrating materials. Given these multi-view projections, three-dimensional tomographic reconstruction aims to recover the radiodensity volume. Traditional CT systems usually require hundreds of projections to obtain a clean three-dimensional body, but this will bring a large radiation dose. Therefore, sparse view CT (SVCT), which aims to use only a few tens of projections while still maintaining high-quality reconstruction, has become an important direction for safer imaging. Traditional fan-beam CT reconstructs three-dimensional bodies from one-dimensional projection arrays slice by slice; in recent years, cone-beam CT has become increasingly popular because it can quickly acquire two-dimensional projection images, thus increasing the demand for direct body reconstruction. Traditional methods can be divided into direct and iterative categories: direct methods calculate analytical solutions but have obvious artifacts; iterative methods express reconstruction as an optimization problem and solve it numerically, which can reduce artifacts but easily over-smooth details.

[0003] Similar to traditional algorithms, CT reconstruction based on deep learning can also be divided into direct regression and optimization. Many works use feedforward networks to directly predict results from projections or low-quality reconstructions, but such direct regression lacks physical constraints; therefore, research is increasingly moving towards optimization frameworks, which are divided into neural representation NR and neural prior NP. NR (neural representation) methods optimize a learnable field through differentiable rendering. NeRF and 3DGS in the CT scenario have been explored, but their performance is limited under sparse views. NP (neural prior) methods combine optimization solvers with pre-trained networks: some use deterministic networks as regularization; the latest SOTA couples an unconditional diffusion model with a local solver into iterative denoising, which can enhance details but often causes inter-slice jitter, hallucinated details, and high inference time.

[0004] In summary, how to introduce a powerful data-driven prior under the premise of maintaining the physical consistency of sparse view projections and reconstructing CT three-dimensional bodies, improve reconstruction quality, eliminate artifacts in under-constrained areas, and also consider efficiency, is a key challenge for current SVCT. SUMMARY

[0005] Therefore, in order to solve the above technical problems, a sparse view three-dimensional tomographic reconstruction method and system based on diffusion model prior and neural representation optimization are provided, which can improve the quality of sparse view three-dimensional tomographic reconstruction and improve the reconstruction efficiency.

[0006] A sparse view three-dimensional tomographic reconstruction method based on diffusion model prior and neural representation optimization, the method comprising:

[0007] Sparse view projection data is obtained, and a current reconstruction volume is obtained by volume rendering through a neural representation model; axial slices containing artifacts are obtained by slicing the current reconstruction volume, and input into a single-step graph generation diffusion model SliceFixer; the axial slices are processed through an encoder in the SliceFixer model to obtain latent variables;

[0008] Multi-modal conditional data is obtained and input into the SliceFixer model, and an artifact pattern in the latent variables and a conditional vector in the multi-modal conditional data are extracted through a U-Net structure in the SliceFixer model; a repaired slice is output based on the artifact pattern and the conditional vector through a decoder;

[0009] During the optimization process of the neural representation model, the SliceFixer model is periodically called to repair the axial slices of the current reconstruction volume, and each repaired slice obtained is stacked into a pseudo-reference volume;

[0010] A three-dimensional structure perception loss between the pseudo-reference volume and the current reconstruction volume is calculated, and the three-dimensional structure perception loss is injected into the optimization of the neural representation model in a weighted form, and three-dimensional tomographic reconstruction is realized based on the optimized neural representation model.

[0011] In one embodiment, sparse view projection data is obtained, and a current reconstruction volume is obtained by volume rendering through a neural representation model, comprising:

[0012] Sparse view projection data is obtained, and the SSIM loss is used as the projection domain loss; based on the projection domain loss and three-dimensional total variation regularization, an iterative process is performed to obtain a neural representation model;

[0013] The spatial range of the object to be reconstructed in three-dimensional space is determined, and a three-dimensional grid is divided within the spatial range;

[0014] Based on the coordinate density field in the three-dimensional grid, voxel values are synthesized according to the volume density field, and a current reconstruction volume is constructed.

[0015] In one embodiment, the multi-modal conditional data includes text prompts for providing semantic guidance and orthogonal X-ray projections for providing three-dimensional structure clues; wherein the text prompts and orthogonal X-ray projections are extracted by a pre-trained feature encoder and input into the SliceFixer model as the multi-modal conditional data.

[0016] In one of the embodiments, the artifact pattern in the latent variable and the condition vector in the multi-modal condition data are extracted by the U-Net structure in the SliceFixer model, and the repaired slice is output by the decoder based on the artifact pattern and the condition vector, including:

[0017] The latent variable, text prompt, and orthogonal X-ray projection are input into the U-Net structure in the SliceFixer model; the text prompt is converted into text features by the encoder in the U-Net structure, the orthogonal X-ray projection is converted into image features by the pre-trained feature encoder, and the text features and image features are fused to obtain a condition vector;

[0018] The input time step is obtained, the time step is converted into a time embedding vector, and the time embedding vector is loaded onto the feature map of each layer of the U-Net encoder and decoder to complete the denoising process;

[0019] The artifact pattern in the latent variable is extracted, and the artifact pattern is mapped based on the condition variable to obtain a repaired latent variable;

[0020] The repaired latent variable is input into the VAE decoder in the U-Net structure, decoded back to the pixel space, and a repaired slice is obtained.

[0021] In one of the embodiments, during the optimization process of the neural representation model, the SliceFixer model is periodically called to repair the axial slices of the current reconstructed body, and the obtained each repaired slice is stacked into a pseudo-reference body, including:

[0022] The current neural representation model is taken as a three-dimensional density field function, the three-dimensional density field function is batch queried in a pre-defined three-dimensional grid, the density values of each grid point are obtained, and a current reconstructed body is obtained;

[0023] The current reconstructed body is divided into each two-dimensional slice, and the text prompt and orthogonal X-ray projection of each two-dimensional slice are determined, the SliceFixer model is called for single-step forward propagation, and a repaired slice is output;

[0024] According to the axial order of each repaired slice, a pseudo-reference body is formed by stacking, and iterative optimization is performed.

[0025] In one of the embodiments, the three-dimensional structure perception loss is the average value of the two-dimensional structure similarity index calculated in the axial, sagittal, and coronal directions.

[0026] In one of the embodiments, a three-dimensional structure perception loss between the pseudo reference body and the current reconstructed body is calculated, and the three-dimensional structure perception loss is injected into the optimization of the neural representation model in a weighted manner, including:

[0027] A two-dimensional structure similarity index calculated in axial, sagittal and coronal directions of the pseudo reference body and the current reconstructed body is obtained respectively, and an average value of the two-dimensional structure similarity index is calculated as the three-dimensional structure perception loss between the pseudo reference body and the current reconstructed body;

[0028] A three-dimensional total variation regularization term is determined, and a total loss function is constructed based on the three-dimensional total variation regularization term and the three-dimensional structure perception loss;

[0029] The gradient of the total loss function with respect to the parameters in the neural representation model is calculated, and the parameters are optimized according to the gradient using an optimizer to adjust the density field inside the neural representation model.

[0030] A sparse view three-dimensional tomographic reconstruction system based on the combination of diffusion model prior and neural representation optimization, the system comprising:

[0031] A SliceFixer model processing module is configured to extract axial slices containing artifacts from a current reconstructed body, and input the axial slices into a single-step graph generation diffusion model SliceFixer, and process the axial slices through an encoder in the SliceFixer model to obtain latent variables;

[0032] A slice repair module is configured to input multi-modal conditional data into the SliceFixer model, extract artifact patterns in the latent variables and conditional vectors in the multi-modal conditional data through a U-Net structure in the SliceFixer model, and output repaired slices based on the artifact patterns and the conditional vectors through a decoder;

[0033] A pseudo reference body construction module is configured to periodically call the SliceFixer model to repair axial slices of a current reconstructed body during the optimization process of the neural representation model, and stack the obtained each repaired slice into a pseudo reference body;

[0034] A model optimization module is configured to obtain sparse view projection data, perform volume rendering through a neural representation model to obtain a current reconstructed body, calculate a three-dimensional structure perception loss between the pseudo reference body and the current reconstructed body, and inject the three-dimensional structure perception loss into the optimization of the neural representation model in a weighted manner, so as to realize three-dimensional tomographic reconstruction based on the optimized neural representation model.

[0035] In one of the embodiments, the model optimization module is further configured to obtain sparse view projection data, take the SSIM loss as a projection domain loss, and iteratively obtain a neural representation model based on the projection domain loss and three-dimensional total variation regularization; determine a spatial range of a three-dimensional tomographic reconstruction object in a three-dimensional space, and divide a three-dimensional grid in the spatial range; reconstruct a body density field based on coordinates in the three-dimensional grid, and synthesize voxel values according to the body density field to obtain a current reconstruction object.

[0036] The sparse view three-dimensional tomographic reconstruction method and system based on the diffusion model prior and the neural representation optimization combination can repair the axial slice of the artifact using the SliceFixer model combined with multi-modal conditional data, can repair the missing details, and can improve the quality of the three-dimensional tomographic reconstruction; the three-dimensional structure perception loss is used for supervision, so that the neural representation model considers the projection error when optimizing, and ensures the coordination of the reconstruction object; and the SliceFixer single-step graph generation diffusion model is used, which can greatly improve the calculation efficiency and improve the reconstruction efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 An application environment diagram of the sparse view three-dimensional tomographic reconstruction method based on the diffusion model prior and the neural representation optimization combination in one of the embodiments;

[0038] Figure 2 A flowchart of the sparse view three-dimensional tomographic reconstruction method based on the diffusion model prior and the neural representation optimization combination in one of the embodiments;

[0039] Figure 3 A schematic diagram of the SliceFixer model architecture in one of the embodiments;

[0040] Figure 4 A schematic diagram of the neural representation optimization framework DiffNR in one of the embodiments;

[0041] Figure 5 A visualization result schematic diagram of the ToothFairy and LUNA16 dataset experiment in one of the embodiments;

[0042] Figure 6 A visualization result schematic diagram of the OOD dataset experiment in one of the embodiments;

[0043] Figure 7 A structural block diagram of the sparse view three-dimensional tomographic reconstruction system based on the diffusion model prior and the neural representation optimization combination in one of the embodiments;

[0044] Figure 8 An internal structure diagram of the computer device in one of the embodiments. DETAILED DESCRIPTION

[0045] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not intended to limit the present application.

[0046] The sparse-view three-dimensional tomographic reconstruction method based on diffusion model prior and neural representation optimization provided by the embodiments of the present application can be applied in an application environment as shown in Figure 1 As shown in Figure 1 The application environment includes a computer device 110. The computer device 110 can obtain sparse-view projection data, perform volume rendering on the sparse-view projection data by a neural representation model to obtain a current reconstruction volume, perform slice extraction on the current reconstruction volume to obtain axial slices containing artifacts, input the axial slices into a single-step graph generation diffusion model SliceFixer, process the axial slices by an encoder in the SliceFixer model to obtain latent variables, obtain multi-modal condition data, input the multi-modal condition data into the SliceFixer model, extract artifact patterns in the latent variables and condition vectors in the multi-modal condition data by a U-Net structure in the SliceFixer model, and output repaired slices based on the artifact patterns and the condition vectors by a decoder. The computer device 110 can periodically call the SliceFixer model to repair the axial slices of the current reconstruction volume in an optimization process of the neural representation model, stack the obtained repaired slices into a pseudo-reference volume, calculate a three-dimensional structure perception loss between the pseudo-reference volume and the current reconstruction volume, inject the three-dimensional structure perception loss into the optimization of the neural representation model in a weighted manner, and perform three-dimensional tomographic reconstruction based on the optimized neural representation model. The computer device 110 can be, but is not limited to, various personal computers, notebook computers, smart phones, robots, unmanned aerial vehicles, tablet computers and the like.

[0047] In one embodiment, as shown in Figure 2 A sparse-view three-dimensional tomographic reconstruction method based on diffusion model prior and neural representation optimization is provided, including the following steps:

[0048] In step 202, sparse-view projection data is obtained, volume rendering is performed on the sparse-view projection data by a neural representation model to obtain a current reconstruction volume, slice extraction is performed on the current reconstruction volume to obtain axial slices containing artifacts, and the axial slices are input into a single-step graph generation diffusion model SliceFixer. The axial slices are processed by an encoder in the SliceFixer model to obtain latent variables.

[0049] Among them, the single-step graph generation diffusion model SliceFixer is a modified model based on the framework of the single-step diffusion model sd-turbo, which is a conditional single-step diffusion enhancer. First, a CT slice with artifacts is obtained from the neural representation (Neural Fields / 3DGaussians). Then, a frozen VAE encoder is used to encode the image into a latent space. A U-Net with an adapter is used to denoise and repair the image in the latent space (single step). The text prompt and X-ray / projection features are injected through Cross-Attention to guide the model to remove artifacts according to organ / scene semantics. Finally, the repaired slice is obtained through VAE decoding.

[0050] The computer device acquires real, directly acquired original two-dimensional image data through a physical scanning device. The physical scanning device can acquire N X-ray projections according to a given uniform angular interval Due to the small number of acquired viewing angles, the obtained projection data is sparse in viewing angle.

[0051] In one embodiment, the provided sparse-view three-dimensional tomographic reconstruction method based on diffusion model prior and neural representation optimization can further include the process of obtaining a current reconstruction, and the specific process includes: acquiring sparse-view projection data, taking SSIM loss as the projection domain loss, and iteratively obtaining a neural representation model based on the projection domain loss and three-dimensional total variation regularization; determining the spatial range of the three-dimensional tomographic reconstruction object in the three-dimensional space, and dividing a three-dimensional grid in the spatial range; reconstructing the body density field based on the coordinates in the three-dimensional grid, synthesizing the voxel values according to the body density field, and constructing the current reconstruction.

[0052] Among them, the current reconstruction is a three-dimensional body data obtained by rendering or querying the current, not fully optimized neural representation model on a three-dimensional grid point set at a certain specific iteration time of the neural representation optimization process.

[0053] The sparse view projection data obtained by the computer device is a constraint condition that can drive the optimization of the neural representation (NR) model, but because of insufficient information, the quality of the current reconstruction body optimized is poor. In the embodiment, the SSIM loss is used as the projection domain loss and the three-dimensional total variation regularization is iteratively trained to obtain the neural representation model. To construct the current reconstruction body, the density value predicted by the neural representation model is used as the value of each voxel, or the alpha value is directly used to form the body data. Specifically, the computer device can explicitly define the bounding box occupied by the three-dimensional tomographic reconstruction in the three-dimensional space, and densely and uniformly divide the three-dimensional grid in the bounding box. Each point (x, y, z) on the grid is a query coordinate, and the granularity of the grid determines the clarity of the current reconstruction body. Then, the computer device can input each coordinate (x, y, z) into the current neural representation model, map the spatial position to the density through the neural representation model, and obtain the original 3D density array, which is used as the current reconstruction body.

[0054] In step 204, the multi-modal conditional data is input into the SliceFixer model, the artifact pattern in the latent variable and the condition vector in the multi-modal conditional data are extracted through the U-Net structure in the SliceFixer model, and the repaired slice is output based on the artifact pattern, the condition vector and the decoder.

[0055] To simultaneously remove artifacts and ensure the true structure of the slice, in an embodiment, multi-modal conditional data can be used as input data, and the multi-modal conditional data includes text prompts for providing semantic guidance and orthogonal X-ray projections for providing three-dimensional structure clues. After the text prompts and the orthogonal X-ray projections are extracted by the pre-trained feature encoder, they are input into the SliceFixer model as multi-modal conditional data. The text prompts (such as “lung CT slice”) and the two orthogonal X-ray projections (biplane) are used jointly, the text provides high-level semantic guidance, and the orthogonal X-ray projections provide global structure clues. In the embodiment, a pre-trained encoder for radiological images can be used to extract image features, which are then aggregated with text embeddings through cross-attention to form the condition input of the diffusion model.

[0056] In one embodiment, a sparse-view 3D tomography reconstruction method based on a combination of diffusion model priors and neural representation optimization may further include a slice restoration process. Specifically, this process includes: inputting latent variables, text cues, and orthogonal X-ray projections into the U-Net structure of the SliceFixer model; converting the text cues into text features using the encoder in the U-Net structure, converting the orthogonal X-ray projections into image features using a pre-trained feature encoder, and fusing the text features and image features to obtain a conditional vector; acquiring the input time step, converting the time step into a temporal embedding vector, and loading the temporal embedding vector onto the feature maps of each layer of the U-Net encoder and decoder to complete denoising; extracting artifact patterns from the latent variables, mapping the artifact patterns based on the conditional variables to obtain the restored latent variables; and inputting the restored latent variables into the VAE decoder in the U-Net structure to decode back into pixel space to obtain the restored slice.

[0057] Previous NR-based methods often perform repair at the projection level and incorporate direct image loss to optimize the 3D model; however, for volume reconstruction, penetrable X-ray projection errors accumulate. Therefore, in this embodiment, a SliceFixer model is proposed, such as... Figure 3 As shown, the aim is to predict the repaired version of an axial slice obtained from an NR (Neural Representation) query. A single-step graph-to-image diffusion model is constructed, in which the VAE encodes the axial slice as a latent variable, the U-Net structure predicts the target latent variable given the input encoding, conditions and denoising time steps, and finally the repaired slice is obtained by VAE decoding.

[0058] Specifically, in this embodiment, such as Figure 3 As shown, the VAE encoder compresses artifact-laden slices into latent variables, which are then passed through the input convolutional layer to increase the number of channels to the same level as the U-Net backbone network, facilitating subsequent processing. Text prompts are converted into text features by a text encoder, and orthogonal X-ray projections are converted into image features by an image encoder pre-trained for medical images. Text and image features are fused in the U-Net's cross-attention layer to obtain a conditional vector. Since the SliceFixer model is a single-step model, the time step is typically a fixed value or a simplified representation. Temporal embedding vectors are transformed using sinusoidal positional encoding or a small MLP, and these vectors are then added to the feature maps of each layer in the U-Net encoder and decoder to complete the denoising process.

[0059] In the embodiment, the U-Net is an encoder-bottleneck-decoder structure with skip connections. Among them, the encoder can gradually extract features from low to high in the latent variable, while compressing the spatial size and increasing the number of channels, and the final output is a highly compressed bottleneck feature; the decoder gradually up-samples based on the features extracted by the encoder and the conditions, and reconstructs the repaired latent variable to restore the spatial details. The final output of the decoder is passed through an output convolutional layer to reduce the number of channels to the same dimension as the input latent variable; the repaired latent variable is sent to the VAE decoder to decode back to the pixel space to obtain the final, high-quality repaired slice. Each repaired slice is stacked into a pseudo-reference volume to guide the next optimization of the NR model.

[0060] In the embodiment, in order to let SliceFixer learn general artifact patterns rather than overfitting a certain NR, the training process of the SliceFixer model is also designed: the reconstruction results of the neural field and the three-dimensional Gaussian are mixed in a 1:1 ratio to construct a training pair; and the 256x256 slice is up-sampled to 512x512 for fine-tuning. Higher input resolution helps improve repair quality with up-sampling and down-sampling. In terms of training objectives, in addition to common adversarial and perceptual loss terms, an SSIM term is added to strengthen structural consistency.

[0061] Step 206, in the optimization process of the neural representation model, periodically call the SliceFixer model to repair the axial slices of the current reconstruction volume, and stack the obtained each repaired slice into a pseudo-reference volume.

[0062] In one embodiment, the sparse view three-dimensional tomographic reconstruction method based on diffusion model prior and neural representation optimization combination provided can further include the process of periodically calling the SliceFixer model for slice repair to obtain a pseudo-reference volume, and the specific process includes: taking the current neural representation model as a three-dimensional density field function, batch querying the three-dimensional density field function in a pre-defined three-dimensional grid, obtaining the density values of each grid point, and obtaining the current reconstruction volume; dividing the current reconstruction volume into two-dimensional slices, determining the text prompts and orthogonal X-ray projections of each two-dimensional slice, calling the SliceFixer model for single-step forward propagation, and outputting the repaired slice; stacking according to the axial order of each repaired slice to form a pseudo-reference volume, and performing iterative optimization.

[0063] The optimization process of the neural representation model needs to be iterated multiple times. The computer device can regard the current neural representation model as a three-dimensional density field function, obtain the density value of each grid point on a predefined three-dimensional grid, and assemble the current reconstruction body. Then, the computer device can cut the three-dimensional body into continuous two-dimensional slices along the axial direction, perform single-step forward propagation through SliceFixer, and output the repaired slice. Finally, the computer device can re-stack the repaired slice in the original axial order to form a pseudo-reference body. In the next optimization iteration, the computer device can again query the new current reconstruction body from the NR model, calculate the three-dimensional structural similarity loss between the new current reconstruction body and the generated pseudo-reference body, add the three-dimensional structural similarity loss to the total loss function with a certain weight, update the parameters of the NR model through back propagation, and continue optimization until the next iteration period comes, and repeat to generate a pseudo-reference body with higher quality to continue guiding optimization.

[0064] In step 208, the three-dimensional structural perception loss between the pseudo-reference body and the current reconstruction body is calculated, and the three-dimensional structural perception loss is injected into the optimization of the neural representation model in a weighted form. The three-dimensional tomographic reconstruction is realized based on the optimized neural representation model.

[0065] The three-dimensional structural perception loss is the average value of the two-dimensional structural similarity indexes calculated in the axial, sagittal and coronal directions.

[0066] In one embodiment, the sparse-view three-dimensional tomographic reconstruction method based on the diffusion model prior combined with neural representation optimization further includes integrating SliceFixer into the process of NR optimization. The specific process includes: obtaining the two-dimensional structural similarity indexes calculated in the axial, sagittal and coronal directions of the pseudo-reference body and the current reconstruction body, respectively, and calculating the average value of the two-dimensional structural similarity indexes as the three-dimensional structural perception loss between the pseudo-reference body and the current reconstruction body; determining a three-dimensional total variation regularization term, constructing a total loss function based on the three-dimensional total variation regularization term and the three-dimensional structural perception loss; calculating the gradient of the total loss function with respect to the parameters in the neural representation model, and using an optimizer to optimize the parameters according to the gradient to adjust the density field inside the neural representation model.

[0067] The traditional pixel-wise L1 loss does not work well here, because the details produced by SliceFixer can not be perfectly consistent with the details naturally generated by the model, and forced alignment can cause blurring. SSIM is a perceptual index that reflects human eye perception of structural information better than L1 loss, comparing two images from three aspects of brightness, contrast, and structure. The computer device can calculate 2D SSIM on three orthogonal planes of axial, sagittal, and coronal, and then take the average, which is more efficient than directly calculating true three-dimensional SSIM, and can effectively capture the consistency of three-dimensional structure. The total loss of the original neural representation model is composed of the basic loss term. Based on the total loss function, through back propagation and parameter update, the density field inside the neural representation model can be adjusted.

[0068] In one embodiment, a sparse-view three-dimensional tomographic reconstruction method based on the combination of a diffusion model prior and neural representation optimization can be applied to, for example, Figure 4 As shown in the neural representation optimization framework DiffNR, the neural representation optimization framework DiffNR is a NR optimization framework with diffusion enhancement, taking the conditional single-step diffusion model SliceFixer as an artifact repairer, periodically generating pseudo-reference bodies and guiding the optimization process of NR by three-dimensional perceptual loss, thereby improving the reconstruction quality and generalization ability of the model without significantly increasing the computational cost.

[0069] In one embodiment, directly using the SliceFixer model as post-processing can easily introduce slice jitter and hallucinated details, and cannot ensure volume consistency. DiffNR embeds SliceFixer into the NR optimization loop to form a closed loop of repair and enhancement, as shown in Figure 4 As shown in the neural representation optimization framework DiffNR, the neural representation optimization framework DiffNR is a NR optimization framework with diffusion enhancement, taking the conditional single-step diffusion model SliceFixer as an artifact repairer, periodically generating pseudo-reference bodies and guiding the optimization process of NR by three-dimensional perceptual loss, thereby improving the reconstruction quality and generalization ability of the model without significantly increasing the computational cost.

[0070] In one embodiment, in order to verify the effectiveness of a sparse view three-dimensional tomographic reconstruction method based on a diffusion model prior combined with neural representation optimization provided in the present application applied in the neural representation optimization framework DiffNR, experiments are performed using two datasets ToothFairy and LUNA16, wherein ToothFairy contains 443 dental CBCTs (393 / 25 / 25, training / validation / test), LUNA16 contains 888 chest CTs (838 / 25 / 25), and SliceFixer models are trained on the respective training sets for reconstruction on the corresponding test sets. The sparse setting of the evaluation view is 36 / 24 / 12. The quantitative results on ToothFairy and LUNA16 are shown in the following table:

[0071]

[0072] The traditional and self-supervised NR artifacts are obvious; the NP (diffusion-iteration) method has higher scores than the traditional and NR methods, but brings hallucinations and significant computational overhead. In comparison, DiffNR continuously improves the two NR backbones: NAF is improved by an average of 2.19 dB, and the average is improved by 5.79 dB; although additional optimization time is introduced, it is still much faster than the diffusion-iteration method. The visualization results are shown in Figure 5 DiffNR can restore finer structures and suppress artifacts.

[0073] During the experiment, the SliceFixer pre-trained on ToothFairy is used to evaluate the proposed 18 diverse data in the test, and the + DiffNR framework is used, and the quantitative results on the OOD dataset are shown in the following table:

[0074]

[0075] It can be seen that DiffNR better suppresses artifacts and hallucinations, indicating that SliceFixer has learned generalizable artifact patterns, and the visualization results of the OOD dataset experiment are shown in Figure 6 .

[0076] The application provides a sparse view three-dimensional tomographic reconstruction method based on diffusion model prior and neural representation optimization, proposes DiffNR, a method for efficiently integrating diffusion prior into the neural representation optimization framework in sparse view three-dimensional tomographic reconstruction; the core module SliceFixer is a single-step diffusion model for NR reconstruction slice artifacts; a pseudo-reference body is periodically generated during the reconstruction process to provide three-dimensional perception regularization; the repair and enhancement strategy avoids frequent query of the diffusion model, significantly improves the reconstruction quality and generalization ability while maintaining good efficiency. Through the SliceFixer diffusion model, the learned text and orthogonal X-ray projection conditions are injected into the reconstruction process, which can repair missing details and generate a three-dimensional body data that is more reasonable in structure and clearer in vision; the repair process is embedded in the optimization cycle of the neural representation, and is supervised by the three-dimensional perception loss, which enables the neural representation model to consider not only the projection error but also the pseudo-reference body as the target during optimization, and finally outputs a reconstruction result with high body consistency; the single-step diffusion model SliceFixer can greatly improve the computational efficiency.

[0077] It should be understood that although each step in the above flowchart is displayed in sequence according to the direction of the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in the above flowchart can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but can be alternately executed with at least part of other steps or sub-steps or stages of other steps.

[0078] In one embodiment, as shown in Figure 7 A sparse view three-dimensional tomographic reconstruction system based on diffusion model prior and neural representation optimization is provided, including a SliceFixer model processing module 710, a slice repair module 720, a pseudo-reference body construction module 730, and a model optimization module 740, wherein:

[0079] The SliceFixer model processing module 710 is configured to extract an axial slice containing artifacts from the current reconstructed body, input the axial slice into the single-step graph generation diffusion model SliceFixer, process the axial slice through the encoder in the SliceFixer model, and obtain a latent variable;

[0080] The slice repair module 720 is configured to input the multi-modal conditional data into the SliceFixer model, extract the artifact pattern in the latent variable and the conditional vector in the multi-modal conditional data through the U-Net structure in the SliceFixer model, and output the repaired slice based on the artifact pattern and the conditional vector through the decoder.

[0081] The pseudo-reference body construction module 730 is configured to periodically call the SliceFixer model to repair the axial slice of the current reconstructed body in the optimization process of the neural representation model, and stack the obtained each repaired slice into a pseudo-reference body.

[0082] The model optimization module 740 is configured to obtain sparse view projection data, perform volume rendering on the neural representation model to obtain a current reconstructed body, calculate a three-dimensional structure perception loss between the pseudo-reference body and the current reconstructed body, and inject the three-dimensional structure perception loss into the optimization of the neural representation model in a weighted manner, so as to realize three-dimensional tomographic reconstruction based on the optimized neural representation model.

[0083] In one embodiment, the model optimization module 740 is further configured to obtain sparse view projection data, and take the SSIM loss as a projection domain loss, and iteratively obtain the neural representation model based on the projection domain loss and the three-dimensional total variation regularization; determine a spatial range of the three-dimensional tomographic reconstruction object in the three-dimensional space, and divide a three-dimensional grid in the spatial range; reconstruct a body density field based on the coordinates in the three-dimensional grid, and synthesize voxel values according to the body density field to obtain a current reconstructed body.

[0084] In one embodiment, the multi-modal conditional data includes a text prompt for providing semantic guidance and orthogonal X-ray projection for providing three-dimensional structure clues; wherein the text prompt and the orthogonal X-ray projection are extracted as multi-modal conditional data into the SliceFixer model after being processed by a pre-trained encoder.

[0085] In one embodiment, the slice repair module 720 is further configured to input the latent variable, the text prompt and the orthogonal X-ray projection into the U-Net structure in the SliceFixer model; convert the text prompt into text features and the orthogonal X-ray projection into image features through the encoder in the U-Net structure, fuse the text features and the image features to obtain a conditional vector; convert the input time step into a time embedding vector, load the time embedding vector onto the feature map of each layer of the U-Net encoder and decoder to complete denoising, extract the artifact pattern in the latent variable, map the artifact pattern based on the conditional variable to obtain a repaired latent variable; input the repaired latent variable into the VAE decoder in the U-Net structure to decode back to the pixel space to obtain a repaired slice.

[0086] In an embodiment, the pseudo-reference body construction module 730 is further configured to obtain the current neural representation model as a three-dimensional density field function, batch query the three-dimensional density field function in a predefined three-dimensional grid, obtain density values of each grid point, and obtain a current reconstruction body; split the current reconstruction body into two-dimensional slices, determine text prompts and orthogonal X-ray projections of each two-dimensional slice, call a SliceFixer model for single-step forward propagation, and output a repaired slice; and stack the repaired slices according to an axial order to form a pseudo-reference body and perform iterative optimization.

[0087] In an embodiment, the three-dimensional structure perception loss is an average of two-dimensional structure similarity indexes calculated in the axial, sagittal, and coronal directions.

[0088] In an embodiment, the pseudo-reference body construction module 730 is further configured to obtain two-dimensional structure similarity indexes calculated in the axial, sagittal, and coronal directions between the pseudo-reference body and the current reconstruction body, respectively, and calculate an average of the two-dimensional structure similarity indexes as a three-dimensional structure perception loss between the pseudo-reference body and the current reconstruction body; determine a three-dimensional total variation regularization term, construct a total loss function based on the three-dimensional total variation regularization term and the three-dimensional structure perception loss; calculate a gradient of the total loss function with respect to parameters in the neural representation model, and use an optimizer to perform parameter optimization according to the gradient to adjust the density field inside the neural representation model.

[0089] In an embodiment, a computer device is provided, which can be a terminal, and an internal structure diagram of the computer device can be as shown in Figure 8 The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is configured to communicate with external terminals through network connections. The computer program is executed by the processor to implement a sparse view three-dimensional tomographic reconstruction method based on diffusion model prior and neural representation optimization combination. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or can be a key, trackball, or touchpad arranged on the shell of the computer device, or can be an external keyboard, touchpad, or mouse, etc.

[0090] Those skilled in the art can understand that, Figure 8The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0091] In one embodiment, a computer device is provided, comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps of the sparse-view three-dimensional tomographic reconstruction method based on the combination of the diffusion model prior and the neural representation optimization when executing the computer program.

[0092] In one embodiment, a computer readable storage medium is provided, storing a computer program, and the processor implementing the steps of the sparse-view three-dimensional tomographic reconstruction method based on the combination of the diffusion model prior and the neural representation optimization when executing the computer program.

[0093] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0094] Each technical feature of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described, however, as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present application.

[0095] The above-described embodiments are merely illustrative of several embodiments of the present application, which are described in more detail and in a specific and detailed manner, but should not be construed as limiting the scope of the patent. It should be noted that for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A sparse view three-dimensional tomographic reconstruction method based on the combination of a diffusion model prior and neural representation optimization, characterized in that, The method comprises: acquiring sparse view projection data, obtaining a current reconstruction volume by volume rendering through a neural representation model; performing slice extraction on the current reconstruction volume to obtain an axial slice containing artifacts, inputting the axial slice into a single-step graph generation diffusion model SliceFixer, processing the axial slice through an encoder in the SliceFixer model to obtain a latent variable; acquiring multi-modal conditional data and inputting the multi-modal conditional data into the SliceFixer model, wherein the multi-modal conditional data comprises a text prompt for providing semantic guidance and orthogonal X-ray projection for providing three-dimensional structure clues; wherein the text prompt and the orthogonal X-ray projection are extracted by a pre-trained feature encoder and input into the SliceFixer model as the multi-modal conditional data; the artifact pattern in the latent variable and the conditional vector in the multi-modal conditional data are extracted through a U-Net structure in the SliceFixer model, and a repaired slice is output through a decoder based on the artifact pattern and the conditional vector, comprising: inputting the latent variable, the text prompt, and the orthogonal X-ray projection into the U-Net structure in the SliceFixer model; converting the text prompt into a text feature through an encoder in the U-Net structure, converting the orthogonal X-ray projection into an image feature through a pre-trained feature encoder, and fusing the text feature and the image feature to obtain a conditional vector; acquiring an input time step, converting the time step into a time embedding vector, and loading the time embedding vector onto the feature map of each layer of the U-Net encoder and decoder to complete denoising; extracting the artifact pattern in the latent variable, mapping the artifact pattern based on the conditional vector to obtain a repaired latent variable; inputting the repaired latent variable into a VAE decoder in the U-Net structure to decode back to the pixel space to obtain a repaired slice; In the optimization process of the neural representation model, the SliceFixer model is periodically called to repair the axial slice of the current reconstruction volume, and the obtained each repaired slice is stacked into a pseudo reference volume; calculating a three-dimensional structure perception loss between the pseudo reference volume and the current reconstruction volume, and injecting the three-dimensional structure perception loss in a weighted form into the optimization of the neural representation model, and realizing three-dimensional tomographic reconstruction based on the optimized neural representation model.

2. The sparse-view 3D tomographic reconstruction method based on diffusion model prior combined with neural representation optimization according to claim 1, characterized in that, acquiring sparse view projection data, obtaining a current reconstruction volume by volume rendering through a neural representation model, comprising: acquiring sparse view projection data, and taking the SSIM loss as the projection domain loss, iteratively based on the projection domain loss and three-dimensional total variation regularization to obtain a neural representation model; determining the spatial range of the three-dimensional tomographic reconstruction object in the three-dimensional space, and dividing a three-dimensional grid in the spatial range; based on the coordinate density field in the three-dimensional grid, synthesizing voxel values according to the volume density field to construct a current reconstruction volume.

3. The sparse-view 3D tomographic reconstruction method based on diffusion model prior combined with neural representation optimization according to claim 1, characterized in that, In the optimization process of the neural representation model, the SliceFixer model is periodically called to repair the axial slices of the current reconstruction body, and each repaired slice is stacked into a pseudo-reference body, including: The current neural representation model is taken as a three-dimensional density field function, and the three-dimensional density field function is batch queried in a predefined three-dimensional grid to obtain the density values of each grid point, thereby obtaining the current reconstruction body; The current reconstruction body is divided into two-dimensional slices, and the text prompt and orthogonal X-ray projection of each two-dimensional slice are determined, and the SliceFixer model is called for single-step forward propagation to output a repaired slice; According to the axial order of each repaired slice, the pseudo-reference body is stacked and iteratively optimized.

4. The sparse-view 3D tomographic reconstruction method based on diffusion model prior combined with neural representation optimization according to claim 1, characterized in that, The three-dimensional structure perception loss is the average value of the two-dimensional structure similarity indexes calculated in the axial, sagittal and coronal directions.

5. The sparse-view 3D tomographic reconstruction method based on diffusion model prior combined with neural representation optimization according to claim 4, characterized in that, The three-dimensional structure perception loss between the pseudo-reference body and the current reconstruction body is calculated, and the three-dimensional structure perception loss is injected into the optimization of the neural representation model in a weighted manner, including: The two-dimensional structure similarity indexes calculated in the axial, sagittal and coronal directions of the pseudo-reference body and the current reconstruction body are obtained respectively, and the average value of the two-dimensional structure similarity indexes is calculated as the three-dimensional structure perception loss between the pseudo-reference body and the current reconstruction body; A three-dimensional total variation regularization term is determined, and a total loss function is constructed based on the three-dimensional total variation regularization term and the three-dimensional structure perception loss; The gradient of the total loss function with respect to the parameters in the neural representation model is calculated, and the optimizer is used to optimize the parameters according to the gradient to adjust the density field inside the neural representation model.

6. A sparse view three-dimensional tomographic reconstruction system based on a combination of a diffusion model prior and neural representation optimization, characterized in that, The system comprises: A SliceFixer model processing module is configured to extract slices from the current reconstruction body to obtain axial slices containing artifacts, input the axial slices into a single-step graph generation diffusion model SliceFixer, and process the axial slices through an encoder in the SliceFixer model to obtain latent variables; The slice repair module is configured to input multi-modal conditional data into the SliceFixer model, wherein the multi-modal conditional data comprises a text prompt for providing semantic guidance and orthogonal X-ray projections for providing three-dimensional structure clues; wherein the text prompt and the orthogonal X-ray projections are extracted by a pre-trained feature encoder and then input into the SliceFixer model as the multi-modal conditional data; the artifact pattern in the latent variable and the conditional vector in the multi-modal conditional data are extracted by a U-Net structure in the SliceFixer model, and a repaired slice is output by a decoder based on the artifact pattern and the conditional vector, including: inputting the latent variable, the text prompt and the orthogonal X-ray projections into the U-Net structure in the SliceFixer model; converting the text prompt into a text feature by an encoder in the U-Net structure, converting the orthogonal X-ray projections into an image feature by a pre-trained feature encoder, fusing the text feature and the image feature to obtain a conditional vector; converting an input time step into a time embedding vector, and loading the time embedding vector onto a feature map of each layer of the U-Net encoder and the decoder to complete denoising processing; extracting the artifact pattern in the latent variable, mapping the artifact pattern based on the conditional vector to obtain a repaired latent variable; inputting the repaired latent variable into a VAE decoder in the U-Net structure to decode back to a pixel space to obtain a repaired slice; The pseudo-reference body construction module is configured to periodically call the SliceFixer model to repair axial slices of a current reconstruction body during the optimization process of the neural representation model, and stack the obtained each repaired slice into a pseudo-reference body. The model optimization module is configured to obtain sparse view projection data, perform volume rendering on the neural representation model to obtain a current reconstruction body, calculate a three-dimensional structure perception loss between the pseudo-reference body and the current reconstruction body, and inject the three-dimensional structure perception loss into the optimization of the neural representation model in a weighted form, so as to realize three-dimensional tomographic reconstruction based on the optimized neural representation model.

7. The sparse view three-dimensional tomographic reconstruction system based on a combination of a diffusion model prior and neural representation optimization of claim 6, wherein, The model optimization module is further configured to obtain sparse view projection data, take the SSIM loss as a projection domain loss, and iteratively perform three-dimensional total variation regularization based on the projection domain loss to obtain a neural representation model; determine a spatial range of a three-dimensional tomographic reconstruction object in a three-dimensional space, and divide a three-dimensional grid in the spatial range; reconstruct a body density field based on coordinates in the three-dimensional grid, synthesize voxel values according to the body density field, and construct a current reconstruction body.

Images