Method and system for generating three-dimensional CT image by using two-dimensional X-ray image
Through a generative adversarial network based on frequency decoupling and diffusion, high-frequency information of two-dimensional X-ray images is extracted, combined with generator and discriminator training, the problem of low quality of three-dimensional CT volume image generation in two-dimensional X-ray images is solved, and the problem of low volume image quality of two-dimensional X-ray images is realized. High-quality image generation and training efficiency improvement.
Patent Information
- Application Number
- CN202411716120.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-27
- Publication Date
- 2024-12-27
AI Technical Summary
The existing two-dimensional X-ray image generation three-dimensional CT volume image method cannot effectively capture high-frequency information, resulting in low generation quality.
A generative adversarial network based on frequency decoupling and diffusion is adopted to extract high-frequency information of two-dimensional X-ray images layer by layer through frequency decoupling blocks, and train them in combination with generators and discriminators to generate high-quality three-dimensional CT images. The network includes frequency decoupling blocks, generators and discriminators. It uses a frequency decoupling cascade module and a depth-separable convolution with a pyramid-like structure. It combines a diffusion module to perform adaptive noise diffusion, and dynamically adjusts the noise intensity during training. .
The quality of 3D CT images generated by two-dimensional X-ray images is significantly improved, the problem of insufficient high-frequency information capture is solved, the training time is shortened, and the model complexity and computing resource requirements are reduced.
Smart Images
Figure CN119206095A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of medical image generation, and more specifically, to a method and system for generating a three-dimensional CT image using a two-dimensional X-ray image. Background Art
[0002] The technology of generating three-dimensional CT images from two-dimensional rays is a medical imaging technology that has gradually emerged in recent years. It aims to overcome the limitations of traditional X-ray and computed tomography (CT) imaging and provide more accurate and effective diagnostic tools.
[0003] Traditional two-dimensional X-ray imaging is to generate a flat image based on the difference in the absorption of X-rays by different tissues through X-rays penetrating the human body. This technology is widely used in clinical practice because it is non-invasive, has a low radiation dose and is low in cost, allowing doctors to quickly assess the health of patients. However, two-dimensional X-ray images have insufficient depth information and blurring caused by overlapping tissues of different densities, making it difficult for doctors to make diagnoses. In order to obtain more three-dimensional and clear anatomical information, computed tomography (CT) has emerged as an advanced imaging technology. CT reconstructs three-dimensional images through X-ray projection data from multiple angles, which can provide detailed cross-sectional information, thereby better displaying the internal structure, especially when analyzing complex anatomical parts, which has great advantages.
[0004] However, CT technology also has some disadvantages, such as high radiation dose and high scanning cost. In CT examination, the amount of radiation received by patients is usually 300 times that of conventional X-ray examination, which limits its application in some cases, especially for children, pregnant women or other patients who are sensitive to radiation. Therefore, although CT can provide high-definition three-dimensional images, its significant radiation risk and economic burden are still factors that medical workers must consider when choosing imaging methods.
[0005] In recent years, with the development of computer image processing technology and deep learning, the research on generating three-dimensional CT images based on two-dimensional X-ray images has been increasing. This technology uses advanced algorithms to reconstruct three-dimensional volume images from information extracted from two-dimensional X-ray images, overcoming the shortcomings of traditional two-dimensional imaging. Using machine learning and deep learning algorithms, the ability to recognize tissue features can be improved, thereby achieving a more accurate description of three-dimensional anatomical structures.
[0006] Although this emerging technology has shown broad application prospects, it still faces some challenges. For example, the existing image reconstruction algorithm still needs a lot of training and optimization based on the individual differences of different patients to ensure that the generated three-dimensional images accurately reflect the real anatomical structure. In addition, the conversion between two-dimensional and three-dimensional images involves complex algorithms, which may lead to information loss or distortion in some cases.
[0007] The prior art provides a magnetic nanoparticle X-ray CT contrast tomography method, device and system, which belong to the field of magnetic nanoparticle imaging. The method comprises: rotating a target object containing magnetic nanoparticles as contrast agents relative to X-rays (with a periodically changing excitation magnetic field parallel to its axis) for one circle, and recording multiple continuous transmission projection images of the target object at each angle; performing image post-processing operations on the multiple transmission projection images at each angle, and then performing back-projection transformation on the magnetic nanoparticle contrast imaging projection images at all angles to obtain tomographic imaging results; the image post-processing operations comprise: time-frequency transformation: converting the third dimension of a three-dimensional time domain matrix to the frequency domain to obtain a three-dimensional frequency domain matrix; feature extraction: selecting a two-dimensional matrix with a frequency of kf along the third dimension of the three-dimensional frequency domain image matrix; k is a positive integer; f is the frequency of the excitation magnetic field.
[0008] However, the existing method of generating three-dimensional CT volume images from two-dimensional X-ray images cannot capture high-frequency information, resulting in low generation quality. Therefore, how to invent a stable and high-quality method of generating three-dimensional CT volume images using two-dimensional X-ray images is a technical problem that urgently needs to be solved in this technical field. Summary of the invention
[0009] In order to solve the problem that the existing method of generating three-dimensional CT volume images from two-dimensional X-ray images cannot capture high-frequency information, resulting in low generation quality, the present invention provides a method and system for generating three-dimensional CT images using two-dimensional X-ray images, which has the characteristic of short training time.
[0010] In order to achieve the above-mentioned purpose of the present invention, the technical scheme adopted is as follows: A method for generating a three-dimensional CT image using a two-dimensional X-ray image comprises the following specific steps: Construct a generative adversarial network based on frequency decoupling and diffusion, including a frequency decoupling block, a generator, and a discriminator; Collecting a two-dimensional X-ray image set including multi-angle two-dimensional X-ray images and its corresponding real three-dimensional CT image set; Input multi-angle two-dimensional X-ray images into a generative adversarial network based on frequency decoupling and diffusion; The frequency decoupling block extracts high-frequency information of multi-angle two-dimensional X-ray images layer by layer and inputs it into the generator to generate three-dimensional CT images; The generated three-dimensional CT image and the image in the corresponding real three-dimensional CT image set are input into the discriminator for identification; Based on the identification results, the losses of the discriminator and the generator are calculated and the weights of the generator and the discriminator are updated; Continue training the generator and discriminator until a preset training threshold is reached; The two-dimensional X-ray image to be generated is input into the trained generative adversarial network based on frequency decoupling and diffusion to generate a three-dimensional CT image.
[0011] Preferably, the generative adversarial network based on frequency decoupling and diffusion includes at least one frequency decoupling block, a generator, and a discriminator; wherein the generator includes an encoder block and a decoder block; the discriminator includes a diffusion module; wherein the encoder block includes at least one pair of encoder networks corresponding to the frequency decoupling block; the decoder block includes at least one side decoder network and one intermediate decoder network corresponding to the encoder network; the frequency decoupling block is connected to the second encoder network in the corresponding pair of encoder networks; the second encoder network is connected to the first encoder network; the first encoder network is connected to its corresponding side decoder; all side decoder networks are connected to the intermediate decoder network; and the intermediate decoder network is also connected to the diffusion module.
[0012] Furthermore, the frequency decoupling block is specifically a frequency decoupling cascade module with a pyramid-like structure; the frequency decoupling cascade module specifically includes a cascaded bilinear interpolation function block, a bilinear interpolation function downsampling block, and a bilinear interpolation function upsampling block.
[0013] Furthermore, the encoder network is divided into: the features output by the first encoder network are input into the first layer of its corresponding side decoder network through converter Conversion-A; the output of the second layer of the side decoder network is fused through converter Conversion-C and input into the first layer of the intermediate decoder; the first encoder network and its corresponding side decoder network are connected by residual connection through converter Conversion-B.
[0014] Furthermore, in the generator, all convolutions used are depth-wise separable convolutions including depth-wise convolutions and point-by-point convolutions.
[0015] Furthermore, multiple frequency decoupling blocks extract high-frequency information of multi-angle two-dimensional X-ray images layer by layer. The specific steps are as follows: Input the images at different angles into their corresponding frequency decoupling blocks respectively, and each angle corresponds to one frequency decoupling block; In each frequency decoupling block, for the input image X, a bilinear interpolation function is performed once through the bilinear interpolation function block to obtain the feature map I0; I0 is downsampled by the bilinear interpolation function downsampling block to obtain the feature map I1; I1 is upsampled by the bilinear interpolation function upsampling block to obtain the feature map I2; the difference between I2 and the original feature map is calculated to obtain a map containing high-frequency information.
[0016] Furthermore, the input generator generates a three-dimensional CT image. The specific steps are as follows: The intermediate features of the multi-angle two-dimensional X-ray image are extracted by a first encoder network corresponding to the frequency decoupling block, and the corresponding second encoder network fuses the high-frequency information and the intermediate features; The first encoder network inputs the fused features into the corresponding adjacent decoder network, and the adjacent decoder simultaneously inputs the decoded 3D features into the intermediate decoder network for fusion to generate a 3D CT image.
[0017] Furthermore, when the input is used in the discriminator for identification, adaptive noise diffusion is also performed on the three-dimensional volume image generated by the diffusion module, specifically: noise is adaptively added to the diffusion chain output y sampled from the conditional distribution q(y|x,t) after the real CT image x and the diffusion step t is added, and the diffusion chain output yg sampled from the conditional distribution q(y|G(z),t)) under the conditions of the generated three-dimensional volume image G(Z) and the diffusion step t.
[0018] Furthermore, adaptive noise diffusion is performed to dynamically adjust the intensity of the diffusion process. The specific steps are as follows: Starting from the original image x0, noise is gradually added in each step t using a predefined βt and variance σ2 until the noise level σ2 is reached in T steps, resulting in image x1. The diffusion process is considered as a Markov chain, expressed as;
[0019] Among them, the β value determines the noise intensity in the diffusion process, and different diffusion steps t correspond to different β values; β t The beta value sequence generated using the sigmoid scheduling strategy:
[0020] in β t For the range β start , β end 】between; According to the Gaussian reparameterization formula, the noise adding function is obtained:
[0021] Use 1-a -t To measure the intensity of noise diffusion; the discriminator starts learning from the original image without noise, and then gradually increases t; during training, a metric r is used to estimate the degree to which the discriminator fits the data d The maximum number of steps T for adaptive adjustment;
[0022]
[0023] Where C is a constant, C = BI / 1000K; B represents the batch size, and parameter I represents the number of batches at which an update is performed; During training, rd is calculated and T is updated every several batches: If r d = 0.5, indicating that the discriminator is balanced and T remains unchanged; r d >0.5, indicating that the discriminator is overfitting and T increases; r d <0.5, indicating that the discriminator is underfitting and T decreases; In diffusion, diffusion step t adopts either uniform distribution or priority distribution. For uniform distribution, diffusion step t is uniformly sampled from {1, 2, ..., T}, and the probability of each step is 1 / T. For priority distribution, the sampling probability of diffusion step t increases linearly with the increase of t, that is: ; Use a named The metric updates the probability parameter p in the diffusion process, thereby adaptively adjusting the maximum step size T; define for:
[0024] When fd=0.5, the discriminator is in equilibrium and p remains unchanged; when fd>0.5, the discriminator is overfitting and p increases; when fd is greater than 0.5, the discriminator is overfitting and the p value increases; when fd is less than 0.5, the discriminator is underfitting and the p value decreases:
[0025] Using the adjusted probability parameter p, the diffusion step T is adaptively adjusted: .
[0026] A system for generating a three-dimensional CT image using a two-dimensional X-ray image, comprising a network construction module, an image collection module, a model training module, and a result generation module; The network building module is used to construct a generative adversarial network based on frequency decoupling and diffusion, including a frequency decoupling block, a generator, and a discriminator; The image collection module is used to collect a two-dimensional X-ray image set including multi-angle two-dimensional X-ray images and its corresponding real three-dimensional CT image set and a two-dimensional X-ray image to be generated; The model training module is used to input the multi-angle two-dimensional X-ray image into the generative adversarial network based on frequency decoupling and diffusion; the frequency decoupling block extracts the high-frequency information of the multi-angle two-dimensional X-ray image layer by layer, and inputs it into the generator to generate a three-dimensional CT image; the generated three-dimensional CT image and the image in the corresponding real three-dimensional CT image set are input into the discriminator for identification; based on the identification result, the loss of the discriminator and the generator is calculated and the weights of the generator and the discriminator are updated; the generator and the discriminator are continuously trained until a preset training threshold is reached; The result generation module is used to input the two-dimensional X-ray image to be generated into a trained generative adversarial network based on frequency decoupling and diffusion to generate a three-dimensional CT image.
[0027] The beneficial effects of the present invention are as follows: The present invention discloses a method for generating a three-dimensional CT image using a two-dimensional X-ray image, using a generative adversarial network based on frequency decoupling and diffusion, including a frequency decoupling block, a generator, and a discriminator; wherein the frequency decoupling block extracts high-frequency information of a multi-angle two-dimensional X-ray image layer by layer, and inputs the high-frequency information into the generator to generate a three-dimensional CT image; the generated three-dimensional CT image and an image in a corresponding real three-dimensional CT image set are input into the discriminator for identification; thereby, the present invention solves the problem that the existing method for generating a three-dimensional CT volume image from a two-dimensional X-ray image cannot capture high-frequency information, resulting in low generation quality. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 The invention is a flowchart of a method for generating a three-dimensional CT image using a two-dimensional X-ray image.
[0029] Figure 2 This is a schematic diagram of the overall network structure of the generative adversarial network based on frequency decoupling and diffusion.
[0030] Figure 3 It is a schematic diagram of the framework process of generating three-dimensional CT images from two-dimensional X-ray images using a generative adversarial network based on frequency decoupling and diffusion.
[0031] Figure 4 It is a schematic diagram of the framework flow of generating three-dimensional CT images from two-dimensional X-ray images in a specific generator. DETAILED DESCRIPTION
[0032] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.
[0033] Example 1 like Figure 1 As shown, a method for generating a three-dimensional CT image using a two-dimensional X-ray image is characterized by comprising the following specific steps: Construct a generative adversarial network based on frequency decoupling and diffusion, including a frequency decoupling block, a generator, and a discriminator; Collecting a two-dimensional X-ray image set including multi-angle two-dimensional X-ray images and its corresponding real three-dimensional CT image set; Input multi-angle two-dimensional X-ray images into a generative adversarial network based on frequency decoupling and diffusion; The frequency decoupling block extracts high-frequency information of multi-angle two-dimensional X-ray images layer by layer and inputs it into the generator to generate three-dimensional CT images; The generated three-dimensional CT image and the image in the corresponding real three-dimensional CT image set are input into the discriminator for identification; Based on the identification results, the losses of the discriminator and the generator are calculated and the weights of the generator and the discriminator are updated; Continue training the generator and discriminator until a preset training threshold is reached; The two-dimensional X-ray image to be generated is input into the trained generative adversarial network based on frequency decoupling and diffusion to generate a three-dimensional CT image.
[0034] like Figure 2 As shown, in a specific embodiment, the generative adversarial network based on frequency decoupling and diffusion includes at least one frequency decoupling block, a generator, and a discriminator; wherein the generator includes an encoder block and a decoder block; the discriminator includes a diffusion module; wherein the encoder block includes at least one pair of encoder networks corresponding to the frequency decoupling block; the decoder block includes at least one side decoder network and one intermediate decoder network corresponding to the encoder network; the frequency decoupling block is connected to the second encoder network in the corresponding pair of encoder networks; the second encoder network is connected to the first encoder network; the first encoder network is connected to the corresponding side decoder; all side decoder networks are connected to the intermediate decoder network; and the intermediate decoder network is also connected to the diffusion module.
[0035] In a specific embodiment, the frequency decoupling block is specifically a frequency decoupling cascade module with a pyramid-like structure; the frequency decoupling cascade module specifically includes a cascaded bilinear interpolation function block, a bilinear interpolation function downsampling block, and a bilinear interpolation function upsampling block.
[0036] In this embodiment, the frequency decoupling cascade module of the pyramid-like structure is mainly based on the Gaussian pyramid and Laplacian pyramid. It is an important part of the generator and mainly extracts image edges and details, that is, high-frequency information. The extracted high-frequency information is input into the second encoder network and fused with the intermediate feature map of the first encoder network layer by layer to avoid the loss of high-frequency information during network transmission. The purpose of the design module is to effectively extract the edge information and detail information of the input X-ray image, extract a feature map containing rich edge information and detail information by converting the image into a grayscale image and drawing on the image pyramid technology, and combine the high-frequency features of the X-ray image with the multi-layer output of the encoder, thereby improving the quality and detail features of the reconstructed image.
[0037] In this embodiment, upsampling and downsampling in the Gaussian pyramid are both achieved by using Gaussian filtering, and the Las pyramid is generated by calculating the difference between the layers of the Gaussian pyramid. The frequency decoupling cascade module of the pyramid-like structure uses bilinear interpolation based on pytorch to achieve upsampling and downsampling. The use of GPU for calculation significantly improves the efficiency of frequency decoupling; the frequency decoupling cascade module does not rely on a specific direction, has stronger insight, and can extract rich high-frequency information at different resolutions. This is especially important for complex medical image analysis, because lesions and structures in medical images may exist in various geometries and geometries. PSFD is based on simple convolution operations and interpolation operations, has high computational efficiency, is easy to implement and optimize in a deep learning framework, and is suitable for large-scale data processing and real-time applications.
[0038] These operations can effectively extract and enhance input features, improve the expressiveness and performance of the model, and provide richer feature expressions for subsequent processing. Experimental results show that this method can effectively extract detailed features of images and play a positive role in generating three-dimensional CT volumes from X-ray images.
[0039] In a specific embodiment, the encoder network is divided into: the features output by the first encoder network are input into the first layer of its corresponding side decoder network through converter Conversion-A; the output of the second layer of the side decoder network is fused through converter Conversion-C and input into the first layer of the intermediate decoder; the first encoder network and its corresponding side decoder network are connected by residual connection through converter Conversion-B.
[0040] In a specific embodiment, the discriminator block is further provided with an adaptive noise diffusion block.
[0041] In a specific embodiment, in the generator, all convolutions used are depth-wise separable convolutions including depth-wise convolutions and point-by-point convolutions.
[0042] In this embodiment, the dual-plane X-ray image generates a CT volume image model based on a generative adversarial network, which works well in the field of medical image generation. However, there are problems such as too many parameters increasing the model complexity and slow calculation speed. In order to solve these problems. In order to balance parameter efficiency and performance in the neural network structure, the standard convolution in the model is replaced with a lightweight convolution to achieve the effect of a lightweight network. The calculation amount and parameter amount of the deep separable convolution are theoretically only 1 / 9 of the standard convolution, which can effectively reduce the complexity of the model, improve the generalization ability, interpretability, and running speed of the model, making the model more efficient and lightweight, and having a wider range of application scenarios. In the training process of the model, the reduction in computational complexity reduces the time required for each iteration cycle, shortens the training time, and the reduction in the number of parameters helps to alleviate the overfitting problem of the model. The main principle of the deep separable convolution is to decompose the standard convolution operation into deep convolution and point-by-point convolution. The deep convolution is responsible for extracting spatial features, and the point-by-point convolution is responsible for extracting channel features, making each step of the operation simpler and more transparent. This separation can help researchers better understand the role of each layer of convolution operation and provide a clearer explanation of the model structure. Depthwise separable convolution improves the generalization ability of the model on different data sets, allowing the generative model to maintain good performance when faced with new, unseen data.
[0043] By replacing the ordinary convolution in the generative model with the depthwise separable convolution, not only the problem of too many model parameters and low computational efficiency is effectively solved, but also the training and reasoning speed of the model is improved, and the generalization ability and interpretability are enhanced. This improvement makes the generative model have a wider application prospect in the field of medical image generation, especially for efficient deployment on resource-constrained devices. The experimental results verify the advantages of the depthwise separable convolution and show that this method significantly optimizes the use of computing resources while maintaining or improving model performance. When batch_size is set to 2, under the same operating environment, it only takes 8.8654H to train 100 epochs based on the deep convolutional neural network, while it takes 15.1612H to train 100 epochs based on the standard convolution. When batch_size is set to 1, under the same operating environment, it only takes 19.692H to train 100 epochs based on the deep separable convolutional deep network, while it takes 25.695H to train 100 epochs based on the standard convolutional network.
[0044] Example 2 More specifically, Figure 3 As shown, a method for generating a three-dimensional CT image using a two-dimensional X-ray image, based on the generative adversarial network based on frequency decoupling and diffusion, includes the following specific steps: Construct a generative adversarial network based on frequency decoupling and diffusion, including a frequency decoupling block, a generator, and a discriminator; Collecting a two-dimensional X-ray image set including multi-angle two-dimensional X-ray images and its corresponding real three-dimensional CT image set; Input multi-angle two-dimensional X-ray images into a generative adversarial network based on frequency decoupling and diffusion; The frequency decoupling block extracts high-frequency information of the multi-angle two-dimensional X-ray image layer by layer, and the corresponding first encoder network extracts the intermediate features of the multi-angle two-dimensional X-ray image, and the corresponding second encoder network fuses the high-frequency information and the intermediate features; The first encoder network inputs the fused features into the corresponding adjacent decoder network, and the adjacent decoder simultaneously inputs the decoded 3D features into the intermediate decoder network for fusion to generate a 3D CT image. The generated three-dimensional CT image and the image in the corresponding real three-dimensional CT image set are input into the discriminator for identification; Based on the identification results, the losses of the discriminator and the generator are calculated and the weights of the generator and the discriminator are updated; Continue training the generator and discriminator until a preset training threshold is reached; The two-dimensional X-ray image to be generated is input into the trained generative adversarial network based on frequency decoupling and diffusion to generate a three-dimensional CT image.
[0045] In a specific embodiment, Figure 4 As shown, multiple frequency decoupling blocks extract high-frequency information of multi-angle two-dimensional X-ray images layer by layer. The specific steps are: Input the images at different angles into their corresponding frequency decoupling blocks respectively, and each angle corresponds to one frequency decoupling block; In this embodiment, there are images at two angles, front and side, and two frequency decoupling blocks corresponding thereto; Figure 4 In the middle, fd corresponds to the frequency decoupling block; In each frequency decoupling block, for the input image X, a bilinear interpolation function is performed once through the bilinear interpolation function block to obtain the feature map I0; I0 is downsampled by the bilinear interpolation function downsampling block to obtain the feature map I1; I1 is upsampled by the bilinear interpolation function upsampling block to obtain the feature map I2; the difference between I2 and the original feature map is calculated to obtain a map containing high-frequency information.
[0046] In this embodiment, the input image X is passed through a bilinear interpolation function, and I0 represents a feature map with a height and width of 128×128. I0 is input to the bilinear interpolation function and downsampled to 64×64, and I1 is used to represent the feature map with a height and width of 64×64. Finally, the feature map I1 is input to the bilinear interpolation function and upsampled to 128×128 and represented by I2.
[0047] In a specific embodiment, considering the use of a generative adversarial network to generate a three-dimensional CT volume image from a two-dimensional X-ray image, a diffusion module is introduced into the GAN structure in combination with a discriminator to achieve the addition of Gaussian mixture distributed instance noise. During training, problems such as mode collapse, mode worship, and training instability may be faced. Experiments have shown that the addition of a diffusion module has a positive effect in the task of generating three-dimensional CT images. Both the real CT image and the generated CT image are gradually noisy through the same adaptive diffusion process. The gradual noisy process is called an adaptive noisy diffusion process, and each diffusion time step has a different noise-data ratio, corresponding to different noise intensities. The length of the forward diffusion chain is adaptively adjusted to balance the noise and data levels. The real image after noisy and the generated image after noisy are input into the discriminator, allowing the discriminator to learn to distinguish the true and false data after noisy. The generator learns from the feedback of the discriminator through back propagation, and the discriminator learns to distinguish the diffused real image from the diffused false image in all diffusion processes. Data-efficient and stable leakage-free GAN training is achieved, thereby improving the output effect of the generator.
[0048] In a specific embodiment, when training the discriminator, the intensity of the diffusion process is dynamically adjusted by adaptive noise diffusion, that is, noise is adaptively added to the diffusion chain output y sampled from the conditional distribution q(y|x,t) after the real CT image x and the diffusion step t is noised, and the diffusion chain output yg sampled from the conditional distribution q(y|G(z),t)) under the conditions of the generated three-dimensional volume image G(Z) and the diffusion step t.
[0049] In this embodiment, a method of updating the time step distribution is used so that the model can give priority to the time step that has an important impact on the generation quality during the training process. In order to improve the performance of the discriminator, a new discriminator optimization strategy is used using the diffusion function and the time step. The goal is to set a task for the discriminator that is both challenging and not too simple to avoid overfitting the image, but also not too difficult to avoid hindering the learning process. The design dynamically adjusts the intensity of the diffusion process through the discriminator's ability to distinguish between real images and generated images, that is, adding an appropriate amount of noise to y and yg. When the time step t is large, the ratio of noise to image is high, and the task is more difficult.
[0050] In a specific embodiment, the adaptive noise diffusion for dynamically adjusting the intensity of the diffusion process comprises the following specific steps: Starting from the original image x0, noise is gradually added in each step t using a predefined βt and variance σ2 until the noise level σ2 is reached in T steps, resulting in image x1. The diffusion process is considered as a Markov chain, expressed as;
[0051] Among them, the β value determines the noise intensity in the diffusion process, and different diffusion steps t correspond to different β values; β t The beta value sequence generated using the sigmoid scheduling strategy:
[0052] in β t For the range β start , β end 】between; According to the Gaussian reparameterization formula, the noise adding function is obtained:
[0053] Use 1-a -t To measure the intensity of noise diffusion; the discriminator starts learning from the original image without noise, and then gradually increases t; during training, a metric r is used to estimate the degree to which the discriminator fits the data d The maximum number of steps T for adaptive adjustment;
[0054]
[0055] Where C is a constant, C = BI / 1000K; B represents the batch size, and parameter I represents the number of batches at which an update is performed; During training, rd is calculated and T is updated every several batches: If r d = 0.5, indicating that the discriminator is balanced and T remains unchanged; r d >0.5, indicating that the discriminator is overfitting and T increases; r d <0.5, indicating that the discriminator is underfitting and T decreases; In diffusion, diffusion step t adopts either uniform distribution or priority distribution. For uniform distribution, diffusion step t is uniformly sampled from {1, 2, ..., T}, and the probability of each step is 1 / T. For priority distribution, the sampling probability of diffusion step t increases linearly with the increase of t, that is: ; Use a file called f d The metric updates the probability parameter p in the diffusion process, thereby adaptively adjusting the maximum step size T; define f d for:
[0056] When fd=0.5, the discriminator is in equilibrium and p remains unchanged; when fd>0.5, the discriminator is overfitting and p increases; when fd is greater than 0.5, the discriminator is overfitting and the p value increases; when fd is less than 0.5, the discriminator is underfitting and the p value decreases:
[0057] Using the adjusted probability parameter p, the diffusion step T is adaptively adjusted: .
[0058] Therefore, the present invention improves the CT generation model based on the generative adversarial network using X-rays, adds a frequency decoupling cascade module with a pyramid structure in the encoder part of the generator, uses a diffusion module noise in the discriminator, and innovatively combines diffusion with GAN to generate medical images. The effects of group convolution and deep convolution on the model parameters and computational complexity are compared, and deep separable convolution is used to replace standard convolution, which has made significant progress in performance and efficiency, and realizes the generation of CT from dual-plane X-rays.
[0059] Example 3 A system for generating a three-dimensional CT image using a two-dimensional X-ray image, comprising a network construction module, an image collection module, a model training module, and a result generation module; The network building module is used to construct a generative adversarial network based on frequency decoupling and diffusion, including a frequency decoupling block, a generator, and a discriminator; The image collection module is used to collect a two-dimensional X-ray image set including multi-angle two-dimensional X-ray images and its corresponding real three-dimensional CT image set and a two-dimensional X-ray image to be generated; The model training module is used to input the multi-angle two-dimensional X-ray image into the generative adversarial network based on frequency decoupling and diffusion; the frequency decoupling block extracts the high-frequency information of the multi-angle two-dimensional X-ray image layer by layer, and inputs it into the generator to generate a three-dimensional CT image; the generated three-dimensional CT image and the image in the corresponding real three-dimensional CT image set are input into the discriminator for identification; based on the identification result, the loss of the discriminator and the generator is calculated and the weights of the generator and the discriminator are updated; the generator and the discriminator are continuously trained until a preset training threshold is reached; The result generation module is used to input the two-dimensional X-ray image to be generated into a trained generative adversarial network based on frequency decoupling and diffusion to generate a three-dimensional CT image.
[0060] Obviously, the above embodiments of the present invention are only examples for clearly explaining the present invention, and are not intended to limit the implementation methods of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention shall be included in the protection scope of the claims of the present invention.
Claims
1. A method for generating a three-dimensional CT image using a two-dimensional X-ray image, characterized in that: The specific steps include: Construct a generative adversarial network based on frequency decoupling and diffusion, including a frequency decoupling block, a generator, and a discriminator; Collecting a two-dimensional X-ray image set including multi-angle two-dimensional X-ray images and its corresponding real three-dimensional CT image set; Input multi-angle two-dimensional X-ray images into a generative adversarial network based on frequency decoupling and diffusion; The frequency decoupling block extracts high-frequency information of multi-angle two-dimensional X-ray images layer by layer and inputs it into the generator to generate three-dimensional CT images; The generated three-dimensional CT image and the image in the corresponding real three-dimensional CT image set are input into the discriminator for identification; Based on the identification results, the losses of the discriminator and the generator are calculated and the weights of the generator and the discriminator are updated; Continue training the generator and discriminator until a preset training threshold is reached; The two-dimensional X-ray image to be generated is input into the trained generative adversarial network based on frequency decoupling and diffusion to generate a three-dimensional CT image.
2. The method for generating a three-dimensional CT image using a two-dimensional X-ray image according to claim 1, characterized in that: The generative adversarial network based on frequency decoupling and diffusion includes at least one frequency decoupling block, a generator, and a discriminator; wherein the generator includes an encoder block and a decoder block; the discriminator includes a diffusion module; wherein the encoder block includes at least one pair of encoder networks corresponding to the frequency decoupling block; the decoder block includes at least one side decoder network and one intermediate decoder network corresponding to the encoder network; the frequency decoupling block is connected to the second encoder network in the corresponding pair of encoder networks; the second encoder network is connected to the first encoder network; the first encoder network is connected to the corresponding side decoder; all side decoder networks are connected to the intermediate decoder network; and the intermediate decoder network is also connected to the diffusion module.
3. The method for generating a three-dimensional CT image using a two-dimensional X-ray image according to claim 2, characterized in that: The frequency decoupling block is specifically a frequency decoupling cascade module of a pyramid-like structure; the frequency decoupling cascade module specifically includes a cascaded bilinear interpolation function block, a bilinear interpolation function downsampling block, and a bilinear interpolation function upsampling block.
4. The method for generating a three-dimensional CT image using a two-dimensional X-ray image according to claim 3, characterized in that: The encoder network is divided into: the features output by the first encoder network are input into the first layer of its corresponding side decoder network through converter Conversion-A; the output of the second layer of the side decoder network is fused through converter Conversion-C and input into the first layer of the intermediate decoder; the first encoder network and its corresponding side decoder network are connected by residual connection through converter Conversion-B.
5. The method of generating a three-dimensional CT image using a two-dimensional X-ray image according to claim 1, characterized in that: In the generator, all convolutions used are depth-wise separable convolutions including depth-wise convolutions and point-by-point convolutions.
6. The method of generating a three-dimensional CT image using a two-dimensional X-ray image according to claim 4, characterized in that: Multiple frequency decoupling blocks extract high-frequency information of multi-angle two-dimensional X-ray images layer by layer. The specific steps are as follows: Input the images at different angles into their corresponding frequency decoupling blocks respectively, and each angle corresponds to one frequency decoupling block; In each frequency decoupling block, for the input image X, a bilinear interpolation function is performed once through the bilinear interpolation function block to obtain the feature map I0; I0 is downsampled by the bilinear interpolation function downsampling block to obtain the feature map I1; I1 is upsampled by the bilinear interpolation function upsampling block to obtain the feature map I2; the difference between I2 and the original feature map is calculated to obtain a map containing high-frequency information.
7. The method for generating a three-dimensional CT image using a two-dimensional X-ray image according to claim 6, characterized in that: The input generator generates a 3D CT image. The specific steps are: The intermediate features of the multi-angle two-dimensional X-ray image are extracted by a first encoder network corresponding to the frequency decoupling block, and the corresponding second encoder network fuses the high-frequency information and the intermediate features; The first encoder network inputs the fused features into the corresponding adjacent decoder network, and the adjacent decoder simultaneously inputs the decoded 3D features into the intermediate decoder network for fusion to generate a 3D CT image.
8. The method for generating a three-dimensional CT image using a two-dimensional X-ray image according to claim 7, characterized in that: When the input is discriminated in the discriminator, adaptive noise diffusion is also used on the three-dimensional volume image generated by the diffusion module. Specifically, noise is adaptively added to the diffusion chain output y sampled from the conditional distribution q(y|x,t) after the real CT image x and the diffusion step t is added, and the diffusion chain output yg sampled from the conditional distribution q(y|G(z),t)) under the generated three-dimensional volume image G(Z) and the diffusion step t.
9. The method for generating a three-dimensional CT image using a two-dimensional X-ray image according to claim 8, characterized in that: Adaptive noise diffusion that dynamically adjusts the intensity of the diffusion process. The specific steps are: Starting from the original image x0, noise is gradually added in each step t using a predefined βt and variance σ2 until the noise level σ2 is reached in T steps, resulting in image x1. The diffusion process is considered as a Markov chain, expressed as; Among them, the β value determines the noise intensity in the diffusion process, and different diffusion steps t correspond to different β values; β t The beta value sequence generated using the sigmoid scheduling strategy: in β t For the range β start , β end 】between; According to the Gaussian reparameterization formula, the noise adding function is obtained: Use 1-a -t To measure the intensity of noise diffusion; the discriminator starts learning from the original image without noise, and then gradually increases t; during training, a metric r is used to estimate the degree to which the discriminator fits the data d The maximum number of steps T for adaptive adjustment; Where C is a constant, C = BI / 1000K; B represents the batch size, and parameter I represents the number of batches at which an update is performed; During training, rd is calculated and T is updated every several batches: If r d = 0.5, indicating that the discriminator is balanced and T remains unchanged; r d >0.5, indicating that the discriminator is overfitting and T increases; r d < 0.5, indicating that the discriminator is underfitting and T decreases; In diffusion, diffusion step t adopts either uniform distribution or priority distribution. For uniform distribution, diffusion step t is uniformly sampled from {1, 2, ..., T}, and the probability of each step is 1 / T. For priority distribution, the sampling probability of diffusion step t increases linearly with the increase of t, that is: ； Use a named The metric updates the probability parameter p in the diffusion process, thereby adaptively adjusting the maximum step size T; define for: When fd=0.5, the discriminator is in equilibrium and p remains unchanged; when fd>0.5, the discriminator is overfitting and p increases; when fd is greater than 0.5, the discriminator is overfitting and the p value increases; when fd is less than 0.5, the discriminator is underfitting and the p value decreases: Using the adjusted probability parameter p, the diffusion step T is adaptively adjusted: 。 10. A system for generating a three-dimensional CT image using a two-dimensional X-ray image, characterized in that: It includes network construction module, image collection module, model training module and result generation module; The network building module is used to construct a generative adversarial network based on frequency decoupling and diffusion, including a frequency decoupling block, a generator, and a discriminator; The image collection module is used to collect a two-dimensional X-ray image set including multi-angle two-dimensional X-ray images and its corresponding real three-dimensional CT image set and a two-dimensional X-ray image to be generated; The model training module is used to input the multi-angle two-dimensional X-ray image into the generative adversarial network based on frequency decoupling and diffusion; the frequency decoupling block extracts the high-frequency information of the multi-angle two-dimensional X-ray image layer by layer, and inputs it into the generator to generate a three-dimensional CT image; the generated three-dimensional CT image and the image in the corresponding real three-dimensional CT image set are input into the discriminator for identification; based on the identification result, the loss of the discriminator and the generator is calculated and the weights of the generator and the discriminator are updated; Continue training the generator and discriminator until a preset training threshold is reached; The result generation module is used to input the two-dimensional X-ray image to be generated into a trained generative adversarial network based on frequency decoupling and diffusion to generate a three-dimensional CT image.