Semi-supervised based ct image synthesis method and device, equipment and storage medium
By combining a semi-supervised generative adversarial network model with supervised and unsupervised methods, and training an image generator and a residual shrinkage discriminator using paired and unpaired datasets, the problems of model overfitting and noise interference in MRI-to-CT images are solved, thereby improving the quality and detail of the synthesized images.
Patent Information
- Application Number
- CN202310535204.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-12
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2043-05-12
AI Technical Summary
Existing methods for synthesizing CT images based on MRI suffer from problems such as easy model overfitting, lack of accurate supervision signals, and susceptibility to noise interference, resulting in poor quality of synthesized images.
A semi-supervised generative adversarial network model is adopted, which combines supervised and unsupervised methods. The image generator and residual shrinkage discriminator are trained using paired and unpaired datasets. The feature learning ability of the discriminator is improved by the residual shrinkage module, and a frequency domain loss term is introduced into the supervised branch network to optimize the image generation performance.
It improves the quality of image synthesis, effectively avoids model overfitting and noise interference, enhances the image synthesis performance of the generator, and improves the detail and overall quality of synthesized CT images.
Smart Images

Figure CN116645436B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of medical image synthesis technology, and in particular to a semi-supervised CT image synthesis method, apparatus, device and storage medium. Background Technology
[0002] With the development of deep learning and computer vision technologies, pseudo-CT (computed tomography) synthesis methods based on deep neural networks and generative adversarial networks have attracted widespread attention. Currently, there are numerous studies on depth-based MRI (magnetic resonance imaging) CT image generation, mainly including supervised and unsupervised methods. Supervised methods often train MRI-to-CT image generation models using large amounts of paired MRI and CT data, optimizing the model by minimizing the difference between the generated CT images and real CT images, thereby learning the correct mapping relationship from MRI to CT. Unsupervised methods, on the other hand, utilize unpaired datasets—images from both MRI and CT modalities—to train deep models to learn the data distributions of both MRI and CT domains, effectively avoiding dependence on paired datasets.
[0003] While significant progress has been made in deep learning-based MRI-to-CT synthesis methods, the following issues still lead to poor image quality: First, supervised methods require a large amount of paired MRI and CT data, and models trained using only paired data are prone to overfitting, resulting in poor performance when faced with new data. Unsupervised methods, while utilizing unpaired datasets, lack supervisory guidance, making it difficult to accurately learn the mapping relationship between the two domains. Second, redundant information or noise in medical images can also affect the model's image generation capabilities. Summary of the Invention
[0004] This application provides a semi-supervised CT image synthesis method, apparatus, device, and storage medium to solve the problems of poor synthesized image quality caused by the model's tendency to overfit, lack of accurate supervision signals, and susceptibility to noise interference when synthesizing CT images based on MRI in related technologies.
[0005] Firstly, a semi-supervised CT image synthesis method is provided, comprising the following steps:
[0006] MRI and CT image data are acquired to construct a training dataset, which includes paired and unpaired datasets;
[0007] A semi-supervised generative adversarial network model is constructed. The semi-supervised generative adversarial network model includes a supervised branch network and an unsupervised branch network that share an image generator and a residual shrinkage discriminator. The residual shrinkage discriminator includes multiple channel-level residual shrinkage modules and a convolutional layer.
[0008] The semi-supervised generative adversarial network model is trained based on the training dataset to update and optimize the image generator and residual shrinkage discriminator, thereby generating an image synthesis model.
[0009] The MRI image to be converted is transformed into a synthetic CT image based on the image synthesis model.
[0010] In some embodiments, the channel-level residual shrinkage module is used to perform absolute value operations and soft thresholding operations on the input features, and to perform residual concatenation between the soft-thresholded features and the input features to obtain residual features.
[0011] In some embodiments, the updated and optimized image generator and residual shrinkage discriminator include:
[0012] The parameters of the image generator G are fixed, and the parameters of the residual shrinkage discriminator D are updated and optimized by minimizing the first loss function used to update the residual shrinkage discriminator D.
[0013] The parameters of the residual shrinkage discriminator D are fixed, and the parameters of the image generator G are updated and optimized by minimizing the second loss function used to update the image generator G.
[0014] In some embodiments, the second loss function includes an adversarial loss term, a pixel loss term, and a frequency domain loss term corresponding to a supervised branch network, as well as an adversarial loss term corresponding to an unsupervised branch network.
[0015] In some embodiments, the second loss function L G The calculation formula is:
[0016]
[0017] In the formula, y s and Y s Let represent the real CT images and the set of real CT images corresponding to the supervised branch network, respectively. L represents a pseudo-CT image synthesized by a supervised branch network. pix L represents the pixel loss term. FFL λ represents the frequency domain loss term. pix and λ FFL The hyperparameter y represents the weights of the loss term. u and Y u Let these represent the real CT images and the set of real CT images corresponding to the unsupervised branch network, respectively. This represents a pseudo-CT image synthesized by an unsupervised branch network. This indicates a loss against the loss.
[0018] In some embodiments, training the semi-supervised generative adversarial network model based on the training dataset includes:
[0019] The supervised branch network is trained based on the paired dataset;
[0020] The unsupervised branch network is trained based on the unpaired dataset.
[0021] Secondly, a semi-supervised CT image synthesis device is provided, comprising:
[0022] A data construction unit is used to acquire MRI image data and CT image data to construct a training dataset, which includes paired and unpaired datasets;
[0023] The model building unit is used to build a semi-supervised generative adversarial network model. The semi-supervised generative adversarial network model includes a supervised branch network and an unsupervised branch network that share an image generator and a residual shrinkage discriminator. The residual shrinkage discriminator includes multiple channel-level residual shrinkage modules and a convolutional layer.
[0024] The model training unit is used to train the semi-supervised generative adversarial network model based on the training dataset to update and optimize the image generator and residual shrinkage discriminator, and generate an image synthesis model.
[0025] An image synthesis unit is used to convert an MRI image to be converted into a synthetic CT image based on the image synthesis model.
[0026] In some embodiments, the channel-level residual shrinkage module is used to perform absolute value operations and soft thresholding operations on the input features, and to perform residual concatenation between the soft-thresholded features and the input features to obtain residual features.
[0027] Thirdly, a semi-supervised CT image synthesis device is provided, comprising: a memory and a processor, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to realize the aforementioned semi-supervised CT image synthesis method.
[0028] Fourthly, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned semi-supervised CT image synthesis method.
[0029] This application provides a semi-supervised CT image synthesis method, apparatus, device, and storage medium, including acquiring MRI image data and CT image data to construct a training dataset, which includes paired and unpaired datasets; constructing a semi-supervised generative adversarial network (GAN) model, which includes a supervised branch network and an unsupervised branch network sharing an image generator and a residual shrinkage discriminator, wherein the residual shrinkage discriminator includes multiple channel-level residual shrinkage modules and a convolutional layer; training the semi-supervised GAN model based on the training dataset to update and optimize the image generator and the residual shrinkage discriminator, generating an image synthesis model; and converting the MRI image to be converted into a synthesized CT image based on the image synthesis model. This application combines supervised and unsupervised methods by using a semi-supervised learning network, effectively avoiding ambiguity that occurs in supervised methods by utilizing the additional information provided by unpaired data. Simultaneously, it uses residual shrinkage modules as the basic building blocks of the discriminator to effectively improve the discriminator's feature learning ability for noisy images, thereby improving the discriminator's ability to distinguish between real and fake images, thus enhancing the image synthesis performance of the generator and improving the quality of the synthesized image. Attached Figure Description
[0030] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0031] Figure 1 A flowchart illustrating a semi-supervised CT image synthesis method provided for embodiments of this application;
[0032] Figure 2 This is a schematic diagram of the structure of MRI image synthesized into CT image according to an embodiment of this application;
[0033] Figure 3 This is a schematic diagram of the structure of a semi-supervised generative adversarial network provided in an embodiment of this application;
[0034] Figure 4 This is a schematic diagram of the structure of the image generator provided in the embodiments of this application;
[0035] Figure 5 This is a schematic diagram of the structure of the depth residual shrinkage discriminator provided in the embodiments of this application;
[0036] Figure 6 This is a schematic diagram of a semi-supervised CT image synthesis device provided in an embodiment of this application. Detailed Implementation
[0037] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0038] This application provides a semi-supervised CT image synthesis method, apparatus, device, and storage medium, which can solve the problems of poor synthesized image quality caused by the model's tendency to overfit, lack of accurate supervision signals, and susceptibility to noise interference when synthesizing CT images based on MRI in related technologies.
[0039] See Figure 1 and Figure 2 As shown, this application provides a semi-supervised CT image synthesis method, including the following steps:
[0040] Step S10: Acquire MRI image data and CT image data to construct a training dataset, which includes paired and unpaired datasets;
[0041] As an example, in this embodiment, MRI and CT image data are acquired and preprocessed to construct a training dataset. Specifically, all MRI and CT images are first sliced along the axial plane according to a preset ratio, and slices containing less than a fixed threshold of information are removed to obtain 2D slice data of MRI and CT. A uniform distribution transformation of intensity values in the range [0, 1000] is preferably applied to all MRI image data, while a uniform distribution transformation of HU values in the range [-700, 1300] is preferably applied to all CT image data. The transformed MRI and CT images are then linearly normalized to the range [-1, 1].
[0042] Then, data augmentation operations are performed: the 2D MRI and 2D CT image data are padded to a size of 286×286, and then randomly cropped to a size of 256×256. Next, the MRI and CT image data are horizontally flipped with a 50% probability. Finally, the MRI and CT image data are randomly rotated by a maximum of 15 degrees, resulting in paired and unpaired datasets for training. The paired datasets refer to paired MRI and CT image datasets. and N represents the number of paired MRI and CT images, while unpaired datasets refer to single-modal MRI image datasets. M represents the number of single-modality MRI images.
[0043] It should be noted that the above implementation process of data preprocessing and data augmentation operations is only a presentation of an example. The specific preprocessing process settings and data augmentation operation settings can be determined according to actual needs and are not limited here.
[0044] Step S20: Construct a semi-supervised generative adversarial network model, which includes a supervised branch network and an unsupervised branch network that share an image generator and a residual shrinkage discriminator. The residual shrinkage discriminator includes multiple channel-level residual shrinkage modules and a convolutional layer.
[0045] As an example, in this embodiment, a semi-supervised generative adversarial network (GAN) model for synthesizing CT images from MRI will be constructed. The semi-supervised GAN model includes a CT image generator G and a depthwise residual shrinkage discriminator D. The image generator is used to convert MRI images into synthesized CT images, while the residual shrinkage discriminator is used to distinguish between real CT images and synthesized pseudo-CT images. See also... Figure 3 As shown, the semi-supervised generative adversarial network model in this embodiment includes two branch networks: a supervised branch network and an unsupervised branch network. The supervised branch network and the unsupervised branch network share an image generator G and a residual shrinkage discriminator D.
[0046] See Figure 4 As shown, the image generator in this embodiment includes a cascaded encoder, converter, and decoder. The encoder is a downsampling process used to extract features related to the source domain image (i.e., the MRI image). The converter is used to integrate the features from the encoder, and the decoder is used to generate CT images based on the features output by the converter. The encoder includes three convolutional blocks, each consisting of a convolutional layer, an instance normalization layer, and a ReLU activation function layer. Feature extraction is performed by the first convolutional block with a kernel size of 7×7 and a stride of 1. Downsampling is performed by the latter two convolutional blocks, each with a kernel size of 3×3 and a stride of 2.
[0047] The converter consists of multiple cascaded residual blocks, each with the same structure: a 3×3 convolutional layer, an instance normalization layer, a ReLU layer, another 3×3 convolutional layer, and an instance normalization layer. The output and input of the residual blocks are then fused using a residual connection. The decoder consists of two deconvolutional blocks with a stride of 2, a 7×7 convolutional layer, and a Tanh activation function, which is used to convert the features into a synthetic image (i.e., a CT image).
[0048] The deep residual shrinking discriminator comprises four channel-level residual shrinking modules and one convolutional layer. The number of channels in the feature map of each layer is {64, 128, 256, 512, 1}. The stride of the first three channel-level residual shrinking modules is 2, meaning the first convolutional layer in the first three modules has a stride of 2, and the remaining convolutional layers have a stride of 1. The stride of the last channel-level residual shrinking module is 1, meaning all convolutional layers in the last channel-level residual shrinking module have a stride of 1. It should be noted that the number of channels in the output feature map of the fully connected layers in the four channel-level residual shrinking modules is equal to the number of channels in the input feature map. Furthermore, the kernel size of the last convolutional layer in the residual shrinking discriminator is 4×4, the stride is 1, the number of channels in the output feature map is 1, and it outputs a binary classification result for image authenticity. This embodiment improves the quality of the synthesized image through the game and adversarial interaction between the generator network and the discriminator network.
[0049] Furthermore, the channel-level residual shrinkage module is used to perform absolute value operations and soft thresholding operations on the input features, and to perform residual connection between the soft-thresholded features and the input features to obtain residual features.
[0050] Exemplary, see Figure 5 As shown, the channel-level residual shrinkage module in this embodiment first extracts features from the input image through a convolutional block including convolutional layers, batch normalization layers, and ReLU activation function layers. It then simplifies the extracted features into a one-dimensional vector using absolute value operations and a global average pooling layer. This vector is then input into a two-layer fully connected network, where the number of neurons in both networks is equal to the number of channels in the input feature map. Finally, a sigmoid activation function is used to expand the output of the fully connected layers to the range (0,1), resulting in a scaling parameter α. c The scaling parameter is calculated as follows:
[0051]
[0052] In the formula, z c Let α be the feature vector of the c-th channel output by the fully connected layer. c This is the scaling parameter corresponding to the c-th channel.
[0053] Next, multiply the mean of the features in each channel by the scaling parameter corresponding to that channel to obtain the channel-level threshold. The calculation process for this threshold is as follows:
[0054] τ c =α c ·average i,j,c |x i,j,c |
[0055] In the above formula, i,j,c are the width, height, and channel index values of the input feature x, respectively, and τc is the threshold for the c-th channel.
[0056] Then use the threshold τ c Channel-level soft thresholding is performed on feature x to filter out relevant noise information. The expression for this soft threshold is as follows:
[0057]
[0058] In the above formula, x is the input feature and τ is the set threshold.
[0059] Finally, the soft-thresholded features are residually concatenated with the input features and used as the output of the channel-level residual shrinkage module, which can effectively reduce the loss of feature information.
[0060] Therefore, this embodiment addresses the common problem of redundant information and noise in medical images by proposing a novel discriminator structure. By using a residual shrinkage module as the basic building block of the discriminator, it effectively enhances the discriminator's feature learning ability for noisy images, thereby improving its ability to distinguish between real and fake images and ultimately improving the image synthesis performance of the generator. Furthermore, the deep residual shrinkage discriminator in this embodiment has high adaptability and can be easily extended to other noisy image generation tasks as a discriminator to improve the quality of the synthesized images.
[0061] Step S30: Train the semi-supervised generative adversarial network model based on the training dataset to update and optimize the image generator and residual shrinkage discriminator, and generate an image synthesis model;
[0062] Exemplary, in this embodiment, the supervised and unsupervised branch networks in the semi-supervised generative adversarial network model are trained using paired and unpaired datasets to alternately optimize the image generator and the residual shrinkage discriminator. The trained network model is obtained based on the final weight parameters of the residual shrinkage discriminator and the image generator. The trained image generator (i.e., Figure 2 The generator network model in the model is used as the prediction model, i.e., the generated image synthesis model. Therefore, this embodiment, considering the problems of supervised methods easily overfitting to training data and unsupervised methods struggling to accurately learn the mapping relationship between two modalities, combines supervised and unsupervised methods to form a dual-branch network. Simultaneously, it trains the model using paired MRI and CT image data and single-modal MRI data to utilize the additional information provided by unpaired data, effectively avoiding ambiguity that occurs in supervised learning, thus solving the problems of model overfitting and lack of accurate supervision signals.
[0063] Furthermore, training the semi-supervised generative adversarial network model based on the training dataset includes:
[0064] The supervised branch network is trained based on the paired dataset;
[0065] The unsupervised branch network is trained based on the unpaired dataset.
[0066] As an example, in this embodiment, the supervised branch network will be trained using paired datasets (i.e., paired MRI image data and CT image data), and the unsupervised branch network will be trained using unpaired datasets (i.e., single-modal MRI image data) to take advantage of the additional information provided by the unpaired data and effectively avoid the ambiguity that occurs in the supervised case.
[0067] Furthermore, the updated and optimized image generator and residual shrinkage discriminator include:
[0068] The parameters of the image generator G are fixed, and the parameters of the residual shrinkage discriminator D are updated and optimized by minimizing the first loss function used to update the residual shrinkage discriminator D.
[0069] The parameters of the residual shrinkage discriminator D are fixed, and the parameters of the image generator G are updated and optimized by minimizing the second loss function used to update the image generator G.
[0070] As an example, in this embodiment, a semi-supervised adversarial training strategy is used to alternately optimize the image generator G and the residual shrinking discriminator D. Specifically, the semi-supervised adversarial training strategy is as follows: first, the parameters of the image generator G are fixed, and the loss function used to update the residual shrinking discriminator D is minimized to update and optimize the parameters of the residual shrinking discriminator D; then, the parameters of the residual shrinking discriminator D are fixed, and the loss function used to update the image generator G is minimized to update and optimize the parameters of the image generator G.
[0071] It should be understood that before model training, the network objective function needs to be constructed, which includes adversarial loss, pixel loss, and frequency domain loss. The adversarial loss is used to calculate the adversarial loss between the image generator and the residual shrinking discriminator in both supervised and unsupervised branch networks. Specifically, the adversarial loss between the image generator G and the residual shrinking discriminator D in the supervised branch network is obtained according to the first calculation formula:
[0072]
[0073] In the formula, Let x represent the adversarial loss term corresponding to the supervised branch network. s and y sLet G(x) represent the input MRI image and the real CT image, respectively, for the supervised branch network. s D(y) represents the pseudo-CT image synthesized by the image generator G under a supervised branch network. s ) represents the residual shrinkage discriminator D for the real CT image y s The discrimination result, D(G(x) s )) is the residual shrinkage discriminator D for the synthesized CT image G(x) s The judgment result of ).
[0074] The adversarial loss between the image generator G and the residual shrinkage discriminator D in the unsupervised branch network is obtained according to the second calculation formula, which is:
[0075]
[0076] In the formula, x represents the adversarial loss term corresponding to the unsupervised branch network. u G(x) represents the input MRI image of the unsupervised branch network. u ) represents the pseudo-CT image synthesized by the image generator G under an unsupervised branch network, D(G(x) u )) is the residual shrinkage discriminator D for the synthesized CT image G(x) u The judgment result of ).
[0077] The pixel loss term is used to calculate the pixel-level distance between real CT images and synthetic pseudo-CT images on the supervised branch network. Specifically, according to the third calculation formula, the L1 loss is calculated for each pixel of each synthetic pseudo-CT image and the corresponding real CT image to obtain the pixel-level distance between the real and pseudo images. The third calculation formula is as follows:
[0078]
[0079] In the formula, ||·||1 represents the L1 norm, and y s This represents the real CT image corresponding to the supervised branch network. This represents a pseudo-CT image synthesized by a supervised branch network.
[0080] It should be understood that currently, pseudo-CT images synthesized using deep networks often suffer from significant information loss in high-frequency regions, resulting in issues such as image blurring and unclear boundaries. To ensure consistency between the input image and the synthesized image in the frequency domain and to improve the model's ability to synthesize frequency components that are difficult to synthesize in the image, a focal frequency loss is introduced as a frequency domain loss in the supervised branch. This establishes constraints on the input image and the synthesized image at the frequency domain level. Specifically, this embodiment will use the frequency domain loss term to calculate the focal frequency loss between the real CT image and the synthesized pseudo-CT image on the supervised branch network.
[0081] In this embodiment, the focal frequency loss is used as the frequency domain loss term, and the calculation process of the focal frequency loss function is as follows:
[0082] First, perform 2D Fourier transforms on the input CT image and the synthesized pseudo-CT image according to the fourth calculation formula to obtain their respective representations in the frequency domain. The fourth calculation formula is as follows:
[0083]
[0084] In the formula, x represents the input image to be converted, W×H represents the size of image x, (p,q) represents the pixel space coordinates of the image, f(p,q) represents the pixel value of the image at pixel space position (p,q); i represents the imaginary unit, e represents the Euler number; F x (u,v) represents the frequency value of the image x in the frequency domain coordinates (u,v) after it has been converted to the frequency domain representation.
[0085] Secondly, the weight value of each coordinate in the frequency domain is calculated according to the fifth calculation formula, which is as follows:
[0086]
[0087] In the formula, y s For the real CT images corresponding to the supervised branch network, This represents a pseudo-CT image synthesized by a supervised branch network. and Let w(u,v) represent the frequency domain representations of the real CT image and the synthetic pseudo CT image corresponding to the supervised branch network, respectively, and w(u,v) represent the weight value at the frequency domain coordinates (u,v).
[0088] Then, the focal frequency loss between the CT image and the synthesized pseudo-CT image is calculated according to the sixth calculation formula to obtain the frequency domain loss term. The sixth calculation formula is as follows:
[0089]
[0090] In the formula, This represents the focal frequency loss function.
[0091] Specifically, the second loss function includes an adversarial loss term, a pixel loss term, and a frequency domain loss term corresponding to the supervised branch network, as well as an adversarial loss term corresponding to the unsupervised branch network. The second loss function L... G The calculation formula is:
[0092]
[0093] In the formula, y sYs and Ys represent the real CT images and the set of real CT images corresponding to the supervised branch network, respectively. L represents a pseudo-CT image synthesized by a supervised branch network. pix L represents the pixel loss term. FFL λ represents the frequency domain loss term. pix and λ FFL The hyperparameter y represents the weights of the loss term. u and Y u Let these represent the real CT images and the set of real CT images corresponding to the unsupervised branch network, respectively. This represents a pseudo-CT image synthesized by an unsupervised branch network. This indicates a loss against the loss.
[0094] In this embodiment, during the alternating optimization of the image generator G and the residual shrinkage discriminator D using a semi-supervised adversarial training strategy, the parameters of the image generator G are first fixed, and the adversarial loss term L used to update the residual shrinkage discriminator D is obtained according to the seventh calculation formula. D (i.e., the first loss function), and then the parameters of the residual shrinkage discriminator D are updated and optimized by minimizing this loss. The seventh calculation formula is:
[0095]
[0096] In the formula, y s and L represents the real CT image and the synthetic pseudo-CT image corresponding to the supervised branch network, respectively. BCE For binary cross-entropy loss and L BCE (p,t)=-tlog(p)-(1-t)log(1-p), where t represents the true value in the actual value vector and p represents the predicted value in the predicted value vector.
[0097] Then, the parameters of the residual shrinkage discriminator D are fixed, and the adversarial loss term used to update the image generator G is obtained according to the eighth calculation formula. The eighth calculation formula is:
[0098]
[0099] In the formula, and These represent the synthesized pseudo-CT images corresponding to the supervised branch network and the unsupervised branch network, respectively.
[0100] Secondly, the loss function L used to update the image generator G is obtained according to the ninth calculation formula. G The parameters of the image generator G are updated and optimized by minimizing this loss function. The ninth calculation formula is:
[0101]
[0102] In the formula, λ pix and λ FFL This represents the hyperparameter that controls the weights of the loss term; the default value can be set to λ. pix =100, λ FFL =1; for the second loss function L G In other words,
[0103] Therefore, this embodiment takes into account the common difficulty in synthesizing high-frequency regions of MRI images in CT image synthesis methods. Frequency domain loss is added to the supervised branch network of the proposed semi-supervised generative adversarial network to reduce the difference between real CT images and synthesized CT images at the frequency domain level, thereby solving the problem of loss of high-frequency details in synthesized images. This improves the model's ability to synthesize difficult-to-synthesize frequency parts of the image, thus effectively improving the details and overall image quality of the synthesized CT image.
[0104] Step S40: Based on the image synthesis model, convert the MRI image to be converted into a synthetic CT image.
[0105] As an example, in this embodiment, the image synthesis model obtained above is used to convert the input MRI image to be converted into a synthetic CT image. It should be noted that the computer language and development framework used in this embodiment can be determined according to actual needs and are not limited here. For example, the preferred computer language is Python, and the preferred development framework is PyTorch.
[0106] Considering the limitations of supervised and unsupervised cross-domain medical image synthesis methods, this embodiment proposes a generative adversarial network based on semi-supervised learning for synthesizing CT images from MRI. It uses a dual-branch network with both supervised and unsupervised training to simultaneously utilize paired MRI and CT image data and single-modal MRI data. In the supervised branch network, a frequency domain loss term is introduced to reduce the difference between the synthesized image and the real image at the frequency domain level. At the same time, a residual shrinkage module is introduced into the discriminator to improve the discriminator's feature learning ability for noisy medical images, thereby effectively improving the detail and overall quality of the synthesized CT image.
[0107] This application also provides a semi-supervised CT image synthesis device, including:
[0108] A data construction unit is used to acquire MRI image data and CT image data to construct a training dataset, which includes paired and unpaired datasets;
[0109] The model building unit is used to build a semi-supervised generative adversarial network model. The semi-supervised generative adversarial network model includes a supervised branch network and an unsupervised branch network that share an image generator and a residual shrinkage discriminator. The residual shrinkage discriminator includes multiple channel-level residual shrinkage modules and a convolutional layer.
[0110] The model training unit is used to train the semi-supervised generative adversarial network model based on the training dataset to update and optimize the image generator and residual shrinkage discriminator, and generate an image synthesis model.
[0111] An image synthesis unit is used to convert an MRI image to be converted into a synthetic CT image based on the image synthesis model.
[0112] Furthermore, the channel-level residual shrinkage module is used to perform absolute value operations and soft thresholding operations on the input features, and to perform residual connection between the soft-thresholded features and the input features to obtain residual features.
[0113] Furthermore, the model training unit is specifically used for:
[0114] The parameters of the image generator G are fixed, and the parameters of the residual shrinkage discriminator D are updated and optimized by minimizing the first loss function used to update the residual shrinkage discriminator D.
[0115] The parameters of the residual shrinkage discriminator D are fixed, and the parameters of the image generator G are updated and optimized by minimizing the second loss function used to update the image generator G.
[0116] Furthermore, the second loss function includes an adversarial loss term, a pixel loss term, and a frequency domain loss term corresponding to the supervised branch network, as well as an adversarial loss term corresponding to the unsupervised branch network.
[0117] Furthermore, the second loss function L G The calculation formula is:
[0118]
[0119] In the formula, y s and Y s Let represent the real CT images and the set of real CT images corresponding to the supervised branch network, respectively. L represents a pseudo-CT image synthesized by a supervised branch network. pix L represents the pixel loss term. FFL λ represents the frequency domain loss term. pix and λ FFL The hyperparameter y represents the weights of the loss term. u and Y uLet these represent the real CT images and the set of real CT images corresponding to the unsupervised branch network, respectively. This represents a pseudo-CT image synthesized by an unsupervised branch network. This indicates a loss against the loss.
[0120] Furthermore, the model training unit is specifically used for:
[0121] The supervised branch network is trained based on the paired dataset;
[0122] The unsupervised branch network is trained based on the unpaired dataset.
[0123] It should be noted that those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the device and each unit described above can be referred to the corresponding process in the aforementioned embodiments of the semi-supervised CT image synthesis method, and will not be repeated here.
[0124] The apparatus provided in the above embodiments can be implemented as a computer program, which can be used in, for example... Figure 6 The image is running on a semi-supervised CT image synthesis device, as shown.
[0125] This application also provides a semi-supervised CT image synthesis device, including: a memory, a processor, and a network interface connected via a system bus. The memory stores at least one instruction, which is loaded and executed by the processor to implement all or part of the steps of the aforementioned semi-supervised CT image synthesis method.
[0126] The network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0127] A processor can be a CPU, or other general-purpose processors, DSPs (Digital Signal Processors), ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor, or any conventional processor. The processor is the control center of a computer device, connecting all parts of the computer device through various interfaces and lines.
[0128] Memory can be used to store computer programs and / or modules. The processor implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, at least one application program required for a function (such as video playback, image playback, etc.), etc.; the data storage area can store data created based on the use of the mobile phone (such as video data, image data, etc.). Furthermore, memory can include high-speed random access memory (RAM), and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, SMC (Smart MediaCard), SD (Secure Digital) cards, flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.
[0129] This application also provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements all or part of the steps of the aforementioned semi-supervised CT image synthesis method.
[0130] The embodiments of this application can implement all or part of the aforementioned processes, or they can be accomplished by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various methods described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, ROM (Read-Only memory), RAM (Random Access memory), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.
[0131] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, servers, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.
[0132] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0133] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.
[0134] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.
Claims
1. A semi-supervised CT image synthesis method, characterized in that, Includes the following steps: MRI and CT image data are acquired to construct a training dataset, which includes paired and unpaired datasets; A semi-supervised generative adversarial network model is constructed. The semi-supervised generative adversarial network model includes a supervised branch network and an unsupervised branch network that share an image generator and a residual shrinkage discriminator. The residual shrinkage discriminator includes multiple channel-level residual shrinkage modules and a convolutional layer. The semi-supervised generative adversarial network model is trained based on the training dataset to update and optimize the image generator and residual shrinkage discriminator, thereby generating an image synthesis model. Based on the image synthesis model, the MRI image to be converted is transformed into a synthetic CT image; The updated and optimized image generator and residual shrinkage discriminator include: Fixed image generator The parameters are determined by updating the residual shrinkage discriminant. The first loss function is minimized to update and optimize the residual shrinkage discriminator. Parameters; Fixed residual shrinkage discriminant The parameters are used to update the image generator. The second loss function is minimized to update and optimize the image generator. The parameters.
2. The semi-supervised CT image synthesis method as described in claim 1, characterized in that: The channel-level residual shrinkage module is used to perform absolute value operations and soft thresholding operations on the input features, and to perform residual concatenation between the soft-thresholded features and the input features to obtain residual features.
3. The semi-supervised CT image synthesis method as described in claim 1, characterized in that, The second loss function includes an adversarial loss term, a pixel loss term, and a frequency domain loss term corresponding to the supervised branch network, as well as an adversarial loss term corresponding to the unsupervised branch network.
4. The semi-supervised CT image synthesis method as described in claim 3, characterized in that, The second loss function The calculation formula is: In the formula, and Let represent the real CT images and the set of real CT images corresponding to the supervised branch network, respectively. This represents a pseudo-CT image synthesized by a supervised branch network. Represents the pixel loss term. Represents the frequency domain loss term. and The hyperparameter representing the weights of the loss term. and Let these represent the real CT images and the set of real CT images corresponding to the unsupervised branch network, respectively. This represents a pseudo-CT image synthesized by an unsupervised branch network. This indicates a loss against the loss.
5. The semi-supervised CT image synthesis method as described in claim 1, characterized in that, The training of the semi-supervised generative adversarial network model based on the training dataset includes: The supervised branch network is trained based on the paired dataset; The unsupervised branch network is trained based on the unpaired dataset.
6. A semi-supervised CT image synthesis device, characterized in that, include: A data construction unit is used to acquire MRI image data and CT image data to construct a training dataset, which includes paired and unpaired datasets; The model building unit is used to build a semi-supervised generative adversarial network model. The semi-supervised generative adversarial network model includes a supervised branch network and an unsupervised branch network that share an image generator and a residual shrinkage discriminator. The residual shrinkage discriminator includes multiple channel-level residual shrinkage modules and a convolutional layer. The model training unit is used to train the semi-supervised generative adversarial network model based on the training dataset to update and optimize the image generator and residual shrinkage discriminator, and generate an image synthesis model. An image synthesis unit is used to convert an MRI image to be converted into a synthesized CT image based on the image synthesis model. Specifically, the model training unit is used to fix the image generator. The parameters are determined by updating the residual shrinkage discriminant. The first loss function is minimized to update and optimize the residual shrinkage discriminator. Parameters; Fixed residual shrinkage discriminant The parameters are used to update the image generator. The second loss function is minimized to update and optimize the image generator. The parameters.
7. The semi-supervised CT image synthesis apparatus as described in claim 6, characterized in that: The channel-level residual shrinkage module is used to perform absolute value operations and soft thresholding operations on the input features, and to perform residual concatenation between the soft-thresholded features and the input features to obtain residual features.
8. A semi-supervised CT image synthesis device, characterized in that, include: A memory and a processor, wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the semi-supervised CT image synthesis method according to any one of claims 1 to 5.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program that, when executed by a processor, implements the semi-supervised CT image synthesis method according to any one of claims 1 to 5.
Citation Information
Patent Citations
Rapid CS-MRI reconstruction method based on deep residual generative adversarial network
CN112164122A
MRI reconstruction using deep learning, generative adversarial network and acquisition signal model
US20190369191A1
