UTE-CT cross-modal image style migration method and device based on cyclic generative adversarial network

Abstract

The invention relates to a UTE-CT cross-modal image style migration method and device based on a cyclic generative adversarial network. The method comprises the following steps: using a U-Net network added with a space channel double attention mechanism module scSE-block as a segmentation network to cut a UTE / CT image; a TransMorph network which is combined with a Transform and a ConvNet is adopted to carry out registration on the UTE / CT image; and carrying out cross-modal style migration on the UTE / CT image by adopting CyclGan. According to the method, the UTE image is converted into the CT-like image, the quality of the UTE image is improved, and the clinical application environment of the UTE is expanded to conventional screening and diagnosis of respiratory system diseases from reexamination and comparison of clearly-diagnosed large focuses. Meanwhile, the CT-like image obtained after migration is more in line with the radiograph reading habit of imaging doctors, multi-angle and multi-mode focus information can be provided for the doctors, and the diagnosis accuracy is further improved. The method is also used for data enhancement to generate diversified and high-quality high-throughput medical images, so that a training data set is enriched, and the generalization ability and performance of a deep learning radiomics model are improved.

Description

Technical Field

[0001] The present invention belongs to the field of medical image analysis, and in particular, relates to a UTE-CT cross-modal image style transfer method and device based on a cyclic generative adversarial network. Background Art

[0002] CT, or computer tomography, is a type of scan that uses a highly collimated X-ray beam to scan a certain thickness of the human body in a cross-sectional plane. It can provide high-definition and high-spatial-resolution images and is one of the gold standards for respiratory system imaging. CT can accurately display the internal structure and lesions of the lungs, including tiny structural changes in blood vessels, airways, masses, nodules, etc. It has extremely high sensitivity and specificity for lung nodule screening, lung cancer diagnosis and staging, and lung infection. However, the ionizing radiation generated by CT scanning limits its clinical application environment, and pregnant women and those preparing for pregnancy should avoid CT scanning. In addition, a large number of studies have shown that the accumulation of ionizing radiation will also increase the probability of occasional malignant transformation of lung nodules and postoperative metastasis of malignant lung cancer, and even induce or aggravate radiation pneumonia. For patients with the above-mentioned diseases who need to undergo regular CT follow-up examinations but need to avoid exposure to ionizing radiation as much as possible, frequent CT scans bring them huge health risks and heavy psychological burdens.

[0003] MRI, or magnetic resonance imaging, is an imaging examination that uses magnetic fields and radio frequency pulses for imaging. Although MRI does not have ionizing radiation, the low proton density of lung tissue and the presence of a large number of gas-tissue interfaces will lead to poor imaging effects, so MRI has rarely been used for respiratory system scanning in previous studies. Ultrashort echo imaging (UTE), a cutting-edge technology of magnetic resonance, fills the blind spot of MRI respiratory system imaging. Magnetic resonance ultrashort echo imaging technology UTE uses hard pulse excitation and directly detects free induction decay, which can image short T2 tissues such as lungs and bone cortex, and has been tentatively used for respiratory system scanning. However, compared with CT, the image resolution of UTE is still low, which is not conducive to the identification of subtle lesions. It is currently only used for review and comparison of larger lesions that have been clearly diagnosed. In addition, the image characteristics (style) of UTE are very different from CT, which does not fit the diagnostic reading habits of radiologists for many years. In summary, how to migrate the image style of UTE to the image style of CT to meet the reading habits of radiologists and increase the image resolution without losing effective image information is an urgent problem to be solved in the industry. Summary of the invention

[0004] The embodiments of the present application provide a UTE-CT cross-modal image style transfer method and device based on a recurrent generative adversarial network. The CT-like images obtained after the transfer will be more in line with the reading habits of imaging physicians, and can provide doctors with multi-angle and multi-modal lesion information, thereby improving the accuracy of diagnosis.

[0005] In a first aspect, an embodiment of the present application provides a UTE-CT cross-modal image style transfer method based on a cyclic generative adversarial network, comprising:

[0006] S101, cutting the UTE and CT images;

[0007] S103, registering the UTE and CT images;

[0008] S105, a cyclic generative adversarial network is used to perform cross-modal style transfer on UTE and CT images.

[0009] In step S101, a U-Net network with a spatial channel dual attention mechanism module scSE-block added is used as a segmentation network to segment the UTE and CT images, including:

[0010] In the feature extraction stage, two 3×3×3 convolutional layers are used each time to extract image features at different levels, and the LeakyReLU activation layer is used for activation, followed by a convolutional layer with a stride of 2 for downsampling;

[0011] In the decoding stage, the feature map after each upsampling is first fused with the shallow features of the same level encoding stage through jump connections, and then passes through two 3×3×3 convolutional layers to further decouple the features. Each convolutional layer is still followed by a LeakyReLU activation layer; then it passes through a spatial channel dual attention mechanism module scSE-block to recalibrate the features; finally, an up-convolution with a step size of 2 is used to increase the size of the feature map and restore the image’s detail information and spatial resolution.

[0012] Among them, the spatial channel dual attention mechanism module scSE-block workflow includes:

[0013] Compress the spatial dimension information of each channel into a single value to form a channel descriptor;

[0014] Use two consecutive 1x1 convolutional layers and Sigmoid activation function to learn the importance weight of each channel;

[0015] Calculate the attention weight of the spatial dimension through a 1x1 convolution layer and a Sigmoid activation function;

[0016] Apply the learned channel attention weights and spatial attention weights to the channel and spatial dimensions of the original feature map respectively to perform feature recalibration;

[0017] The feature maps that have been recalibrated by attention are fused to obtain the final feature representation, which can be passed to the next layer of the network.

[0018] In step S101, the Dice loss coefficient is used as the objective function to optimize each training sample:

[0019]

[0020] Among them, one group is the real segmentation area, and the other group is the predicted segmentation area; |PS∩GT| calculates the overlapping part of the two groups of segmentation areas, that is, the number of voxels shared by the two groups of segmentation areas, |PS| represents the number of voxels in the predicted segmentation area, and |GT| represents the number of voxels in the real segmentation area.

[0021] In step S101, the Dice coefficient, Recall and Precision are used as evaluation indicators. The Dice coefficient is used to evaluate the overlap between the true label and the predicted label; Recall reflects how many voxels in the true label are correctly predicted; Precision reflects how many of the predicted segmentation results belong to the true label. The specific formula is as follows:

[0022]

[0023] Where, T TP represents the number of voxels correctly classified into the target area, F FP represents the number of voxels misclassified into the target region, F FN Represents the number of voxels that are misclassified into the background area.

[0024] In step S103, the UTE and CT images are registered using a TransMorph network combining Transformer and ConvNet, including:

[0025] Before image registration, the two sets of images are aligned through an affine transformation network as a preprocessing stage of image registration. The affine network architecture is an improved Swin Transformer, which takes two 3D images as input and generates 12 affine parameters: three rotation angles, three translation parameters, three scaling parameters, and three shearing parameters.

[0026] The channel features of adjacent token groups of 2x2x2 are added together to realize the Patch Merging layer to merge adjacent image blocks. While merging image blocks, the Patch Merging layer doubles the number of channels of the features. Through continuous PatchMerging operations, Swin Transformer can create a series of feature maps of different resolutions.

[0027] In the encoding stage of the network, the two sets of input images are first divided into non-overlapping 3D blocks, and the size of each patch is 2×P×P×P, where the size of P is a preset value; then each patch is flattened and used as a token, and each token is projected into a feature space of arbitrary dimensions through a linear mapping; then the image is encoded through several consecutive Patchmerging and Swin Transformer, and the output dimension of the last level of the encoder is:

[0028]

[0029] The decoder consists of consecutive upsampling layers and convolutional layers, where the convolution kernel size is 3×3; each upsampled feature map in the decoding stage is connected to the corresponding feature map in the encoding path through a skip connection, and then passes through two consecutive convolutional layers in sequence; when the image is input, the image is directly downsampled and feature extracted, and the original and downsampled image pairs are used as input to capture local information and generate high-resolution feature maps;

[0030] The overall loss function for network training consists of two parts: one part calculates the similarity between the registered floating image and the fixed image, and the other part maintains the smoothness of the deformation field:

[0031] L(I f ,I m ,Φ)=L sim (I f ,I m ,Φ)+λR(Φ)

[0032] Where I f refers to a fixed image, I m refers to the floating image, Φ is the generated deformation field, L sim is the image similarity measure, and R(Φ) is the deformation field regularization;

[0033] L sim Two similarity measures were used to evaluate the similarity between two image regions;

[0034] Assess the quality of the registered images.

[0035] In step S105, a cyclic generative adversarial network is used as the backbone network for style transfer, and three loss functions are used, namely, adversarial loss L adv , cycle-consistent loss L cycle and the similarity constraint loss term L sc ;

[0036] L adv The least squares loss LSGAN is used for calculation. The specific calculation formula of the discriminator is as follows:

[0037]

[0038] In the formula, x represents the real image in the target domain, y represents the image in the source domain, D(x)-1 represents the probability of judging the real image as fake, D(F(y)) represents the probability of judging the generated image as real, and E x ,E y It represents the distribution of target domain images and source domain images. The calculation formula of the generator part is as follows:

[0039] L adv =E y [(D(F(y))-1) 2 ]

[0040] L cycle It is an indirect structural constraint term between the input real image and the output pseudo image. It calculates the difference between the pseudo source domain image generated after the cycle and the real source domain image. The calculation process is as follows:

[0041] L cycle =E UTE [||F(G(UTE))-UTE||1]+E CT [||G(F(CT))-CT||1]

[0042] L sc In order to ensure the accuracy of the generated pseudo CT (sCT), a direct similarity constraint is imposed on the sCT image and the input UTE image, and the similarity measure in multimodal image registration, the normalized mutual information, is introduced as the similarity constraint term L sc , the function of the loss term consists of two parts of the corresponding two generators:

[0043]

[0044] In the formula, the loss term of the generator G is calculated as follows:

[0045]

[0046] H(UTE) is the information entropy of the UTE image, H(G UTE-CT (UTE)) is the information entropy of the generated sCT image, H(G UTE-CT (UTE),UTE) is the joint entropy of the two images, and similarly we can get

[0047] In a second aspect, the present application provides a UTE-CT cross-modal image style transfer device based on a cyclic generative adversarial network, comprising:

[0048] A cutting unit, used for cutting UTE and CT images;

[0049] A registration unit, used for registering UTE and CT images;

[0050] Transfer unit, used to perform cross-modal style transfer between UTE and CT images using a recurrent generative adversarial network.

[0051] In a third aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of any of the above methods when executed by a processor.

[0052] In a fourth aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above methods when executing the program.

[0053] The UTE-CT cross-modal image style transfer method and device based on the cyclic generative adversarial network in the embodiment of the present application have the following beneficial effects:

[0054] This application realizes the conversion of UTE images into CT-like images, aiming to improve the quality of UTE images and expand the clinical application environment of UTE from the review and comparison of clearly diagnosed large lesions to the routine screening and diagnosis of respiratory diseases. At the same time, the CT-like images obtained after the migration will be more in line with the reading habits of imaging physicians, and can provide doctors with multi-angle and multi-modal lesion information, thereby improving the accuracy of diagnosis. In addition, this technology will also be used for data enhancement to generate diverse, high-quality high-throughput medical images to enrich the training data set and thus improve the generalization ability and performance of deep learning imaging omics models. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a schematic diagram of the process of the UTE-CT cross-modal image style transfer method based on the cyclic generative adversarial network in the embodiment of the present application;

[0056] Figure 2 This is a diagram of the segmented network structure in this application;

[0057] Figure 3 This is the structure diagram of the registration network and the structure diagram of the Swin Transformer in this application;

[0058] Figure 4 Align the network structure diagram in this application;

[0059] Figure 5 This is the CycleGan network model diagram in this application;

[0060] Figure 6This is the structure diagram of the generator in this application;

[0061] Figure 7 This is a schematic diagram of the structure of the UTE-CT cross-modal image style transfer device based on a cyclic generative adversarial network in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The present application is further described below in conjunction with the accompanying drawings and embodiments.

[0063] In the following introduction, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. The following introduction provides multiple embodiments of the present invention, and different embodiments can be replaced or combined, so this application can also be considered to include all possible combinations of the same and / or different embodiments recorded. Therefore, if one embodiment includes features A, B, and C, and another embodiment includes features B and D, then this application should also be considered to include embodiments containing one or more of all other possible combinations of features A, B, C, and D, even though the embodiment may not be clearly recorded in the following text.

[0064] like Figure 1 As shown, the UTE-CT cross-modal image style transfer method based on the cyclic generative adversarial network of the present application includes: S101, cutting the UTE and CT images; S103, registering the UTE and CT images; S105, using the cyclic generative adversarial network to perform cross-modal style transfer on the UTE and CT images. The present application realizes the conversion of UTE images into CT-like images, aiming to improve the quality of UTE images and expand the clinical application environment of UTE from the review and comparison of large lesions that have been clearly diagnosed to the routine screening and diagnosis of respiratory diseases. At the same time, the CT-like images obtained after the migration will be more in line with the reading habits of imaging physicians, and can provide doctors with multi-angle and multi-modal lesion information, thereby improving the accuracy of diagnosis. In addition, this technology will also be used for data enhancement to generate diversified, high-quality high-throughput medical images to enrich the training data set and thus improve the generalization ability and performance of the deep learning imaging omics model. The following is a detailed introduction.

[0065] This application proposes a cross-modal image style transfer method for CT-like UTE images using a cyclic generative adversarial network (CycleGan). Since chest images contain moving organs such as lungs and heart, the relative positions of tissue structures in UTE and CT images are affected by many factors such as patient posture, breathing, heartbeat, etc., and direct migration will seriously affect the accuracy of the model. Therefore, in order to solve the above problems, different from previous solutions, the inventors use the following three steps for image processing: 1. The U-Net network with the added spatial channel dual attention mechanism module scSE-block is used as the segmentation network to cut the UTE / CT images; 2. The TransMorph network that combines Transformer and ConvNet is used to align the UTE / CT images; 3. CycleGan is used to perform cross-modal style transfer on UTE / CT images.

[0066] 1. Segmentation of CT images and UTE images

[0067] The purpose of medical image segmentation is to obtain the segmentation labels of the region of interest and to propose boundary information to further perform tasks such as image analysis and computer-aided diagnosis. After segmenting and extracting the region of interest, it will not only reduce the impact of irrelevant regions on style transfer, greatly improving the transfer effect, but also reduce the amount of data for subsequent processing, reduce model parameters, and improve performance. This application uses the U-Net network with the spatial channel dual attention mechanism module scSE-block as the segmentation network. The network structure is as follows: Figure 2 As shown, the specific steps are as follows:

[0068] (1) Encoding stage: In the feature extraction stage, two 3×3×3 convolutional layers are used each time to extract image features at different levels, and a LeakyReLU activation layer is used for activation. Subsequently, a convolutional layer with a stride of 2 is used for downsampling to compress the feature dimension and increase the network depth.

[0069] (2) Decoding stage: In the decoding stage, the feature map after each upsampling is first fused with the shallow features of the same level encoding stage through skip connections, and then passes through two 3×3×3 convolutional layers (each convolutional layer is still followed by a LeakyReLU activation layer) to further decouple the features and improve the nonlinear mapping ability of the network. Subsequently, the features are recalibrated through a spatial channel dual attention mechanism module scSE-block. Finally, an up-convolution with a stride of 2 is used to increase the size of the feature map and restore the image's detail information and spatial resolution.

[0070] (3) ScSE-block: Spatial-channel dual attention mechanism module is an attention mechanism module for convolutional neural networks. It combines the spatial attention and channel attention mechanisms to improve the network's ability to screen and enhance features, thereby improving model performance. Its workflow usually includes the following steps:

[0071] Squeeze operation: Through operations such as global average pooling, the spatial dimension information of each channel is compressed into a single value to form a channel descriptor.

[0072] Excitation operation: Use two consecutive 1x1 convolutional layers and Sigmoid activation function to learn the importance weights of each channel.

[0073] Spatial attention weight calculation: The attention weight of the spatial dimension is calculated through a 1x1 convolution layer and a Sigmoid activation function.

[0074] Feature recalibration: The learned channel attention weights and spatial attention weights are applied to the channel and spatial dimensions of the original feature map respectively to perform feature recalibration.

[0075] Feature fusion: The feature maps that have been recalibrated by attention are fused to obtain the final feature representation, which can be passed to the next layer of the network.

[0076] (4) Application of global skip connection structure: Skip connections play a vital role in the U-Net network structure. They are essential to the performance and functionality of U-Net. By directly connecting the feature maps of shallower and deeper layers, it helps the gradient to back propagate more efficiently during training. And it allows the network to directly access the high-resolution features learned in the encoding (downsampling) stage in the decoding (upsampling) stage. This helps the network recover the detailed information of the image and improve the accuracy of segmentation.

[0077] (5) Loss function: Use the Dice loss coefficient as the objective function to optimize each training sample.

[0078]

[0079] Among them, one group is the real segmentation area (Ground Truth, GT), and the other group is the predicted segmentation area (Predicted Segmentation, PS). |PS∩GT| calculates the overlapping parts of the two groups of segmentation areas, that is, the number of voxels shared by the two groups of segmentation areas, while |PS| represents the number of voxels in the predicted segmentation area, and |GT| represents the number of voxels in the real segmentation area.

[0080] (6) Evaluate the quality of the segmented image: Use the Dice coefficient, Recall rate and Precision accuracy as evaluation indicators. The Dice coefficient is used to evaluate the overlap between the true label and the predicted label; Recall reflects how many voxels in the true label are correctly predicted; Precision reflects how many of the predicted segmentation results belong to the true label. The specific formula is as follows:

[0081]

[0082] Among them, T TP represents the number of voxels correctly classified into the target area, F FP represents the number of voxels misclassified into the target region, F FN Represents the number of voxels that are misclassified into the background area.

[0083] II. Registration of CT images and UTE images

[0084] Medical image registration is a key technology in the field of medical image analysis. Specifically, for the two images to be registered, one image (Moving Image) is mapped to the other image (Fixed Image) so that the points with the same anatomical significance on the two images correspond to each other, thereby performing information matching and fusion. Through the registration and fusion of multimodal images such as computed tomography (CT) and magnetic resonance ultrashort echo sequence images (UTE), the complementary information of different modalities can be utilized to analyze diseases from different aspects. Since the registration speed and the quality of the registration results will seriously affect the subsequent quantitative analysis, it is of great clinical significance to achieve fast and accurate multimodal registration. This application uses the TransMorph network that combines Transformer and ConvNet as the segmentation network. The network structure and the structure diagram of Swin Transformer are shown in the following figure. Figure 3 As shown, the specific steps are as follows:

[0085] (1) Image alignment: Before image registration, the two sets of images are aligned through the Affine Transformation Network as a preprocessing stage for image registration, which is conducive to the subsequent more complex DIR process (deformable image registration). The affine network architecture used here is an improved Swin Transformer, which takes two 3D images as input and generates 12 affine parameters: three rotation angles, three translation parameters, three scaling parameters, and three shearing parameters. The structural diagram of the alignment network is shown in Figure 4 shown.

[0086] (2) Patch Merging: The Patch Merging layer merges adjacent image patches, thereby reducing the resolution of the feature map. This is achieved by adding the channel features of adjacent token groups of 2x2x2, effectively halving the size of the feature map in each dimension. And while merging image patches, the Patch Merging layer doubles the number of feature channels, keeping the total number of parameters stable. Through continuous Patch Merging operations, the Swin Transformer is able to create a series of feature maps of different resolutions, which helps capture the multi-scale information of the image.

[0087] (3) Encoding stage: In the encoding stage of the network, the two sets of input images are first divided into non-overlapping 3D blocks, and the size of each patch is 2×P×P×P, where the size of P is generally set to 4. Each patch is then flattened and used as a token, and each token is projected into a feature space of arbitrary dimensions through a linear mapping. The image is then encoded through several consecutive patch merging and Swin Transformer, and the output dimension of the last level of the encoder is Wherein H represents height, W represents width, and L represents length.

[0088] (4) Decoding stage: The decoder consists of consecutive upsampling layers and convolutional layers, where the convolution kernel size is 3×3. Each upsampled feature map in the decoding stage is connected to the corresponding feature map in the encoding path through a skip connection, and then passes through two consecutive convolutional layers in sequence. Due to the nature of the patch operation, the Transformer encoder can only provide a maximum of The high-resolution feature map is not available, and local information cannot be aggregated at lower layers. To address this shortcoming, this application directly downsamples and extracts features from images when input, and uses the original and downsampled image pairs as input to capture local information and generate high-resolution feature maps. The outputs of these layers are connected to the feature maps in the decoder to generate deformation fields.

[0089] (5) Loss function: The overall loss function of network training consists of two parts: one part calculates the similarity between the registered floating image and the fixed image, and the other part maintains the smoothness of the deformation field.

[0090] L(I f ,I m ,Φ)=L sim (I f ,I m ,Φ)+λR(Φ)

[0091] Among them I frefers to a fixed image, I m refers to the floating image, Φ is the generated deformation field, L sim is the image similarity measure, and R(Φ) is the deformation field regularization, which maintains the smoothness of the deformation field.

[0092] (6) Image similarity measurement: L sim Two similarity metrics are used, one is MSE (mean square error), which is I f and I m The mean of the squared differences in voxel values ​​between .

[0093]

[0094] where p represents the voxel position and Ω represents the image domain.

[0095] Another metric used is LNCC (Local Normalized Cross Correlation), which is used to evaluate the similarity between two image regions.

[0096]

[0097] in Indicates that the size of voxel p is n 3 The average voxel value in the local window, generally n=9.

[0098] (7) Evaluate the quality of the registered image: Use Hausdorff Distance (HdDist95), structural similarity (SSIM) and mean absolute error (MSE) as evaluation indicators. HdDist95 is used to measure the maximum distance between two sets of points, which refers to the maximum surface distance between the registered image and the reference image; SSIM is used to quantify the structural similarity between the two images.

[0099] 3. Cross-modal style transfer between UTE images and CT images

[0100] In the medical field, accurate dose planning is essential to ensure treatment effectiveness and protect surrounding healthy tissue. To achieve this goal, doctors usually rely on high-resolution CT images to assess the location and size of the region of interest, as well as the structure of the surrounding tissue. However, the acquisition of CT images involves the use of ionizing radiation, which not only increases the economic burden on patients, but also may pose additional risks to their health. UTE images provide an alternative method that does not require radiation, which can display the region of interest and surrounding tissue with higher soft tissue contrast. However, the physical properties of UTE images are different from those of CT images, and the resolution is lower, which makes it complicated to use UTE images directly for dose planning. Therefore, achieving style transfer from UTE images to CT images has important clinical and social significance in the medical field, and it helps to improve the safety, accuracy and accessibility of treatment. This application uses a cyclic generative adversarial network (CycleGan) as the backbone network for style transfer, and the specific steps are as follows:

[0101] (1) CyeleGAN: CycleGAN is an innovative variant of the Generative Adversarial Network (GAN) that can achieve effective conversion from source domain UTE images to target domain CT images in the absence of paired samples. Figure 5 As shown in the figure, the core goal of CycleGAN is to convert UTE images into CT images by learning a mapping function G, and then use another generator F to inversely convert the CT images back to UTE images. This process is achieved by introducing a cycle consistency loss, that is, ensuring that F(G(UTE)) is close to the original UTE image. In order to minimize the difference between the input UTE image and the image converted back by F, the generator G must produce a CT image that is highly similar to the original image. Otherwise, when the image generated by G is too different from the input image, the image inversely converted by F will also be significantly different from the original UTE image. The CycleGAN framework contains two pairs of generators and discriminators, and its innovation lies in the use of the concept of cycle consistency. This cyclic process ensures the consistency between the original UTE image and its corresponding pseudo-UTE image, forming a cycle that starts from the original image and eventually returns to the starting point.

[0102] In addition, in previous style transfer, paired training data is always required for the model to learn. However, in nature, it may be very difficult or costly to obtain paired image data. The emergence of CycleGAN solves this problem well.

[0103] (2) Generator: Figure 6 For the generator structure, such as Figure 6As shown in Figure 1, in CycleGAN, a convolutional neural network is combined with a residual network to form a generator, which is generally implemented in the form of a codec. This residual network structure helps to better preserve and transfer the semantic information of the image, thereby achieving higher quality results in image conversion tasks.

[0104] In the encoding stage, the image is first input through a convolution layer with a convolution kernel size of 7x7 for image enhancement. The main purpose is to extract the edge information of the image and improve the image resolution. Then, repeated convolution layers with a convolution kernel size of 3×3 are used to extract image features. Each convolution operation contains an instance normalization layer and a ReLu activation function. The encoding part uses maximum pooling to reduce the feature size. After one pooling layer, the image size is reduced by 1 times, the number of feature channels is increased by 1 times, and the encoded feature map enters the residual connection block. Finally, in the decoding stage, upsampling operations are used to gradually restore the size of the feature map. Through two upsampling operations, the feature map is enlarged to the original size, and finally the synthesized CT image is output through convolution.

[0105] (3) Discriminator: The discriminator in CycleGan adopts the form of PatchGan, which judges the true or false of the input image in units of local image blocks and outputs a matrix. Each value in the matrix represents the probability of the current patch being judged as true in the original receptive field, and finally takes the average value as a judgment result. Its structure is relatively simple, using 5 convolutional layers, including 4 convolutions and instance normalization processing and LeakyRelu as the activation function. Finally, the output layer uses the sigmoid function to determine whether the output is a real CT image.

[0106] (4) Loss function: Three loss functions are used, namely, adversarial loss (L adv ), cycle consistency loss (L cycle ) and the similarity constraint loss term (L sc ).

[0107] L adv LSGAN (least square loss) is used for calculation. The specific calculation formula of the discriminator is as follows:

[0108]

[0109] Where x represents the real image in the target domain, y represents the image in the source domain, D(x)-1 represents the probability of judging the real image as fake, and D(F(y)) represents the probability of judging the generated image as real. x ,E y It represents the distribution of target domain images and source domain images. The calculation formula of the generator part is as follows:

[0110] L adv=E y [(D(F(y))-1) 2 ]

[0111] L cycle It is an indirect structural constraint term between the input real image and the output pseudo image. It calculates the difference between the pseudo source domain image generated after the loop and the real source domain image. The specific calculation process is as follows:

[0112] L cycle =E UTE [||F(G(UTE))-UTE||1]+E CT [||G(F(CT))-CT||1]

[0113] L sc In order to ensure the accuracy of the generated pseudo CT (sCT), a direct similarity constraint is imposed on the sCT image and the input UTE image, and the normalized mutual information (NMI), a similarity measure commonly used in multimodal image registration, is introduced as the similarity constraint term L sc The function of this loss term consists of two parts corresponding to the two generators:

[0114]

[0115] Among them, the loss term of the generator G is calculated as follows:

[0116]

[0117] H(UTE) is the information entropy of the UTE image, H(G UTE-CT (UTE)) is the information entropy of the generated sCT image, H(G UTE-CT (UTE),UTE) is the joint entropy of the two images. Similarly, we can get

[0118] (5) Evaluation indicators: Peak signal-to-noise ratio (PSNR), structural similarity (SSIM) and mean absolute error (MSE) are used as evaluation indicators. PSNR (unit: dB) indicates the quality of CT image signal reconstruction; SSIM is an evaluation indicator for evaluating the overall similarity between the real CT image and the sCT image, ranging from [0, 1]. The larger the value, the more similar the two images are; MSE is the sum of the absolute value differences between the synthesized sCT image and the real CT image, and then the average is taken.

[0119] like Figure 7As shown, the present application also provides a UTE-CT cross-modal image style transfer device based on a cyclic generative adversarial network, including: a cutting unit 201, used to cut UTE and CT images; a registration unit 202, used to register UTE and CT images; a migration unit 203, used to perform cross-modal style transfer on UTE and CT images using a cyclic generative adversarial network.

[0120] In the present application, the embodiment of the UTE-CT cross-modal image style transfer device based on a cyclic generative adversarial network is basically similar to the embodiment of the UTE-CT cross-modal image style transfer method based on a cyclic generative adversarial network. For relevant matters, please refer to the introduction of the embodiment of the UTE-CT cross-modal image style transfer method based on a cyclic generative adversarial network.

[0121] The present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above methods when executing the program.

[0122] The embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of the UTE-CT cross-modal image style transfer method based on a cyclic generative adversarial network are implemented. The computer-readable storage medium may include, but is not limited to, any type of disk, including a floppy disk, an optical disk, a DVD, a CD-ROM, a microdrive, and a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory device, a magnetic card or an optical card, a nanosystem (including a molecular memory IC), or any type of medium or device suitable for storing instructions and / or data.

[0123] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A UTE-CT cross-modal image style transfer method based on a cyclic generative adversarial network, characterized in that: include: S101, cutting the UTE and CT images; S103, registering the UTE and CT images; S105, a cyclic generative adversarial network is used to perform cross-modal style transfer on UTE and CT images.

2. The UTE-CT cross-modal image style transfer method based on a cyclic generative adversarial network according to claim 1 is characterized in that: In step S101, a U-Net network with a spatial channel dual attention mechanism module scSE-block added is used as a segmentation network to segment the UTE and CT images, including: In the feature extraction stage, two 3×3×3 convolutional layers are used each time to extract image features at different levels, and the LeakyReLU activation layer is used for activation, followed by a convolutional layer with a stride of 2 for downsampling; In the decoding stage, the feature map after each upsampling is first fused with the shallow features of the same level encoding stage through jump connections, and then passes through two 3×3×3 convolutional layers to further decouple the features. Each convolutional layer is still followed by a LeakyReLU activation layer; then it passes through a spatial channel dual attention mechanism module scSE-block to recalibrate the features; finally, an up-convolution with a step size of 2 is used to increase the size of the feature map and restore the image’s detail information and spatial resolution.

3. The UTE-CT cross-modal image style transfer method based on a cyclic generative adversarial network according to claim 2 is characterized in that: The spatial channel dual attention mechanism module scSE-block workflow includes: Compress the spatial dimension information of each channel into a single value to form a channel descriptor; Use two consecutive 1x1 convolutional layers and Sigmoid activation function to learn the importance weight of each channel; Calculate the attention weight of the spatial dimension through a 1x1 convolution layer and a Sigmoid activation function; Apply the learned channel attention weights and spatial attention weights to the channel and spatial dimensions of the original feature map respectively to perform feature recalibration; The feature maps that have been recalibrated by attention are fused to obtain the final feature representation, which can be passed to the next layer of the network.

4. The UTE-CT cross-modal image style transfer method based on a cyclic generative adversarial network according to claim 2 or 3, characterized in that: In step S101, the Dice loss coefficient is used as the objective function to optimize each training sample: Among them, one group is the real segmentation area, and the other group is the predicted segmentation area; |PS∩GT| calculates the overlapping part of the two groups of segmentation areas, that is, the number of voxels shared by the two groups of segmentation areas, |PS| represents the number of voxels in the predicted segmentation area, and |GT| represents the number of voxels in the real segmentation area.

5. The UTE-CT cross-modal image style transfer method based on a cyclic generative adversarial network according to claim 2 or 3, characterized in that: In step S101, the Dice coefficient, Recall and Precision are used as evaluation indicators. The Dice coefficient is used to evaluate the overlap between the true label and the predicted label; Recall reflects how many voxels in the true label are correctly predicted; Precision reflects how many of the predicted segmentation results belong to the true label. The specific formula is as follows: Where, T TP represents the number of voxels correctly classified into the target area, F FP represents the number of voxels misclassified into the target region, F FN Represents the number of voxels that are misclassified into the background area.

6. The UTE-CT cross-modal image style transfer method based on a cyclic generative adversarial network according to any one of claims 1 to 3, characterized in that: In step S103, the UTE and CT images are registered using a TransMorph network that combines Transformer and ConvNet, including: Before image registration, the two sets of images are aligned through an affine transformation network as a preprocessing stage of image registration. The affine network architecture is an improved Swin Transformer, which takes two 3D images as input and generates 12 affine parameters: three rotation angles, three translation parameters, three scaling parameters, and three shearing parameters. The channel features of adjacent token groups of 2x2x2 are added together to realize the Patch Merging layer to merge adjacent image blocks. While merging image blocks, the Patch Merging layer doubles the number of channels of the features. Through continuous PatchMerging operations, Swin Transformer can create a series of feature maps of different resolutions. In the encoding stage of the network, the two sets of input images are first divided into non-overlapping 3D blocks, and the size of each patch is 2×P×P×P, where the size of P is a preset value; then each patch is flattened and used as a token, and each token is projected into a feature space of arbitrary dimensions through a linear mapping; then the image is encoded through several consecutive Patch merging and Swin Transformer, and the output dimension of the last level of the encoder is: The decoder consists of consecutive upsampling layers and convolutional layers, where the convolution kernel size is 3×3; each upsampled feature map in the decoding stage is connected to the corresponding feature map in the encoding path through a skip connection, and then passes through two consecutive convolutional layers in sequence; when the image is input, the image is directly downsampled and feature extracted, and the original and downsampled image pairs are used as input to capture local information and generate high-resolution feature maps; The overall loss function for network training consists of two parts: one part calculates the similarity between the registered floating image and the fixed image, and the other part maintains the smoothness of the deformation field: L(I f ,AND m ,Φ)=L sim (AND f ,AND m ,Φ)+λR(Φ) In the formula, I f refers to a fixed image, I m refers to the floating image, Φ is the generated deformation field, L sim is the image similarity measure, and R(Φ) is the deformation field regularization; L sim Two similarity measures were used to evaluate the similarity between two image regions; Evaluate the quality of registered images.

7. The UTE-CT cross-modal image style transfer method based on a cyclic generative adversarial network according to any one of claims 1 to 3, characterized in that: In step S105, a cyclic generative adversarial network is used as the backbone network for style transfer, and three loss functions are used, namely, adversarial loss L adv , cycle-consistent loss L cycle and the similarity constraint loss term L sc ; L adv The least squares loss LSGAN is used for calculation. The specific calculation formula of the discriminator is as follows: In the formula, x represents the real image in the target domain, y represents the image in the source domain, D(x)-1 represents the probability of judging the real image as fake, D(F(y)) represents the probability of judging the generated image as real, and E x ,E y It represents the distribution of target domain images and source domain images. The calculation formula of the generator part is as follows: L adv =E y [(D(F(y))-1) 2 ] L cycle It is an indirect structural constraint term between the input real image and the output pseudo image. It calculates the difference between the pseudo source domain image generated after the cycle and the real source domain image. The calculation process is as follows: L cycle =E UTE [||F(G(UTE))-UTE||1]+E CT [||G(F(CT))-CT||1] L sc In order to ensure the accuracy of the generated pseudo CT (sCT), a direct similarity constraint is imposed on the sCT image and the input UTE image, and the similarity measure in multimodal image registration, the normalized mutual information, is introduced as the similarity constraint term L sc , the function of the loss term is composed of two parts of the corresponding two generators composition: Among them, the loss term of the generator G is calculated as follows: H(UTE) is the information entropy of the UTE image, H(G UTE-CT (UTE)) is the information entropy of the generated sCT image, H(G UTE-CT (UTE),UTE) is the joint entropy of the two images, and similarly we can get 8. A UTE-CT cross-modal image style transfer device based on a cyclic generative adversarial network, characterized in that: include: A cutting unit, used for cutting UTE and CT images; A registration unit, used for registering UTE and CT images; Transfer unit, used to perform cross-modal style transfer between UTE and CT images using a recurrent generative adversarial network.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images