Progressive medical image cross-modality generation method and device based on deep learning deterministic guidance

Abstract

The application discloses a kind of progressive medical image cross-modal generation method and device of certainty guidance based on deep learning, comprising: obtaining real CT image and real MR image photographed to same part and constructing sample data, with CycleGAN as benchmark, by increasing multiscale MR organ region discriminator and / or multiscale MR lesion region discriminator, form cross-modal generation framework;When constructing loss function, increase the adversarial loss function based on MR organ region and / or the adversarial loss function based on MR lesion region, form the total loss function of cross-modal generation framework;Utilize total loss function to optimize the parameter of cross-modal generation framework, so that the progressive adversarial generation network of single discriminator to multiple different complexity discriminators is used, with the purpose of certainty guidance, focus on organ region and / or target region such as lesion region, generate higher quality image.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of computer technology and medical image cross-modal generation, specifically relating to a deterministically guided progressive medical image cross-modal generation method and apparatus based on deep learning. Background Technology

[0002] Medical imaging is a powerful diagnostic and research tool that creates visual representations of anatomical structures and is widely used in disease diagnosis and surgical planning. In current clinical practice, computed tomography (CT) and magnetic resonance imaging (MR) are the most commonly used. Because CT and various MR modalities provide complementary information, the effective integration of these different modalities can help physicians make more informed decisions. Due to the difficulty in obtaining paired multimodal images, there is a growing need in clinical practice to develop multimodal image generation techniques to assist in clinical diagnosis and treatment.

[0003] Medical image generation can be divided into traditional machine learning methods and deep learning methods. Traditional machine learning methods rely on explicit feature representations, such as random forests and k-nearest neighbors, which iteratively optimize feature representations. Recently, convolutional neural networks have been widely used in various image generation tasks, and state-of-the-art performance has been achieved through generative adversarial networks.

[0004] Existing methods have proposed conditional generative adversarial networks (GANs) with deterministic outputs. These architectures typically employ L1 / L2-based loss functions and learn deterministic mappings without explicitly modeling robustness to outliers or prediction uncertainties, leading to performance degradation when encountering unseen out-of-distribution patterns during testing. While these methods can deliver synthetic images with high visual quality, the image content may still deviate significantly from the corresponding ground truth. This can lead to overconfidence or misinterpretation and have negative consequences, particularly in the medical field.

[0005] Existing GAN architectures typically focus on generating the entire image, which often leads to deformation of the target region without prior knowledge. The generation quality of local target regions is poor, becoming blurry or even having additional unreasonable textures.

[0006] In existing GAN architectures, the discriminator typically uses a single-scale discriminator (Markov discriminator). This is because in cross-modal medical image generation, the data distribution differences between different modalities are significant. Applying multi-scale discriminators used in natural images to medical images often leads to modality collapse or gradient vanishing phenomena. A stronger discriminator also means higher quality generated results.

[0007] Recent research has shown that using the idea of ​​multi-discriminator ensemble can produce higher quality results. However, the stepwise generation of high-quality images is still unguided, without paying special attention to regions where the generated results are poor. Summary of the Invention

[0008] In view of the above, the purpose of this invention is to provide a method and apparatus for progressive medical image generation based on deterministic guidance using deep learning. It employs a progressive adversarial generative network that ranges from a single discriminator to multiple discriminators of different complexities, with deterministic guidance as its goal, focusing on target regions such as organ regions and / or lesion regions to generate higher quality images.

[0009] To achieve the above-mentioned objectives, this embodiment provides a deterministically guided progressive cross-modal generation method for medical images based on deep learning, including deterministic guidance of the generation process from CT images to MR images, comprising the following steps:

[0010] Acquire real CT images and real MR images of the same area, and preprocess the real CT images and real MR images to construct image pairs containing real CT and real MR, MR organ region masks, MR lesion region masks, and form sample data;

[0011] Based on CycleGAN, a cross-modal generation framework is formed by adding a multi-scale MR organ region discriminator for identifying MR organ regions and / or a multi-scale MR lesion region discriminator for identifying MR lesion regions.

[0012] Based on the original loss function of CycleGAN, an adversarial loss function based on MR organ region and / or an adversarial loss function based on MR lesion region is added to form the total loss function of cross-modal generation framework;

[0013] The sample data is input into the cross-modal generation framework, and the parameters of the cross-modal generation framework are optimized using the total loss function. After parameter optimization, the generator contained in CycleGAN is used for cross-modal generation of medical images.

[0014] In one embodiment, the preprocessing of CT and MR images includes: rigid registration and normalization of real CT and real MR images.

[0015] In one embodiment, the CycleGAN includes a first generator, a second generator, and a first global discriminator, wherein the first generator is used to generate MR based on real CT, the second generator is used to generate reconstructed CT based on the generated MR, and the first global discriminator is used to distinguish MR based on the input generated MR and real MR.

[0016] In one embodiment, the input of the multi-scale MR organ region discriminator includes an MR organ region mask, a generated MR obtained by CycleGAN based on real CT, and a real MR. It is used to identify MR organ regions based on the MR organ region mask and the generated MR to obtain a first MR organ region identification result, and is also used to identify MR organ regions based on the MR organ region mask and the real MR to obtain a second MR organ region identification result.

[0017] In one embodiment, the adversarial loss function based on MR organ regions includes: an L2 loss constructed based on the identification results of the first MR organ region, and an L2 loss constructed based on the identification results of the first MR organ region and the identification results of the second MR organ region.

[0018] In one embodiment, the multi-scale MR organ region discriminator performs MR organ region identification based on different image scales, and averages the identification results at different image scales as the final identification result output by the multi-scale MR organ region discriminator. The final identification result includes a first MR organ region identification result and a second MR organ region identification result.

[0019] In one embodiment, the multi-scale MR organ region discriminator identifies MR organ regions based on two image scales, wherein the first image scale is the original image size, and the second image scale is half the size of the original image obtained by downsampling the original image.

[0020] In one embodiment, the input of the multi-scale MR lesion region discriminator includes an MR lesion region mask, a generated MR obtained by CycleGAN based on real CT, and a real MR. It is used to identify MR lesion regions based on the MR lesion region mask and the generated MR to obtain a first MR lesion region identification result, and is also used to identify MR lesion regions based on the MR lesion region mask and the real MR to obtain a second MR lesion region identification result.

[0021] In one embodiment, the adversarial loss function based on the MR lesion region includes: an L2 loss constructed based on the identification result of the first MR lesion region, and an L2 loss constructed based on the identification result of the first MR lesion region and the identification result of the second MR lesion region.

[0022] In one embodiment, the multi-scale MR lesion region discriminator identifies MR lesion regions based on different image scales, and averages the identification results at different image scales as the final identification result output by the multi-scale MR lesion region discriminator. The final identification result includes the identification result of the first MR lesion region and the identification result of the second MR lesion region.

[0023] In one embodiment, the multi-scale MR lesion region discriminator identifies MR lesion regions based on three image scales, wherein the first image scale is the original image size, the second image scale is half the size of the original image obtained by downsampling the original image, and the third image scale is one-quarter the size of the original image obtained by downsampling the original image.

[0024] In one embodiment, the method further includes deterministic guidance of the MR image to CT image generation process, comprising the following steps:

[0025] Construct CT organ region masks and CT lesion region masks from CT images and supplement them into the sample data;

[0026] In the constructed cross-modal generation framework, a multi-scale CT organ region discriminator for identifying CT organ regions and / or a multi-scale CT lesion region discriminator for identifying CT lesion regions are also added, forming a new cross-modal generation framework;

[0027] Based on the original total loss function of the cross-modal generation framework, an adversarial loss function based on CT organ regions and / or an adversarial loss function based on CT lesion regions is added to form a new total loss function;

[0028] The sample data is input into the new cross-modal generation framework, and the parameters of the new cross-modal generation framework are optimized using a new total loss function. After parameter optimization, the generator contained in CycleGAN is used for cross-modal generation of medical images.

[0029] In one embodiment, the CycleGAN further includes a second global discriminator, wherein the second generator is further configured to generate a CT based on the real MR, the first generator is further configured to generate a reconstructed MR based on the generated CT, and the second global discriminator is configured to distinguish the CT based on the input generated CT and the real CT.

[0030] In one embodiment, the input of the multi-scale CT organ region discriminator includes a CT organ region mask, a generated CT obtained by CycleGAN based on real MR, and a real CT. It is used to identify CT organ regions based on the CT organ region mask and the generated CT to obtain a first CT organ region identification result, and is also used to identify CT organ regions based on the CT organ region mask and the real CT to obtain a second CT organ region identification result.

[0031] The adversarial loss function based on CT organ regions includes: L2 loss constructed based on the first CT organ region identification result, and L2 loss constructed based on the first CT organ region identification result and the second CT organ region identification result.

[0032] In one embodiment, the input of the multi-scale CT lesion region discriminator includes a CT lesion region mask, a generated CT obtained by CycleGAN based on real MR, and a real CT. It is used to identify CT lesion regions based on the CT lesion region mask and the generated CT to obtain a first CT lesion region identification result. It is also used to identify CT lesion regions based on the CT lesion region mask and the real CT to obtain a second CT lesion region identification result.

[0033] The adversarial loss function based on CT lesion regions includes: L2 loss constructed based on the identification results of the first CT lesion region, and L2 loss constructed based on the identification results of the first CT lesion region and the identification results of the second CT lesion region.

[0034] In one embodiment, the multi-scale CT organ region discriminator performs CT organ region identification based on different image scales, and averages the identification results at different image scales as the final identification result output by the multi-scale CT organ region discriminator. The final identification result includes a first CT organ region identification result and a second CT organ region identification result.

[0035] The multi-scale CT lesion region discriminator identifies CT lesion regions based on different image scales, and averages the identification results of different image scales as the final identification result output by the multi-scale CT lesion region discriminator. The final identification result includes the identification result of the first CT lesion region and the identification result of the second CT lesion region.

[0036] In one embodiment, the lesion region mask is calculated as follows:

[0037] Find the four points on the far left, far right, far top, and far bottom of the original lesion area mask, and calculate the center point based on these four points. Then calculate the farthest distance from the center point to the edge of the original lesion area mask. Finally, take the center point as the center point and the farthest distance plus the adjustment parameter as the radius as the new lesion area mask. The lesion area mask includes the MR lesion area mask and the CT lesion area mask.

[0038] To achieve the above-mentioned objectives, the embodiments also provide an apparatus for a deterministically guided progressive medical image cross-modal generation method based on deep learning, including a data acquisition and processing unit, a cross-modal generation framework construction unit, a loss function construction unit, and a parameter optimization unit.

[0039] The data acquisition and processing unit is used to acquire real CT images and real MR images of the same location, and to preprocess the real CT images and real MR images to construct image pairs containing real CT and real MR, MR organ region masks, MR lesion region masks, and form sample data.

[0040] The cross-modal generative framework building unit is used to form a cross-modal generative framework based on CycleGAN by adding a multi-scale MR organ region discriminator for identifying MR organ regions and / or a multi-scale MR lesion region discriminator for identifying MR lesion regions.

[0041] The loss function construction unit is used to add an adversarial loss function based on MR organ regions and / or an adversarial loss function based on MR lesion regions to the original loss function of CycleGAN, forming the total loss function of the cross-modal generation framework;

[0042] The parameter optimization unit is used to input sample data into the cross-modal generation framework and optimize the parameters of the cross-modal generation framework using the total loss function. After parameter optimization, the generator contained in CycleGAN is used for cross-modal generation of medical images.

[0043] To achieve the above-mentioned objectives, an embodiment also provides a computing device, including a memory, a processor, and a computer program stored in the memory and executed on the processor, wherein the processor executes the computer program to implement the above-mentioned progressive medical image cross-modal generation method.

[0044] To achieve the above-mentioned objectives, the embodiments also provide a computer-readable storage medium having a computer program stored thereon, which, when processed and executed, implements the above-mentioned progressive medical image cross-modal generation method.

[0045] Compared with the prior art, the beneficial effects of the present invention include at least the following:

[0046] 1. Learn multimodal medical image generation without relying on paired data to improve the local generation quality of target regions such as organ regions and / or lesion regions.

[0047] 2. By progressively adding different numbers of discriminators to CycleGAN, multi-scale discriminators are used to guide the generator to focus on the target region without causing gradient vanishing or mode collapse, resulting in higher quality images that can better serve downstream tasks.

[0048] 3. Furthermore, the method proposed in this invention can be easily combined with any other medical image generation algorithm to improve performance and enhance the generation quality of the target region without changing the original algorithm's network structure. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a flowchart of a deterministically guided, progressive medical image cross-modal generation method based on deep learning, provided in the embodiments.

[0051] Figure 2 This is a schematic diagram of the generator provided in the embodiment;

[0052] Figure 3 This is a schematic diagram of the network structure of the residual module in the generator provided in the embodiment;

[0053] Figure 4 This is a schematic diagram of the CycleGAN process for generating MR images from CT images, provided in the embodiment.

[0054] Figure 5 This is a schematic diagram of the cross-modal generation framework formed by adding a multi-scale MR organ region discriminator to CycleGAN, as provided in the embodiment.

[0055] Figure 6 This is a schematic diagram of the cross-modal generation framework provided in the embodiment, which simultaneously adds a multi-scale MR organ region discriminator and a multi-scale MR lesion region discriminator to CycleGAN.

[0056] Figure 7 This is a flowchart of another deterministically guided, progressive medical image cross-modal generation method based on deep learning, provided in the embodiments;

[0057] Figure 8 This is a schematic diagram of the structure of the deterministically guided, progressive medical image cross-modal generation device based on deep learning provided in the embodiment. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0059] The example uses private hospital data, including real MR images, real CT images, and masks for lesion areas (MR lesion area masks) and organ areas (MR organ area masks) in the real MR images, as well as masks for lesion areas (CT lesion area masks) and organ areas (CT organ area masks) in the real CT images. These masks were annotated by doctors. Both the real CT and MR images include: plain phase (NC), arterial phase (ART), portal venous phase (PV), and delayed phase (DL). The data format for MR and CT images is nii, and the data format for the masks is nrrd.

[0060] Based on the above data sources, this embodiment provides a deterministically guided, progressive cross-modal generation method for medical images based on deep learning, including deterministic guidance of the generation process from CT images to MR images, such as... Figure 1 As shown, it includes the following steps:

[0061] S1. Acquire real CT images and real MR images of the same location, and preprocess the real CT images and real MR images to construct image pairs containing real CT and real MR, MR organ region masks, MR lesion region masks, and form sample data.

[0062] In this embodiment, since the obtained data is three-dimensional continuous data, preprocessing of the CT and MR images is required when constructing the sample data. This preprocessing includes: adjusting window width and level, rigid registration, normalization, and data selection. Adjusting window width and level refers to adjusting the window width and level of the CT and MR images to adjust their original dimensions. Preferably, for CT images, based on the doctor's prior knowledge, the window width is set to (-110, 190), using the `np.clip` method from the NumPy library. For MR images, the `estimate_sigma` and `nlmeans` methods from the Dipy library are used to obtain denoised images.

[0063] Rigid registration involves registering CT and MR images. Specifically, the CT image of the same location is used as the unregistered image, and the MR image is used as the target image. Based on the lesion region masks in both the MR and CT images, the transformation relationship between the lesion regions in the two images is calculated. This transformation relationship is then applied to the entire CT image to obtain the registered CT image. Affine registration using Dipy can be employed in this method.

[0064] Normalization refers to the normalization process applied to the registered CT and MR images. Specifically, for CT images, linear normalization is used to normalize pixel values ​​to [-1, 1]. For MR images, z-score is first applied, followed by linear normalization to normalize pixel values ​​to [-1, 1]. The normalized CT and MR images then form an image pair.

[0065] Data selection refers to choosing images with clearly defined lesion areas from a large number of images. This can be done by using the slice with the largest lesion area mask as a reference, and then selecting the lesion area mask closest to the reference to construct the sample data. Specifically, four slices above and four slices below the reference, for a total of nine slices, can be selected as lesion area masks for constructing the sample data. It should be noted that the lesion area masks here include both MR and CT lesion area masks.

[0066] In this embodiment, the lesion region mask is redefined from the original lesion region mask annotated by the doctor. The specific calculation process includes finding the leftmost, rightmost, topmost, and bottommost points in the original lesion region mask, which are A(X) points respectively. left ,Y A ), B(X right ,Y B ), E(X) E ,Y top ), F(X F ,Y bottom And calculate the center point based on these four points. Then, the farthest distance D from the center point C to the edge of the original lesion area mask is calculated. Finally, a circular area with the center point as the center point and the farthest distance plus the adjustment parameter γ as the radius is used as the new lesion area mask. In this embodiment, the value of the adjustment parameter γ is 1-3 pixels, preferably 2 pixels.

[0067] The formula for calculating the farthest distance D is:

[0068] D = max(distance) (C,A) distance (C,B) distance (C,E) distance (C,F) ).

[0069] S2, based on CycleGAN, forms a cross-modal generation framework by adding a multi-scale MR organ region discriminator for identifying MR organ regions and / or a multi-scale MR lesion region discriminator for identifying MR lesion regions.

[0070] CycleGAN includes a first generator and a second generator, a first global discriminator and a second global discriminator. The first generator and the second generator have the same structure and are used to generate modal CT images and MR images from each other. The two discriminators have the same structure and are used to distinguish between real and fake whole CT images and whole MR images, respectively.

[0071] In the embodiment, the first generator and the second generator are built based on Unet, such as Figure 2 As shown, the sub-modules in the network structure are: ResConv, Down, Up, and OutConv. The residual network ResConv is as follows: Figure 3 As shown, the convolutional layers use a kernel size of 3 and padding of 1. The Down submodule contains a max-pooling network and a residual network ResConv. The Up submodule contains an upsampling layer and ResConv. The output of the upsampling layer is twice the input, using a bilinear upsampling algorithm, aligning the corner pixels of the input with the output tensor. The Outconv submodule contains a convolutional layer with a kernel size of 1. The last layer of both generators contains three heads, outputting the generated image, the α (scale map), and the β (shape map) required by the zero-mean generalized Gaussian loss function.

[0072] The first and second global discriminators employ single-scale discriminators, such as Markov discriminators. The convolutional layers have a kernel size of 4, padding of 1, a stride of 2, an activation function of LeakyReLU, and a negative slope of 0.2.

[0073] When CycleGAN is used for CT image to MR image generation, such as Figure 4 As shown, the first generator (G: A→B) is used to generate data based on the real CT (x... A ) to generate MR The second generator (G:B→A) is used to obtain the reconstructed CT from the generated MR. First overall discriminant (D) whole ) is used to generate MR and true MR(x) based on the input. B ) to perform MR identification.

[0074] like Figure 5 As shown, in Figure 4 Based on the CycleGAN shown, a multi-scale MR organ region discriminator is added to identify MR organ regions. A cross-modal generation framework is formed, which focuses on the generation of organ regions through training.

[0075] in, The input includes MR organ region masks. CycleGAN generates MR based on real CT scans. Real MR(x) B ), used as a basis and The identification of MR organ regions yielded the results of the first MR organ region identification. Also used as a basis and x B The identification of the MR organ region yielded the results of the second MR organ region identification.

[0076] In the embodiments, Structure and D whole The same, the difference is Capable of multi-scale discrimination, i.e. MR organ region identification is performed based on different image scales, and the identification results at different image scales are averaged to obtain the final identification result output by the multi-scale MR organ region discriminator. The final identification result includes... In one implementation method MR organ region identification is performed based on two image scales, where the first image scale is the original image size, and the second image scale is half the size of the original image obtained by downsampling the original image.

[0077] like Figure 6 As shown, in Figure 5 Based on the CycleGAN shown, a multi-scale MR lesion region discriminator is added to further identify MR lesion regions. A cross-modal generation framework is formed, which, through training, focuses on the generation of MR lesion regions while also focusing on MR organ regions.

[0078] in, The input includes an MR lesion region mask. CycleGAN generates MR based on real CT scans. Real MR(x) B ), used as a basis and The identification of the MR lesion region yielded the identification result of the first MR lesion region. Also used as a basis and x B The identification of the MR lesion region yielded the results of the identification of the second MR lesion region.

[0079] In the embodiments, Structure and D whole The same, the difference is Capable of multi-scale discrimination, i.e. MR lesion regions are identified based on different image scales, and the average of the identification results at different image scales is used as the final identification result output by the multi-scale MR lesion region discriminator. The final identification result includes... In one implementation method The identification of MR lesion regions is based on three image scales: the first image scale is the original image size, the second image scale is half the size of the original image obtained by downsampling the original image, and the third image scale is one-quarter the size of the original image obtained by downsampling the original image.

[0080] Of course, it can also be done in such cases Figure 4 Based on the CycleGAN shown, by adding A cross-modal generative framework is formed, which is trained to focus on the generation of MR lesion regions, specifically targeting... The structure and uses are detailed in the above description and will not be repeated here.

[0081] S3, based on the original loss function of CycleGAN, adds an adversarial loss function based on MR organ regions and / or an adversarial loss function based on MR lesion regions to form the total loss function of the cross-modal generation framework.

[0082] In this embodiment, during the process of generating MR images from CT images, the original loss function of CycleGAN includes a loss function based on a zero-mean generalized Gaussian distribution, a cycle consistency loss, and an adversarial loss function. The loss function based on the zero-mean generalized Gaussian distribution... Represented as:

[0083]

[0084] The cycle consistency loss is expressed as:

[0085]

[0086] The adversarial loss function between the generator and the discriminator is expressed as:

[0087]

[0088]

[0089] Where K represents the sum of all pixels in the data, and Γ represents the gamma function.

[0090] Original loss function

[0091] The added adversarial loss function based on MR organ regions includes: L2 loss constructed based on the first MR organ region identification result. L2 loss constructed based on the first MR organ region identification results and the second MR organ region identification results They are represented as follows:

[0092]

[0093]

[0094] Adversarial loss function based on MR organ region

[0095] The added adversarial loss function based on MR lesion region includes: L2 loss constructed based on the identification result of the first MR lesion region. L2 loss constructed based on the identification results of the first MR lesion region and the identification results of the second MR lesion region. They are represented as follows:

[0096]

[0097]

[0098] Adversarial loss function based on MR lesion region

[0099] The total loss function is L = L1 + L2 + L3.

[0100] S4. Input the sample data into the cross-modal generation framework and optimize the parameters of the cross-modal generation framework using the total loss function. After parameter optimization, the generator contained in CycleGAN is used for cross-modal generation of medical images.

[0101] Specifically, during the training process, sample data is input into the cross-modal generation framework. Each batch (batch size) has 1 element. The adaptive moment estimation (Adam) optimizer is used, and the parameters are optimized using the total loss function L constructed by S3. After parameter optimization, the first and second generators included in CycleGAN are used for cross-modal generation of medical images.

[0102] Of course, in specific training, progressive training can be adopted. On the basis of training CycleGAN with L1, the multi-scale MR organ region discriminator can be trained using the adversarial loss function L2 based on MR organ regions. Training is then performed, followed by the use of the L3 adversarial loss function based on MR lesion regions to train the multi-scale MR lesion region discriminator. Conduct training.

[0103] In practical applications, mismatched real CT images and real MR images often occur. To address this issue, based on S1-4 above, a deterministic guided process for generating CT images from MR images is introduced, such as... Figure 7 As shown, the specific steps include:

[0104] S1' constructs CT organ region masks and CT lesion region masks from CT images and adds them to the sample data.

[0105] The CT organ region mask is obtained by the doctor's annotation, and the CT lesion region mask is calculated using the above method.

[0106] S2', in the constructed cross-modal generation framework, also adds a multi-scale CT organ region discriminator for identifying CT organ regions and / or a multi-scale CT lesion region discriminator for identifying CT lesion regions, forming a new cross-modal generation framework.

[0107] When a deterministic guided process is added to the generation of MR images into CT images, in CycleGAN, the second generator is also used to generate images based on the true MR (x) B ) to generate CT The first generator is also used to obtain reconstructed MR based on the generated CT. The second overall discriminator is used to determine the generated CT and the real CT based on the input (x). A ) to perform CT differentiation.

[0108] In such Figure 5 In the cross-modal generation framework shown, a multi-scale CT organ region discriminator is added for identifying CT organ regions. A novel cross-modal generation framework is developed, which focuses on the generation of MR organ regions and CT organ regions through training.

[0109] in, The input includes CT organ region masks. CycleGAN generates CT based on real MR images. Real CT (x) A ), used as a basis and The identification of organ regions on CT scans yielded the results of the first CT organ region identification. Also used as a basis and x AThe identification of organ regions by MR was used to obtain the identification results of organ regions by CT.

[0110] In the embodiments, Structure and D whole The same, the difference is It can perform multi-scale discrimination, that is, identify MR organ regions based on different image scales, and average the discrimination results at different image scales as the final output of the multi-scale MR organ region discriminator. and In one implementation, Two image scales can be used for CT organ region identification, where the first image scale is the original image size, and the second image scale is half the size of the original image obtained by downsampling the original image.

[0111] In such Figure 6 In the cross-modal generation framework shown, by simultaneously adding and a multi-scale CT lesion region discriminator for identifying CT lesion regions A new cross-modal generation framework is formed, which focuses on the generation of MR organ regions, CT organ regions, and more importantly, MR lesion regions and CT lesion regions through training.

[0112] in, The input includes a CT lesion area mask. CycleGAN generates CT based on real MR images. Real CT (x) A ), used as a basis and The identification of lesion regions on CT scans yielded the results of the first CT lesion region identification. Also used as a basis and x A The identification of lesion areas on CT scans yielded the results of the identification of the second lesion area on CT scans.

[0113] In the embodiments, Structure and D whole The same, the difference is It can perform multi-scale discrimination, that is, identify CT lesion regions based on different image scales, and average the discrimination results at different image scales as the output of the multi-scale CT lesion region discriminator. and In one implementation, Three image scales can be used to identify CT lesion areas. The first image scale is the original image size, the second image scale is half the size of the original image obtained by downsampling the original image, and the third image scale is one-quarter the size of the original image obtained by downsampling the original image.

[0114] Of course, you can also add only to CycleGAN In the resulting cross-modal generation framework, additional methods for identifying CT lesion regions are added. A novel cross-modal generative framework is developed, which, through training, focuses on the generation of lesion regions in both MR and CT modes, specifically targeting... The structure and uses are detailed in the above description and will not be repeated here.

[0115] S3' adds an adversarial loss function based on CT organ regions and / or an adversarial loss function based on CT lesion regions to the original total loss function of the cross-modal generation framework, forming a new total loss function.

[0116] In this embodiment, during the process of generating CT images from MR images, the original loss function of CycleGAN also includes a loss function with a zero-mean generalized Gaussian distribution. Cyclic consistency loss and adversarial loss function and

[0117]

[0118]

[0119]

[0120]

[0121] The original loss function also includes

[0122] In this embodiment, the added adversarial loss function based on CT organ regions includes: L2 loss constructed based on the first CT organ region identification result. L2 loss constructed based on the organ region identification results of the first CT scan and the organ region identification results of the second CT scan. They are represented as follows:

[0123]

[0124]

[0125] Adversarial loss function based on CT organ regions

[0126] In this embodiment, the added adversarial loss function based on the CT lesion region includes: L2 loss constructed based on the identification results of the first CT lesion region. L2 loss constructed based on the identification results of the first CT lesion region and the identification results of the second CT lesion region. They are represented as follows:

[0127]

[0128]

[0129] Adversarial loss function based on CT lesion region

[0130] The new total loss function is L' = L + L'1 + L'2 + L3'.

[0131] S4' inputs the sample data into the new cross-modal generation framework and optimizes the parameters of the new cross-modal generation framework using the new total loss function. After parameter optimization, the generator contained in CycleGAN is used for cross-modal generation of medical images.

[0132] Specifically, during the training process, sample data is input into the new cross-modal generation framework, namely x. A , x B , The input is fed into a new cross-modal generation framework with a batch size of 1. The adaptive moment estimation (Adam) optimizer is used and the parameters are optimized using a new total loss function L' constructed by S3'. After parameter optimization, the first and second generators contained in CycleGAN are used for cross-modal generation of medical images.

[0133] Of course, in specific training, progressive training can be adopted. On the basis of training CycleGAN with L1, L2 and L2′ can be used to train the multi-scale MR organ region discriminator. and multi-scale CT organ region discriminator Training, and then using L3 and L3′ to discriminate multiscale MR lesion regions. and Conduct training.

[0134] The embodiment also provides performance evaluation of the first and second generators with optimized parameters, specifically including the following four evaluation metrics, two of which are for the entire image and the other two for the lesion region: peak signal-to-noise ratio (TPSNR) based on the lesion region, structural similarity based on the lesion region (TSSIM), learning-based perceptual similarity (LPIPS), and Frechet Inception distance (FID).

[0135] Using sample data from 60 patients at different stages (ART, PV, NC), and employing the four assessment indicators mentioned above, the results are as follows:

[0136]

[0137]

[0138] Based on the same inventive concept, the embodiments also provide a deterministically guided, progressive medical image cross-modal generation device based on deep learning, such as... Figure 7 As shown, it includes a data acquisition and processing unit 710, a cross-modal generation framework construction unit 720, a loss function construction unit 730, and a parameter optimization unit 740.

[0139] The data acquisition and processing unit 710 is used to acquire real CT images and real MR images taken of the same part, and to preprocess the real CT images and real MR images to construct image pairs containing real CT and real MR, MR organ region masks, MR lesion region masks, and form sample data.

[0140] The cross-modal generative framework building unit 720 is used to form a cross-modal generative framework based on CycleGAN by adding a multi-scale MR organ region discriminator for identifying MR organ regions and / or a multi-scale MR lesion region discriminator for identifying MR lesion regions.

[0141] The loss function construction unit 730 is used to add an adversarial loss function based on MR organ regions and / or an adversarial loss function based on MR lesion regions to the original loss function of CycleGAN, forming the total loss function of the cross-modal generation framework;

[0142] The parameter optimization unit 740 is used to input sample data into the cross-modal generation framework and optimize the parameters of the cross-modal generation framework using the total loss function. After parameter optimization, the generator contained in CycleGAN is used for cross-modal generation of medical images.

[0143] In this embodiment, the data acquisition and processing unit 710 is also used to construct CT organ region masks and CT lesion region masks from CT images and supplement them to the sample data;

[0144] The cross-modal generation framework building unit 720 is also used to add a multi-scale CT organ region discriminator for identifying CT organ regions and / or a multi-scale CT lesion region discriminator for identifying CT lesion regions to the constructed cross-modal generation framework, forming a new cross-modal generation framework;

[0145] The loss function construction unit 730 is also used to add an adversarial loss function based on CT organ regions and / or an adversarial loss function based on CT lesion regions to the total loss function of the cross-modal generation framework, forming a new total loss function;

[0146] The parameter optimization unit 740 is also used to input sample data into the new cross-modal generation framework and optimize the parameters of the new cross-modal generation framework using the new total loss function. After parameter optimization, the generator contained in CycleGAN is used for cross-modal generation of medical images.

[0147] It should be noted that the deterministically guided progressive medical image cross-modal generation device provided in the above embodiments should be illustrated using the above-described division of functional units when performing progressive medical image cross-modal generation. The functions described above can be assigned to different functional units as needed, that is, the internal structure of the terminal or server can be divided into different functional units to complete all or part of the functions described above. Furthermore, the deterministically guided progressive medical image cross-modal generation method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the deterministically guided progressive medical image cross-modal generation method embodiments, which will not be repeated here.

[0148] Based on the same inventive concept, the embodiment also provides a computing device, including a memory, a processor, and a computer program stored in the memory and executed on the processor, wherein the processor executes the computer program to implement the above-described progressive medical image cross-modal generation method.

[0149] In practical applications, the memory can be volatile memory located at the local end, such as RAM; it can also be non-volatile memory, such as ROM, FLASH, floppy disks, hard disks, etc.; or it can be a remote storage cloud. The processor can be a central processing unit (CPU), microprocessor (MPU), digital signal processor (DSP), or field-programmable gate array (FPGA), meaning that these processors can be used to implement the progressive medical image cross-modal generation method steps.

[0150] Based on the same inventive concept, the embodiment also provides a computer-readable storage medium having a computer program stored thereon, which, when processed and executed, implements the above-described progressive medical image cross-modal generation method.

[0151] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A deterministically guided, progressive cross-modal generation method for medical images based on deep learning, characterized in that, This includes deterministic guidance for the process of generating MR images from CT images, comprising the following steps: Acquire real CT images and real MR images of the same area, and preprocess the real CT images and real MR images to construct image pairs containing real CT and real MR, MR organ region masks, MR lesion region masks, and form sample data; Based on CycleGAN, a cross-modal generation framework is formed by adding a multi-scale MR organ region discriminator for identifying MR organ regions and / or a multi-scale MR lesion region discriminator for identifying MR lesion regions. The inputs of the multi-scale MR organ region discriminator include an MR organ region mask, generated MR data obtained by CycleGAN based on real CT, and real MR data. This discriminator is used to identify MR organ regions based on the MR organ region mask and generated MR data to obtain a first MR organ region identification result, and also to identify MR organ regions based on the MR organ region mask and real MR data to obtain a second MR organ region identification result. Similarly, the inputs of the multi-scale MR lesion region discriminator include an MR lesion region mask, generated MR data obtained by CycleGAN based on real CT, and real MR data. This discriminator is used to identify MR lesion regions based on the MR lesion region mask and generated MR data to obtain a first MR lesion region identification result, and also to identify MR lesion regions based on the MR lesion region mask and real MR data to obtain a second MR lesion region identification result. Based on the original loss function of CycleGAN, adversarial loss functions based on MR organ regions and / or adversarial loss functions based on MR lesion regions are added to form the total loss function of the cross-modal generation framework. Among them, the adversarial loss function based on MR organ regions includes: L2 loss constructed based on the identification results of the first MR organ region. L2 loss constructed based on the first MR organ region identification results and the second MR organ region identification results The adversarial loss function based on MR lesion region includes: L2 loss constructed based on the identification result of the first MR lesion region. L2 loss constructed based on the identification results of the first MR lesion region and the identification results of the second MR lesion region. The sample data is input into the cross-modal generation framework, and the parameters of the cross-modal generation framework are optimized using the total loss function. After parameter optimization, the generator contained in CycleGAN is used for cross-modal generation of medical images.

2. The method for deterministically guided progressive cross-modal generation of medical images based on deep learning according to claim 1, characterized in that, Preprocessing of CT and MR images includes rigid registration and normalization of real CT and MR images.

3. The method for deterministically guided progressive cross-modal generation of medical images based on deep learning according to claim 1, characterized in that, The CycleGAN includes a first generator, a second generator, and a first global discriminator. The first generator is used to generate MR based on real CT, the second generator is used to generate reconstructed CT based on the generated MR, and the first global discriminator is used to distinguish MR based on the input generated MR and real MR.

4. The method for deterministically guided progressive cross-modal generation of medical images based on deep learning according to claim 1, characterized in that, The multi-scale MR organ region discriminator identifies MR organ regions based on different image scales, and averages the identification results at different image scales as the final identification result output by the multi-scale MR organ region discriminator. The final identification result includes the first MR organ region identification result and the second MR organ region identification result.

5. The method for deterministically guided progressive cross-modal generation of medical images based on deep learning according to claim 4, characterized in that, The multi-scale MR organ region discriminator identifies MR organ regions based on two image scales, where the first image scale is the original image size and the second image scale is half the size of the original image obtained by downsampling the original image.

6. The method for deterministically guided progressive cross-modal generation of medical images based on deep learning according to claim 1, characterized in that, The multi-scale MR lesion region discriminator identifies MR lesion regions based on different image scales, and averages the identification results at different image scales as the final identification result output by the multi-scale MR lesion region discriminator. The final identification result includes the identification result of the first MR lesion region and the identification result of the second MR lesion region.

7. The method for deterministically guided progressive cross-modal generation of medical images based on deep learning according to claim 1, characterized in that, The multi-scale MR lesion region discriminator identifies MR lesion regions based on three image scales, where the first image scale is the original image size, the second image scale is half the size of the original image obtained by downsampling the original image, and the third image scale is one-quarter the size of the original image obtained by downsampling the original image.

8. The method for deterministically guided progressive cross-modal generation of medical images based on deep learning according to claim 1, characterized in that, It also includes deterministic guidance for the process of generating CT images from MR images, comprising the following steps: Construct CT organ region masks and CT lesion region masks from CT images and supplement them into the sample data; In the constructed cross-modal generation framework, a multi-scale CT organ region discriminator for identifying CT organ regions and / or a multi-scale CT lesion region discriminator for identifying CT lesion regions are also added, forming a new cross-modal generation framework; Based on the original total loss function of the cross-modal generation framework, an adversarial loss function based on CT organ regions and / or an adversarial loss function based on CT lesion regions is added to form a new total loss function; The sample data is input into the new cross-modal generation framework, and the parameters of the new cross-modal generation framework are optimized using a new total loss function. After parameter optimization, the generator contained in CycleGAN is used for cross-modal generation of medical images.

9. The method for deterministically guided progressive cross-modal generation of medical images based on deep learning according to claim 8, characterized in that, The CycleGAN also includes a second global discriminator, wherein the second generator is further used to generate a CT based on the real MR, the first generator is further used to generate a reconstructed MR based on the generated CT, and the second global discriminator is used to distinguish the CT based on the input generated CT and the real CT.

10. The method for deterministically guided progressive cross-modal generation of medical images based on deep learning according to claim 8, characterized in that, The inputs of the multi-scale CT organ region discriminator include a CT organ region mask, a generated CT obtained by CycleGAN based on real MR, and a real CT. It is used to identify CT organ regions based on the CT organ region mask and the generated CT to obtain a first CT organ region identification result. It is also used to identify CT organ regions based on the CT organ region mask and the real CT to obtain a second CT organ region identification result. The adversarial loss function based on CT organ regions includes: L2 loss constructed based on the first CT organ region identification result. L2 loss constructed based on the organ region identification results of the first CT scan and the organ region identification results of the second CT scan.

11. The method for deterministically guided progressive cross-modal generation of medical images based on deep learning according to claim 8, characterized in that, The inputs of the multi-scale CT lesion region discriminator include a CT lesion region mask, a generated CT obtained by CycleGAN based on real MR, and a real CT. It is used to identify CT lesion regions based on the CT lesion region mask and the generated CT to obtain a first CT lesion region identification result. It is also used to identify CT lesion regions based on the CT lesion region mask and the real CT to obtain a second CT lesion region identification result. The adversarial loss function based on CT lesion region includes: L2 loss constructed based on the identification result of the first CT lesion region. L2 loss constructed based on the identification results of the first CT lesion region and the identification results of the second CT lesion region.

12. The method for deterministically guided progressive cross-modal generation of medical images based on deep learning according to claim 8, characterized in that, The multi-scale CT organ region discriminator identifies CT organ regions based on different image scales, and averages the identification results at different image scales as the final identification result output by the multi-scale CT organ region discriminator. The final identification result includes the first CT organ region identification result and the second CT organ region identification result. The multi-scale CT lesion region discriminator identifies CT lesion regions based on different image scales, and averages the identification results of different image scales as the final identification result output by the multi-scale CT lesion region discriminator. The final identification result includes the identification result of the first CT lesion region and the identification result of the second CT lesion region.

13. The method for deterministically guided progressive cross-modal generation of medical images based on deep learning according to claim 1 or 8, characterized in that, The method for calculating the lesion area mask is as follows: Find the four points on the far left, far right, far top, and far bottom of the original lesion area mask, and calculate the center point based on these four points. Then calculate the farthest distance from the center point to the edge of the original lesion area mask. Finally, take the center point as the center point and the farthest distance plus the adjustment parameter as the radius as the new lesion area mask. The lesion area mask includes the MR lesion area mask and the CT lesion area mask.

14. A deterministically guided progressive medical image cross-modal generation device based on deep learning, characterized in that, It includes a data acquisition and processing unit, a cross-modal generation framework construction unit, a loss function construction unit, and a parameter optimization unit. The data acquisition and processing unit is used to acquire real CT images and real MR images of the same location, and to preprocess the real CT images and real MR images to construct image pairs containing real CT and real MR, MR organ region masks, MR lesion region masks, and form sample data. The cross-modal generation framework construction unit is used as a reference to form a cross-modal generation framework by adding a multi-scale MR organ region discriminator for identifying MR organ regions and / or a multi-scale MR lesion region discriminator for identifying MR lesion regions, based on CycleGAN. The inputs of the multi-scale MR organ region discriminator include an MR organ region mask, generated MR data obtained by CycleGAN based on real CT, and real MR data. It is used to identify MR organ regions based on the MR organ region mask and generated MR data to obtain a first MR organ region identification result, and also to identify MR organ regions based on the MR organ region mask and real MR data to obtain a second MR organ region identification result. Similarly, the inputs of the multi-scale MR lesion region discriminator include an MR lesion region mask, generated MR data obtained by CycleGAN based on real CT, and real MR data. It is used to identify MR lesion regions based on the MR lesion region mask and generated MR data to obtain a first MR lesion region identification result, and also to identify MR lesion regions based on the MR lesion region mask and real MR data to obtain a second MR lesion region identification result. The loss function construction unit is used to add adversarial loss functions based on MR organ regions and / or adversarial loss functions based on MR lesion regions to the original loss function of CycleGAN, forming the total loss function of the cross-modal generation framework. The adversarial loss function based on MR organ regions includes: L2 loss constructed based on the identification results of the first MR organ region. L2 loss constructed based on the first MR organ region identification results and the second MR organ region identification results The adversarial loss function based on MR lesion region includes: L2 loss constructed based on the identification result of the first MR lesion region. L2 loss constructed based on the identification results of the first MR lesion region and the identification results of the second MR lesion region. The parameter optimization unit is used to input sample data into the cross-modal generation framework and optimize the parameters of the cross-modal generation framework using the total loss function. After parameter optimization, the generator contained in CycleGAN is used for cross-modal generation of medical images.

15. The deterministically guided progressive medical image cross-modal generation device based on deep learning as described in claim 14, characterized in that, The data acquisition and processing unit is also used to construct CT organ region masks and CT lesion region masks from CT images and supplement them to the sample data; The cross-modal generation framework construction unit is also used to add a multi-scale CT organ region discriminator for identifying CT organ regions and / or a multi-scale CT lesion region discriminator for identifying CT lesion regions to the constructed cross-modal generation framework, forming a new cross-modal generation framework; The loss function construction unit is also used to add adversarial loss functions based on CT organ regions and / or adversarial loss functions based on CT lesion regions to the total loss function of the cross-modal generation framework, forming a new total loss function; The parameter optimization unit is also used to input sample data into the new cross-modal generation framework and optimize the parameters of the new cross-modal generation framework using the new total loss function. After parameter optimization, the generator contained in CycleGAN is used for cross-modal generation of medical images.

16. A computing device comprising a memory, a processor, and a computer program stored in the memory and executed on the processor, characterized in that, When the processor executes the computer program, it implements the progressive medical image cross-modal generation method according to any one of claims 1-13.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is processed and executed, it implements the progressive medical image cross-modal generation method according to any one of claims 1-13.

Images