Mutual synthesis method, system and related device of CT image and MRI image
Through the mutual synthesis method of CT graph and MRI graph, network generator and discriminator, combined with conditional semantic generator and symmetric cross attention processing, the shortcomings of convolution operation and Transformer model in medical image synthesis are solved, and high quality is achieved The image synthesis is adapted to different material integrity, has strong adaptability and high efficiency.
Patent Information
- Application Number
- CN202411281160.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-13
- Publication Date
- 2024-12-27
- Estimated Expiration
- 2044-09-13
AI Technical Summary
In the prior art, convolution operation limits the expression ability of long-distance spatially dependent features in medical image synthesis, resulting in poor synthesis quality. The Transformer model is too dependent on high-quality source data and has poor adaptability, especially in the absence of CT maps or Effective synthesis cannot be achieved during MRI graphs.
A mutual synthesis method of CT graph and MRI graph is proposed. Through a network generator and discriminator, a conditional semantic generator and symmetric cross attention processing, simulated images are generated and optimized, and peak signal-to-noise ratio and feature similarity control is carried out. Synthetic quality, adapt to the integrity of different materials.
With or without CT or MRI, high-quality image synthesis is achieved, with strong adaptability, high image synthesis quality and high efficiency, and can improve the quality of the synthetic image through noise processing and feature mapping when data is missing.
Smart Images
Figure QLYQS_1 
Figure QLYQS_6
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image synthesis, and in particular to a mutual synthesis method, system and related device of CT images and MRI images. Background Art
[0002] Computed tomography (CT) and magnetic resonance imaging (MRI) play a very important role in the field of medical image analysis. Among them, synthesizing MRI images based on CT images can enhance the visualization of soft tissues and better observe brain tissues, spinal cord, etc. At the same time, converting MRI images to CT images helps to evaluate bone structures, avoid multiple CT scans, and reduce patients' radiation exposure.
[0003] Currently, convolution operations perform excellently in medical image synthesis tasks, but convolution operations limit the expressive power of contextual features that reflect long-distance spatial dependencies, resulting in poor quality of medical image synthesis and the inability to display the lesion area well in the synthesized image. Using the Transformer model for medical imaging synthesis can effectively capture the long-distance dependencies between different regions in the image and solve the shortcomings of the convolution operation. However, the Transformer model is overly dependent on a large amount of high-quality source data (target data). When CT images or MRI images are missing, medical imaging synthesis cannot be achieved based on the Transformer model, and its adaptability is poor. Summary of the invention
[0004] The purpose of the present invention is to provide a method, system and related device for synthesizing CT images and MRI images.
[0005] The technical solution of the present invention is as follows:
[0006] A method for synthesizing a CT image and an MRI image, comprising the following operations:
[0007] S1. If there is a CT image and an MRI image, the purpose is to generate an MRI composite image or a CT composite image, and S2 is executed; if there is a CT image but no MRI image, the purpose is to generate an MRI composite image, and S3 is executed; if there is no CT image but an MRI image, the purpose is to generate a CT composite image, and S4 is executed;
[0008] S2. If the purpose is to generate an MRI composite image, the CT image is processed by a generator whose network parameters are standard MRI image parameters to obtain a simulated MRI image; the simulated MRI image and the MRI image are processed by a discriminator whose network parameters are standard MRI image parameters, and when the peak signal-to-noise ratio of the simulated MRI image, the mean square error and the structural similarity between the simulated MRI image and the MRI image are respectively less than a first signal-to-noise ratio threshold, the mean square error and the first similarity threshold, the simulated MRI image at this time is output as an MRI composite image; if the purpose is to generate a CT composite image, the MRI image is processed by a generator and a discriminator whose network parameters are standard CT image parameters to obtain a CT composite image;
[0009] S3, the CT image is processed by a generator whose network parameters are standard MRI image parameters to obtain a simulated MRI image; noise is added to the CT image several times to obtain a CT noise image; the CT noise image and the simulated MRI image are processed by a conditional semantic generator respectively to obtain a CT noise semantic image and a simulated MRI semantic image; based on the simulated MRI semantic image, the CT noise semantic image is subjected to several denoising and feature mapping processes to obtain an optimized simulated MRI image; the optimized simulated MRI image and the simulated MRI image are processed by a discriminator, and when the peak signal-to-noise ratio of the optimized simulated MRI image and the feature similarity between the optimized simulated MRI image and the simulated MRI image are respectively less than a second signal-to-noise ratio threshold and a second similarity threshold, the optimized simulated MRI image at this time is output as an MRI synthetic image;
[0010] S4. The MRI image is processed by a generator, a conditional semantic generator and a discriminator whose network parameters are the standard CT image parameters to obtain a CT synthetic image.
[0011] The operations processed by the conditional semantic generator in S3 are specifically as follows: the CT noise image or the simulated MRI image is processed by convolution and downsampling to obtain the first downsampling image; the first downsampling image is processed by convolution and downsampling to obtain the second downsampling image; and so on to obtain the Nth downsampling image; N is a positive integer; the Nth downsampling image is processed by symmetric cross attention to obtain the Nth cross attention image; the Nth cross attention image is processed by convolution and upsampling, and then spliced with the Nth downsampling image to obtain the Nth fused feature image; the Nth fused feature image is processed by convolution and upsampling, and then spliced with the N-1th downsampling image to obtain the N-1th fused feature image; and so on to obtain the CT noise semantic image or the simulated MRI semantic image feature image.
[0012] The operation of symmetric cross attention processing is specifically as follows: obtaining the channel mapping map and the spatial mapping map of the Nth down-sampling image; the channel mapping map and the spatial mapping map are processed by pre-attention to obtain the pre-attention map; the pre-attention map and the spatial mapping map are processed by post-attention to obtain the Nth cross attention map; the operation of pre-attention processing is as follows: the channel mapping map and the spatial mapping map are processed by Fourier transform respectively to obtain the channel Fourier map and the spatial Fourier map; the channel Fourier map and the spatial Fourier map are respectively multiplied element by element with the Fourier space key item weight map and the Fourier space query item weight map, and then fused to obtain the Fourier fusion map; the Fourier fusion map is processed by filtering and inverse Fourier transform to obtain a fused feature map; the fused feature map is processed by weighting and multi-layer perceptron to obtain the pre-attention map; in the post-attention processing, the pre-attention Fourier map and the spatial Fourier map are respectively multiplied element by element with the Fourier space query item weight map and the Fourier space key item weight map.
[0013] The filtering operation can be achieved through several convolution processes and nonlinear processes.
[0014] The method for obtaining feature similarity in S3 is: after pooling the optimized simulated MRI image and the simulated MRI image respectively, the average value of the pixel difference at the corresponding position is obtained to obtain the pooled mean difference; based on the pooled mean difference, the feature covariance matrix of the optimized simulated MRI image and the feature covariance matrix of the simulated MRI image, the feature similarity is obtained.
[0015] In S3, the step size is greater than 1 in the process of adding noise to the CT image several times.
[0016] In S1, the CT image or the MRI image is normalized and converted into a two-dimensional slice image to obtain a CT two-dimensional slice image or an MRI two-dimensional slice image, which is used to perform the operations in S2, or / and S3, or / and S4.
[0017] A system for synthesizing CT images and MRI images, used to implement the above-mentioned method for synthesizing CT images and MRI images, comprising:
[0018] The data integrity judgment module is used to execute the first composite image generation module if there is a CT image and an MRI image, and the purpose is to generate an MRI composite image or a CT composite image; if there is a CT image but no MRI image, and the purpose is to generate an MRI composite image, execute the second composite image generation module; if there is no CT image but an MRI image, and the purpose is to generate a CT composite image, execute the third composite image generation module;
[0019] The first synthetic image generation module is used for, if the purpose is to generate an MRI synthetic image, the CT image is processed by a generator whose network parameters are standard MRI image parameters to obtain a simulated MRI image; the simulated MRI image and the MRI image are processed by a discriminator whose network parameters are standard MRI image parameters, and when the peak signal-to-noise ratio of the simulated MRI image, the mean square error and the structural similarity between the simulated MRI image and the MRI image are respectively less than a first signal-to-noise ratio threshold, the mean square error and the first similarity threshold, the simulated MRI image at this time is output as an MRI synthetic image; if the purpose is to generate a CT synthetic image, the MRI image is processed by a generator and a discriminator whose network parameters are standard CT image parameters to obtain a CT synthetic image;
[0020] The second synthetic image generation module is used for processing the CT image by a generator whose network parameters are standard MRI image parameters to obtain a simulated MRI image; adding noise to the CT image several times to obtain a CT noise image; the CT noise image and the simulated MRI image are respectively processed by a conditional semantic generator to obtain a CT noise semantic image and a simulated MRI semantic image; performing denoising and feature mapping processing on the CT noise semantic image several times based on the simulated MRI semantic image to obtain an optimized simulated MRI image; the optimized simulated MRI image and the simulated MRI image are processed by a discriminator, and when the peak signal-to-noise ratio of the optimized simulated MRI image and the feature similarity between the optimized simulated MRI image and the simulated MRI image are respectively less than a second signal-to-noise ratio threshold and a second similarity threshold, the optimized simulated MRI image at this time is output as an MRI synthetic image;
[0021] The third synthetic image generation module is used for the MRI image to be processed by a generator whose network parameters are standard CT image parameters, a conditional semantic generator and a discriminator to obtain a CT synthetic image.
[0022] A device for synthesizing CT images and MRI images comprises a processor and a memory, wherein the processor implements the above-mentioned method for synthesizing CT images and MRI images when executing a computer program stored in the memory.
[0023] A computer-readable storage medium is used to store a computer program, wherein the computer program implements the above-mentioned method for synthesizing CT images and MRI images when executed by a processor.
[0024] The beneficial effects of the present invention are:
[0025] The present invention provides a mutual synthesis method of CT images and MRI images. When there are both CT images and MRI images, an image with MRI characteristics is generated from CT data and an image with CT characteristics is generated from MRI data to obtain a simulated MRI image and a simulated CT image. Then, the real MRI image and CT image, which are synthesis targets, are adversarially distinguished from the simulated MRI image and the simulated CT image respectively, and the synthesis quality is controlled based on the peak signal-to-noise ratio, the mean square error and the structural domain similarity to achieve mutual synthesis of the CT image and the MRI image. When there is no CT image or MRI image, taking the synthesized MRI image as an example, an image with MRI characteristics is generated from the CT image to obtain a simulated MRI image. RI image; then the CT noise image and the simulated MRI image are processed by the conditional semantic generator respectively, and the features of the simulated MRI semantic image are mapped to the CT noise semantic image in the denoising process to obtain an optimized simulated MRI image; finally, the simulated MRI image is used as the target image, and the optimized simulated MRI image and the simulated MRI image are processed by the discriminator, and the image quality is controlled by the peak signal-to-noise ratio and feature similarity to obtain an MRI synthetic image; based on the above method, a CT image can be generated based on the MRI image when a CT image is missing; the synthesis method provided by the present invention can provide different image synthesis methods according to the degree of completeness of the material, has strong adaptability, and has high image synthesis quality and high efficiency. DETAILED DESCRIPTION
[0026] This embodiment provides a method for synthesizing a CT image and an MRI image, including the following operations:
[0027] S1. If there is a CT image and an MRI image, the purpose is to generate an MRI composite image or a CT composite image, and S2 is executed; if there is a CT image but no MRI image, the purpose is to generate an MRI composite image, and S3 is executed; if there is no CT image but an MRI image, the purpose is to generate a CT composite image, and S4 is executed;
[0028] S2. If the purpose is to generate an MRI composite image, the CT image is processed by a generator whose network parameters are standard MRI image parameters to obtain a simulated MRI image; the simulated MRI image and the MRI image are processed by a discriminator whose network parameters are standard MRI image parameters, and when the peak signal-to-noise ratio of the simulated MRI image, the mean square error and the structural similarity between the simulated MRI image and the MRI image are respectively less than a first signal-to-noise ratio threshold, the mean square error and the first similarity threshold, the simulated MRI image at this time is output as an MRI composite image; if the purpose is to generate a CT composite image, the MRI image is processed by a generator and a discriminator whose network parameters are standard CT image parameters to obtain a CT composite image;
[0029] S3, the CT image is processed by a generator whose network parameters are standard MRI image parameters to obtain a simulated MRI image; noise is added to the CT image several times to obtain a CT noise image; the CT noise image and the simulated MRI image are processed by a conditional semantic generator respectively to obtain a CT noise semantic image and a simulated MRI semantic image; based on the simulated MRI semantic image, the CT noise semantic image is subjected to several denoising and feature mapping processes to obtain an optimized simulated MRI image; the optimized simulated MRI image and the simulated MRI image are processed by a discriminator, and when the peak signal-to-noise ratio of the optimized simulated MRI image and the feature similarity between the optimized simulated MRI image and the simulated MRI image are respectively less than a second signal-to-noise ratio threshold and a second similarity threshold, the optimized simulated MRI image at this time is output as an MRI synthetic image;
[0030] S4. The MRI image is processed by a generator, a conditional semantic generator and a discriminator whose network parameters are the standard CT image parameters to obtain a CT synthetic image.
[0031] S1. If there is a CT image and an MRI image, the purpose is to generate an MRI composite image or a CT composite image, and S2 is executed.
[0032] When the synthetic data is complete, including both CT images and MRI images, and the purpose is to generate an MRI synthetic image or a CT synthetic image, the operation in S2 should be performed to achieve mutual synthesis of the CT image and the MRI image.
[0033] In order to improve the quality of image synthesis, the CT image or MRI image is normalized and converted into a two-dimensional slice image to obtain a CT two-dimensional slice image or a MRI two-dimensional slice image, which is used to perform the operations in S2, or / and S3, or / and S4. Specifically, the original Dicom format CT image and MRI image collected by the hospital are converted into Nifit format using MRIcron software, and the Nifti format CT image and MRI image are normalized and converted into two-dimensional slices in PNG format to obtain a CT two-dimensional slice image or a MRI two-dimensional slice image, which is stored as a MAT file for performing the operations in S2, or / and S3, or / and S4.
[0034] S2. If the purpose is to generate an MRI synthetic image, the CT image is processed by a generator whose network parameters are standard MRI image parameters to obtain a simulated MRI image; the simulated MRI image and the MRI image are processed by a discriminator whose network parameters are standard MRI image parameters, and when the peak signal-to-noise ratio of the simulated MRI image, the mean square error and the structural similarity between the simulated MRI image and the MRI image are respectively less than a first signal-to-noise ratio threshold, the mean square error and the first similarity threshold, the simulated MRI image at this time is output as an MRI synthetic image; if the purpose is to generate a CT synthetic image, the MRI image is processed by a generator and a discriminator whose network parameters are standard CT image parameters to obtain a CT synthetic image.
[0035] When the synthetic data is complete, in order to achieve the mutual synthesis of CT images and MRI images, a generator whose network parameters are standard MRI image parameters and standard CT image parameters is used to generate images with MRI features from CT data and to generate images with CT features from MRI data, so as to obtain simulated MRI images and simulated CT images. Then, the real MRI images and CT images are adversarially distinguished from the simulated MRI images and simulated CT images, respectively. Based on the peak signal-to-noise ratio, mean square error and structural domain similarity, the image reconstruction quality and feature expression ability of the simulated MRI images are controlled, and the synthesis quality of the simulated MRI images is controlled to achieve the mutual synthesis of CT images and MRI images, and obtain MRI synthetic images and CT synthetic images.
[0036] When there are both CT images and MRI images, the specific operation of generating an MRI composite image based on the CT image is as follows.
[0037] First, the CT image is processed by a generator whose network parameters are standard MRI image parameters, and an image with MRI characteristics is generated from the CT data without performing additional MRI scans to obtain a simulated MRI image. The generator whose network parameters are standard MRI image parameters can be obtained by training the generator with an MRI data set formed by several standard MRI images.
[0038] Then, in order to improve the quality of the synthesized image, the simulated MRI image and the real MRI image as the synthesis target are processed by a discriminator whose network parameters are the parameters of the standard MRI image, and adversarial distinction is made between the simulated MRI image and the MRI image, thereby making the semantic expression ability of the simulated MRI image closer to that of the MRI image.
[0039] Finally, when the peak signal-to-noise ratio of the simulated MRI image, the mean square error and the structural similarity between the simulated MRI image and the MRI image are respectively less than the first signal-to-noise ratio threshold, the mean square error and the first similarity threshold, the semantic expression ability of the simulated MRI image meets the requirements, and the simulated MRI image at this time is output as an MRI synthetic image.
[0040] The peak signal-to-noise ratio is the ratio of the maximum pixel value of the simulated MRI image to the average value of the sum of squares of pixel errors of the simulated MRI image and the CT image.
[0041] When there are both CT images and MRI images, the process of generating a CT composite image based on the MRI image is similar to the above process of generating an MRI composite image, and the specific operations are as follows.
[0042] Firstly, the MRI image is processed by a generator whose network parameters are the standard CT image parameters, and an image with CT features is generated from the MRI data to obtain a simulated CT image.
[0043] Then, in order to improve the quality of the synthesized image, the simulated CT image and the real CT image as the synthesis target are processed by a discriminator whose network parameters are the parameters of the standard CT image, and adversarial distinction is made between the simulated CT image and the CT image, thereby making the semantic expression ability of the simulated CT image and the CT image closer.
[0044] Finally, when the peak signal-to-noise ratio of the simulated CT image, the mean square error and the structural similarity between the simulated CT image and the CT image are respectively less than the first signal-to-noise ratio threshold, the mean square error and the first similarity threshold, the semantic expression ability of the simulated CT image meets the requirements, and the simulated CT image at this time is output as a CT synthetic image.
[0045] S3, the CT image is processed by a generator whose network parameters are the parameters of a standard MRI image to obtain a simulated MRI image; noise is added to the CT image several times to obtain a CT noise image; the CT noise image and the simulated MRI image are processed by a conditional semantic generator respectively to obtain a CT noise semantic image and a simulated MRI semantic image; based on the simulated MRI semantic image, the CT noise semantic image is subjected to several denoising and feature mapping processes to obtain an optimized simulated MRI image; the optimized simulated MRI image and the simulated MRI image are processed by a discriminator, and when the peak signal-to-noise ratio of the optimized simulated MRI image and the feature similarity between the optimized simulated MRI image and the simulated MRI image are respectively less than a second signal-to-noise ratio threshold and a second similarity threshold, the optimized simulated MRI image at this time is output as an MRI composite image.
[0046] When the synthetic data is incomplete and there is no MRI image as the target synthetic image, in order to generate MRI images based on CT images, an image with MRI features is generated from the CT image to obtain a simulated MRI image; then the CT noise image and the simulated MRI image are processed by the conditional semantic generator respectively, and the features of the simulated MRI semantic image are mapped to the CT noise semantic image in the denoising process to obtain an optimized simulated MRI image; finally, the simulated MRI image is used as the target image, and the optimized simulated MRI image and the simulated MRI image are processed by the discriminator, and the image quality is controlled by the peak signal-to-noise ratio and feature similarity to obtain an MRI synthetic image.
[0047] Firstly, the CT image is processed by a generator whose network parameters are standard MRI image parameters, so that when the MRI image is missing, an image with MRI characteristics can be generated from the CT data to obtain a simulated MRI image.
[0048] At the same time, noise is added to the CT image several times, and the step size in the process of adding noise several times is greater than 1 and much greater than 1, so as to enhance the quality and efficiency of image generation and obtain a CT noise image.
[0049] The operation of adding several noises can be achieved by the following formula:
[0050] ,
[0051] x t For the t CT noise map of time, γ t is the noise variance, x t-k For the t-k CT noise map of time, k is the step length, k >1, the ratio of k to 1 is not less than 10.
[0052] Noise variance γ t It can be obtained by the following formula:
[0053] ,
[0054] , are the minimum and maximum values of the noise control index, T For total time.
[0055] The noise transfer probability in the process of adding several noises can be realized by the following formula:
[0056] ,
[0057] q(x t |x t-k ) For the t Time and t-k The noise transition probability in time, I is the unit covariance matrix, is a normal distribution.
[0058] Then, the CT noise image and the simulated MRI image are processed by the conditional semantic generator respectively to obtain the CT noise semantic image and the simulated MRI semantic image.
[0059] Among them, the network parameters processed by the conditional semantic generator are standard CT image parameters, and the operations processed by the conditional semantic generator are specifically as follows: the CT noise image or the simulated MRI image is processed by convolution and downsampling to obtain the first downsampling image; the first downsampling image is processed by convolution and downsampling to obtain the second downsampling image; and so on to obtain the Nth downsampling image; N is a positive integer; the Nth downsampling image is processed by symmetric cross attention to obtain the Nth cross attention image; the Nth cross attention image is processed by convolution and upsampling, and then spliced with the Nth downsampling image to obtain the Nth fused feature map; the Nth fused feature map is processed by convolution and upsampling, and then spliced with the N-1th downsampling image to obtain the N-1th fused feature map; and so on to obtain a CT noise semantic map or a simulated MRI semantic map feature map.
[0060] The operations of the above-mentioned symmetric cross-attention processing are specifically as follows: obtaining the channel mapping map and the spatial mapping map of the Nth down-sampling image; the channel mapping map and the spatial mapping map are processed by the pre-attention to obtain the pre-attention map; the pre-attention map and the spatial mapping map are processed by the post-attention to obtain the Nth cross-attention map.
[0061] The channel mapping map and the space mapping map are respectively the pixel value distribution map of each position point on the target channel in the Nth down-sampling map, and the pixel value distribution map of each channel at the target position point.
[0062] The operation of pre-attention processing is as follows: the channel mapping map and the space mapping map are processed by Fourier transform respectively to obtain the channel Fourier map and the space Fourier map; the channel Fourier map and the space Fourier map are respectively multiplied element by element with the Fourier space key item weight map and the Fourier space query item weight map, and then fused (which can be achieved by element-by-element multiplication) to obtain the Fourier fusion map; the Fourier fusion map is filtered and processed by inverse Fourier transform to obtain a fused feature map; the fused feature map is weighted and processed by a multi-layer perceptron to obtain a pre-attention map.
[0063] The process of post-attention processing is similar to that of pre-attention processing, and both include the steps of Fourier transform processing, fusion processing, filtering processing, inverse Fourier transform processing, weighted processing and multi-layer perceptron processing. However, in the post-attention processing, the pre-attention Fourier map (obtained by Fourier transform processing of the pre-attention map) and the spatial Fourier map (obtained by Fourier transform processing of the spatial mapping map) are respectively multiplied element-by-element with the Fourier space query item weight map and the Fourier space key item weight map.
[0064] The above filtering operation can be achieved through several convolution processes and nonlinear processing (preferably Sigmoid function).
[0065] Next, the CT noise semantic map is subjected to several denoising and feature mapping processes based on the simulated MRI semantic map. During the denoising process, the features of the simulated MRI semantic map are mapped to the CT noise semantic map to obtain an optimized simulated MRI image.
[0066] The noise transfer probability during several denoising processes can be realized by the following formula:
[0067] ,
[0068] p(X t |X t-k ) For the denoising process t Time and t-k The noise transition probability in time, X t For the t Time-optimized simulation MRI images, For the t The CT noise semantic map of time and the t-k The noise feature value of the temporal CT noise semantic map, X t-k For the t-k Time-optimized simulation MRI images, k is the step length, k >1, the ratio of k to 1 is not less than 10, is the denoising noise variance, I is the unit covariance matrix, is a normal distribution.
[0069] Among them, the noise characteristic value This can be achieved through the following formula:
[0070] ,
[0071] in, For the t The time-corrected noise balance factor, , α t =1-γ t , γ t is the noise variance in the noise adding process, For the t-k Time-corrected noise balance factor, α r For the r A noise balance factor, X 0 is the simulated MRI semantic graph at the initial input, X t For thet Time-optimized simulation MRI image.
[0072] Subsequently, the optimized simulated MRI images and the simulated MRI images are processed by the discriminator to perform adversarial distinction between the optimized simulated MRI images and the simulated MRI images, thereby making the semantic expression capabilities of the optimized simulated MRI images and the CT images closer.
[0073] Finally, when the peak signal-to-noise ratio of the optimized simulated MRI image and the feature similarity between the optimized simulated MRI image and the simulated MRI image are respectively less than the second signal-to-noise ratio threshold and the second similarity threshold, the optimized simulated MRI image at this time is output as the MRI composite image.
[0074] Among them, the method for obtaining feature similarity is to obtain the average value of pixel differences at corresponding positions after pooling the optimized simulated MRI image and the simulated MRI image respectively, and obtain the pooled mean difference; based on the pooled mean difference, the feature covariance matrix of the simulated MRI image and the feature covariance matrix of the simulated MRI image are optimized to obtain the feature similarity.
[0075] The feature similarity can be obtained by the following formula:
[0076] ,
[0077] F is the feature similarity ,μ X -μ Y is the pooled mean difference, ‖ ‖ For paradigm processing, tr( ) is the sum function of the matrix diagonal elements, M X To optimize the covariance matrix of the simulated MRI image, M Y is the covariance matrix of the simulated MRI image.
[0078] S4. The MRI image is processed by a generator whose network parameters are the standard CT image parameters, a conditional semantic generator, and a discriminator to obtain a CT synthetic image.
[0079] If there is no CT image as the synthesis target, but there is an MRI image, in order to generate a CT synthetic image, similar operations to those in S3 above are performed, that is, the MRI image is processed by a generator whose network parameters are standard CT image parameters, a conditional semantic generator, and a discriminator to obtain an MRI synthetic image.
[0080] Specifically, the MRI image is processed by a generator whose network parameters are standard CT image parameters to obtain a simulated CT image; noise is added to the MRI image several times to obtain an MRI noise image; the MRI noise image and the simulated CT image are respectively processed by a conditional semantic generator (the network parameters are standard MRI image parameters) to obtain an MRI noise semantic image and a simulated CT semantic image; based on the simulated CT semantic image, the MRI noise semantic image is subjected to several denoising and feature mapping processes to obtain an optimized simulated CT image; the optimized simulated CT image and the simulated CT image are processed by a discriminator, and when the peak signal-to-noise ratio of the optimized simulated CT image, the mean square error and the structural similarity between the optimized simulated CT image and the simulated CT image are respectively less than the first signal-to-noise ratio threshold, the mean square error and the first similarity threshold, the optimized simulated CT image at this time is output as a CT synthetic image.
[0081] This embodiment further provides a system for synthesizing a CT image and an MRI image, which is used to implement the above-mentioned method for synthesizing a CT image and an MRI image, including:
[0082] The data integrity judgment module is used to execute the first composite image generation module if there is a CT image and an MRI image, and the purpose is to generate an MRI composite image or a CT composite image; if there is a CT image but no MRI image, and the purpose is to generate an MRI composite image, execute the second composite image generation module; if there is no CT image but an MRI image, and the purpose is to generate a CT composite image, execute the third composite image generation module;
[0083] The first synthetic image generation module is used for, if the purpose is to generate an MRI synthetic image, the CT image is processed by a generator whose network parameters are standard MRI image parameters to obtain a simulated MRI image; the simulated MRI image and the MRI image are processed by a discriminator whose network parameters are standard MRI image parameters, and when the peak signal-to-noise ratio of the simulated MRI image, the mean square error and the structural similarity between the simulated MRI image and the MRI image are respectively less than a first signal-to-noise ratio threshold, the mean square error and the first similarity threshold, the simulated MRI image at this time is output as an MRI synthetic image; if the purpose is to generate a CT synthetic image, the MRI image is processed by a generator and a discriminator whose network parameters are standard CT image parameters to obtain a CT synthetic image;
[0084] The second synthetic image generation module is used for processing the CT image by a generator whose network parameters are standard MRI image parameters to obtain a simulated MRI image; adding noise to the CT image several times to obtain a CT noise image; the CT noise image and the simulated MRI image are respectively processed by a conditional semantic generator to obtain a CT noise semantic image and a simulated MRI semantic image; performing denoising and feature mapping processing on the CT noise semantic image several times based on the simulated MRI semantic image to obtain an optimized simulated MRI image; the optimized simulated MRI image and the simulated MRI image are processed by a discriminator, and when the peak signal-to-noise ratio of the optimized simulated MRI image and the feature similarity between the optimized simulated MRI image and the simulated MRI image are respectively less than a second signal-to-noise ratio threshold and a second similarity threshold, the optimized simulated MRI image at this time is output as an MRI synthetic image;
[0085] The third synthetic image generation module is used for the MRI image to be processed by a generator whose network parameters are standard CT image parameters, a conditional semantic generator and a discriminator to obtain a CT synthetic image.
[0086] This embodiment further provides a device for synthesizing a CT image and an MRI image, comprising a processor and a memory, wherein the processor implements the above-mentioned method for synthesizing a CT image and an MRI image when executing a computer program stored in the memory.
[0087] This embodiment further provides a computer-readable storage medium for storing a computer program, wherein the computer program implements the above-mentioned method for synthesizing CT images and MRI images when executed by a processor.
[0088] The present embodiment provides a mutual synthesis method of a CT image and an MRI image. When both a CT image and an MRI image are available, an image with MRI features is generated from the CT data and an image with CT features is generated from the MRI data to obtain a simulated MRI image and a simulated CT image. Then, the real MRI image and CT image, which are synthesis targets, are adversarially distinguished from the simulated MRI image and the simulated CT image, respectively. The synthesis quality is controlled based on the peak signal-to-noise ratio, the mean square error and the structural domain similarity to achieve mutual synthesis of the CT image and the MRI image. When there is no CT image or MRI image, taking the synthesized MRI image as an example, an image with MRI features is generated from the CT image to obtain a simulated MRI image. RI image; then the CT noise image and the simulated MRI image are processed by the conditional semantic generator respectively, and the features of the simulated MRI semantic image are mapped to the CT noise semantic image in the denoising process to obtain an optimized simulated MRI image; finally, the simulated MRI image is used as the target image, and the optimized simulated MRI image and the simulated MRI image are processed by the discriminator, and the image quality is controlled by the peak signal-to-noise ratio and feature similarity to obtain an MRI synthetic image; based on the above method, a CT image can be generated based on the MRI image when a CT image is missing; the synthesis method provided by the present invention can provide different image synthesis methods according to the degree of completeness of the material, has strong adaptability, and has high image synthesis quality and high efficiency.
Claims
1. A method for synthesizing a CT image and an MRI image, characterized in that: The following operations are included: S1. If there is a CT image or an MRI image, and the purpose is to generate an MRI composite image or a CT composite image, execute S2; If there is a CT image but no MRI image, and the purpose is to generate an MRI composite image, execute S3; If there is no CT image but there is an MRI image, the purpose is to generate a CT composite image, and S4 is executed; S2. If the purpose is to generate an MRI composite image, the CT image is processed by a generator whose network parameters are standard MRI image parameters to obtain a simulated MRI image. The simulated MRI image and the MRI image are processed by a discriminator whose network parameters are standard MRI image parameters. When the peak signal-to-noise ratio of the simulated MRI image, the mean square error and the structural similarity between the simulated MRI image and the MRI image are respectively less than a first signal-to-noise ratio threshold, the mean square error and the first similarity threshold, the simulated MRI image at this time is output as the MRI composite image; If the purpose is to generate a CT synthetic image, the MRI image is processed by a generator and a discriminator whose network parameters are standard CT image parameters to obtain a CT synthetic image; S3, the CT image is processed by a generator whose network parameters are standard MRI image parameters to obtain a simulated MRI image, and a plurality of noises are added to the CT image to obtain a CT noise image; The CT noise map and the simulated MRI map are processed by a conditional semantic generator to obtain a CT noise semantic map and a simulated MRI semantic map, and the CT noise semantic map is subjected to several denoising and feature mapping processes based on the simulated MRI semantic map. During the denoising process, the features of the simulated MRI semantic map are mapped into the CT noise semantic map to obtain an optimized simulated MRI map. The noise transfer probability during several denoising processes is realized by the following formula: ， p(X t |X t-k ) For the denoising process t Time and tk The noise transition probability in time, X t For the t Time-optimized simulation MRI images, For the t The CT noise semantic map of time and the tk The noise feature value of the temporal CT noise semantic map, X t-k For the tk Time-optimized simulation MRI images, k is the step length, is the denoising noise variance, I is the unit covariance matrix, is a normal distribution; Noise characteristic value This is achieved through the following formula: ， in, For the t The time-corrected noise balance factor, , α t =1-γ t , γ t is the noise variance in the noise adding process, For the tk Time-corrected noise balance factor, α r For the r A noise balance factor, X 0 is the simulated MRI semantic graph at the initial input, X t For the t Time-optimized simulation MRI images; The optimized simulated MRI image and the simulated MRI image are processed by a discriminator, and when the peak signal-to-noise ratio of the optimized simulated MRI image and the feature similarity between the optimized simulated MRI image and the simulated MRI image are respectively less than a second signal-to-noise ratio threshold and a second similarity threshold, the optimized simulated MRI image at this time is output as an MRI composite image; S4. The MRI image is processed by a generator whose network parameters are standard CT image parameters, a conditional semantic generator and a discriminator to obtain a CT synthetic image.
2. The method for synthesizing CT images and MRI images according to claim 1, characterized in that: In S3, the operations processed by the conditional semantic generator are specifically: The CT noise image or the simulated MRI image is subjected to convolution and downsampling processing to obtain a first downsampling image, the first downsampling image is subjected to convolution and downsampling processing to obtain a second downsampling image, and so on to obtain an Nth downsampling image, where N is a positive integer; The Nth down-sampled image is subjected to symmetrical cross-attention processing to obtain an Nth cross-attention image; After the Nth cross-attention map is processed by convolution and upsampling, it is spliced with the Nth downsampling map to obtain the Nth fused feature map. After the Nth fused feature map is processed by convolution and upsampling, it is spliced with the N-1th downsampling map to obtain the N-1th fused feature map, and so on to obtain a CT noise semantic map or a simulated MRI semantic map feature map.
3. The method for synthesizing CT images and MRI images according to claim 2, characterized in that: The operation of the symmetric cross attention processing is specifically as follows: Obtain a channel mapping map and a spatial mapping map of the Nth down-sampled image, subject the channel mapping map and the spatial mapping map to pre-attention processing to obtain a pre-attention map, and subject the pre-attention map and the spatial mapping map to post-attention processing to obtain the Nth cross-attention map; The operation of the pre-attention processing is as follows: the channel mapping map and the space mapping map are respectively processed by Fourier transformation to obtain a channel Fourier map and a space Fourier map, the channel Fourier map and the space Fourier map are respectively multiplied element by element with the Fourier space key item weight map and the Fourier space query item weight map, and then fused to obtain a Fourier fusion map, the Fourier fusion map is filtered and processed by inverse Fourier transformation to obtain a fusion feature map, and the fusion feature map is weighted and processed by a multi-layer perceptron to obtain a pre-attention map; In the post-attention processing, the pre-attention Fourier map and the spatial Fourier map are element-wise multiplied with the Fourier space query item weight map and the Fourier space key item weight map, respectively.
4. The method for combining CT images and MRI images according to claim 3, characterized in that: The filtering operation can be implemented through several convolution processes and nonlinear processes.
5. The method for combining CT images and MRI images according to claim 1, characterized in that: In S3, the method for obtaining feature similarity is: After the optimized simulated MRI image and the simulated MRI image are respectively subjected to pooling processing, the average value of the pixel difference at the corresponding position is obtained to obtain the pooled mean difference, and the feature similarity is obtained based on the pooled mean difference, the feature covariance matrix of the optimized simulated MRI image and the feature covariance matrix of the simulated MRI image.
6. The method for combining CT images and MRI images according to claim 1, characterized in that: In S3, the step size is greater than 1 in the process of adding noise to the CT image several times.
7. The method for combining CT images and MRI images according to claim 1, characterized in that: In S1, the CT image or MRI image is normalized and converted into a two-dimensional slice image to obtain a CT two-dimensional slice image or an MRI two-dimensional slice image for performing the operations in S2, or / and S3, or / and S4.
8. A system for synthesizing CT images and MRI images, used to implement the method for synthesizing CT images and MRI images according to claim 1, characterized in that: include: A data integrity judgment module is used to execute a first composite image generation module if there is a CT image and an MRI image, and the purpose is to generate an MRI composite image or a CT composite image; if there is a CT image but no MRI image, and the purpose is to generate an MRI composite image, execute a second composite image generation module; if there is no CT image but an MRI image, and the purpose is to generate a CT composite image, execute a third composite image generation module; A first synthetic image generating module is used for, if the purpose is to generate an MRI synthetic image, the CT image is processed by a generator whose network parameters are standard MRI image parameters to obtain a simulated MRI image, the simulated MRI image and the MRI image are processed by a discriminator whose network parameters are standard MRI image parameters, when the peak signal-to-noise ratio of the simulated MRI image, the mean square error and the structural similarity between the simulated MRI image and the MRI image are respectively less than a first signal-to-noise ratio threshold, the mean square error and the first similarity threshold, the simulated MRI image at this time is output as an MRI synthetic image, if the purpose is to generate a CT synthetic image, the MRI image is processed by a generator and a discriminator whose network parameters are standard CT image parameters to obtain a CT synthetic image; A second synthetic image generating module is used for processing the CT image by a generator whose network parameters are standard MRI image parameters to obtain a simulated MRI image, adding noise to the CT image several times to obtain a CT noise image, the CT noise image and the simulated MRI image are respectively processed by a conditional semantic generator to obtain a CT noise semantic image and a simulated MRI semantic image, performing several denoising and feature mapping processes on the CT noise semantic image based on the simulated MRI semantic image to obtain an optimized simulated MRI image, the optimized simulated MRI image and the simulated MRI image are processed by a discriminator, and when the peak signal-to-noise ratio of the optimized simulated MRI image and the feature similarity between the optimized simulated MRI image and the simulated MRI image are respectively less than a second signal-to-noise ratio threshold and a second similarity threshold, outputting the optimized simulated MRI image at this time as an MRI synthetic image; The third synthetic image generation module is used for processing the MRI image through a generator whose network parameters are standard CT image parameters, a conditional semantic generator and a discriminator to obtain a CT synthetic image.
9. A device for synthesizing CT images and MRI images, characterized in that: The method comprises a processor and a memory, wherein the processor implements the mutual synthesis method of the CT image and the MRI image as claimed in any one of claims 1 to 7 when executing the computer program stored in the memory.
10. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein when the computer program is executed by a processor, the method for synthesizing a CT image and an MRI image according to any one of claims 1 to 7 is implemented.