Cross-modal conversion method and system for medical image

Abstract

The invention discloses a cross-modal conversion method of a medical image. The method comprises the steps of obtaining existing multi-modal medical image data and performing preprocessing to construct a training data set; for each type of medical image modality, adopting the training data set to train a corresponding auto-encoder; constructing a cross-modal conversion initial model of the medical image based on a variational diffusion model by combining an auto-encoder, and training to obtain a trained cross-modal conversion model of the medical image; and carrying out actual cross-modal conversion of the medical image by adopting the cross-modal conversion model of the medical image. The invention also discloses a system for realizing the cross-modal conversion method of the medical image. According to the method, cross-modal conversion of the medical image is realized, data sets do not need to be paired, and meanwhile, the reliability is higher and the accuracy is better.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular relates to a cross-modal conversion method and system for medical images. Background Art

[0002] Modal conversion of medical images has always been a crucial task in both basic medical research and clinical practice. Modal conversion aims to convert source modality images into target modality images (e.g., MRI to CT). This modality conversion solution not only provides complementary information between modalities but also reduces the need for multiple examinations, significantly facilitating medical professionals.

[0003] At present, commonly used medical image modality conversion schemes include those based on strong supervisory signals and those based on unsupervised learning. Among them, medical image modality conversion schemes based on strong supervisory signals mostly rely on paired datasets (i.e., image data of the same object under different modalities) for model training, so that the model can fully learn the strong mapping relationship from source modality images to target modality images; however, in real practice, it is extremely difficult to obtain paired datasets for medical images. Existing medical image modality conversion schemes based on unsupervised learning are mostly implemented based on GAN. Although they do not require paired datasets to train the model, such schemes are not only deeply affected by gradient vanishing and mode collapse, but also have the defect of limited adversarial learning ability. Therefore, the quality of the final synthesized images of such schemes is poor. Summary of the Invention

[0004] One of the objectives of the present invention is to provide a cross-modal conversion method for medical images with high reliability and good accuracy.

[0005] A second objective of the present invention is to provide a system for implementing the cross-modal conversion method of medical images.

[0006] The cross-modal conversion method of medical images provided by the present invention comprises the following steps:

[0007] S1. Acquire existing multimodal medical image data;

[0008] S2. Preprocessing the multimodal medical image data obtained in step S1 to construct a training dataset;

[0009] S3. Based on the medical image data obtained in step S1, for each type of medical image modality, use the training dataset constructed in step S2 to train the corresponding autoencoder;

[0010] S4. Combining the autoencoder obtained in step S3, and based on the variational diffusion model, construct an initial model for cross-modal conversion of medical images;

[0011] S5. Using the training dataset constructed in step S2, the initial model of cross-modal conversion of medical images constructed in step S4 is trained to obtain a trained cross-modal conversion model of medical images;

[0012] S6. Use the cross-modal conversion model of medical images obtained in step S5 to perform actual cross-modal conversion of medical images.

[0013] Step S3, based on the medical image data obtained in step S1, uses the training dataset constructed in step S2 for each type of medical image modality to train the corresponding autoencoder, specifically including the following steps:

[0014] Based on the medical image data obtained in step S1, for each type of medical image modality, the training dataset constructed in step S2 is used to train the corresponding autoencoder;

[0015] The trained autoencoder includes the encoder part of the autoencoder and the decoder part of the autoencoder.

[0016] In step S4, the autoencoder obtained in step S3 is combined to construct an initial model for cross-modal conversion of medical images based on the variational diffusion model, which specifically includes the following steps:

[0017] The constructed initial model for cross-modal conversion of medical images includes the encoder part of the autoencoder, the variational diffusion model backbone network, the decoder part of the autoencoder, the generator and the discriminator;

[0018] The processing of the initial model for cross-modal conversion of medical images includes:

[0019] The input unpaired data is (x a ,x b ), where x a is A modal data, x b is the B-mode data;

[0020] x b Through the first generator G B→A Generate A-modal synthetic image x' a ;x' a The first latent space representation z' is obtained through the encoder part of the autoencoder a ; z' a Latent space representation of the noisy image at time step t compared to the true B-modality data Perform splicing to obtain the first splicing feature Will Input the variational diffusion model backbone network to generate the first latent space representation of the predicted image Will Then the corresponding first generated image is obtained through the decoder part of the autoencoder; at the same time, x' a Also through the second generator G A→B Generate B-mode reconstructed image

[0021] x a Through the second generator G A→B Generate B-mode synthetic image x' b ;x' b The second latent space representation z' is obtained through the encoder part of the autoencoder b ; z' b The latent space representation of the noisy image at time step t with the true A modality data Perform splicing to obtain the second splicing feature Will Input the variational diffusion model backbone network to generate the second latent space representation of the predicted image Will Then the corresponding second generated image is obtained through the decoder part of the autoencoder; at the same time, x' b Also through the first generator G B→A Generate A-mode reconstructed image

[0022] In addition, the first discriminator D A With the first generator G B→A Correspondingly, the second discriminator D B With the second generator G A→B Corresponding; D A The input includes A modality synthetic image x' a and A modal data x a ;D A The output of is the judgment result of the input image, including whether the input image is an image synthesized by the generator or the input image is a real image.

[0023] The processing of the variational diffusion model backbone network also includes the following steps:

[0024] The first concatenated feature of the input Get the first latent space representation z' a ; Latent space representation of the noisy image at time step t based on the real B-modality data From pure noise samples First, perform step-by-step denoising according to the following rules:

[0025]

[0026] In the formula is the potential space representation of the noise image of the B-modal data obtained in the s-th step at a time step of s; α s is the noise mean corresponding to the time step s; is the output of the DiM network; σ s is the noise standard deviation corresponding to the time step s; α t is the noise mean corresponding to the time step t; σ t is the noise standard deviation corresponding to the time step t; s is the current step number and T is the total number of sampling steps, t is the time step;

[0027] Second splicing feature for input Get the second latent space representation z' b ; Latent space representation of the noise image at time step t based on the real A modality data From pure noise samples First, perform step-by-step denoising according to the following rules:

[0028]

[0029] In the formula is the potential space representation of the noise image of the A modality data obtained in the s-th step at a time step of s;

[0030] Finally, the first latent space representation and the second latent space representation As the output of the variational diffusion model backbone network.

[0031] The structure of the variational diffusion model backbone network specifically includes the following parts:

[0032] The input image features are divided into several patches; then a 3×3 convolution is performed to obtain the convolution image features; in the obtained convolution image features, a padding flag to be learned is added at the end of each row and each column; in the convolution image features with the added padding flags, a time step padding flag and a corresponding padding flag to be learned are added to obtain the input features;

[0033] Process the input features through the Mamba network layer to obtain processed features;

[0034] For the processing features, we first delete the added time step padding flag and the padding flag to be learned, then perform 3×3 convolution processing, and then fuse the obtained patches to obtain the final output of the variational diffusion model backbone network.

[0035] The Mamba network layer processing process includes the following steps:

[0036] The Mamba network layer includes an input Mamba module and N Mamba processing modules. The input Mamba module and the N Mamba processing modules are connected in series. The output of the i-th Mamba processing module and the output of the Ni-th Mamba processing module are added together and used as the input of the N-i+1-th Mamba processing module. The input Mamba module is used to learn the global information between different patches. The Mamba processing module is used to learn the global information between patches under different scanning modes.

[0037] The Mamba processing modules all have the same structure. The Mamba processing modules include a channel scanning module and a Mamba module. The channel scanning module is used to control the order in which input image features are input to the Mamba module, and the Mamba module is used to learn global information between patches. The channel scanning module randomly selects one of the following four scanning modes and is used to control the order in which input image features are input to the Mamba module:

[0038] Method 1: The time step padding flag and the corresponding padding flag to be learned are added to the front of the first row of the convolution image feature with the padding flag added. The scanning order is: starting from the time step padding flag and the corresponding padding flag to be learned, starting from the first row, and scanning in order of rows;

[0039] Method 2: The time step padding flag and the corresponding padding flag to be learned are added to the front of the first row of the convolution image feature with the padding flag added. The scanning order is: starting from the time step padding flag and the corresponding padding flag to be learned, starting from the first column, and scanning in order of columns;

[0040] Method 3: The time step padding flag and the corresponding padding flag to be learned are added to the end of the last row of the convolution image features with the padding flag added. The scanning order is: starting from the time step padding flag and the corresponding padding flag to be learned, starting from the last row, and scanning in order of rows;

[0041] Method 4: The time step padding flag and the corresponding padding flag to be learned are added to the end of the last row of the convolution image feature with the padding flag added. The scanning order is: starting from the time step padding flag and the corresponding padding flag to be learned, starting from the last column, and scanning in order of columns.

[0042] The training described in step S5 specifically includes the following steps:

[0043] The first loss is calculated using the following formula:

[0044]

[0045] In the formula is the first loss value; is the loss function corresponding to the discriminator; D B (x b ) is the discriminator corresponding to the B modality; is the loss function corresponding to the generator; G A→B (x a ) is the generator corresponding to the B mode;

[0046] The second loss is calculated using the following formula:

[0047]

[0048] In the formula is the second loss value; is the loss function corresponding to the discriminator; D A (x a ) is the discriminator corresponding to modality A; is the loss function corresponding to the generator; G B→A (x b ) is the generator corresponding to the A mode;

[0049] The third loss is calculated using the following formula:

[0050]

[0051] In the formula is the third loss value; is the cycle loss corresponding to mode A; is the cycle loss representation corresponding to the B mode; || ||1 is the L1 loss representation;

[0052] The fourth loss is calculated using the following formula:

[0053]

[0054] In the formula is the fourth loss value; is the identity loss corresponding to the B modality; is the identity loss corresponding to modality A;

[0055] The fifth loss is calculated using the following formula:

[0056]

[0057] In the formula is the fifth loss value; λ cyc is the cycle consistency loss weight; λ id is the identity loss weight;

[0058] The sixth loss is calculated using the following formula:

[0059]

[0060] In the formula is the sixth loss value; w(λ t ) is a predefined weighting function; It is represented by L2 loss; is the DiM network representation corresponding to the B-mode; is the DiM network representation corresponding to mode A;

[0061] Finally, the total loss function L during training is

[0062] The present invention also provides a system for implementing the cross-modal conversion method of medical images, comprising a data acquisition module, a data processing module, a pre-training module, a model construction module, a model training module and a cross-modal conversion module; the data acquisition module, the data processing module, the pre-training module, the model construction module, the model training module and the cross-modal conversion module are connected in series in sequence; the data acquisition module is used to acquire existing multi-modal medical image data and upload the data information to the data processing module; the data processing module is used to pre-process the acquired multi-modal medical image data according to the received data information to construct a training data set, and upload the data information to the pre-training module; the pre-training module is used to, according to the received data information and the acquired medical image data, for each type of medical image Image modality, using the training data set constructed in step S2 to train the corresponding autoencoder, and upload the data information to the model construction module; the model construction module is used to construct an initial model of cross-modal conversion of medical images based on the variational diffusion model in combination with the obtained autoencoder according to the received data information, and upload the data information to the model training module; the model training module is used to train the constructed initial model of cross-modal conversion of medical images according to the received data information using the constructed training data set, obtain the trained cross-modal conversion model of medical images, and upload the data information to the cross-modal conversion module; the cross-modal conversion module is used to perform actual cross-modal conversion of medical images according to the received data information and the obtained cross-modal conversion model of medical images.

[0063] The cross-modal conversion method and system for medical images provided by the present invention not only achieves cross-modal conversion of medical images through the training of autoencoders for each image modality and the construction and training of corresponding cross-modal conversion models for medical images, but also does not require paired data sets and has higher reliability and better accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 Schematic diagram of the process of the present invention.

[0065] Figure 2 Schematic diagram of qualitative comparison between the method of the present invention and existing solutions on the IXI dataset.

[0066] Figure 3 Schematic diagram of qualitative comparison between the method of the present invention and existing solutions on the Pelvis dataset.

[0067] Figure 4 Schematic diagram of the functional modules of the system of the present invention. DETAILED DESCRIPTION

[0068] like Figure 1 The figure shows a flow chart of the method of the present invention: The cross-modal conversion method of medical images disclosed in the present invention comprises the following steps:

[0069] S1. Acquire existing multimodal medical image data;

[0070] S2. Preprocessing the multimodal medical image data obtained in step S1 to construct a training dataset;

[0071] S3. Based on the medical image data obtained in step S1, for each type of medical image modality, the training dataset constructed in step S2 is used to train the corresponding autoencoder; specifically, the steps include:

[0072] Based on the medical image data obtained in step S1, for each type of medical image modality, the training dataset constructed in step S2 is used to train the corresponding autoencoder;

[0073] The trained autoencoder includes the encoder part of the autoencoder and the decoder part of the autoencoder;

[0074] In the specific implementation, the autoencoder is implemented using a VAE network, and the corresponding encoder part of the autoencoder and the decoder part of the autoencoder are obtained;

[0075] S4. Combining the autoencoder obtained in step S3, and based on the variational diffusion model, construct an initial model for cross-modal conversion of medical images; specifically, the steps include:

[0076] The constructed initial model for cross-modal conversion of medical images includes the encoder part of the autoencoder, the variational diffusion model backbone network, the decoder part of the autoencoder, the generator and the discriminator;

[0077] The processing of the initial model for cross-modal conversion of medical images includes:

[0078] The input unpaired data is (x a ,x b ), where x a is A modal data, xb is the B-mode data;

[0079] x b Through the first generator G B→A Generate A-modal synthetic image x' a ;x' a The first latent space representation z' is obtained through the encoder part of the autoencoder a ; z' a Latent space representation of the noisy image at time step t compared to the true B-modality data Perform splicing to obtain the first splicing feature Will Input the variational diffusion model backbone network to generate the first latent space representation of the predicted image Will Then the corresponding first generated image is obtained through the decoder part of the autoencoder; at the same time, x' a Also through the second generator G A→B Generate B-mode reconstructed image

[0080] x a Through the second generator G A→B Generate B-mode synthetic image x' b ;x' b The second latent space representation z' is obtained through the encoder part of the autoencoder b ; z' b The latent space representation of the noisy image at time step t with the true A modality data Perform splicing to obtain the second splicing feature Will Input the variational diffusion model backbone network to generate the second latent space representation of the predicted image Will Then the corresponding second generated image is obtained through the decoder part of the autoencoder; at the same time, x' b Also through the first generator G B→A Generate A-mode reconstructed image

[0081] In addition, the first discriminator D A With the first generator G B→A Correspondingly, the second discriminator D B With the second generator G A→B Corresponding; D A The input includes A modality synthetic image x' a and A modal data x a ;D A The output of is the judgment result of the input image, including whether the input image is an image synthesized by the generator or the input image is a real image;

[0082] In specific implementation, the processing of the variational diffusion model backbone network also includes the following steps:

[0083] The first concatenated feature of the input Get the first latent space representation z' a ; Latent space representation of the noisy image at time step t based on the real B-modality data From pure noise samples First, perform step-by-step denoising according to the following rules:

[0084]

[0085] In the formula is the potential space representation of the noise image of the B-modal data obtained in the s-th step at a time step of s; α s is the noise mean corresponding to the time step s; is the output of the DiM network; σ s is the noise standard deviation corresponding to the time step s; α t is the noise mean corresponding to the time step t; σ t is the noise standard deviation corresponding to the time step t; s is the current step number and T is the total number of sampling steps, t is the time step;

[0086] Second splicing feature for input Get the second latent space representation z' b ; Latent space representation of the noise image at time step t based on the real A modality data From pure noise samples First, perform step-by-step denoising according to the following rules:

[0087]

[0088] In the formula is the potential space representation of the noise image of the A modality data obtained in the s-th step at a time step of s;

[0089] Finally, the first latent space representation and the second latent space representation As the output of the variational diffusion model backbone network;

[0090] The structure of the variational diffusion model backbone network specifically includes the following parts:

[0091] The input image features are divided into several patches; then a 3×3 convolution is performed to obtain the convolution image features; in the obtained convolution image features, a padding flag to be learned is added at the end of each row and each column; in the convolution image features with the added padding flags, a time step padding flag and a corresponding padding flag to be learned are added to obtain the input features;

[0092] Process the input features through the Mamba network layer to obtain processed features;

[0093] For processing features, we first remove the added time step padding flags and the padding flags to be learned, then perform a 3×3 convolution, and then fuse the obtained patches to obtain the final output of the variational diffusion model backbone network.

[0094] The Mamba network layer processing process includes the following steps:

[0095] The Mamba network layer includes an input Mamba module and N Mamba processing modules. The input Mamba module and the N Mamba processing modules are connected in series, and the output of the i-th Mamba processing module and the output of the Ni-th Mamba processing module are superimposed and used as the input of the N-i+1-th Mamba processing module. The input Mamba module is used to learn the global information between different patches. The Mamba processing module is used to learn the global information between patches under different scanning modes. The Mamba network layer is implemented based on the state-space model (SSM) and has a linear recursive structure, thereby realizing sequence modeling. At the same time, the computational complexity of the Mamba network layer increases linearly or logarithmically with the sequence length, effectively alleviating the problem of quadratic computational complexity in the self-attention mechanism. This skip connection solves the problem that the training target is sensitive to low-level features.

[0096] The Mamba processing modules all have the same structure. The Mamba processing modules include a channel scanning module and a Mamba module. The channel scanning module is used to control the order in which input image features are input to the Mamba module, and the Mamba module is used to learn global information between different patches. The channel scanning module randomly selects one of the following four scanning modes and is used to control the order in which input image features are input to the Mamba module:

[0097] Method 1: The time step padding flag and the corresponding padding flag to be learned are added to the front of the first row of the convolution image feature with the padding flag added. The scanning order is: starting from the time step padding flag and the corresponding padding flag to be learned, starting from the first row, and scanning in order of rows;

[0098] Method 2: The time step padding flag and the corresponding padding flag to be learned are added to the front of the first row of the convolution image feature with the padding flag added. The scanning order is: starting from the time step padding flag and the corresponding padding flag to be learned, starting from the first column, and scanning in order of columns;

[0099] Method 3: The time step padding flag and the corresponding padding flag to be learned are added to the end of the last row of the convolution image features with the padding flag added. The scanning order is: starting from the time step padding flag and the corresponding padding flag to be learned, starting from the last row, and scanning in order of rows;

[0100] Method 4: The time step padding flag and the corresponding padding flag to be learned are added to the end of the last row of the convolution image feature with the padding flag added. The scanning order is: starting from the time step padding flag and the corresponding padding flag to be learned, starting from the last column, and scanning in column order;

[0101] S5. Using the training dataset constructed in step S2, the initial model of cross-modal conversion of medical images constructed in step S4 is trained to obtain a trained cross-modal conversion model of medical images;

[0102] In practice, the training includes the following steps:

[0103] The first loss is calculated using the following formula:

[0104]

[0105] In the formula is the first loss value; is the loss function corresponding to the discriminator; D B (x b ) is the discriminator corresponding to the B modality; is the loss function corresponding to the generator; G A→B (x a ) is the generator corresponding to the B mode;

[0106] The second loss is calculated using the following formula:

[0107]

[0108] In the formula is the second loss value; is the loss function corresponding to the discriminator; D A (x a ) is the discriminator corresponding to modality A; is the loss function corresponding to the generator; G B→A (x b ) is the generator corresponding to the A mode;

[0109] The third loss is calculated using the following formula:

[0110]

[0111] In the formula is the third loss value; is the cycle loss corresponding to mode A; is the cycle loss corresponding to mode B; || ||1 is the L1 loss; the fourth loss is calculated using the following formula:

[0112]

[0113] In the formula is the fourth loss value; is the identity loss corresponding to the B modality; is the identity loss corresponding to modality A;

[0114] The fifth loss is calculated using the following formula:

[0115]

[0116] In the formula is the fifth loss value; λ cyc is the cycle consistency loss weight; λ id is the identity loss weight;

[0117] The sixth loss is calculated using the following formula:

[0118]

[0119] In the formula is the sixth loss value; w(λ t ) is a predefined weighting function; It is represented by L2 loss; is the DiM network representation corresponding to the B-mode; is the DiM network representation corresponding to mode A;

[0120] Finally, the total loss function L during training is

[0121] S6. Using the cross-modal conversion model of medical images obtained in step S5, perform actual cross-modal conversion of medical images;

[0122] In specific implementation, the generator and discriminator used in the training process are no longer needed in actual application; the generator and discriminator are only used in the training process of the model.

[0123] The method of the present invention is further described below with reference to an embodiment:

[0124] The proposed method is compared with existing solutions on the multi-contrast MRI dataset (IXI) and the cross-modal MRI-CT dataset (Pelvis). Table 1 summarizes the experimental results of different synthesis methods for bidirectional conversion of T1 and PD modalities on the multi-contrast MRI IXI dataset. Table 2 summarizes the experimental results of different synthesis methods for MRI T2 to CT modality conversion on the cross-modal MRI-CT Pelvis dataset. The corresponding solutions are evaluated using the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) metrics.

[0125] Table 1 Schematic diagram of comparative experimental results of IXI dataset

[0126]

[0127] Table 2 Schematic diagram of comparative experimental results of Pelvis dataset

[0128]

[0129] It can be seen from Table 1 and Table 2 that the method of the present invention shows better performance than the existing GAN-based method and Diffusion-based method.

[0130] Table 3 summarizes the number of parameters, single epoch training time, and 3D image inference time of different synthesis methods on the IXI dataset:

[0131] Table 3 Comparison results of computational complexity

[0132]

[0133] It can be seen from Table 3 that the method of the present invention can better balance the synthesis effect and computational complexity, and is significantly better than the existing Diffusion-based method in terms of training time and inference time.

[0134] Figure 2 and Figure 3 The following is a table showing the qualitative comparison results of the method of the present invention with the existing solutions on the IXI dataset and the Pelvis dataset. Figure 2 and Figure 3 It can be seen that the method of the present invention can capture and generate more accurate anatomical texture information.

[0135] like Figure 4The figure shows a schematic diagram of the functional modules of the system of the present invention: the system disclosed in the present invention for realizing the cross-modal conversion method of the medical image comprises a data acquisition module, a data processing module, a pre-training module, a model construction module, a model training module and a cross-modal conversion module; the data acquisition module, the data processing module, the pre-training module, the model construction module, the model training module and the cross-modal conversion module are sequentially connected in series; the data acquisition module is used to acquire existing multi-modal medical image data and upload the data information to the data processing module; the data processing module is used to pre-process the acquired multi-modal medical image data according to the received data information to construct a training data set, and upload the data information to the pre-training module; the pre-training module is used to pre-process the acquired multi-modal medical image data according to the received data information to construct a training data set, and upload the data information to the pre-training module; the pre-training module is used to pre-process the acquired medical image data according to the received data information For each type of medical image modality, the training data set constructed in step S2 is used to train the corresponding autoencoder, and the data information is uploaded to the model construction module; the model construction module is used to construct an initial cross-modal conversion model of medical images based on the received data information and the obtained autoencoder based on the variational diffusion model, and upload the data information to the model training module; the model training module is used to train the constructed initial cross-modal conversion model of medical images based on the received data information and the constructed training data set, obtain the trained cross-modal conversion model of medical images, and upload the data information to the cross-modal conversion module; the cross-modal conversion module is used to perform actual cross-modal conversion of medical images based on the received data information and the obtained cross-modal conversion model of medical images.

Claims

1. A cross-modal conversion method for medical images, comprising the following steps: S1. Acquire existing multimodal medical image data; S2. Preprocessing the multimodal medical image data obtained in step S1 to construct a training dataset; S3. Based on the medical image data obtained in step S1, for each type of medical image modality, use the training dataset constructed in step S2 to train the corresponding autoencoder; S4. Combining the autoencoder obtained in step S3, and based on the variational diffusion model, construct an initial model for cross-modal conversion of medical images; S5. Using the training dataset constructed in step S2, the initial model of cross-modal conversion of medical images constructed in step S4 is trained to obtain a trained cross-modal conversion model of medical images; S6. Use the cross-modal conversion model of medical images obtained in step S5 to perform actual cross-modal conversion of medical images.

2. The cross-modal conversion method of medical images according to claim 1, characterized in that Step S3, based on the medical image data obtained in step S1, uses the training dataset constructed in step S2 for each type of medical image modality to train the corresponding autoencoder, specifically including the following steps: Based on the medical image data obtained in step S1, for each type of medical image modality, the training dataset constructed in step S2 is used to train the corresponding autoencoder; The trained autoencoder includes the encoder part of the autoencoder and the decoder part of the autoencoder.

3. The cross-modal conversion method of medical images according to claim 2, characterized in that In step S4, the autoencoder obtained in step S3 is combined to construct an initial model for cross-modal conversion of medical images based on the variational diffusion model, which specifically includes the following steps: The constructed initial model for cross-modal conversion of medical images includes the encoder part of the autoencoder, the variational diffusion model backbone network, the decoder part of the autoencoder, the generator and the discriminator; The processing of the initial model for cross-modal conversion of medical images includes: The input unpaired data is (x a ,x b ), where x a is A modal data, x b is the B-mode data; x b Through the first generator G B→A Generate A-modal synthetic image x' a ;x' a The first latent space representation z' is obtained through the encoder part of the autoencoder a ; z' a Latent space representation of the noisy image at time step t compared to the true B-modality data Perform splicing to obtain the first splicing feature Will Input the variational diffusion model backbone network to generate the first latent space representation of the predicted image Will Then the corresponding first generated image is obtained through the decoder part of the autoencoder; at the same time, x' a Also through the second generator G A→B Generate B-mode reconstructed image x a Through the second producer G A→B Generate B-mode synthetic image x' b ;x' b The second latent space representation z' is obtained through the encoder part of the autoencoder b ; z' b The latent space representation of the noisy image at time step t with the true A modality data Perform splicing to obtain the second splicing feature Will Input the variational diffusion model backbone network to generate the second latent space representation of the predicted image Will Then the corresponding second generated image is obtained through the decoder part of the autoencoder; at the same time, x' b Also through the first generator G B→A Generate A-mode reconstructed image In addition, the first discriminator D A With the first generator G B→A Correspondingly, the second discriminator D B With the second generator G A→B Corresponding; D A The input includes A modality synthetic image x' a and A modal data x a ;D A The output of is the judgment result of the input image, including whether the input image is an image synthesized by the generator or the input image is a real image.

4. The cross-modal conversion method of medical images according to claim 3, characterized in that The processing of the variational diffusion model backbone network also includes the following steps: The first concatenated feature of the input Get the first latent space representation z' a ; Latent space representation of the noisy image at time step t based on the real B-modality data From pure noise samples First, perform step-by-step denoising according to the following rules: In the formula is the potential space representation of the noise image of the B-modal data obtained in the s-th step at a time step of s; α s is the noise mean corresponding to the time step s; is the output of the DiM network; σ s is the noise standard deviation corresponding to the time step s; α t is the noise mean corresponding to the time step t; σ t is the noise standard deviation corresponding to the time step t; s is the current step number and T is the total number of sampling steps, t is the time step; Second splicing feature for input Get the second latent space representation z' b ; Latent space representation of the noise image at time step t based on the real A modality data From pure noise samples First, perform step-by-step denoising according to the following rules: In the formula is the potential space representation of the noise image of the A-modal data obtained in the s-th step with a time step of s, and the rest is the same as above; Finally, the first latent space representation and the second latent space representation As the output of the variational diffusion model backbone network.

5. The cross-modal conversion method of medical images according to claim 4, characterized in that The structure of the variational diffusion model backbone network specifically includes the following parts: The input image features are divided into several patches; then a 3×3 convolution is performed to obtain the convolution image features; in the obtained convolution image features, a padding flag to be learned is added at the end of each row and each column; in the convolution image features with the added padding flags, a time step padding flag and a corresponding padding flag to be learned are added to obtain the input features; Process the input features through the Mamba network layer to obtain processed features; For the processing features, we first delete the added time step padding flag and the padding flag to be learned, then perform 3×3 convolution processing, and then fuse the obtained patches to obtain the final output of the variational diffusion model backbone network.

6. The cross-modal conversion method of medical images according to claim 5, characterized in that The Mamba network layer processing process includes the following steps: The Mamba network layer includes an input Mamba module and N Mamba processing modules. The input Mamba module and the N Mamba processing modules are connected in series. The output of the i-th Mamba processing module and the output of the Ni-th Mamba processing module are added together and used as the input of the N-i+1-th Mamba processing module. The input Mamba module is used to learn the global information between different patches. The Mamba processing module is used to learn the global information between patches under different scanning modes. The Mamba processing modules all have the same structure. The Mamba processing modules include a channel scanning module and a Mamba module. The channel scanning module is used to control the order in which input image features are input to the Mamba module, and the Mamba module is used to learn global information between different patches. The channel scanning module randomly selects one of the following four scanning modes and is used to control the order in which input image features are input to the Mamba module: Method 1: The time step padding flag and the corresponding padding flag to be learned are added to the front of the first row of the convolution image feature with the padding flag added. The scanning order is: starting from the time step padding flag and the corresponding padding flag to be learned, starting from the first row, and scanning in order of rows; Method 2: The time step padding flag and the corresponding padding flag to be learned are added to the front of the first row of the convolution image feature with the padding flag added. The scanning order is: starting from the time step padding flag and the corresponding padding flag to be learned, starting from the first column, and scanning in order of columns; Method 3: The time step padding flag and the corresponding padding flag to be learned are added to the end of the last row of the convolution image features with the padding flag added. The scanning order is: starting from the time step padding flag and the corresponding padding flag to be learned, starting from the last row, and scanning in order of rows; Method 4: The time step padding flag and the corresponding padding flag to be learned are added to the end of the last row of the convolution image feature with the padding flag added. The scanning order is: starting from the time step padding flag and the corresponding padding flag to be learned, starting from the last column, and scanning in order of columns.

7. The cross-modal conversion method of medical images according to claim 6, characterized in that The training described in step S5 specifically includes the following steps: The first loss is calculated using the following formula: In the formula is the first loss value; is the loss function corresponding to the discriminator; D B (x b ) is the discriminator corresponding to the B modality; is the loss function corresponding to the generator; G A→B (x a ) is the generator corresponding to the B mode; The second loss is calculated using the following formula: In the formula is the second loss value; is the loss function corresponding to the discriminator; D A (x a ) is the discriminator corresponding to modality A; is the loss function corresponding to the generator; G B→A (x b ) is the generator corresponding to the A mode; The third loss is calculated using the following formula: In the formula is the third loss value; is the cycle loss corresponding to mode A; is the cycle loss representation corresponding to the B mode; || ||1 is the L1 loss representation; The fourth loss is calculated using the following formula: In the formula is the fourth loss value; is the identity loss corresponding to the B modality; is the identity loss corresponding to modality A; The fifth loss is calculated using the following formula: In the formula is the fifth loss value; λ cyc is the cycle consistency loss weight; λ id is the identity loss weight; The sixth loss is calculated using the following formula: In the formula is the sixth loss value; w(λ t ) is a predefined weighting function; It is represented by L2 loss; is the DiM network representation corresponding to the B-mode; is the DiM network representation corresponding to mode A; Finally, the total loss function L during training is 8. A system for implementing the cross-modal conversion method of medical images according to any one of claims 1 to 7, characterized in that It includes a data acquisition module, a data processing module, a pre-training module, a model construction module, a model training module and a cross-modal conversion module; the data acquisition module, the data processing module, the pre-training module, the model construction module, the model training module and the cross-modal conversion module are connected in series in sequence; the data acquisition module is used to acquire existing multimodal medical image data and upload the data information to the data processing module; The data processing module is used to pre-process the acquired multimodal medical image data based on the received data information to construct a training data set and upload the data information to the pre-training module; The pre-training module is used to train the corresponding autoencoder for each type of medical image modality based on the received data information and the acquired medical image data using the training data set constructed in step S2, and upload the data information to the model construction module; the model construction module is used to construct an initial model of cross-modal conversion of medical images based on the variational diffusion model according to the received data information and the obtained autoencoder, and upload the data information to the model training module; The model training module is used to train the constructed cross-modal conversion initial model of medical images using the constructed training data set based on the received data information, obtain the trained cross-modal conversion model of medical images, and upload the data information to the cross-modal conversion module; The cross-modal conversion module is used to perform actual cross-modal conversion of medical images based on the received data information and the cross-modal conversion model of medical images.

Images