Medical image modality conversion method, system and storage medium based on contrast learning

Abstract

The application discloses a medical image modality conversion method and system based on contrast learning and a storage medium. The method comprises the following steps: acquiring source target image data; analyzing the source target image data to obtain content features and modality features of the source target image data; converting the modality features of the source target image data into target modality features; and merging and decoding the target modality features and the content features of the source target image data to obtain target modality images. The application converts from source modality images to target modality images by using a contrast learning-based method, thereby avoiding repeated scanning of the same anatomical structure of the same individual. The application converts in the modality space, can avoid ambiguity in the image space conversion of the previous method, and can further improve the conversion efficiency and precision.

Description

Technical Field

[0001] This application relates to the fields of data processing and data transmission, and more specifically, to a method, system, and storage medium for medical image modality conversion based on contrastive learning. Background Technology

[0002] Medical imaging refers to the techniques and processes used to obtain images of the internal structures of the human body or parts of the human body non-invasively for medical or research purposes. Common medical imaging methods include X-rays, computed tomography (CT), ultrasound, magnetic resonance imaging (MRI), and positron emission tomography (PET). Different modalities of medical imaging data have their own advantages and disadvantages in disease diagnosis, treatment, and prognosis assessment. Clinically, multiple modalities of images may be needed for some patients for a more comprehensive examination; however, a large number of repetitive imaging scans can impose an economic and psychological burden on patients. Because medical images of different modalities of the same patient are correlated, and modalities themselves have specific attributes, medical image modality conversion has become a solution for obtaining multiple modalities of images. By training a modality-to-modality conversion model, it is possible to obtain images of another modality after acquiring medical images of a patient in a specific modality through modality conversion.

[0003] Early medical image modality transformation (MAD) methods largely employed traditional approaches. These methods relied on specific prior information to construct a mapping between the source and target modalities, establishing a relationship between images or extracted features. However, this mapping was often difficult to generalize to other modalities due to their respective prior assumptions. In recent years, the rise of deep learning and its widespread application in various image processing fields, including natural image style transfer, has brought new insights into medical image MAD. In image style transfer, images can retain their original content while being assigned the style of other images by the network. This is very similar to medical image MAD, which also requires preserving the anatomical structures of the original images and combining them with the imaging attributes of the target modality to synthesize new transformed data. Deep learning-based medical image MAD methods include U-shaped deep convolutional neural networks, which typically use supervised learning to learn the mapping from source to target modalities. Another approach is based on generative adversarial networks (GANs), which can achieve source-to-target modality transformation without labeled data pairs, using unsupervised learning.

[0004] The two types of medical image modality conversion methods currently used mostly employ direct image-to-image conversion. For example, in the U-net model, the input source modality image is encoded and decoded to obtain the target modality image. The generated target image is constrained to be consistent with the labeled images in the training set using a norm, i.e., image-to-image consistency. In GAN-based models, the goal is also to generate a target image that can fool the discriminator, misleading it into misidentifying it as real rather than generated image data. This type of method also directly learns end-to-end image-to-image conversion. This image-to-image conversion has two drawbacks: first, the image space is a low-dimensional distribution within a high-dimensional space, making it relatively difficult to learn point-to-point mappings directly in a high-dimensional space; second, because images contain both content and style information, which are intertwined and difficult to separate, the original style information can easily be mixed in during style conversion.

[0005] Therefore, the existing technology has defects and urgently needs improvement. Summary of the Invention

[0006] In view of the above problems, the purpose of this invention is to provide a method, system, and storage medium for medical image modality transformation based on contrastive learning. This invention achieves effective mapping learning with limited training data by learning the mapping between modal features. Furthermore, by decoupling the initial image, this invention effectively avoids the retention of modal attributes of the source image in the target image.

[0007] The first aspect of this invention provides a medical image modality conversion method based on contrastive learning, comprising:

[0008] Acquire source target image data;

[0009] Based on the analysis of the source target image data, the content features and modal features of the source target image data are obtained;

[0010] The modal features of the source target image data are converted into target modal features;

[0011] The target modal features are merged and decoded with the content features of the source target image data to obtain the target modal image.

[0012] This plan also includes:

[0013] Construct an initial medical image modality conversion model;

[0014] Acquire paired CT-MRI image data;

[0015] The paired CT-MRI image data are input into the initial medical image modality conversion model for optimization training to obtain a preset medical image modality conversion model.

[0016] In this scheme, the step of inputting the paired CT-MRI image data into the initial medical image modality conversion model for optimization training specifically involves:

[0017] Multiple loss functions are obtained by calculating the paired CT-MRI image data according to a preset method;

[0018] The weight scores of the multiple loss functions are obtained by multiplying the weight coefficients of the corresponding loss functions.

[0019] The weight scores of the multiple loss functions are summed to obtain the total loss function;

[0020] The initial medical image modality conversion model is optimized and trained based on the total loss function.

[0021] The multiple loss functions include structural contrast loss function, modal contrast loss function, and image consistency loss function.

[0022] In this solution, the step of analyzing the source target image data to obtain the content features and modal features of the source target image data includes:

[0023] Decouple the source target image data to obtain the content features and modal features of the source target image data;

[0024] The decoupling is expressed by the following formula:

[0025] [C x M x ] = Encoder(x)

[0026] Where x is the input source modal image, Encoder is the encoder to be trained, and C x M represents the content features obtained from image x after being encoded. x The obtained modal features.

[0027] In this scheme, the conversion of the modal features of the source target image data into target modal features includes:

[0028] The modal features of the source target image data are transformed using a modal transformation network to obtain the target modal features;

[0029] The feature transformation is expressed by the following formula:

[0030] M y =Transfer(M x ),

[0031] Where Transfer is a mode converter, My M represents the modal features corresponding to the target mode. x The modal features of the source modal image.

[0032] In this scheme, the step of merging and decoding the target modal features with the content features of the source target image data to obtain the target modal image includes:

[0033] The content features of the source target image data are merged with the target modal features to obtain merged feature data;

[0034] The merged feature data is decoded to obtain the target modal image;

[0035] The merging and decoding is expressed by the following formula:

[0036] y = Decoder(M y C x )

[0037] Where x is the input source modal image, y is the generated target modal image, and Decoder is the decoder.

[0038] A second aspect of the present invention provides a medical image modality conversion system based on contrastive learning, comprising a memory and a processor. The memory includes a program for a medical image modality conversion method based on contrastive learning. When the program for the medical image modality conversion method based on contrastive learning is executed by the processor, it performs the following steps:

[0039] Acquire source target image data;

[0040] Based on the analysis of the source target image data, the content features and modal features of the source target image data are obtained;

[0041] The modal features of the source target image data are converted into target modal features;

[0042] The target modal features are merged and decoded with the content features of the source target image data to obtain the target modal image.

[0043] This plan also includes:

[0044] Construct an initial medical image modality conversion model;

[0045] Acquire paired CT-MRI image data;

[0046] The paired CT-MRI image data are input into the initial medical image modality conversion model for optimization training to obtain a preset medical image modality conversion model.

[0047] In this scheme, the step of inputting the paired CT-MRI image data into the initial medical image modality conversion model for optimization training specifically involves:

[0048] Multiple loss functions are obtained by calculating the paired CT-MRI image data according to a preset method;

[0049] The weight scores of the multiple loss functions are obtained by multiplying the weight coefficients of the corresponding loss functions.

[0050] The weight scores of the multiple loss functions are summed to obtain the total loss function;

[0051] The initial medical image modality conversion model is optimized and trained based on the total loss function.

[0052] The multiple loss functions include structural contrast loss function, modal contrast loss function, and image consistency loss function.

[0053] In this solution, the step of analyzing the source target image data to obtain the content features and modal features of the source target image data includes:

[0054] Decouple the source target image data to obtain the content features and modal features of the source target image data;

[0055] The decoupling is expressed by the following formula:

[0056] [C x M x ] = Encoder(x)

[0057] Where x is the input source modal image, Encoder is the encoder to be trained, and C x M represents the content features obtained from image x after being encoded. x The obtained modal features.

[0058] In this scheme, the conversion of the modal features of the source target image data into target modal features includes:

[0059] The modal features of the source target image data are transformed using a modal transformation network to obtain the target modal features;

[0060] The feature transformation is expressed by the following formula:

[0061] M y =Transfer(M x ),

[0062] Where Transfer is a mode converter, M yM represents the modal features corresponding to the target mode. x The modal features of the source modal image.

[0063] In this scheme, the step of merging and decoding the target modal features with the content features of the source target image data to obtain the target modal image includes:

[0064] The content features of the source target image data are merged with the target modal features to obtain merged feature data;

[0065] The merged feature data is decoded to obtain the target modal image;

[0066] The merging and decoding is expressed by the following formula:

[0067] y = Decoder(M y C x )

[0068] Where x is the input source modal image, y is the generated target modal image, and Decoder is the decoder.

[0069] A third aspect of the present invention provides a computer-readable storage medium including a medical image modality conversion method program based on contrastive learning, wherein when the medical image modality conversion method program based on contrastive learning is executed by a processor, it implements the steps of the medical image modality conversion method based on contrastive learning as described in any of the preceding claims.

[0070] This invention discloses a medical image modality conversion method, system, and storage medium based on contrastive learning. The method includes: acquiring source and target image data; analyzing the source and target image data to obtain content features and modal features; converting the modal features of the source and target image data into target modal features; and merging and decoding the target modal features with the content features of the source and target image data to obtain a target modal image. This invention uses a contrastive learning-based method to convert source and target modal images, thereby avoiding repeated scanning of the same anatomical structure of the same individual. This invention performs the conversion in modal space, avoiding the ambiguity of previous methods in image space conversion, and further improving conversion efficiency and accuracy. Attached Figure Description

[0071] Figure 1 A flowchart of the medical image modality conversion method based on contrastive learning of the present invention is shown;

[0072] Figure 2 A flowchart of the pre-set medical image modality conversion model training method of the present invention is shown;

[0073] Figure 3 A flowchart of the target modal image acquisition method of the present invention is shown;

[0074] Figure 4 A block diagram of the medical image modality conversion system based on contrastive learning of the present invention is shown. Detailed Implementation

[0075] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0076] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0077] Figure 1 A flowchart of the medical image modality conversion method based on contrastive learning of the present invention is shown.

[0078] like Figure 1 As shown, this invention discloses a medical image modality conversion method based on contrastive learning, including:

[0079] S102, acquire source target image data;

[0080] S104, Analyze the source target image data to obtain the content features and modal features of the source target image data;

[0081] S106, convert the modal features of the source target image data into target modal features;

[0082] S108, the target modal features and the content features of the source target image data are merged and decoded to obtain the target modal image.

[0083] According to embodiments of the present invention, this patent proposes a medical image modality conversion method based on contrastive learning. It utilizes the structural similarity between different modalities of data within the same anatomical structure and the modal similarity between different anatomical structures within the same modality to construct a contrastive loss function for inter-modal mapping learning, ultimately obtaining the target modality image. Specifically, by pre-setting a medical image modality conversion model, the input source-target image data is first decoupled into content features and modal features. Then, by learning a modality-to-modality mapping network, the source modal features of the source-target image data are converted into target modal features. Finally, the target modal features and the content features of the source-target image data are re-encoded to obtain the target image. This patent uses CT modality conversion to MRI as an example for illustration, but in practical applications, it is not limited to CT to MRI conversion; it can also be MRI to CT, MRI T1 contrast to T2 contrast, PET to CT, and other conversion methods.

[0084] In practical applications, the first step is to prepare N pairs of pre-matched MRI-CT image training data. Where, x i For CT images, y i For x i With well-matched MRI data, the goal of modality transformation is to learn a modality from CT data x i To MRI data y i The mapping network enables the input of new CT image data. The corresponding MRI data can be obtained through this mapping network. This avoids repeated MRI scans, saving costs. This can be expressed as a formula:

[0085]

[0086] in The CT image to be converted for The input is the MRI image generated by the transfer network, where Encoder(·) is the attribute encoding network, Transfer(·) is the modality transfer network, and Decoder(·) is the attribute decoding network.

[0087] According to an embodiment of the present invention, it further includes:

[0088] Construct an initial medical image modality conversion model;

[0089] Acquire paired CT-MRI image data;

[0090] The paired CT-MRI image data are input into the initial medical image modality conversion model for optimization training to obtain a preset medical image modality conversion model.

[0091] It should be noted that the paired CT-MRI image data pairs are used for network training. Each image data pair contains several images of the same anatomical structure and several images of different anatomical structures. The initial medical image modality conversion model comprises three sub-networks: an attribute encoding network, a modality conversion network, and an attribute decoding network. The three sub-networks are optimized and trained by calculating three loss functions: structural contrast loss function, modality contrast loss function, and image consistency loss function, thereby obtaining the preset medical image modality conversion model.

[0092] Figure 2 A flowchart of the pre-set medical image modality conversion model training method of the present invention is shown.

[0093] like Figure 2 As shown in the embodiment of the present invention, the step of inputting the paired CT-MRI image data into the initial medical image modality conversion model for optimization training specifically includes:

[0094] S202, calculate multiple loss functions based on the paired CT-MRI image data according to a preset method;

[0095] S204, Multiply the plurality of loss functions by the weight coefficients of the corresponding loss functions respectively to obtain the weight scores of the plurality of loss functions;

[0096] S206, The weight scores of the multiple loss functions are summed to obtain the total loss function;

[0097] S208, Optimize and train the initial medical image modality conversion model according to the total loss function;

[0098] The multiple loss functions include structural contrast loss function, modal contrast loss function, and image consistency loss function.

[0099] It should be noted that the multiple loss functions include the structural contrast loss function, the modal contrast loss function, and the image consistency loss function. Different loss functions assign different image weights to the model training and optimization. Therefore, after calculating the three loss functions, the total loss function is obtained by combining the weight coefficients of their corresponding loss functions. The calculation method of the total loss function is expressed by the formula:

[0100] L total =λ1*L cc +λ2*L cm +λ3*L rec

[0101] Among them, L totalLet L be the total loss function. cc Let L be the structural contrastive loss function, λ1 be the weight coefficients of the structural contrastive loss function, and L be the weight coefficients of the structural contrastive loss function. cm Let L be the modal contrast loss function, λ² be the weighting coefficients of the modal contrast loss function, and L be the modal contrast loss function. rec Let λ be the image consistency loss function, and λ3 be the weighting coefficient of the image consistency loss function.

[0102] According to an embodiment of the present invention, the step of analyzing the source target image data to obtain the content features and modal features of the source target image data includes:

[0103] Decouple the source target image data to obtain the content features and modal features of the source target image data;

[0104] The decoupling is expressed by the following formula:

[0105] [C x M x ] = Encoder(x)

[0106] Where x is the input source modal image, Encoder is the encoder to be trained, and C x M represents the content features obtained from image x after being encoded. x The obtained modal features.

[0107] It should be noted that after acquiring the source target image data, the acquired source target image data x is first input into the encoder network attribute encoding network Encoder(·) through a preset medical image modality transformation model. This decouples the input source target image data into corresponding content features C. x and modal features M x .

[0108] According to an embodiment of the present invention, converting the modal features of the source target image data into target modal features includes:

[0109] The modal features of the source target image data are transformed using a modal transformation network to obtain the target modal features;

[0110] The feature transformation is expressed by the following formula:

[0111] M y =Transfer(M x ),

[0112] Where Transfer is a mode converter, M y M represents the modal features corresponding to the target mode. x The modal features of the source modal image.

[0113] It should be noted that after obtaining the content features and modal features of the source target image data, the modal features of the source target image data are input into the modal conversion network Transfer(·). The modal features of the source target image data are processed by the modal conversion network, and feature conversion is performed on the modal features of the source target image data to convert the modal features of the source target image data into target modal features.

[0114] Figure 3 A flowchart of the target modal image acquisition method of the present invention is shown.

[0115] like Figure 3 As shown in the embodiment of the present invention, the step of merging and decoding the target modal features with the content features of the source target image data to obtain the target modal image includes:

[0116] S302, merge the content features of the source target image data with the target modal features to obtain merged feature data;

[0117] S304, Decode the merged feature data to obtain the target modal image;

[0118] The merging and decoding is expressed by the following formula:

[0119] y = Decoder(M y C x )

[0120] Where x is the input source modal image, y is the generated target modal image, and Decoder is the decoder.

[0121] It should be noted that after obtaining the target modal image, the content features of the target modal image and the source target image data are used as input to the attribute decoding network Decoder(·). The target modal image is then re-encoded, combined and decoded by the attribute decoding network.

[0122] According to an embodiment of the present invention, the method for calculating the structural contrast loss function is as follows:

[0123] The calculation method of the structural contrast loss function is expressed by the following formula:

[0124]

[0125] Where u∈P means that u and x belong to the same anatomical structure P, u is one of the images corresponding to anatomical structure P, v∈Q means that Q is another anatomical structure different from P, and v is one of the images corresponding to anatomical structure Q. Sim(·,·) is used to measure the similarity of input variables, and a cosine function can be used. τ is a temperature coefficient used to adjust the distance between samples.

[0126] It should be noted that the image structure and content of different individuals with the same anatomical structure are similar, meaning their content features are similar. However, the image content corresponding to different anatomical structures is different. Therefore, a contrastive loss function can be defined to increase the similarity of the same anatomical structure in the content encoding space and increase the distance and difference between different anatomical structures in the content encoding space. This can be calculated using a structural contrastive loss function. This loss function maximizes the similarity of images belonging to the same anatomical structure projected into the content space, while minimizing the similarity of content features corresponding to different anatomical structures.

[0127] According to an embodiment of the present invention, the method for calculating the modal contrast loss function is as follows:

[0128] The modal contrast loss function is calculated using the following formula:

[0129]

[0130] Where M1 is the source mode, M2 is the target mode, and x * x belongs to the M1 mode, y belongs to the M2 mode, and τ is a temperature coefficient used to adjust the distance between samples.

[0131] It should be noted that modal features extracted from image data of the same modality are similar, while modal features extracted from different modalities are different. These differences can be calculated using a modal contrast loss function. This loss function can narrow the gap between modal features extracted from images of the same modality and widen the gap between modal features extracted from images of different modalities.

[0132] According to an embodiment of the present invention, the method for calculating the modal contrast loss function is as follows:

[0133] The method for calculating the image consistency loss function is expressed by the following formula:

[0134] L rec =‖y * -y‖1

[0135] Where y * This is another modal image data that is known to be paired with x.

[0136] It should be noted that,

[0137] When the source target image data x is input, the network will eventually generate the corresponding target modal image y. This generated data y must be compared with x in the training dataset. i Matching y i To maintain consistency, the L1 norm can be used for consistency constraints.

[0138] Figure 4 A block diagram of the medical image modality conversion system based on contrastive learning of the present invention is shown.

[0139] like Figure 4 As shown, a second aspect of the present invention provides a medical image modality conversion system 4 based on contrastive learning, including a memory 41 and a processor 42. The memory includes a medical image modality conversion method program based on contrastive learning. When the contrastive learning-based medical image modality conversion method program is executed by the processor, it performs the following steps:

[0140] Acquire source target image data;

[0141] Based on the analysis of the source target image data, the content features and modal features of the source target image data are obtained;

[0142] The modal features of the source target image data are converted into target modal features;

[0143] The target modal features are merged and decoded with the content features of the source target image data to obtain the target modal image.

[0144] According to embodiments of the present invention, this patent proposes a medical image modality conversion method based on contrastive learning. It utilizes the structural similarity between different modalities of data within the same anatomical structure and the modal similarity between different anatomical structures within the same modality to construct a contrastive loss function for inter-modal mapping learning, ultimately obtaining the target modality image. Specifically, by pre-setting a medical image modality conversion model, the input source-target image data is first decoupled into content features and modal features. Then, by learning a modality-to-modality mapping network, the source modal features of the source-target image data are converted into target modal features. Finally, the target modal features and the content features of the source-target image data are re-encoded to obtain the target image. This patent uses CT modality conversion to MRI as an example for illustration, but in practical applications, it is not limited to CT to MRI conversion; it can also be MRI to CT, MRI T1 contrast to T2 contrast, PET to CT, and other conversion methods.

[0145] In practical applications, the first step is to prepare N pairs of pre-matched MRI-CT image training data. Where, x i For CT images, y i For x i With well-matched MRI data, the goal of modality transformation is to learn a modality from CT data x i To MRI data y i The mapping network enables the input of new CT image data. The corresponding MRI data can be obtained through this mapping network. This avoids repeated MRI scans, saving costs. This can be expressed as a formula:

[0146]

[0147] in The CT image to be converted for The input is the MRI image generated by the transfer network, where Encoder(·) is the attribute encoding network, Transfer(·) is the modality transfer network, and Decoder(·) is the attribute decoding network.

[0148] According to an embodiment of the present invention, it further includes:

[0149] Construct an initial medical image modality conversion model;

[0150] Acquire paired CT-MRI image data;

[0151] The paired CT-MRI image data are input into the initial medical image modality conversion model for optimization training to obtain a preset medical image modality conversion model.

[0152] It should be noted that the paired CT-MRI image data pairs are used for network training. Each image data pair contains several images of the same anatomical structure and several images of different anatomical structures. The initial medical image modality conversion model comprises three sub-networks: an attribute encoding network, a modality conversion network, and an attribute decoding network. These three sub-networks are optimized and trained by calculating three loss functions: structural contrast loss function, modality contrast loss function, and image consistency loss function, thereby obtaining the preset medical image modality conversion model.

[0153] According to an embodiment of the present invention, the step of inputting the paired CT-MRI image data into the initial medical image modality conversion model for optimization training specifically includes:

[0154] Multiple loss functions are obtained by calculating the paired CT-MRI image data according to a preset method;

[0155] The weight scores of the multiple loss functions are obtained by multiplying each of the loss functions by its corresponding weight coefficient.

[0156] The weight scores of the multiple loss functions are summed to obtain the total loss function;

[0157] The initial medical image modality conversion model is optimized and trained based on the total loss function.

[0158] The multiple loss functions include structural contrast loss function, modal contrast loss function, and image consistency loss function.

[0159] It should be noted that the multiple loss functions include the structural contrast loss function, the modal contrast loss function, and the image consistency loss function. Different loss functions assign different image weights to the model training and optimization. Therefore, after calculating the three loss functions, the total loss function is obtained by combining the weight coefficients of their corresponding loss functions. The calculation method of the total loss function is expressed by the formula:

[0160] L total =λ1*L cc +λ2*L cm +λ3*L rec

[0161] Among them, L total Let L be the total loss function. cc Let L be the structural contrastive loss function, λ1 be the weight coefficients of the structural contrastive loss function, and L be the weight coefficients of the structural contrastive loss function. cm Let L be the modal contrast loss function, λ² be the weighting coefficients of the modal contrast loss function, and L be the modal contrast loss function. rec Let λ be the image consistency loss function, and λ3 be the weighting coefficient of the image consistency loss function.

[0162] According to an embodiment of the present invention, the step of analyzing the source target image data to obtain the content features and modal features of the source target image data includes:

[0163] Decouple the source target image data to obtain the content features and modal features of the source target image data;

[0164] The decoupling is expressed by the following formula:

[0165] [C x M x ] = Encoder(x)

[0166] Where x is the input source modal image, Encoder is the encoder to be trained, and C x M represents the content features obtained from image x after being encoded. x The obtained modal features.

[0167] It should be noted that after acquiring the source target image data, the acquired source target image data x is first input into the encoder network attribute encoding network Encoder(·) through a preset medical image modality transformation model. This decouples the input source target image data into corresponding content features C. x and modal features M x .

[0168] According to an embodiment of the present invention, converting the modal features of the source target image data into target modal features includes:

[0169] The modal features of the source target image data are transformed using a modal transformation network to obtain the target modal features;

[0170] The feature transformation is expressed by the following formula:

[0171] M y =Transfer(M x ),

[0172] Where Transfer is a mode converter, M y M represents the modal features corresponding to the target mode. x The modal features of the source modal image.

[0173] It should be noted that after obtaining the content features and modal features of the source target image data, the modal features of the source target image data are input into the modal conversion network Transfer(·). The modal features of the source target image data are processed by the modal conversion network, and feature conversion is performed on the modal features of the source target image data to convert the modal features of the source target image data into target modal features.

[0174] According to an embodiment of the present invention, the step of merging and decoding the target modal features with the content features of the source target image data to obtain the target modal image includes:

[0175] The content features of the source target image data are merged with the target modal features to obtain merged feature data;

[0176] The merged feature data is decoded to obtain the target modal image;

[0177] The merging and decoding is expressed by the following formula:

[0178] y = Decoder(M y C x )

[0179] Where x is the input source modal image, y is the generated target modal image, and Decoder is the decoder.

[0180] It should be noted that after obtaining the target modal image, the content features of the target modal image and the source target image data are used as input to the attribute decoding network Decoder(·). The target modal image is then re-encoded, combined and decoded by the attribute decoding network.

[0181] According to an embodiment of the present invention, the method for calculating the structural contrast loss function is as follows:

[0182] The calculation method of the structural contrast loss function is expressed by the following formula:

[0183]

[0184] Where u∈P means that u and x belong to the same anatomical structure P, u is one of the images corresponding to anatomical structure P, v∈Q means that Q is another anatomical structure different from P, and v is one of the images corresponding to anatomical structure Q. Sim(·,·) is used to measure the similarity of input variables, and a cosine function can be used. τ is a temperature coefficient used to adjust the distance between samples.

[0185] It should be noted that the image structure and content of different individuals with the same anatomical structure are similar, meaning their content features are similar. However, the image content corresponding to different anatomical structures is different. Therefore, a contrastive loss function can be defined to increase the similarity of the same anatomical structure in the content encoding space and increase the distance and difference between different anatomical structures in the content encoding space. This can be calculated using a structural contrastive loss function. This loss function maximizes the similarity of images belonging to the same anatomical structure projected into the content space, while minimizing the similarity of content features corresponding to different anatomical structures.

[0186] According to an embodiment of the present invention, the method for calculating the modal contrast loss function is as follows:

[0187] The modal contrast loss function is calculated using the following formula:

[0188]

[0189] Where M1 is the source mode, M2 is the target mode, and x * x belongs to the M1 mode, y belongs to the M2 mode, and τ is a temperature coefficient used to adjust the distance between samples.

[0190] It should be noted that modal features extracted from image data of the same modality are similar, while modal features extracted from different modalities are different. These differences can be calculated using a modal contrast loss function. This loss function can narrow the gap between modal features extracted from images of the same modality and widen the gap between modal features extracted from images of different modalities.

[0191] According to an embodiment of the present invention, the method for calculating the modal contrast loss function is as follows:

[0192] The method for calculating the image consistency loss function is expressed by the following formula:

[0193] Lrec =‖y * -y‖1

[0194] Where y * This is another modal image data that is known to be paired with x.

[0195] It should be noted that,

[0196] When the source target image data x is input, the network will eventually generate the corresponding target modal image y. This generated data y must be compared with x in the training dataset. i Matching y i To maintain consistency, the L1 norm can be used for consistency constraints.

[0197] A third aspect of the present invention provides a computer-readable storage medium including a medical image modality conversion method program based on contrastive learning, wherein when the medical image modality conversion method program based on contrastive learning is executed by a processor, it implements the steps of the medical image modality conversion method based on contrastive learning as described in any of the preceding claims.

[0198] This invention discloses a medical image modality conversion method, system, and storage medium based on contrastive learning. The method includes: acquiring source and target image data; analyzing the source and target image data to obtain content features and modal features; converting the modal features of the source and target image data into target modal features; and merging and decoding the target modal features with the content features of the source and target image data to obtain a target modal image. This invention uses a contrastive learning-based method to convert source and target modal images, thereby avoiding repeated scanning of the same anatomical structure of the same individual. This invention performs the conversion in modal space, avoiding the ambiguity of previous methods in image space conversion, and further improving conversion efficiency and accuracy.

[0199] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0200] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0201] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0202] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0203] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

Claims

1. A medical image modality conversion method based on contrastive learning, characterized in that, include: Construct an initial medical image modality conversion model; Acquire paired CT-MRI image data; The paired CT-MRI image data is input into the initial medical image modality conversion model for optimization training to obtain a preset medical image modality conversion model. Specifically, this includes: calculating multiple loss functions on the paired CT-MRI image data according to a preset method; multiplying each of the multiple loss functions by its corresponding weight coefficient to obtain a weight score for each loss function; summing the weight scores of the multiple loss functions to obtain a total loss function; and optimizing the initial medical image modality conversion model based on the total loss function. The multiple loss functions include a structural contrast loss function, a modality contrast loss function, and an image consistency loss function. Acquire source target image data; Based on the analysis of the source target image data, the content features and modal features of the source target image data are obtained; The modal features of the source target image data are converted into target modal features; The target modal features are merged and decoded with the content features of the source target image data to obtain the target modal image.

2. The medical image modality conversion method based on contrastive learning according to claim 1, characterized in that, The step of analyzing the source target image data to obtain the content features and modal features of the source target image data includes: Decouple the source target image data to obtain the content features and modal features of the source target image data; The decoupling is expressed by the following formula: ； in The source modal image is the input. For the encoder to be trained, For images Content features obtained after encoder The obtained modal features.

3. The medical image modality conversion method based on contrastive learning according to claim 1, characterized in that, The step of converting the modal features of the source target image data into target modal features includes: The modal features of the source target image data are transformed using a modal transformation network to obtain the target modal features; The feature transformation is expressed by the following formula: ； in For mode converters, The modal features corresponding to the target modality. The modal features of the source modal image.

4. The medical image modality conversion method based on contrastive learning according to claim 1, characterized in that, The step of merging and decoding the target modal features with the content features of the source target image data to obtain the target modal image includes: The content features of the source target image data are merged with the target modal features to obtain merged feature data; The merged feature data is decoded to obtain the target modal image; The merging and decoding is expressed by the following formula: ； in For the generated image of the target modality, The modal features corresponding to the target modality. For images Content features obtained after encoder For decoders.

5. A medical image modality conversion system based on contrastive learning, characterized in that, The system includes a memory and a processor. The memory contains a program for a medical image modality conversion method based on contrastive learning. When executed by the processor, the program for the medical image modality conversion method based on contrastive learning performs the following steps: Acquire source target image data; Based on the analysis of the source target image data, the content features and modal features of the source target image data are obtained; The modal features of the source target image data are converted into target modal features; The target modal features are merged and decoded with the content features of the source target image data to obtain the target modal image.

6. The medical image modality conversion system based on contrastive learning according to claim 5, characterized in that, The step of analyzing the source target image data to obtain the content features and modal features of the source target image data includes: Decouple the source target image data to obtain the content features and modal features of the source target image data; The decoupling is expressed by the following formula: ； in The source modal image is the input. For the encoder to be trained, For images Content features obtained after encoder The obtained modal features.

7. The medical image modality conversion system based on contrastive learning according to claim 6, characterized in that, The step of converting the modal features of the source target image data into target modal features includes: The modal features of the source target image data are transformed using a modal transformation network to obtain the target modal features; The feature transformation is expressed by the following formula: ； in For mode converters, The modal features corresponding to the target modality. The modal features of the source modal image.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a medical image modality conversion method program based on contrastive learning, which, when executed by a processor, implements the steps of the medical image modality conversion method based on contrastive learning as described in any one of claims 1 to 4.

Images