Brain MRI image translation method and device based on semi-supervised generative adversarial network

Abstract

The invention provides a brain MRI image translation method and device based on a semi-supervised generative adversarial network, and relates to the technical field of computer vision. The method comprises the following steps: acquiring a first modal brain MRI image as a condition image; constructing a semi-supervised generative adversarial network, and introducing a weight coefficient to design a loss function; the semi-supervised generative adversarial network comprises a generator and a discriminator, and the generator is used for generating a target image according to an input condition image, namely a second modal brain MRI image; the discriminator is used for distinguishing the real target image and the generated target image; and optimizing parameters of the generator and the discriminator in an iterative training mode to obtain a trained semi-supervised generative adversarial network for brain MRI image translation. According to the method, the MRI image of the missing mode can be effectively generated, the precision of the generated image can be improved when paired registered and unregistered images are processed, and the calculation efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a brain MRI image translation method and device based on a semi-supervised generative adversarial network. Background Art

[0002] In recent years, with the rapid development of science and technology, magnetic resonance imaging (MRI) technology has made significant progress. Its operating principle is mainly based on the nuclear magnetic resonance phenomenon. By precisely controlling the direction and intensity of the static magnetic field and combining it with a specific gradient field, the magnetic resistance signals generated by compounds such as fat and water in the magnetic field form responses of different frequencies. These signals are processed carefully and finally converted into clear and visible images, providing a solid basis for medical diagnosis.

[0003] In medical practice, doctors can obtain detailed information about patients from all directions and angles by adopting multimodal MRI imaging technology, thereby achieving accurate diagnosis of the disease and tailoring personalized treatment plans for patients. Brain MRI images include multiple modes such as T1-weighted, T2-weighted, FLAIR, DWI and fMRI. These technologies can fully reveal the patient's skull structure, brain tissue morphology, cerebrovascular status and the functional status of the nervous system. For example, T1-weighted imaging technology plays a vital role in tissue structure positioning and anatomical research due to its excellent clarity. T2-weighted imaging technology, due to its unique advantages, can accurately show the distribution of cerebrospinal fluid and soft tissue, especially in the monitoring of nervous system and lumbar diseases, showing significant effects. FLAIR imaging technology can accurately depict the boundaries between white matter and gray matter in the brain, and DWI imaging technology can quickly identify tiny ischemic or hemorrhagic lesions, providing important help for the diagnosis of diseases such as stroke. fMRI imaging technology can directly detect brain activity and neural connections without drug intervention.

[0004] However, in practical applications, the full utilization of multimodal medical imaging data faces many challenges. First, the acquisition of multimodal data is often restricted by factors such as patient discomfort, high scanning costs, and long scanning times, resulting in insufficient or incomplete multimodal medical imaging data for a single patient. Secondly, technical limitations in the imaging process, such as image artifacts caused by patient movement, low resolution caused by rapid imaging, and data corruption, also affect image quality. In addition, the acquisition of some modality images may require the use of specific contrast agents, which increases the risk to patients. Therefore, how to maximize the utilization of medical information under limited data modalities has become an important issue that needs to be solved urgently.

[0005] MRI is widely used in clinical applications because it can provide rich soft tissue contrast. Through specific MRI pulse sequence design, multiple contrast effects can be produced even for the same anatomical region. For example, T1-weighted imaging can clearly distinguish gray matter from white matter, while T2-weighted imaging specializes in identifying fluid and cortical tissue. In this way, multiple contrast images of the same subject greatly enrich the amount of information in clinical diagnosis and medical research. However, due to cost, time constraints or patient (especially children and the elderly) cooperation limitations, it is not always feasible to fully obtain all contrast images. Sometimes only contrasts with shorter scanning times can be selected for image acquisition, which may cause some images to be unusable for subsequent diagnosis due to noise or interference.

[0006] MRI image translation aims to convert images from one imaging modality to another to meet specific clinical and research needs (such as converting T1-weighted images to T2-weighted images), thereby providing a more comprehensive understanding of the disease when a specific imaging modality cannot be directly obtained. This conversion technology greatly improves the availability and flexibility of imaging data, providing more possibilities for early detection, diagnosis and treatment planning of diseases.

[0007] At the same time, in order to overcome the problem of scarce data for generating multimodal MRI images, the Generative Adversarial Network (GAN) came into being. GAN is a deep learning architecture whose core components include the Generator network and the Discriminator network. Through the competition and collaboration training mechanism between the two networks, GAN can generate highly realistic sample data, showing broad application prospects in multiple fields.

[0008] However, standard GAN models usually require a large amount of paired data for supervised learning, which faces the challenge of data scarcity in the application of multimodal MRI image generation. Summary of the invention

[0009] In view of the above problems, the purpose of the present invention is to provide a method and device for brain MRI image translation based on a semi-supervised generative adversarial network. By constructing a semi-supervised generative adversarial network, for labeled data points, a standard cross entropy loss function is used to accurately quantify the difference between the prediction and the true label; and for unlabeled data points, a consistency regularization strategy is used, such as minimizing the mean square error between the prediction results or calculating the KL divergence between the prediction distributions, to ensure that the model can maintain stable prediction performance when facing unknown data. In order to integrate the losses of supervised learning and unsupervised learning, the present invention introduces weight coefficients and constructs a comprehensive semi-supervised learning loss function, which fully considers the role of labeled data and unlabeled data, thereby ensuring the comprehensiveness and accuracy of model training.

[0010] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0011] In one aspect, a method for translating brain MRI images based on a semi-supervised generative adversarial network is provided, the method comprising the following steps:

[0012] S1, obtaining a first modality brain MRI image as a conditional image;

[0013] S2. Construct a semi-supervised generative adversarial network and introduce weight coefficients to design the loss function;

[0014] The semi-supervised generative adversarial network includes a generator and a discriminator, wherein the generator is used to generate a target image according to an input conditional image, and the target image is a second modality brain MRI image; and the discriminator is used to distinguish between a real target image and a generated target image;

[0015] S3. Optimize the parameters of the generator and the discriminator through iterative training to obtain the trained semi-supervised generative adversarial network;

[0016] S4. Input the first modality brain MRI image to be translated into the trained semi-supervised generative adversarial network to obtain a translated second modality brain MRI image.

[0017] Optionally, the semi-supervised generative adversarial network is a pixel-based generative adversarial network, and the total loss function includes an adversarial loss function, a pixel-level loss function, and a perceptual loss function;

[0018] The pixel-generated adversarial network includes a generator, a discriminator, and a pre-trained VGG16 network;

[0019] The generator adopts a convolutional neural network architecture, with conditional images and random noise as input, and generates a target image through a convolution layer, a batch normalization layer, and an activation function layer;

[0020] The discriminator adopts a convolutional neural network architecture, with the input being the generated target image and the real image, and outputs a authenticity score through a convolutional layer, an activation function layer, a pooling layer, and a fully connected layer, which is used to distinguish the real target image from the generated target image;

[0021] The pre-trained VGG16 network is used for perceptual loss calculation.

[0022] Optionally, the adversarial loss function is calculated as follows:

[0023]

[0024] Where x represents the input conditional image, y represents the real target image, z is random noise, D is the discriminator, G is the generator, D(x,y) represents the output value of the discriminator D under the input x and the real target image y, G(x,z) represents the target image generated by the generator G under the input x and noise z, E is the expected symbol, E x,y is the expected value on the input x and the target image y, E x,z is the expected value on the input x and noise z;

[0025] The pixel-level loss function is calculated as follows:

[0026]

[0027] E x,y,z Represents the expected value on the input x, target image y and noise z;

[0028] The perceptual loss function is calculated as follows:

[0029]

[0030] Where V(y) is the feature map of the real target image y extracted from the pre-trained VGG16 network, and G(x) is the target image generated by the generator G based on the input x;

[0031] The total loss function of the pixel-generated adversarial network is:

[0032]

[0033] where λ and λ perc are the weights of the pixel-level loss function and the perceptual loss function, respectively.

[0034] Optionally, the semi-supervised generative adversarial network is a cyclic generative adversarial network, and the total loss function includes a cycle consistency loss function and an adversarial loss function;

[0035] The recurrent generative adversarial network includes: a first generator, a first discriminator, a second generator, and a second discriminator; wherein the first generator is used to translate the second modality brain MRI image into the first modality brain MRI image, and the second generator is used to translate the first modality brain MRI image into the second modality brain MRI image; the first discriminator is used to distinguish between the real first modality brain MRI image and the generated first modality brain MRI image, and the second discriminator is used to distinguish between the real second modality brain MRI image and the generated second modality brain MRI image.

[0036] Optionally, the first generator is G x , the second generator is G y , the first discriminator is Dx , the second discriminator is D y , the first modality brain MRI image is x, the second modality brain MRI image is y, and the image generated by the first generator is G x (y), the image generated by the second generator is G y (x);

[0037] The cycle consistency loss function is calculated as follows:

[0038]

[0039] By combining adjacent sections, the cycle consistency loss function becomes:

[0040]

[0041] Among them, k represents the mode, x k and k denote the input image and the corresponding generated image of the kth modality respectively, Respectively represent the input images x of different modalities k and generate image y k Expected value;

[0042] The adversarial loss function is calculated as follows:

[0043]

[0044]

[0045] The total loss function of the cyclic generative adversarial network is:

[0046]

[0047] where λ cycle is the weight of the cycle consistency loss function.

[0048] Optionally, the optimizing the parameters of the generator and the discriminator by iterative training specifically includes:

[0049] During the training process, the parameters of the generator are first fixed and the parameters of the discriminator are updated so that the discriminator can better distinguish between the real target image and the generated target image; then the parameters of the discriminator are fixed and the parameters of the generator are updated so that the target image generated by the generator can be closer to the real target image.

[0050] On the other hand, a brain MRI image translation device based on a semi-supervised generative adversarial network is provided, for implementing any of the above methods, the device comprising:

[0051] An image acquisition module, used for acquiring a first modality brain MRI image as a conditional image;

[0052] Model building module, used to build a semi-supervised generative adversarial network and introduce weight coefficients to design the loss function;

[0053] The semi-supervised generative adversarial network includes a generator and a discriminator, wherein the generator is used to generate a target image according to an input conditional image, and the target image is a second modality brain MRI image; and the discriminator is used to distinguish between a real target image and a generated target image;

[0054] The model training module is used to optimize the parameters of the generator and the discriminator through iterative training to obtain the trained semi-supervised generative adversarial network;

[0055] The image translation module is used to input the first modality brain MRI image to be translated into the trained semi-supervised generative adversarial network to obtain the translated second modality brain MRI image.

[0056] In another aspect, an electronic device is provided, the electronic device comprising:

[0057] processor;

[0058] A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are loaded and executed by the processor, the steps of the brain MRI image translation method based on the semi-supervised generative adversarial network are implemented.

[0059] On the other hand, a computer-readable storage medium is provided, in which a program code is stored. The program code can be called by a processor to execute the steps of the brain MRI image translation method based on a semi-supervised generative adversarial network as described above.

[0060] The beneficial effects brought about by the technical solution provided by the present invention include at least:

[0061] The present invention proposes a novel brain MRI image translation method based on a semi-supervised generative adversarial network, which can generate high-quality multimodal MRI images by combining supervised and unsupervised learning strategies. Experimental results show that this method performs well in the multimodal MRI image translation task, not only improving the quality of the generated images, but also significantly enhancing the robustness and generalization ability of the model in the absence of a large amount of annotated data. Specifically, the method of the present invention is significantly superior to the traditional method in the quality of the generated MRI images. The adversarial loss function and the cycle consistency loss function are used to effectively capture the mapping relationship between different modalities, and the generated images have high consistency in details and structures. In the case of limited annotated data, the semi-supervised generative adversarial network demonstrates a strong learning ability, can make full use of unlabeled data, and improves the robustness and generalization ability of the model, which is of great significance for data scarcity in practical applications. By introducing the combination of PGAN and CGAN, not only the accuracy of the generated images is improved, but also the computational efficiency is excellent. When processing paired registered and unregistered images, the model shows efficient image generation capabilities. The present invention provides a new solution for the efficient translation of multimodal medical images, which can effectively generate MRI images of missing modalities without increasing additional scanning costs and time, thereby improving the practicality and efficiency of multimodal imaging in medical diagnosis and treatment planning. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0063] Figure 1 is a flowchart of a brain MRI image translation method based on a semi-supervised generative adversarial network provided by an embodiment of the present invention;

[0064] Figure 2 is a schematic diagram of a PGAN model provided by an embodiment of the present invention;

[0065] Figure 3 is a schematic diagram of a CGAN model provided by an embodiment of the present invention;

[0066] Figure 4 It is a structural schematic diagram of a brain MRI image translation device based on a semi-supervised generative adversarial network provided by an embodiment of the present invention;

[0067] Figure 5 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the word "exemplary" is used to present concepts in a specific way.

[0070] The embodiment of the present invention provides a method for translating brain MRI images based on a semi-supervised generative adversarial network. The method can be implemented by an electronic device, which can be a terminal or a server. Figure 1 As shown, the processing flow of the method may include the following steps:

[0071] S1. Acquire a first modality brain MRI image as a conditional image.

[0072] MRI image translation refers to converting MRI images from one imaging modality to another to meet specific clinical and research needs. For example, for brain MRI images, T1-weighted images (first-modality brain MRI images) are converted to T2-weighted images (second-modality brain MRI images), thereby providing a more comprehensive understanding of the disease when a specific imaging modality cannot be directly obtained.

[0073] S2. Construct a semi-supervised generative adversarial network and introduce weight coefficients to design the loss function.

[0074] Among them, the semi-supervised generative adversarial network includes a generator and a discriminator, the generator is used to generate a target image based on the input conditional image, and the target image is a second modality brain MRI image; the discriminator is used to distinguish between the real target image (i.e., the real second modality brain MRI image) and the generated target image (i.e., the generated second modality brain MRI image).

[0075] Semi-supervised learning is an efficient machine learning strategy that can make full use of unlabeled data for learning when the labeled data is limited. In the scenario of multimodal MRI image translation, the semi-supervised generative adversarial network can not only use a small number of paired multimodal images for learning, but also learn the cross-modal mapping relationship from a large number of single-modal images, thereby effectively improving the training efficiency of the model and the quality of the generated images.

[0076] As an optional embodiment of the present invention, the semi-supervised generative adversarial network is a pixel generative adversarial network PGAN (PixelGAN). PGAN is a deep learning model for multimodal MRI image generation. By introducing pixel-level loss and perceptual loss in the generation and discrimination process, the quality and detail retention of the generated image are significantly improved.

[0077] The pixel-generated adversarial network model reference Figure 2 As shown in the figure, it includes a generator, a discriminator, and a pre-trained VGG16 network. The generator adopts a convolutional neural network architecture, with conditional images and random noise as input, and generates a target image through convolutional layers, batch normalization layers, and activation function layers; the discriminator adopts a convolutional neural network architecture, with generated target images and real images as input, and outputs a authenticity score through convolutional layers, activation function layers, pooling layers, and fully connected layers to distinguish between real target images and generated target images; the pre-trained VGG16 network is used for perceptual loss calculation.

[0078] Specifically, in the PGAN architecture, the generator (G) is responsible for generating a target image (e.g., a T2-weighted image) from an input conditional image (e.g., a T1-weighted image). The generator uses a convolutional neural network (CNN) architecture, with the input being a conditional image and random noise. The generator gradually maps the input to the space of the target image through a series of convolutional layers, batch normalization layers, and activation function layers to generate a high-quality target image. During the generation process, the generator not only needs to learn the global structure of the input image, but also needs to retain detail information to ensure that the generated image is visually indistinguishable from the real image.

[0079] The discriminator (D) is responsible for distinguishing between the real target image and the generated target image. Its input includes the real target image and the generated target image, and through a series of convolutional layers and activation function layers, it outputs a scalar representing the authenticity score of the image. The discriminator also uses a convolutional neural network architecture to extract the features of the input image through multiple convolutional layers, and adds an activation function layer after each convolutional layer to introduce nonlinear capabilities. In addition, the discriminator uses pooling layers (such as maximum pooling or average pooling) to gradually reduce the spatial size of the feature map, thereby improving the computational efficiency of the model. Finally, the discriminator outputs an authenticity score through a fully connected layer, which is used to distinguish between the real target image and the generated target image.

[0080] The core of PGAN is the adversarial training mechanism, that is, the generator and the discriminator are continuously optimized through mutual confrontation. The goal of the generator is to generate target images that are as realistic as possible to deceive the discriminator; while the goal of the discriminator is to maximize the score of the real target image and minimize the score of the generated target image, so as to distinguish the real target image from the generated target image.

[0081] Furthermore, the pixel-generated adversarial network PGAN uses a variety of loss functions in the process of generating images to ensure the quality and detail retention of the generated images. The loss functions of PGAN mainly include adversarial loss, pixel-wise loss and perceptual loss. The reasonable design and combination of these loss functions enable PGAN to generate high-quality and detail-rich images in multimodal MRI image applications.

[0082] In the embodiment of the present invention, the goal of the adversarial loss function is to train the generator G to generate a target image that is indistinguishable from a real image, while training the discriminator D to distinguish. The adversarial loss function is calculated as follows:

[0083]

[0084] Where x represents the input conditional image, y represents the real target image, z is random noise, D is the discriminator, and G is the generator; D(x, y) represents the output value of the discriminator D under the input x and the real target image y. The output of the discriminator D is a probability value, indicating whether the input x matches y; G(x, z) represents the target image generated by the generator G under the input x and noise z. The generator G tries to generate a target image that is as similar as possible to the real target image y; E is the expected symbol, E x,y is the expected value on the input x and the target image y, E x,z is the expected value on the input x and the noise z.

[0085] The pixel-level loss function (also called pixel-level L1 loss function) measures the difference between the generated target image and the real target image at the pixel level. The pixel-level loss function is calculated as follows:

[0086]

[0087] E x,y,z Represents the expected value on the input x, target image y and noise z.

[0088] The perceptual loss function evaluates the similarity between the generated target image and the real target image by measuring the difference between the feature maps in the pre-trained VGG16 network. The perceptual loss function is calculated as follows:

[0089]

[0090] Where V(y) is the feature map of the real target image y extracted from the pre-trained VGG16 network, and G(x) is the target image generated by the generator G based on the input x.

[0091] Combining the above loss functions, the total loss function of the pixel-generated adversarial network is:

[0092]

[0093] where λ and λ perc are the weights of the pixel-level loss function and the perceptual loss function, respectively.

[0094] PGAN generates high-quality, detail-rich images in multimodal MRI image synthesis by rationally designing and combining adversarial loss, pixel-level loss, and perceptual loss. The design and optimization of these loss functions are crucial to improving the performance of the PGAN model, ensuring high visual and structural similarity of the generated images, thereby improving the diagnostic value of medical images.

[0095] As another optional embodiment of the present invention, the semi-supervised generative adversarial network is a cyclic generative adversarial network CGAN (CycleGAN). The core architecture of the cyclic generative adversarial network covers two key components: the generator and the discriminator. Among them, the core function of the generator is to synthesize the target image based on the preset conditional image and random noise. At the same time, the discriminator plays the role of a discriminator, responsible for accurately distinguishing the real target image from the target image produced by the generator. In the process of mutual adversarial training, these two network components jointly pursue the optimization goal: the generator continues to improve the quality of its generated images until the discriminator can hardly distinguish the subtle differences between the real target image and the generated target image.

[0096] The design of the generator usually adopts the convolutional neural network (CNN) architecture. Its input includes conditional images and random noise, and through a series of convolutional layers, batch normalization layers, and activation function layers, the input is gradually mapped to the space of the target image. In order to generate high-quality images, the generator needs to capture the detailed information of the input image and generate high-quality images that are very close to the real target image. In the specific implementation, the generator first splices the input conditional image and random noise together, gradually extracts features through the convolution layer, and then restores the feature map to the resolution of the target image through the deconvolution layer. In this process, the generator not only needs to learn the global structure of the input image, but also needs to retain the detailed information to ensure that the generated image is visually the same as the real image.

[0097] The discriminator also uses the framework of convolutional neural networks, and its input includes real target images and generated target images. After multiple levels of convolution operations and activation function processing, the discriminator finally outputs a scalar value for evaluating the authenticity of the image. The core goal of the discriminator is to optimize its performance, that is, to increase the score of the real target image as much as possible, while reducing the score of the generated target image to ensure its accuracy in distinguishing between real and generated images. In order to enhance the discriminator's discriminative ability, a multi-scale discriminator is usually introduced into the discriminator's structure, that is, the input image is discriminated at different scales. This method can capture the detailed information of the image at different scales, thereby improving the accuracy of the discriminator.

[0098] In the embodiment of the present invention, the structure of the cyclic generative adversarial network is referenced Figure 3 As shown, it is composed of two generators and two discriminators, which are respectively used to process the conversion task between images of different contrasts. Specifically, it includes: a first generator, a first discriminator, a second generator, and a second discriminator; wherein, the first generator is used to translate the second modality brain MRI image into the first modality brain MRI image, and the second generator is used to translate the first modality brain MRI image into the second modality brain MRI image; the first discriminator is used to distinguish between the real first modality brain MRI image and the generated first modality brain MRI image, and the second discriminator is used to distinguish between the real second modality brain MRI image and the generated second modality brain MRI image.

[0099] Taking the T1-weighted image and T2-weighted image in the brain MRI image as an example, the generator G(T1) converts the T2-weighted image into a T1-weighted image, and the generator G(T2) converts the T1-weighted image into a T2-weighted image. The discriminator D(T1) distinguishes between the real T1-weighted image and the generated T1-weighted image, and the discriminator D(T2) distinguishes between the real T2-weighted image and the generated T2-weighted image.

[0100] Furthermore, CGAN is used to learn the mapping relationship between unregistered images, and the total loss function includes the cycle consistency loss function and the adversarial loss function.

[0101] Specifically, set the first generator to be G x , the second generator is G y , the first discriminator is D x , the second discriminator is D y , the first modality brain MRI image is x, the second modality brain MRI image is y, and the image generated by the first generator is G x (y), the image generated by the second generator is G y (x).

[0102] G x Try to generate G similar to x x (y), and D x Try to G x (y) is distinguished from the real x. Similarly, G y Try to generate y similar to G y (x), and D y Try to G y (x) is distinguished from the true y.

[0103] In order to ensure the consistency between the input image and the target image, a cycle consistency loss is introduced, which is calculated as follows:

[0104]

[0105] By combining adjacent sections, the cycle consistency loss function becomes:

[0106]

[0107] Among them, k represents the mode, x k and k denote the input image and the corresponding generated image of the kth modality respectively, Respectively represent the input images x of different modalities k and generate image y k expected value.

[0108] The two sets of adversarial loss functions are calculated as follows:

[0109]

[0110]

[0111] Combining the above loss functions, the total loss function of the cyclic generative adversarial network is:

[0112]

[0113] where λ cycle is the weight of the cycle consistency loss function.

[0114] S3. Optimize the parameters of the generator and the discriminator through iterative training to obtain a trained semi-supervised generative adversarial network.

[0115] During the training process, the parameters of the generator are first fixed and the parameters of the discriminator are updated so that the discriminator can better distinguish between the real target image and the generated target image; then the parameters of the discriminator are fixed and the parameters of the generator are updated so that the target image generated by the generator can be closer to the real target image to deceive the discriminator. Through this iterative training method, the generator and the discriminator are optimized together, and finally the generator can generate high-quality images that are very close to the real image.

[0116] S4. Input the first modality brain MRI image to be translated into the trained semi-supervised generative adversarial network to obtain a translated second modality brain MRI image.

[0117] Based on the trained semi-supervised generative adversarial network, for newly acquired first-modality brain MRI images, the generator can generate high-quality images that are very close to the real second-modality brain MRI images, thereby realizing brain multi-modal MRI image translation.

[0118] Accordingly, an embodiment of the present invention also provides a brain MRI image translation device based on a semi-supervised generative adversarial network, such as Figure 4 As shown, the device comprises:

[0119] An image acquisition module 201 is used to acquire a first modality brain MRI image as a conditional image;

[0120] A model building module 202 is used to build a semi-supervised generative adversarial network and introduce weight coefficients to design a loss function;

[0121] The semi-supervised generative adversarial network includes a generator and a discriminator, wherein the generator is used to generate a target image according to an input conditional image, and the target image is a second modality brain MRI image; and the discriminator is used to distinguish between a real target image and a generated target image;

[0122] A model training module 203 is used to optimize the parameters of the generator and the discriminator through iterative training to obtain a trained semi-supervised generative adversarial network;

[0123] The image translation module 204 is used to input the first modality brain MRI image to be translated into the trained semi-supervised generative adversarial network to obtain a translated second modality brain MRI image.

[0124] For ease of explanation, Figure 4 Only the main components of the device are shown. The device of this embodiment can be used to perform Figure 1 The technical solution of the method embodiment shown has similar implementation principles and technical effects, which will not be repeated here.

[0125] The brain MRI image translation method and device based on the semi-supervised generative adversarial network provided by the present invention can generate high-quality multimodal MRI images by combining supervised and unsupervised learning strategies. Experimental results show that this method performs well in the multimodal MRI image translation task, not only improving the quality of the generated image, but also significantly enhancing the robustness and generalization ability of the model in the absence of a large amount of annotated data. Specifically, the present invention is significantly superior to the traditional method in the quality of the generated MRI image. The adversarial loss function and the cycle consistency loss function are used to effectively capture the mapping relationship between different modalities, and the generated image has high consistency in details and structure. In the case of limited annotated data, the semi-supervised generative adversarial network demonstrates a strong learning ability, can make full use of unlabeled data, and improves the robustness and generalization ability of the model, which is of great significance for the situation of data scarcity in practical applications. By introducing the combination of PGAN and CGAN, not only the accuracy of the generated image is improved, but also the computational efficiency is excellent. When processing paired registered and unregistered images, the model shows efficient image generation capabilities.

[0126] The present invention provides a new solution for the efficient translation of multimodal medical images, which can effectively generate MRI images of missing modalities without increasing additional scanning costs and time, thereby improving the practicality and efficiency of multimodal imaging in medical diagnosis and treatment planning.

[0127] In an exemplary embodiment, the present invention further provides an electronic device, the electronic device comprising:

[0128] processor;

[0129] A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are loaded and executed by the processor, the steps of the brain MRI image translation method based on the semi-supervised generative adversarial network are implemented.

[0130] Figure 5 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 5 As shown, the electronic device 300 may include a processor 3001 and a memory 3002. Optionally, the electronic device 300 may further include a transceiver 3003. The processor 3001, the memory 3002, and the transceiver 3003 may be connected, for example, via a communication bus. The memory 3002 stores computer-readable instructions, which, when executed by the processor 3001, implement the steps of the brain MRI image translation method based on the semi-supervised generative adversarial network as described above.

[0131] In a specific implementation, as an embodiment, the processor 3001 may include one or more CPUs, such as Figure 5CPU0 and CPU1 are shown in FIG.

[0132] In a specific implementation, as an embodiment, the electronic device 300 may also include multiple processors, such as Figure 5 3001 and processor 3004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0133] The memory 3002 is used to store the software program for executing the solution of the present invention, and the execution is controlled by the processor 3001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0134] The transceiver 3003 is used to communicate with a network device or a terminal device.

[0135] Optionally, the transceiver 3003 may include a receiver and a transmitter, wherein the receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0136] Optionally, the transceiver 3003 may be integrated with the processor 3001 , or may exist independently and be coupled to the processor 3001 via an interface circuit of the electronic device 300 , which is not specifically limited in the embodiment of the present invention.

[0137] It should be noted that Figure 5 The structure of the electronic device 300 shown in the figure does not constitute a limitation on the electronic device. The actual electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. In addition, the technical effects of the electronic device 300 can refer to the technical effects of the above-mentioned method embodiment, which will not be repeated here.

[0138] In an exemplary embodiment, the present invention further provides a computer-readable storage medium, wherein at least one instruction is stored in the computer-readable storage medium, and the at least one instruction is loaded and executed by a processor to implement the steps of the brain MRI image translation method based on a semi-supervised generative adversarial network as described above. For example, the computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0139] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0140] References in the specification to "one embodiment", "an embodiment", "an exemplary embodiment", "some embodiments", etc. indicate that the described embodiments may include a particular feature, structure, or characteristic, but not every embodiment may include the particular feature, structure, or characteristic. In addition, when a particular feature, structure, or characteristic is described in conjunction with an embodiment, it should be within the knowledge of a person skilled in the relevant art to implement such feature, structure, or characteristic in conjunction with other embodiments (whether or not explicitly described).

[0141] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0142] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0143] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0144] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0145] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0146] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0147] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0148] The present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention. In order to make the public have a thorough understanding of the present invention, specific details are described in detail in the following preferred embodiments of the present invention, but those skilled in the art can fully understand the present invention without the description of these details. In addition, in order to avoid unnecessary confusion about the essence of the present invention, well-known methods, processes, procedures, components and circuits are not described in detail.

[0149] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A brain MRI image translation method based on a semi-supervised generative adversarial network, characterized in that: The following steps are involved: S1, obtaining a first modality brain MRI image as a conditional image; S2. Construct a semi-supervised generative adversarial network and introduce weight coefficients to design the loss function; The semi-supervised generative adversarial network includes a generator and a discriminator, wherein the generator is used to generate a target image according to an input conditional image, and the target image is a second modality brain MRI image; and the discriminator is used to distinguish between a real target image and a generated target image; S3. Optimize the parameters of the generator and the discriminator through iterative training to obtain the trained semi-supervised generative adversarial network; S4. Input the first modality brain MRI image to be translated into the trained semi-supervised generative adversarial network to obtain a translated second modality brain MRI image.

2. The method for brain MRI image translation based on semi-supervised generative adversarial network according to claim 1, characterized in that: The semi-supervised generative adversarial network is a pixel-level generative adversarial network, and the total loss function includes an adversarial loss function, a pixel-level loss function, and a perceptual loss function; The pixel-generated adversarial network includes a generator, a discriminator, and a pre-trained VGG16 network; The generator adopts a convolutional neural network architecture, with conditional images and random noise as input, and generates a target image through a convolution layer, a batch normalization layer, and an activation function layer; The discriminator adopts a convolutional neural network architecture, with the input being the generated target image and the real image, and outputs a authenticity score through a convolutional layer, an activation function layer, a pooling layer, and a fully connected layer, which is used to distinguish the real target image from the generated target image; The pre-trained VGG16 network is used for perceptual loss calculation.

3. The method for brain MRI image translation based on semi-supervised generative adversarial network according to claim 2, characterized in that: The adversarial loss function is calculated as follows: ； Where x represents the input conditional image, y represents the real target image, z is random noise, D is the discriminator, G is the generator, D(x,y) represents the output value of the discriminator D under the input x and the real target image y, G(x,z) represents the target image generated by the generator G under the input x and noise z, E is the expected symbol, E x,y is the expected value on the input x and the target image y, E x,z is the expected value on the input x and noise z; The pixel-level loss function is calculated as follows: ； E x,y,z Represents the expected value on the input x, target image y and noise z; The perceptual loss function is calculated as follows: ； Where V(y) is the feature map of the real target image y extracted from the pre-trained VGG16 network, and G(x) is the target image generated by the generator G based on the input x; The total loss function of the pixel-generated adversarial network is: ； where λ and λ perc are the weights of the pixel-level loss function and the perceptual loss function, respectively.

4. The method for brain MRI image translation based on semi-supervised generative adversarial network according to claim 1, characterized in that: The semi-supervised generative adversarial network is a cyclic generative adversarial network, and the total loss function includes a cycle consistency loss function and an adversarial loss function; The recurrent generative adversarial network includes: a first generator, a first discriminator, a second generator, and a second discriminator; wherein the first generator is used to translate the second modality brain MRI image into the first modality brain MRI image, and the second generator is used to translate the first modality brain MRI image into the second modality brain MRI image; the first discriminator is used to distinguish between the real first modality brain MRI image and the generated first modality brain MRI image, and the second discriminator is used to distinguish between the real second modality brain MRI image and the generated second modality brain MRI image.

5. The method for brain MRI image translation based on semi-supervised generative adversarial network according to claim 4, characterized in that: The first generator is G x , the second generator is G y , the first discriminator is D x , the second discriminator is D y , the first modality brain MRI image is x, the second modality brain MRI image is y, and the image generated by the first generator is G x (y), the image generated by the second generator is G y (x); The cycle consistency loss function is calculated as follows: ； By combining adjacent sections, the cycle consistency loss function becomes: ； Among them, k represents the mode, x k and k denote the input image and the corresponding generated image of the kth modality respectively, Respectively represent the input images x of different modalities k and generate image y k Expected value; The adversarial loss function is calculated as follows: ； ； The total loss function of the cyclic generative adversarial network is: ； where λ cycle is the weight of the cycle consistency loss function.

6. The method for brain MRI image translation based on semi-supervised generative adversarial network according to claim 1, characterized in that: The optimization of the parameters of the generator and the discriminator by iterative training specifically includes: During the training process, the parameters of the generator are first fixed and the parameters of the discriminator are updated so that the discriminator can better distinguish between the real target image and the generated target image; then the parameters of the discriminator are fixed and the parameters of the generator are updated so that the target image generated by the generator can be closer to the real target image.

7. A brain MRI image translation device based on a semi-supervised generative adversarial network, the device being used to implement the method according to any one of claims 1 to 6, characterized in that: The device comprises: An image acquisition module, used for acquiring a first modality brain MRI image as a conditional image; Model building module, used to build a semi-supervised generative adversarial network and introduce weight coefficients to design loss functions; The semi-supervised generative adversarial network includes a generator and a discriminator, wherein the generator is used to generate a target image according to an input conditional image, and the target image is a second modality brain MRI image; and the discriminator is used to distinguish between a real target image and a generated target image; The model training module is used to optimize the parameters of the generator and the discriminator through iterative training to obtain the trained semi-supervised generative adversarial network; The image translation module is used to input the first modality brain MRI image to be translated into the trained semi-supervised generative adversarial network to obtain the translated second modality brain MRI image.

8. An electronic device, characterized in that: The electronic device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are loaded and executed by the processor, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, which can be called by a processor to execute the method according to any one of claims 1 to 6.

Images