AI-based medical image conversion method and device

Abstract

The present invention relates to a medical image conversion method, and more particularly, to an image conversion method for converting medical images using a generative adversarial network (GAN). According to one embodiment of the present invention, there is provided a medical image conversion method that enables medical staff to perform medical examinations, diagnoses, and treatments with accurate information through conversion between medical images using a learning model that performs machine learning, thereby enabling patients to receive appropriate medical services.

Description

[Technical Field]

[0001] One aspect of the present invention relates to a method for converting medical images using artificial intelligence, and more particularly, to an image conversion method and apparatus for converting medical images using a generative adversarial network (GAN). [Background technology]

[0002] The material described in this section merely provides background information for embodiments of the present invention and may not constitute prior art.

[0003] Medical image conversion has recently emerged as an important issue in the field of imaging medicine, where medical staff can diagnose and treat patients.

[0004] For the diagnosis and treatment of patients with cerebral hemorrhage in the emergency room and tumors, medical staff can use non-enhanced CT images, which do not use contrast enhancers, and enhanced CT images, which do use contrast enhancers.

[0005] Contrast-enhanced CT images are a type of medical image taken to search for blood vessels, various organs, and cancers using a contrast agent to increase tissue contrast.

[0006] However, contrast agents often cause side effects, and in severe cases, can lead to death due to cardiac arrest, shock, and other causes.

[0007] Since side effects caused by contrast agents cannot be predicted in advance, if image conversion could convert non-contrast-enhanced CT or MR images into contrast-enhanced CT or MR images, it would be possible to prevent such side effects in advance and reduce medical costs associated with contrast agents.

[0008] In addition, in cases where MR images are available but X-ray irradiation is not possible and CT images are not available, such as pregnant women, or when CT images are available but MR images cannot be taken due to time, cost, or implants, medical staff can still use image-to-image conversion to determine tissue boundaries or the location of tumors without the use of contrast agents.

[0009] In addition, conversion between MR images, for example, conversion from T1 image to T2 image, or conversion from T1 image to diffusion image, can help medical staff better understand information about lesions.

[0010] Meanwhile, medical image conversion using an AI learning model has recently become an issue. That is, a learning model is trained to convert medical images, and after the learning is complete, the model converts medical images according to the needs of medical staff.

[0011] Since such learning models perform learning through generative adversarial networks (GANs), it is necessary to collect training data that constitutes a paired data set.

[0012] Here, a paired data set refers to a pair of training data required for training a learning model. For example, to generate a synthetic MR image based on an original CT image, the original CT image and an original MR image that matches the original CT image are required.

[0013] However, in the process of collecting training data, it is very difficult to obtain a paired dataset that is consistent with each other due to time and space constraints, as patients may have restrictions on medical imaging and may not be able to experience both CT and MR imaging at the same time.

[0014] Therefore, problems such as the slice levels of the input video and the target video not matching accurately occur.

[0015] In some cases, the input video may be secured but the target video may not be secured, making it impossible to secure the paired data set.

[0016] It is urgent to present a learning model with a new learning method that enables the learning model to generate accurate output images even in such cases.

[0017] The above-mentioned background art is technical information that the inventor possessed for the purpose of deriving the embodiments of the present invention or that he acquired in the process of deriving the embodiments of the present invention, and cannot necessarily be said to be publicly known art that was disclosed to the general public prior to the filing of the embodiments of the present invention. Summary of the Invention [Problem to be solved by the invention]

[0018] One aspect of the present invention has been proposed to solve the above-mentioned problems, and an object of the present invention is to provide an artificial intelligence learning model that compares an output image with an input image and reflects this in a learning model so that the output image accurately reflects the input image.

[0019] Another object of the present invention is to provide a medical image conversion method or apparatus using a new learning method that enables a learning model to generate an accurate output image when the input image and the target image are not aligned or when the input image is secured but the target image is not secured.

[0020] Another object of the present invention is to provide a medical image conversion method or device that enables medical staff to perform medical examinations, diagnoses, and treatments with accurate information through conversion between medical images, thereby enabling patients to receive appropriate medical services.

[0021] The technical problems that the present invention aims to achieve are not limited to the technical problems mentioned above, and other technical problems not mentioned above will be clearly understood by those skilled in the art to which the present invention pertains from the following description. [Means for solving the problem]

[0022] In order to achieve the above-mentioned object, one aspect of the present invention is a medical image conversion method for converting medical images using a generative adversarial network (GAN), comprising:

[0023] a first step of receiving an input of a first image selected from a paired dataset including a first image and a second image;

[0024] a second step of generating a third image based on the first image, the third image corresponding to the first image and having a different domain from the first image;

[0025] a third step of comparing the third image with the second image and training a learning model taking into account the comparison results; and

[0026] a fourth step of comparing the third image with the first image and training the training model in consideration of the comparison result.

[0027] Here, the fourth step may be characterized by measuring image pattern similarity between the third image and the first image, and feeding back the measured value to the learning model.

[0028] The image pattern similarity may include a Pearson Correlation Coefficient (PCC) or Mutual Information (MI) according to an embodiment.

[0029] The third step may be characterized in that the matching rate between the third image and the second image is calculated and the calculated value is fed back to the learning model, where the matching rate is different from the image pattern similarity.

[0030] Another aspect of the present invention provides a learning unit that receives an input of a first image and generates a third image based on the first image;

[0031] a first comparing unit that compares the third image generated by the learning unit with a second image to generate a first comparison value and feeds the first comparison value back to the learning unit; and

[0032] and a second comparison unit that compares the third image generated by the learning unit with the first image to generate a second comparison value and feeds back the second comparison value to the learning unit.

[0033] The first comparison value and the second comparison value may be generated in different ways.

[0034] The first comparison unit may compare a difference between the second image and the third image that belong to the same domain.

[0035] The first comparison unit may calculate a matching rate between the second image and the third image, and feed the result back to the learning unit.

[0036] The second comparison unit may compare a difference between the first image and the third image belonging to different domains.

[0037] The second comparison unit may calculate an image pattern similarity between the first image and the third image, and feed the result back to the learning unit.

[0038] Another aspect of the present invention is an image conversion device for converting medical images using a generative adversarial network (GAN), comprising:

[0039] It is possible to provide an image conversion device including at least one of a learning model that receives an input image and generates an output image corresponding to the input image based on the input image, and a comparison unit that compares the output image with the input image and feeds back the comparison value to the learning model.

[0040] The learning model may be characterized in that it performs learning by comparing the output image with ground truth separately from the comparison unit.

[0041] The comparison unit may measure image pattern similarity between the input image and the output image and input the result to the learning model.

[0042] The image pattern similarity may include a Pearson Correlation Coefficient (PCC) or Mutual Information (MI). [Effects of the Invention]

[0043] As described above, according to one embodiment of the present invention, an artificial intelligence learning model can be provided that compares an output image with an input image and reflects this in a learning model so that the output image accurately reflects the input image.

[0044] According to another embodiment of the present invention, there is provided a medical image conversion method or apparatus using a new learning method that enables a learning model to generate an accurate output image when the input image and the target image are not aligned or when the input image is secured but the target image is not secured.

[0045] According to another embodiment of the present invention, a medical image conversion method or device can be provided that enables medical staff to perform medical examinations, diagnoses, and treatments with accurate information through conversion between medical images, thereby enabling patients to receive appropriate medical services.

[0046] In addition, the present invention has various effects such as excellent versatility depending on the embodiment, and such effects can be clearly confirmed in the description of the embodiments below. [Brief explanation of the drawings]

[0047] The following drawings attached to this specification illustrate one embodiment of the present invention and, together with the above detailed description of the invention, serve to further understand the technical concept of the present invention. Therefore, the present invention should not be interpreted as being limited to the matters depicted in such drawings.

[0048] [Figure 1] FIG. 1 illustrates a medical image conversion method according to one embodiment of the present invention. [Figure 2] FIG. 2 illustrates an artificial intelligence learning model according to one embodiment of the present invention. [Figure 3] FIG. 3 shows an embodiment of calculating the match rate between the output image and the target image. [Figure 4] FIG. 4 illustrates an artificial intelligence learning model according to another embodiment of the present invention. [Figure 5] FIG. 5 shows the result of measuring the image pattern similarity according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0049] The advantages and features of the present invention, and methods for achieving them, will become clearer with reference to the following detailed description of the embodiments accompanied by the accompanying drawings. However, the present invention is not limited to the following embodiments, and may be embodied in various different forms, and includes all modifications, equivalents, and alternatives within the spirit and technical scope of the present invention. The following embodiments are provided to ensure a complete disclosure of the present invention and to fully convey the scope of the invention to those skilled in the art. In describing the present invention, if a detailed description of related prior art is considered to obscure the gist of the present invention, such a detailed description will be omitted.

[0050] The terms used in this application are merely used to describe specific embodiments and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise.

[0051] In this application, the terms "comprise" or "have" are intended to specify the presence of a specified feature, number, step, operation, component, part, or combination thereof, and should be understood as not precluding the presence or additional possibility of one or more other features, numbers, steps, operations, components, parts, or combinations thereof. Terms such as "first," "second," etc. may be used to describe various components, but the components should not be limited by these terms. These terms are used only to distinguish one component from another.

[0052] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, identical or corresponding components will be given the same drawing numbers, and duplicate descriptions thereof will be omitted.

[0053] In this specification, the terms "input image" and "input video" may have substantially the same meaning, the terms "output image" and "output image" may have substantially the same meaning, and the terms "target image" and "target video" may have the same meaning.

[0054] FIG. 1 illustrates a medical image conversion method according to an embodiment of the present invention, and FIG. 2 illustrates an artificial intelligence learning model according to an embodiment of the present invention.

[0055] According to one embodiment of the present invention, there is provided a medical image conversion method for converting medical images using a generative adversarial network (GAN).

[0056] The medical image conversion method according to this embodiment includes a first step (S100) of receiving an input of a first image 110 selected from a paired data set including a first image 110 and a second image 120, and a second step (S110) of generating a third image 130 corresponding to the first image 110 and having a different domain from the first image 110 based on the first image 110.

[0057] The method may include a third step (S120) of comparing the third image 130 with the second image 120 and training a learning model taking into account the comparison result, and a fourth step (S130) of comparing the third image 130 with the first image 110 and training the learning model taking into account the comparison result.

[0058] The first image 110, the second image 120, and the third image 130 may represent medical images. Here, the medical images may include magnetic resonance (MR) images, such as 2D MR, 3D MR, 2D streaming MR, 4D MR, 4D volumetric MR, and 4D cine MR; functional MR images, such as fMR, DCE-MR, and diffusion MR; computed tomography (CT) images, such as 2D CT, conebeam CT, 3D CT, and 4D CT; ultrasound images, such as 2D ultrasound, 3D ultrasound, and 4D ultrasound; positron emission tomography (PET) images; X-ray images; fluoroscopic images; radiotherapy portal images; single-photon emission computed tomography (SPECT) images; and computer-generated synthetic images, such as pseudo-CT.

[0059] In addition, the medical image may include medical image data, such as a training image, a ground truth image, a contoured image, a dose image, and the like.

[0060] Depending on the embodiment, the medical image may include an image acquired from an image acquisition device, a computer-generated image, etc. The image acquisition device may include, for example, an MR imaging device, a CT imaging device, a PET imaging device, an ultrasound imaging device, a fluoroscopic device, a SPECT imaging device, an integrated linear accelerator and MR imaging device, etc. In addition, the image acquisition device is not limited to the above examples and may include various medical imaging devices within the scope of the technical concept for acquiring a medical image of a patient.

[0061] The paired data set may include a set of training data including images of an object, for example, a specific part of a patient.

[0062] For example, a CT imaging device and an MR imaging device can be used to take CT images and MR images of a specific part of a patient, such as the head, respectively, and in this case, the CT images and MR images can form a paired data set.

[0063] When non-enhanced CT images and MR images are taken of a specific area of ​​a patient, the non-enhanced CT images and MR images can form a paired data set, and when contrast-enhanced CT images and MR images are taken of a specific area of ​​a patient, the contrast-enhanced CT images and MR images can form a paired data set.

[0064] When T1 and T2 images are taken of a specific part of a patient, the T1 and T2 images can form a paired data set.

[0065] Here, the specific part of a patient may refer to a specific part of the same patient, or may refer to a specific part of different patients if the parts being imaged are the same.

[0066] For example, CT images and MR images of the same specific area of ​​a patient can form a paired data set, and CT images and MR images of the same area of ​​different patients can also form a paired data set.

[0067] What does it mean for two images to have different domains? For example, if the contrast of the patterns contained in the two corresponding images differs between the two images, or if the intensity distribution or pattern differs between the two images, then the domains are different.

[0068] In addition, images taken with different imaging devices have different domains. For example, CT images and MR images are taken with CT imaging devices and MR imaging devices, respectively, so they have different domains. Similarly, there are different domains between ultrasound images and CT images, between ultrasound images and MR images, and between MR images and PET images.

[0069] Furthermore, non-contrast-enhanced images and contrast-enhanced images have different intensity distributions for the same region, and therefore have different domains.

[0070] In the case of MR images, T1 images and T2 images are the same MR images, but they have different intensity distributions, so they have different domains. In other words, if there is a difference in the signal, sequence, or image creation process that is the basis for image creation between two images created through reconstruction based on signals provided by an individual, the two images with the difference have different domains. In addition, the domains are different between the examples of medical images mentioned above.

[0071] To explain a medical image conversion method according to an embodiment of the present invention, an example will be described in which an image acquisition device acquires a first image 110 and a second image 120 of the same part of the same patient and obtains paired data sets. In this embodiment, the first image 110 is an original CT image, the second image 120 is an original MR image, and the third image 130 is a combined MR image.

[0072] The first step (S100) may include a step of receiving, as input to a learning model, a first image 110 selected from a paired dataset including a first image 110 and a second image 120. Here, the first image 110 may include an input image obtained by processing an original CT image using learning data.

[0073] The manufacturer of the software medical device that performs the medical image conversion method according to this embodiment can determine which of the first image 110 and the second image 120 to select as the input image in the paired data set including the first image 110 and the second image 120. In this case, the image to be selected can be automatically selected by a selection algorithm or manually selected by an operator.

[0074] A manufacturer can create a paired dataset by acquiring original CT images using a CT imaging device and original MR images using an MR imaging device, or can select the original CT images as input images in the paired dataset using an algorithm or manually.

[0075] The second step (S110) may include a step in which a learning model that receives an input of a first image 110 generates a third image 130 based on the first image 110, corresponding to the first image 110 and having a different domain from the first image 110.

[0076] The learning model can receive the original CT image as input and generate a synthetic MR image based on the original CT image.

[0077] The process in which the learning model generates the third image 130 from the first image 110, which has a domain different from that of the first image 110, may be performed through the following two processes according to an embodiment. However, the process in which the learning model generates the third image 130 is not limited to the following two processes.

[0078] The first process is composed of one or more layers that receive a first image 110 acquired by an image acquisition device and perform convolution operations to extract high-dimensional features smaller than the first image 110, and this can be expressed as an artificial neural network that extracts features. Such an artificial neural network that extracts features receives input of 2D or 3D actual image data acquired by an image acquisition device and performs convolution operations to extract image features from the original image.

[0079] To exemplarily explain this process, a convolution layer extracts image features through a filter and a pooling layer strengthens the features and reduces the size of the image. Image features can be extracted by repeating convolution and pooling.

[0080] In a convolution layer configured with multiple layers according to an embodiment, the first convolution layer may extract features of an input image provided as an input and output a feature map from it, the pooling layer may receive the feature map as an input and output a resultant image in which the features are enhanced and the image size is reduced, and the output of the pooling layer may again be input to the second convolution layer.

[0081] The second step is composed of one or more layers that perform deconvolution operations to generate image data of different modalities from the result of the first step, and can be expressed as an artificial neural network that generates images of different modalities. Such an artificial neural network that generates images of different modalities can generate two-dimensional or three-dimensional image data of different modalities by performing deconvolution operations from the results of the convolution operation performed by the artificial neural network that extracts features in the first step.

[0082] The third step (S120) may include comparing the third image 130 with the second image 120 and training a learning model based on the comparison results. If the original CT image is determined as the input image from among the original CT image and the original MR image included in the paired dataset, the original MR image may be determined as the target image. Here, the second image 120 may correspond to ground truth. After generating the third image 130, the learning model may compare the generated third image 130 with the second image 120 and perform training to reduce the difference between the two. That is, the learning model may continuously compare the synthetic MR image with the original MR image to determine whether the synthetic MR image has been generated to be close to the original MR image, thereby training the synthetic MR image to be the same as the original MR image. According to an embodiment, the method may include applying a loss function to the algorithm to calculate a loss value between the third image 130 and the second image 120, and updating the parameters of the learning model based on the loss value.

[0083] According to an embodiment, in the step of calculating the loss value, the calculation of the loss value for updating the learning model may be performed for each iteration. That is, the learning model may calculate the loss value between the synthetic MR image and the original MR image for each iteration while performing learning.

[0084] When the number of repetitions includes a first repetition interval and a second repetition interval that are connected sequentially, depending on the embodiment, if the absolute value of the change between the first loss value calculated in the first repetition interval and the second loss value calculated in the second repetition interval is smaller than a reference value, the learning model may regard this as overfitting and terminate learning.

[0085] Once the learning model has sufficiently learned medical image conversion, it can complete the learning without further training. The learning model that has completed the learning in this way will now have the ability to generate medical images from third images 130 that have a different domain from the first images 110 for various first images 110 each having a different data distribution.

[0086] According to an embodiment of the present invention, the learning model 200 may be a GAN model and may include a generator and a classifier. The generator may perform image generation learning, and the classifier may perform image segmentation learning.

[0087] Referring to FIG. 2, in a learning model 200 to which a GAN model is applied, when an original CT image 110 is input as an input image to a generator 210, the generator 210 performs learning to generate a synthetic MR image 130 through the above-mentioned convolution operation, and a classifier 220 can perform learning to distinguish between the original MR image 120, i.e., the actual medical image, and the synthetic MR image 130.

[0088] According to an embodiment, a cycle-GAN model (not shown) may be applied to the learning model 200 according to this embodiment. In this case, the learning model 200 may be configured to include a first generator 210, a first division molecule 220, a second generator, and a second division molecule. When the original CT image 110 is input to the first generator 210 as an input image, the first generator 210 performs learning to generate a synthetic MR image 130 through the convolution operation described above, and the first division molecule 220 may perform learning to distinguish between the original MR image 120, i.e., the actual medical image, and the synthetic MR image. Furthermore, when the synthetic MR image 130 generated by the first generator 210 is input to the second generator, the second generator may perform learning to generate a synthetic CT image based on the synthetic MR image 130, and the second division molecule may perform learning to distinguish the synthetic CT image from the original CT image.

[0089] Here, the original CT image 110 corresponds to the first image 110 in the above-described medical image conversion method, the synthesized MR image 130 corresponds to the third image 130, and the original MR image 120 corresponds to the second image 120.

[0090] According to an embodiment, the third step (S120) may include a step of calculating a match rate between the third image 130 and the second image 120 and feeding the calculated value back to the learning model. Here, the match rate is different from the image pattern similarity described above. Calculating the match rate here is a method for determining the degree of difference between two images belonging to the same domain, and is therefore different from the image pattern similarity described above. The third step (S120) and the fourth step (S130) each compare different comparison objects using different comparison methods and feed the results back to the learning model.

[0091] FIG. 3 shows an embodiment of calculating the match rate between the output image and the target image.

[0092] In some embodiments, if a lesion area is displayed in the same location on the composite MR image and the original MR image, the matching rate can be calculated by comparing the size of the corresponding lesion area. For example, the positions of tumors and other lesions can be displayed on the composite MR image and the original MR image, respectively, and then the matching rate can be calculated by switching between binary images.

[0093] In this case, the match rate can be calculated using evaluation indices such as the Hausdorff distance or the Dice similarity coefficient, or can be quantitatively calculated based on the difference in the centers of mass of the two lesions. If the target and output images are CT images, the match rate between the two images can be determined by comparing the calculated radiation doses for the lesions. Depending on the embodiment, the difference between the synthesized and original MR images can be quantified using MAE, RMSE, SSIM, PSNR, etc. (hereinafter referred to as performance indices) for determination.

[0094] Referring to Figure 3, the process of aligning the lesion area between the synthesized MR image 130, which is the output image, and the original MR image 120, which is the target image, and then applying the Hausdorff distance measurement method to compare the contours of each lesion and calculate the matching rate is shown.

[0095] FIG. 4 illustrates an artificial intelligence learning model according to another embodiment of the present invention.

[0096] The fourth step (S130) may include a step of comparing the third image 130 with the first image 110 and learning the learning model in consideration of the comparison result. The third image 130 includes an output image 130 that the learning model receives the input image 110, converts it into a medical image, and outputs.

[0097] If the learning model outputs a synthetic MR image 130 based on an original CT image 110 but belonging to a domain different from the domain to which the original CT image 110 belongs, the output image 130 must reflect the original CT image 110.

[0098] The purpose of medical image conversion according to the present invention is to generate an output image 130 that accurately reflects an input image 110. However, since the learning model 200 is trained to reduce the difference between the output image 130 and the target image 120 during the learning process, the output image 130 may not accurately reflect the original CT image 110. In other words, there is a problem of reduced accuracy of the output image 130.

[0099] To solve this problem, the medical image conversion method according to an embodiment of the present invention can perform a process of comparing the third image 130 with the first image 110 and feeding back the comparison result to the learning model. That is, the input image 110 and the output image 130 are compared to measure the degree to which the output image 130 reflects the input image, and the result is input to the learning process of the learning model 200.

[0100] The third image 130 and the first image 110, which are the objects of comparison in the fourth step (S130), are images in different domains. That is, the third image 130 is an MR image, and the first image 110 is a CT image, so they are in different domains. This can be broadly classified as the third image 130 and the second image 120, which are the objects of comparison in the third step (S120), being images that belong to the same domain. Since the synthetic MR image and the original MR image are compared in the third step (S120), images that belong to the same domain are used as the objects of comparison. Therefore, the input values ​​that the learning model receives as feedback in the fourth step (S130) and the input values ​​that the learning model receives as feedback in the third step (S120) are different in nature.

[0101] In some embodiments, the fourth step (S130) may involve measuring the image pattern similarity between the third image 130 and the first image 110 and feeding back the measured value to the learning model. Because the domains of the synthetic MR image and the original CT image are different, the intensity patterns of the same region of interest or organ in the two images may differ. For example, if a specific region A appears as a white area in the CT image, the same region A may appear as a black area in the MR image. Therefore, to compare the differences between the two images in different domains, the image pattern similarity between the two images may be measured, and the measured value may be reflected in the learning model to update the parameters of the learning model.

[0102] One of the various methods for measuring image pattern similarity according to the present embodiment is to use the Pearson Correlation Coefficient (PCC). Alternatively, Mutual Information (MI) may be used. However, this is only one of the various methods for measuring image pattern similarity according to the present embodiment, and various other methods may be used.

[0103] Another embodiment of the present invention includes a learning unit 200 that receives a first image 110 and generates a third image 130 based on the first image 110;

[0104] a first comparing unit (300) that compares the third image (130) generated by the learning unit (200) with the second image (120) to generate a first comparison value and feeds back the first comparison value to the learning unit (200); and

[0105] An artificial intelligence learning model 100 may be provided that includes a second comparison unit 400 that compares the third image 130 generated by the learning unit 200 with the first image 110 to generate a second comparison value and feeds back the second comparison value to the learning unit 200.

[0106] In the AI ​​learning model 100 according to this embodiment, the learning unit 200 receives a first image 110 as an input image and can generate a third image 130 based on the first image 110. Depending on the embodiment, the first image 110 and the third image 130 may include images belonging to the same domain or images belonging to different domains. Depending on the embodiment, the first image 110 may represent an input image, and the third image 130 may represent an output image.

[0107] The AI ​​learning model 100 according to this embodiment may implement the medical image conversion method according to the above-described embodiment when the first image 110 and the third image 130 are images belonging to different domains.

[0108] The AI ​​learning model 100 according to this embodiment may include a first comparison unit 300 and a second comparison unit 400. The third image 130 and the second image 120, which are comparison targets of the first comparison unit 300, may include images of the same modality. The third image 130 and the first image 110, which are comparison targets of the second comparison unit 400, may include images of different modalities.

[0109] According to the embodiment, the first comparison unit 300 can compare the output image with the target image and feed back a first comparison value to the learning unit 200, so that the learning unit 200 can function to learn by reflecting the first comparison value, and the second comparison unit 400 can compare the output image with the input image and feed back a second comparison value to the learning unit 200, so that the learning unit 200 can function to learn by reflecting the second comparison value.

[0110] The learning unit 200 may receive the first comparison value and the second comparison value and perform learning. Depending on the embodiment, the first comparison value and the second comparison value may be generated in different ways.

[0111] According to an embodiment, the first comparison unit 300 may calculate a match rate between the second image 120 and the third image 130 and feed back the resultant value to the learning unit 200. For example, the first comparison value may include a result derived by calculating a match rate between images of the same modality.

[0112] According to an embodiment, the second comparing unit 400 may calculate the image pattern similarity between the first image 110 and the third image 130 and feed back the resultant value to the learning unit 200. For example, the second comparing value may include a result derived by calculating the image pattern similarity between images of different modalities. The first comparing unit 300 may be configured to perform the third step (S120) of the above-described medical image conversion method, and the second comparing unit 400 may be configured to perform the fourth step (S130).

[0113] The following mathematical formula 1 is a mathematical formula for calculating mutual information (MI).

[0114]

number

[0115] Equation 1 is a mathematical formula for calculating the interdependence information (MI) between two medical images X and Y. The interdependence information (MI) can refer to the amount of information that indicates how the information held by medical image X and the information held by medical image Y are related to each other.

[0116] Depending on the embodiment, the capital letter X and the capital letter Y may respectively refer to the original CT image 110 and the composite T2 image 130. The lowercase letter x may represent the intensity value per pixel of the original CT image 110, and the lowercase letter y may represent the intensity value per pixel of the composite T2 image 130.

[0117] In this case, the interdependence information (MI(X;Y)) expressed by Equation 1 may indicate the image pattern similarity between the original CT image 110 and the synthesized T2 image 130.

[0118] More specifically, Equation 1 can indicate how closely the intensity (y) distribution of a specific region in the composite T2 image 130 correlates with the intensity distribution (x) of the original CT image 110. For example, a bright area in the composite T2 image 130 may be dark in the original CT image 110, and Equation 1 can indicate how consistently the intensity distribution corresponds.

[0119] In this embodiment, self-feedback, which feeds back values ​​calculated using methods such as interdependent information (MI) indicating image pattern similarity, can include feedback to ensure that the output image generated by the learning model fully expresses the information in the input image.

[0120] In the process of obtaining a paired dataset including an input image and a target image, even if a CT imaging device and an MR imaging device image the same part of the same patient, the input image and the target image may not be perfectly aligned due to time constraints and differences in imaging settings. That is, when a patient acquires a CT image using a CT imaging device and then acquires an MR image using an MR imaging device again, the images are acquired at different times and in different spaces, which creates a problem in that perfectly aligned CT images and MR images cannot be obtained.

[0121] In this way, if the slices of the input image and the slices of the target image do not perfectly slice-match in the paired dataset, the output image generated by the learning model does not unconditionally follow the target image, but is generated based on the input image, which is called self-feedback.

[0122] In other words, self-feedback does not allow the learning model to learn so that the output image is simply identical to the target image when the input image and target image are not accurately aligned, but rather allows the learning model to learn so that information such as tissues and lesions that appear in the input image are fully represented in other output images.

[0123] Explain self-feedback compared to comparative feedback.

[0124] Comparative feedback is a method of feeding back the calculated value of the matching rate between the third image 130 and the second image 120 in the third step (S120) to the learning model.

[0125] Self-feedback and comparative feedback differ in the comparison objects used for feedback. Self-feedback compares input images with output images, while comparative feedback compares output images with target images. In other words, self-feedback compares images belonging to different domains, while comparative feedback compares images belonging to the same domain.

[0126] Comparative feedback simply looks at the difference in intensity patterns between the two images being compared, whereas self-feedback looks at how similar the two images are.

[0127] For example, in the case of original MR images and synthetic MR images, since the domain of the image information is the same, the matching rate between the two images can be quantified by matrix calculation between the generated synthetic MR image and the original MR image, and reflected in the learning model through comparative feedback.

[0128] However, when converting an original CT image into a synthetic MR image, the two pieces of image information belong to different image domains, so the similarity cannot be quantified simply by matrix operations between the images.

[0129] This is because the original CT image and the synthetic MR image are images from different domains, so a region that appears dark in the CT image may appear bright in the MR image, or may appear dark similarly to the CT image. Therefore, in order to reflect the similarity between the two images in the feedback process of the learning model, a calculation technique other than simple matrix calculation must be applied.

[0130] The image pattern similarity will be described more specifically with reference to FIG.

[0131] FIG. 5 shows the results of calculating image pattern similarity according to one embodiment of the present invention. Here, the image pattern similarity is calculated using Equation 1. FIG. 5(a) shows the learning model receiving an original CT image as input and outputting various output images. Here, the output images are synthetic T2 images. FIG. 5(b) shows the calculated image pattern similarity of the output images.

[0132] Figure 5 shows a brain image. Figure 5 shows the comparison of one slice from the original CT image with the entire synthetic T2 slice. For slices 8, 12, 20, 24, 28, 32, 36, 40, and 44 of the original CT image (the input image), the learning model compared them with the target image to generate output T2 images: slices 8, 12, 20, 24, 28, 32, 36, 40, and 44 of the output T2 images to correspond to the input image.

[0133] Here, the original CT image displayed as the input image in Figure 5 is the 12th slice image, and the 10 composite T2 images displayed as the output image represent the 8th, 12th, 20th, 24th, 28th, 32nd, 36th, 40th, and 44th slices, respectively.

[0134] 5, it can be seen that the 12th slice of the original CT image (which is the input image) has the highest similarity measurement value with the 12th slice of the output T2 image (a), which is the output image. It can also be seen that the 12th slice of the original CT image (which is the input image) has a slightly lower similarity measurement value with the 16th slice of the output T2 image (b) or the 20th slice of the output T2 image (c), which are the output images. The 16th slice of the output image (b) and the 20th slice of the output T2 image (c) clearly show the ventricular structure, which is not visible in the 12th slice of the input image, and the ventricular structure, which is not visible in the 12th slice of the input image, cannot be seen in the 12th slice of the output image alone.

[0135] In the end, the similarity between the 12th slice of the original CT image (the input image) and the 12th slice of the composite T2 image (the output image) is high, while the similarity is low for the rest of the images. This shows that the slice-level match rate can be calculated using pattern similarity.

[0136] That is, the anatomical structure is most similar between the 12th slice of the original CT image and the 12th slice of the output synthesized T2 image.

[0137] Image pattern similarity compares the correlation of anatomical structures between two images. Even in medical images showing the same anatomical structure, the intensity pattern varies depending on the type of image. The output images, slice 16 (b) and slice 20 (c), show ventricular structures that do not appear in the input image, so the two images do not accurately reflect the image pattern of the original image and show low similarity.

[0138] The present invention solves the problem of artificially generating structures that are not reflected in the AI ​​learning model by checking whether the image generated by the computer through the above-mentioned mathematical formula 1 is fully converted based on the information of the original image.

[0139] Image pattern similarity may mean, for example, that the volumes of gray matter and white matter, the thickness of the skull, the diameter of the skull, etc. are similar.

[0140] Even though the synthetic MR image is generated by a learning model, if there is no slice alignment between the original MR image (target image) and the original CT image (input image), the learning model will generate an inaccurate synthetic MR image, so self-feedback has been added to prevent this.

[0141] That is, FIG. 5 shows that an output image that reflects a target image whose level matches that of an input image has a high similarity to the input image, and that an output image that reflects a target image whose level does not match that of the input image has a low similarity to the input image.

[0142] In the case of the brain, since it is fixed by the skull and is rigid, there is no major problem with matching the input image and the target image, but in the case of the abdomen, the position and shape of the organs change significantly every time the patient breathes. In other words, it is difficult to perfectly match the CT image and the MR image because the internal organs are highly fluid depending on their condition.

[0143] Therefore, even if the learning model performs well, the slice levels of the target image and the input image may not match exactly, so the learning model must recognize through self-feedback whether the output image created based on the input image accurately represents the texture of the input image.

[0144] Figure 5 shows several T2 images that have already been compared with the target image, showing how similar they are to the original CT image.

[0145] As a result, it is shown that there is a high degree of similarity between the input image slices and the target image slices at the same level. Although Figure 5 corresponds to a brain, the conclusion that there is a high degree of similarity between the input image slices and the target image slices at the same level can be used in the present invention for the abdomen or other organs where matching is difficult.

[0146] That is, while image similarity does not need to be used significantly in brain images, images of the abdomen may not match or may not have paired data. According to an embodiment of the present invention, even in such cases, the image pattern similarity measurement value between the input image and the output image is fed back to the learning model to update the learning model, thereby solving the problem of reduced similarity of the output image caused by mismatch between the input image and the target image or by the components of paired data.

[0147] Another embodiment of the present invention is an image conversion device for converting medical images using a generative adversarial network (GAN), comprising:

[0148] It is possible to provide an image conversion device including: a learning model that receives an input image and generates an output image corresponding to the input image based on the input image; and a comparison unit that compares the output image with the input image and feeds back the comparison value to the learning model.

[0149] Here, the input image may be substantially the same as the first image 110 in the medical image conversion method according to the above embodiment, and the output image may be substantially the same as the third image 130.

[0150] According to an embodiment, the learning model may perform learning by comparing an output image with ground truth separately from the comparison unit, and the comparison unit may measure and compare image pattern similarities between an input image and an output image.

[0151] Here, the image pattern similarity may include Pearson Correlation Coefficient (PCC) or Mutual Information (MI).

[0152] For example, in this embodiment, the learning model may receive an input of an original CT image and generate a synthetic MR image from the original CT image using a GAN model. The comparison unit may compare the original CT image provided as input to the learning model with the synthetic MR image output by the learning model, measure the image pattern similarity between the two, and input the result to the learning model.

[0153] The above-described embodiments of the present invention may be implemented in the form of a computer program that can be executed by various components on a computer, and such a computer program may be recorded on a computer-readable medium, which may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions, such as ROMs, RAMs, and flash memories.

[0154] Meanwhile, the computer program may be one specially designed and constructed for the present invention, or one that is known and available to those skilled in the art of computer software. Examples of the computer program include not only machine language code, such as that produced by a compiler, but also high-level language code that can be executed by a computer using an interpreter, etc.

[0155] In the present specification (particularly in the claims), the use of the term "said" and similar indicators may be applicable to both the singular and the plural. Furthermore, when a range is described in the present invention, it includes inventions to which individual values ​​within the range are applied (unless otherwise specified), and is the same as if each individual value constituting the range were described in the detailed description of the invention.

[0156] Unless explicitly stated or stated to the contrary, steps constituting the method of the present invention may be performed in any suitable order. The present invention is not necessarily limited to the order of steps described above. The use of all examples or exemplary terms (e.g., etc.) in the present invention is merely for the purpose of describing the present invention in detail, and the scope of the present invention is not limited by the claims. Furthermore, those skilled in the art will recognize that various modifications, combinations, and variations can be made within the scope of the appended claims or their equivalents, depending on design conditions and factors.

[0157] Therefore, the concept of the present invention should not be limited to the above-described embodiments, and not only the scope of the claims described below, but also all scopes equivalent to or modified equivalently from the scope of the claims belong to the scope of the concept of the present invention. [Explanation of symbols]

[0158] 100 AI learning models 110 First Image 120 Second Image 130 Third Image 200 Learning Department 210 Generator 220 ward molecule 300 First Comparison Section 400 Second Comparison Section

Claims

1. A medical image conversion method for converting medical images using a generative adversarial network (GAN), a first step of receiving an input of a first image selected from a paired dataset including a first image and a second image; a second step of generating a third image corresponding to the first image and having a different domain from the first image based on the first image; a third step of comparing the third image with the second image and training a learning model taking into account the comparison results; and a fourth step of comparing the third image with the first image and training the learning model in consideration of the comparison result. A medical image conversion method comprising:

2. The fourth step measures the image pattern similarity between the third image and the first image and feeds this measurement back to the learning model. The medical image conversion method according to claim 1 .

3. The image pattern similarity includes Pearson Correlation Coefficient (PCC) or Mutual Information (MI). The medical image conversion method according to claim 2 .

4. The third step calculates a match rate between the third image and the second image, and feeds this calculated value back to the learning model, where the match rate is different from the image pattern similarity. The medical image conversion method according to claim 2 .

5. a learning unit that receives an input of a first image and generates a third image based on the first image; a first comparison unit that compares the third image generated by the learning unit with a second image to generate a first comparison value and feeds the first comparison value back to the learning unit; and a second comparing unit that compares the third image generated by the learning unit with the first image to generate a second comparison value and feeds the second comparison value back to the learning unit. An artificial intelligence learning model characterized by:

6. The first comparison value and the second comparison value are generated in different ways. The artificial intelligence learning model of claim 5.

7. The first comparison unit compares the difference between the second image and the third image, which belong to the same domain. The artificial intelligence learning model of claim 5.

8. The first comparison unit calculates a matching rate between the second image and the third image, and feeds the result back to the learning unit. The artificial intelligence learning model of claim 7.

9. The second comparison unit compares the difference between the first image and the third image, which belong to different domains. The artificial intelligence learning model of claim 5.

10. The second comparison unit calculates an image pattern similarity between the first image and the third image, and feeds the result back to the learning unit. The artificial intelligence learning model of claim 9.

11. In an image conversion device that converts medical images using a generative adversarial network (GAN), A learning model that receives an input image and generates an output image corresponding to the input image based on the input image; and a comparison unit that compares the output image with the input image and feeds back the comparison value to the learning model. A video conversion device characterized by:

12. If ground truth is available, the learning model performs learning by comparing the output image with the ground truth, separate from the comparison unit. The video conversion device according to claim 11.

13. The comparison unit measures image pattern similarity between the input image and the output image and inputs the result to the learning model. The video conversion device according to claim 11.

14. The image pattern similarity includes Pearson Correlation Coefficient (PCC) or Mutual Information (MI). The video conversion device according to claim 13.

Images