Learning device, method and program, image generating device, method and program, trained model, virtual image, and recording medium

Abstract

In the present invention, a processor inputs a first training image having a first feature to a generator, which is a generation model, to generate a virtual training image having a second feature. By performing multiple kinds of observation condition conversion processing on a second training image, multiple kinds of converted training images having different observation conditions are derived. By performing multiple kinds of observation condition conversion processing on the virtual training image, multiple kinds of converted virtual training images having different observation conditions are derived. The generation model is trained using an assessment result relating to the multiple kinds of converted training images and the multiple kinds of converted virtual training images.

Description

[Technical field]

[0001] The present disclosure relates to a learning device, a method, and a program, an image generating device, a method, and a program, a trained model, a virtual image, and a recording medium. [Background technology]

[0002] Generative Adversarial Networks (GANs) have been proposed that alternately train a generator that creates data and a discriminator that identifies data. For example, "Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio "Generative Adversarial Net work " arXiv:1406.2661 " describes research on GANs. GANs make it possible to learn a generative model that generates highly accurate data that matches the characteristics of the training data. In addition, JP2020-038410A describes technology for classification, object detection, face detection, and image generation using GANs.

[0003] Additionally, technology that applies GANs is being researched. "Image-to-Image Translation with Conditional Adversarial Networks" by Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros, CVPR 2016, describes a method for learning pairs of input and output images using GANs. The method described in the paper by Phillip Isola et al. makes it possible to learn a generative model that does not require much effort in adjusting parameters.

[0004] One problem with GAN image generation is that it is difficult to learn when there is diversity in the input data. For example, in medical images, when the parts of the input images and the slice conditions are diverse, the features of the generated images are averaged. Even when dealing with diverse input data in the paper by Phillip Isola et al., the features of the training data are averaged.

[0005] In response to this, Mehdi Mirza and Simon Osindero "Conditional Generative Adversarial Nets", arXiv:1411.1784 describes a method that uses categorical information of data for GAN training. According to the method described in the paper by Mehdi Mirza et al., the generative model can be adjusted based on categorical information.

[0006] Meanwhile, in the medical field, advances in various modalities such as CT (Computed Tomography) devices and MRI (Magnetic Resonance Imaging) devices have made it possible to perform image diagnosis using higher quality medical images.

[0007] Incidentally, the signal value (CT value) of each pixel in a CT image is a numerical representation of the absorption rate of X-rays in the human body. Specifically, the CT value is preset such that water is 0 and air is -1000 (unit: HU). In addition, since the range of possible values ​​of the CT value is very wide, such as -1000 to 4000, it is difficult to display all parts included in the CT image in an easy-to-see manner. For this reason, when displaying a CT image, observation conditions of the image are set so that a specific part is easy to see, and the density and contrast of the CT image are converted according to the set observation conditions. As the observation conditions when displaying a CT image, a window level (WL) and a window width (WW) are used. The WL is the CT value that is the center of the part to be observed in the gradation that the display can display when displaying a CT image on a display. The WW is the width between the lower limit and the upper limit of the CT value of the part to be observed. As such observation conditions, observation conditions according to the part, such as mediastinal conditions that make it easy to observe bones, lung field conditions that make it easy to observe lung fields, and brain conditions that make it easy to observe the brain, are known. Further, not only CT images but also other medical images such as MRI images or radiological images obtained by simple radiography are displayed under observation conditions that make it easy to observe a specific part.

[0008] In addition, various tasks have been proposed to generate virtual images by converting medical images using the above-mentioned GANs, such as converting the resolution of medical images to high resolution, enhancing or attenuating specific signal components, and removing artifacts. Summary of the Invention [Problem to be solved by the invention]

[0009] Here, when displaying a CT image, various observation conditions are set as described above, and the CT image is converted and displayed so as to have a density and contrast according to the set observation conditions. When a virtual image is generated using GAN with CT images under such various observation conditions as input images, a natural virtual image is generated for a CT image under a specific observation condition, but when the observation conditions are different, the generated virtual image may be unnatural. For example, a CT image with a WW narrower than that under a specific observation condition has a higher contrast than a CT image under a specific observation condition. As a result, the boundaries of structures included in the CT image may appear unnatural, or noise may be noticeable. This phenomenon occurs not only in medical images, but also in photographic images in which a subject is optically captured by a camera.

[0010] For this reason, it is possible to use the observation conditions of the input image as category information in the method described in the paper by Mehdi Mirza et al. However, in the configuration of the GAN described in the paper by Mehdi Mirza et al., it is necessary to input category information to the generator, which requires significant modifications to the network architecture and causes problems in that the interface of the generator becomes complicated.

[0011] The present disclosure has been made in consideration of the above circumstances, and aims to make it possible to generate natural images under actually expected image viewing conditions without making significant modifications to the network architecture. [Means for solving the problem]

[0012] A learning device according to the present disclosure is a learning device that learns a generative model for generating a virtual image having a second feature different from an input image having a first feature, the learning device comprising: at least one processor; The processor obtains a first training image having a first feature and a second training image having a second feature; A first learning image having a first feature is input to a generator that is a generative model, thereby generating a learning virtual image having a second feature; performing a plurality of types of observation condition conversion processing on the second learning image to derive a plurality of types of converted learning images having different observation conditions; A plurality of types of conversion processing for observation conditions is performed on the learning virtual image to derive a plurality of types of converted learning virtual images having different observation conditions; A generative model is trained using evaluation results for a plurality of types of transformation training images and a plurality of types of transformation training virtual images.

[0013] The differences between the first feature and the second feature include differences in image quality and the presence or absence of structures included, etc. The differences in image quality include differences in resolution, the presence or absence of noise, differences in the degree of emphasis of specific signal components, and differences in expression format, etc.

[0014] In addition, in the learning device according to the present disclosure, the generative model has a discriminator that outputs a discrimination result as to whether each of the multiple types of transformation learning images and the multiple types of transformation learning virtual images is a real image or a virtual image generated by a generator, The processor may be configured to learn the generative model using the discrimination result by the discriminator as an evaluation result.

[0015] In addition, in the learning device according to the present disclosure, the generation model has a discriminator that outputs a discrimination result as to whether each of the multiple types of transformation learning images and the multiple types of transformation learning virtual images is a real image or a virtual image generated by a generator; The processor derives a difference between the transformation learning image and the transformation learning virtual image that have been subjected to the same type of observation condition processing; The discrimination results and differences obtained by the discriminator may be used as evaluation results to train a generative model.

[0016] In addition, in the learning device according to the present disclosure, when the learning images are medical images, The processor further inputs body part information representing a body part of the subject included in the learning image to the discriminator; The discriminator may output a discrimination result that reflects the site information.

[0017] In addition, in the learning device according to the present disclosure, the processor derives a difference between a transformation learning image and a transformation learning virtual image that have been subjected to the same type of observation condition processing, The difference may be used as an evaluation result to train a generative model.

[0018] In addition, in the learning device according to the present disclosure, the processor normalizes the first learning image, generating a normalized training virtual image by inputting the normalized first training image to a generator; As the plurality of types of observation condition conversion processes, a plurality of types of scale conversion processes are performed on the second learning image to derive a plurality of types of converted learning images; A plurality of types of transformed training virtual images may be derived by inversely normalizing the training virtual image and performing a plurality of types of scale transformation processing on the inversely normalized training virtual image.

[0019] Furthermore, in the learning device according to the present disclosure, the first learning image and the second learning image may have the same image content.

[0020] "The same image content" means that the images include the same part of the same subject and the body positions included in the images are the same.

[0021] The generative model according to the present disclosure is trained by the learning device according to the present disclosure.

[0022] The image generating apparatus according to the present disclosure comprises at least one processor; and a generative model according to the present disclosure, The processor generates a virtual image having a second feature by inputting an image having a first feature into the generative model.

[0023] The virtual image according to the present disclosure is generated by the image generating device according to the present disclosure.

[0024] The recording medium according to the present disclosure has a virtual image according to the present disclosure recorded thereon.

[0025] Another learning device according to the present disclosure includes at least one processor, The processor Obtaining a virtual image generated by an image generating device according to the present disclosure; A trained model is constructed by performing machine learning using virtual images as training data.

[0026] A learning method according to the present disclosure is a learning method for learning a generative model for generating a virtual image having a second feature different from an input image having a first feature, the method comprising: Obtaining a first learning image having a first feature and a second learning image having a second feature; A first learning image having a first feature is input to a generator that is a generative model, thereby generating a learning virtual image having a second feature; performing a plurality of types of observation condition conversion processing on the second learning image to derive a plurality of types of converted learning images having different observation conditions; A plurality of types of conversion processing for observation conditions is performed on the learning virtual image to derive a plurality of types of converted learning virtual images having different observation conditions; A generative model is trained using evaluation results for a plurality of types of transformation training images and a plurality of types of transformation training virtual images.

[0027] The image generation method according to the present disclosure generates a virtual image having a second feature by inputting an image having a first feature into a generative model according to the present disclosure.

[0028] Another learning method according to the present disclosure includes acquiring a virtual image generated by an image generating device according to the present disclosure, A trained model is constructed by performing machine learning using virtual images as training data.

[0029] The learning method, image generation method, and other learning methods according to the present disclosure may be provided as a program for causing a computer to execute the method. Effect of the Invention

[0030] According to the present disclosure, natural images can be generated under actual expected image viewing conditions without significant modification of the network architecture and parameter tuning. [Brief description of the drawings]

[0031] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of a medical information system to which a learning device and an image generating device according to a first embodiment of the present disclosure are applied. [Diagram 2] FIG. 1 is a diagram showing a schematic configuration of an image generation system including a learning device and an image generation device according to a first embodiment; [Diagram 3] Functional configuration diagram of an image generation system according to a first embodiment [Figure 4] FIG. 1 is a conceptual diagram illustrating a learning process of a generative model according to a first embodiment; [Diagram 5] A schematic diagram of the process performed by the generator [Figure 6] Diagram to explain CT value [Figure 7] Diagram to explain WW and WL [Figure 8] A diagram showing an example of a derived transformation learning image. [Figure 9] FIG. 1 shows a display screen for a target image and a virtual image. [Figure 10] 1 is a flowchart showing a learning process performed in the first embodiment. [Figure 11] 1 is a flowchart showing an image generation process performed in the first embodiment. [Figure 12] FIG. 13 is a conceptual diagram showing a learning process of a generative model in the second embodiment; [Figure 13] FIG. 13 is a diagram for explaining the difference between a transformation learning image TRk that has been subjected to scale transformation processing and a transformation learning virtual image TVk; [Figure 14] 11 is a flowchart showing a learning process performed in the second embodiment. [Figure 15] FIG. 13 is a conceptual diagram showing a learning process of a generative model in the third embodiment. [Figure 16] 11 is a flowchart showing a learning process performed in the third embodiment. [Figure 17] FIG. 13 is a conceptual diagram showing a learning process of a generative model in the fourth embodiment; [Figure 18] FIG. 13 is a conceptual diagram showing a learning process of a generative model in the fifth embodiment. [Figure 19] FIG. 1 shows another learning device according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. Fig. 1 is a hardware configuration diagram showing an overview of a medical information system to which a learning device and an image generating device according to a first embodiment of the present disclosure are applied. As shown in Fig. 1, in the medical information system, a computer 1 including a learning device and an image generating device according to the first embodiment, an imaging device 2, and an image storage server 3 are connected in a communicable state via a network 4.

[0033] The computer 1 includes the learning device and image generating device according to the first embodiment, and constitutes an image generating system described later by installing the learning program and image generating program according to the first embodiment. The computer 1 may be a workstation or personal computer operated directly by a doctor who performs a diagnosis, or may be a server computer connected to them via a network. The learning program and image generating program are stored in a storage device of a server computer connected to a network, or in a network storage, in a state accessible from the outside, and are downloaded and installed in the computer 1 used by the doctor upon request. Alternatively, they are distributed by being recorded on a recording medium such as a DVD (Digital Versatile Disc) or a CD-ROM (Compact Disc Read Only Memory), and are installed in the computer 1 from the recording medium.

[0034] The imaging device 2 is a device that generates a three-dimensional image representing a diagnostic target part of a subject by imaging the part, and specifically, is a CT device, an MRI device, a PET (Positron Emission Tomography) device, etc. The three-dimensional image generated by the imaging device 2 and consisting of a plurality of slice images is transmitted to the image storage server 3 and stored therein.

[0035] The image storage server 3 is a computer that stores and manages various data, and is equipped with a large-capacity external storage device and database management software. The image storage server 3 communicates with other devices via a wired or wireless network 4, and transmits and receives image data and the like. Specifically, various data including image data of three-dimensional images generated by the imaging device 2 is acquired via the network, and stored and managed in a recording medium such as a large-capacity external storage device. The storage format of the image data and the communication between the devices via the network 4 are based on protocols such as DICOM (Digital Imaging and Communication in Medicine). The image storage server 3 also stores learning images, which will be described later.

[0036] Next, a learning device and an image generating device according to the first embodiment will be described. FIG. 2 describes a hardware configuration of an image generating system 20 including a learning device and an image generating device according to the first embodiment. As shown in FIG. 2, the image generating system 20 according to the first embodiment includes a CPU (Central Processing Unit) 11, a non-volatile storage 13, and a memory 16 as a temporary storage area. The image generating system 20 also includes a display 14 such as a liquid crystal display, an input device 15 such as a keyboard and a mouse, and a network I / F (InterFace) 17 connected to a network 4. The CPU 11, the storage 13, the display 14, the input device 15, the memory 16, and the network I / F 17 are connected to a bus 18. The CPU 11 is an example of a processor in the present disclosure.

[0037] Storage 13 is realized by a hard disk drive (HDD), a solid state drive (SSD), a flash memory, etc. A learning program 12A and an image generating program 12B are stored in storage 13 as a storage medium. CPU 11 reads out learning program 12A and image generating program 12B from storage 13, expands them in memory 16, and executes the expanded learning program 12A and image generating program 12B.

[0038] Next, the functional configuration of the image generation system according to the first embodiment will be described. FIG. 3 is a diagram showing the functional configuration of the image generation system according to the first embodiment. FIG. 4 is a conceptual diagram showing the learning process of the generation model in the first embodiment. As shown in FIG. 3, the image generation system 20 according to the first embodiment includes an information acquisition unit 21, a learning unit 22, a derivation unit 23, a generation unit 24, and a display control unit 25. Then, the CPU 11 executes the learning program 12A and the image generation program 12B, so that the CPU 11 functions as the information acquisition unit 21, the learning unit 22, the derivation unit 23, the generation unit 24, and the display control unit 25. The information acquisition unit 21, the learning unit 22, and the derivation unit 23 constitute the learning device according to the first embodiment, and the information acquisition unit 21, the generation unit 24, and the display control unit 25 constitute the image generation device according to the first embodiment.

[0039] The image generating device included in the image generating system 20 according to the first embodiment generates a virtual image having a second characteristic from an input image having a first characteristic. In the first embodiment, the image generating device executes a task of generating a high-resolution virtual image from a relatively low-resolution input image, for example, as described in "Virtual Thin Slice: 3D Conditional GAN-based Super-resolution for CT Slice Interval, Akira Kudo et al., 30 Aug 2019, arXiv:1908.11506." In this case, the first characteristic is the relatively low resolution, and the second characteristic is the high resolution.

[0040] The information acquisition unit 21 acquires a CT image of the subject from the image storage server 3 as a target image G0 for increasing the resolution via the network I / F 17. The information acquisition unit 21 also acquires a learning image TR0 from the image storage server 3. The learning image TR0 is a CT image acquired by the imaging device 2. The target image G0 and the virtual image GV0 have a resolution set in the imaging device 2. Here, since a CT image is a three-dimensional image, the resolution of the target image G0 and the virtual image GV0 is determined by the slice interval of the sectional images constituting the CT image and the number of pixels in the vertical and horizontal directions of each sectional image.

[0041] The information acquisition unit 21 includes a resolution reduction unit 21A as shown in FIG. 4. The resolution reduction unit 21A reduces the resolution of the learning image TR0 to derive a low-resolution learning image TL0. Specifically, the resolution reduction unit 21A derives the low-resolution learning image TL0 by thinning out slices of the learning image TR0 and downsampling the tomographic image included in the learning image TR0. Here, the learning image TL0 has a low-resolution feature, i.e., a first feature. Moreover, the learning image TR0 has a high-resolution feature, i.e., a second feature, compared to the learning image TL0. The low-resolution learning image TL0 is an example of a first learning image, and the learning image TR0 with the original resolution before being reduced in resolution is an example of a second learning image. For this reason, in the following description, the low-resolution learning image TL0 will be referred to as the first learning image TL0, and the original learning image TR0 will be referred to as the second learning image TR0. The second learning image TR0 is input to the derivation unit 23, which will be described later.

[0042] Since the first learning image TL0 is a lower-resolution version of the second learning image TR0, the first learning image TL0 has the same image content as the second learning image TR0. Here, the same image content means that the image quality is different but the same parts of the same subject are included and the body positions included in the images are the same.

[0043] The learning unit 22 learns a generative model 30 for increasing the resolution of an input image to generate a high-resolution virtual image. For the sake of explanation, FIG. 4 illustrates the generative model 30 included in the learning unit 22. As shown in FIG. 4, the generative model 30 has a generator 31 and a discriminator 32. The generator 31 has an encoder 33 and a decoder 34. In the first embodiment, the generator 31 and the discriminator 32 configure a generative adversarial network (GAN).

[0044] The encoder 33 constituting the generator 31 is a convolutional neural network (CNN), which is a type of multi-layer neural network in which multiple processing layers are hierarchically connected, and derives a feature map that represents the features of the input image.

[0045] A convolutional neural network consists of multiple convolution layers. The convolution layer performs convolution processing on the input image using various kernels, and outputs a feature map consisting of feature data obtained by the convolution processing. The kernel has a size of n×n pixels (e.g., n=3), and a weight is set for each element. Specifically, a weight is set like a differential filter that emphasizes the edges of the input image. The convolution layer applies the kernel to the entire input image or the feature map output from the previous processing layer while shifting the pixel of interest of the kernel. Furthermore, the convolution layer applies an activation function such as a sigmoid function to the convolved value, and outputs a feature map.

[0046] The decoder 34 derives a virtual image in which the resolution of the input image is increased based on the feature map derived by the encoder 33. The decoder 34 has multiple deconvolution layers. The deconvolution layers perform the same processing as the convolution layers of the encoder 33, but derive a high-resolution virtual image by applying a kernel for deconvolution calculation to the feature map while upsampling the input feature map.

[0047] Fig. 5 is a diagram showing a schematic diagram of the process performed by the generator 31. As shown in Fig. 5, when a relatively low-resolution image 40 is input to the generator 31, the generator 31 increases the resolution of the image 40 and outputs a high-resolution virtual image 41.

[0048] In the first embodiment, the learning unit 22 has a normalization unit 22A, which normalizes an image input to the generator 31 in order to efficiently learn the generative model 30. The normalization is, for example, a process of converting the range of signal values ​​that the input image can have into a predetermined range. In the first embodiment, the normalization unit 22A normalizes the range of signal values ​​that the CT image can have, for example, to a signal value in the range of 0 to 1.

[0049] When learning the generative model 30, the learning unit 22 normalizes the first training image TL0 by the normalization unit 22A. Then, the learning unit 22 inputs the normalized first training image TL0 to the generator 31 to derive a training virtual image TV0 with increased resolution. Since the first training image TL0 input to the generator 31 is normalized, the training virtual image TV0 output from the generator 31 is also normalized. Since the training virtual image TV0 has increased resolution, it has a second characteristic. The training virtual image TV0 is input to the derivation unit 23.

[0050] The derivation unit 23 performs a plurality of types of observation condition conversion processes on the second learning image TR0 to derive a plurality of types of transformed learning images TRk (k=1 to n; n is the number of observation conditions) having different observation conditions. The derivation unit 23 also performs the same observation condition conversion processes as the plurality of observation condition conversion processes performed on the second learning image TR0 on the learning virtual image TV0 to derive a plurality of types of transformed learning virtual images TVk having different observation conditions. The derivation unit 23 has an inverse normalization unit 23A, which inversely normalizes the learning virtual image TV0 to derive the transformed learning virtual image TVk. Note that inverse normalization refers to converting a signal value normalized to be in the range of 0 to 1, for example, into the range of the original signal value that a CT image can take.

[0051] Here, in the first embodiment, the derivation unit 23 performs a scale conversion process as an observation condition conversion process. The scale conversion process will be described below. Each pixel value (hereinafter, referred to as a CT value) of a CT image targeted in the first embodiment is a numerical representation of the X-ray absorption rate in the human body. Specifically, as shown in FIG. 6, the CT value is determined according to the composition constituting the human body, such that water is 0 and air is -1000. Also, the range of values ​​that the CT value can take is very wide, such as -1000 to 4000. On the other hand, the range of gradations that the display 14 can reproduce is about 0 to 1024. For this reason, it is difficult to display all gradations included in a CT image on the display 14.

[0052] Therefore, when a CT image is displayed, the observation conditions of the image are set so that a specific part is easily seen, and the density and contrast of the CT image are converted based on the set observation conditions. Specifically, a window level (WL) and a window width (WW) are set as the observation conditions. WL is the central CT value of the part to be observed in the gradation that the display 14 can display when the CT image is displayed on the display 14. WW is the signal width between the lower limit and the upper limit of the CT value of the part to be observed.

[0053] FIG. 7 is a diagram showing examples of WW and WL. The values ​​shown in FIG. 7 are CT values ​​in units of HU. As shown in FIG. 7, the mediastinal conditions under which bones are easily observed have a WL of 20 to 60, and a WW of 300 to 400. Moreover, the lung field conditions under which the lung field is easily observed have a WL of -770 to -550, and a WW of 1000 to 1500. Moreover, the observation conditions under which the brain is easily observed have a WL of 20 to 60, and a WW of 60 to 120.

[0054] In the first embodiment, a plurality of viewing conditions consisting of combinations of WW and WL are set. For example, five types of viewing conditions, (WW,WL)=(4096,0), (1600,-600), (400,0), (200,50), and (80,40), are set as combinations of WW and WL. The derivation unit 23 derives a plurality of types of transformation learning virtual images TVk and a plurality of types of transformation learning images TRk by converting the learning virtual image TV0 and the second learning image TR0 according to each of the plurality of viewing conditions. When deriving the plurality of types of transformation learning virtual images TVk and the plurality of types of transformation learning images TRk, the derivation unit 23 performs a scale conversion process. Note that the scale conversion process for the learning virtual image TV0 and the second learning image TR0 is the same, so here, the scale conversion process for the learning image TR0 will be described.

[0055] The derivation unit 23 clips CT values ​​outside the range of the derived observation conditions for the CT values ​​of each pixel of the learning image TR0. For example, when the observation conditions are (WW, WL)=(400, 0), the derivation unit 23 sets a CT value range of 400 with 0 as the center as WL in the learning image TR0. In this case, the CT value range is -200 to 200. The derivation unit 23 then clips CT values ​​outside the set range. That is, the range of CT values ​​smaller than -200 is converted to -200, and the range of CT values ​​larger than 200 is converted to 200.

[0056] Then, the derivation unit 23 performs a scale conversion process by deriving the pixel value of each pixel of the training image TR0 according to the following formula (1) using the maximum and minimum CT values ​​in the set range. In formula (1), p_min and p_max are the minimum and maximum CT values ​​under the set observation conditions, respectively, pb_i is the pixel value of pixel i in the training image TR0 before the scale conversion process, and pp_i is the pixel value of pixel i in the training image TR0 after the scale conversion process. The pp_i derived by formula (1) is normalized to the range of 0 to 1. pp_i=(pb_i-p_min) / (p_max-p_min) (1)

[0057] The scale conversion process is not limited to that performed by the above formula (1). For example, scale conversion process may be performed by the following formula (2) within the range of the minimum and maximum CT values ​​under the set observation conditions, so that the average value becomes 0 and the standard deviation becomes 1. In formula (2), μ is the average value of the signal values ​​within the range of the minimum and maximum CT values ​​under the set observation conditions, and σ is the standard deviation of the signal values ​​within that range. pp_i=(pb_i-μ) / σ (2)

[0058] Furthermore, the scale conversion process can be performed using any known method, such as a method using the median or quantile within the range of the minimum and maximum CT values ​​under the set observation conditions.

[0059] The derivation unit 23 derives a plurality of types of transformation learning virtual images TVk and a plurality of types of transformation learning images TRk by performing scale conversion on the training virtual image TV0 and the second training image TR0 under each of a plurality of types of observation conditions. Fig. 8 is a diagram showing an example of the derived transformation learning images. As shown in Fig. 8, the derivation unit 23 derives five types of transformation learning images TR1 to TR5 having different densities and contrasts from the training image TR0.

[0060] Returning to the learning unit 22, the discriminator 32 is, for example, a convolutional neural network, which discriminates whether the input image is a real image or a virtual image generated by the generator 31, and outputs a discrimination result RF0. Here, the real image is not an image generated by the generator 31, but an image acquired by photographing a subject with the photographing device 2, and an image obtained by applying a scale conversion process to the image thus acquired by the derivation unit 23. In contrast, the virtual image is a virtual image generated from the real image by the generator 31, and an image obtained by applying a scale conversion process to the virtual image by the derivation unit 23.

[0061] Therefore, when the discriminator 32 judges that the input real image is a real image, the discrimination result RF0 is correct, and when it judges that the input virtual image is a virtual image, the discrimination result RF0 is incorrect. Also, when the discriminator 32 judges that the input virtual image is a real image, the discrimination result RF0 is incorrect, and when it judges that the input virtual image is a virtual image, the discrimination result RF0 is correct.

[0062] In the first embodiment, when a conversion learning virtual image TVk is input to the discriminator 32 during learning, if the discrimination result RF0 is a virtual image, the discrimination result RF0 is correct, and if the discrimination result is a real image, the discrimination result RF0 is incorrect. Also, when a conversion learning image TRk, which is a real image, is input to the discriminator 32 during learning, if the discrimination result RF0 is a real image, the discrimination result RF0 is correct, and if the discrimination result is a virtual image, the discrimination result RF0 is incorrect.

[0063] The learning unit 22 derives a loss based on the discrimination result RF0 output by the discriminator 32. This loss is called an adversarial loss. In this embodiment, the adversarial loss in the discriminator 32 is defined as a loss L0.

[0064] In the first embodiment, the learning unit 22 trains the discriminator 32 so that the discrimination result RF0 as to whether an input image is a real image or a virtual image generated by the generator 31 is correct. That is, the discriminator 32 is trained so as to maximize the loss L0. The learning unit 22 also derives a high-resolution virtual image that resembles the real image from the input image, and trains the generator 31 so that the discriminator 32 determines the discrimination result RF0 to be incorrect.

[0065] As the learning progresses, the generator 31 and the discriminator 32 mutually improve their accuracy, and the generator 31 becomes able to generate high-resolution CT images that are closer to real CT images and cannot be identified by the discriminator 32. In the first embodiment, the generator 31 in the trained generative model 30 is applied to the generating unit 24 described later.

[0066] In the first embodiment, a discrimination result RF0 corresponding to the number of set observation conditions is output from the discriminator 32. During learning, the generator 31 and the discriminator 32 may be trained by using losses L0 derived based on the individual discrimination results RF0, or the generator 31 and the discriminator 32 may be trained by deriving a sum of losses L0 derived based on the individual discrimination results RF0 and using the sum of losses L0.

[0067] The generation unit 24 is configured to apply a trained generative model 30, and when a target image G0 to be processed is input to the image generation system 20, the trained generative model 30 increases the resolution of the target image G0 to derive a high-resolution virtual image GV0. Note that although the target image G0 is a CT image, the density and contrast are converted based on specific observation conditions, making it easier to see specific structures included in the CT image.

[0068] The display control unit 25 displays a high-resolution virtual image GV0 generated from the target image G0 on the display 14. Fig. 9 is a diagram showing a display screen of the high-resolution virtual image GV0. As shown in Fig. 9, the display screen 50 displays the high-resolution virtual image GV0 including the head as a subject and the target image G0 to be processed. As shown in Fig. 9, it can be seen that the virtual image GV0 has a higher resolution than the target image G0.

[0069] Next, the processing performed in the first embodiment will be described. Fig. 10 is a flowchart showing the learning processing performed in the first embodiment. It is assumed that the learning image (i.e., the second learning image TR0) is acquired from the image storage server 3 by the information acquisition unit 21 and stored in the storage 13. First, the resolution reduction unit 21A of the information acquisition unit 21 reduces the resolution of the second learning image TR0 to derive the first learning image TL0 (step ST1).

[0070] Next, the normalization unit 22A of the learning unit 22 normalizes the first learning image TL0 (step ST2), and inputs the normalized first learning image TL0 to the generator 31. The generator 31 derives a learning virtual image TV0 (step ST3). Meanwhile, the derivation unit 23 performs multiple types of scale conversion processing on the second learning image TR0 to derive multiple types of transformed learning images TRk (step ST4). In addition, the derivation unit 23 performs multiple types of scale conversion processing on the learning virtual image TV0 to derive multiple types of transformed learning virtual images TVk (step ST5).

[0071] Then, the learning unit 22 inputs a plurality of types of transformation training images TRk and a plurality of types of transformation training virtual images TVk to the discriminator 32. The discriminator 32 outputs a discrimination result RF0 (step ST6).

[0072] Next, the learning unit 22 derives a loss L0 based on the discrimination result RF0 (step ST7). Furthermore, the learning unit 22 learns the generator 31 and the discriminator 32 based on the loss L0 (step ST8). Then, the process returns to step ST1, and the next learning image is obtained from the storage 13, and the processes of steps ST1 to ST8 are repeated. As a result, a trained generative model 30 is constructed.

[0073] Although the learning unit 22 repeats learning so as to maximize the loss L0, it may repeat learning a predetermined number of times.

[0074] Next, the image generation process performed in the first embodiment will be described. FIG. 11 is a flowchart of the image generation process performed in the first embodiment. The information acquisition unit 21 acquires a target image G0 to be processed (step ST11), and the generation unit 24 inputs the target image G0 to the trained generative model 30, thereby increasing the resolution of the target image G0 and deriving a high-resolution virtual image GV0 (step ST12). Then, the display control unit 25 displays the virtual image GV0 on the display 14 (step ST13), and the process ends. Note that when the target image G0 is input to the trained generative model 30, the generation unit 24 normalizes the target image G0. In this case, since the virtual image GV0 output from the trained generative model 30 is normalized, the generation unit 24 performs inverse normalization to derive the virtual image GV0.

[0075] In this manner, in the first embodiment, when training the generative model 30, a plurality of types of transformation training images TRk having different training conditions are derived by performing a plurality of types of observation condition conversion processing on the second training image TR0, and a plurality of types of transformation training virtual images TVk having different training conditions are derived by performing a plurality of types of observation condition conversion processing on the training virtual image TV0 output from the generator 31. Then, a discrimination result RF0 as to whether the plurality of types of transformation training images TRk and the plurality of types of transformation training virtual images TVk are real images or virtual images is output from the discriminator 32, and the generator 31 and the discriminator 32 are trained based on the discrimination result RF0.

[0076] Therefore, as the learning progresses, the generator 31 and the discriminator 32 improve each other's accuracy, and the discriminator 32 can discriminate whether an image is a real image or a virtual image with a higher accuracy regardless of the observation conditions of the image input. Furthermore, the generator 31 can generate a high-resolution virtual image closer to a real image that is not discriminated by the discriminator 32 regardless of the observation conditions of the input image. In particular, in the first embodiment, since a plurality of types of transformation learning images TRk and a plurality of types of transformation learning virtual images TVk with different observation conditions are input to the discriminator 32, a high-resolution virtual image closer to a real image can be generated regardless of the observation conditions of the image input. Furthermore, in the first embodiment, only an image is input to the generator 31, and no category information such as the observation conditions is input as described in the literature of Mehdi Mirza et al. Therefore, according to the first embodiment, a natural image can be generated under the observation conditions of the image that are actually assumed, without making significant modifications to the network architecture.

[0077] Next, a second embodiment of the present disclosure will be described. The configurations of the learning device and image generating device according to the second embodiment are the same as those of the learning device and image generating device according to the first embodiment, and only the processing performed during learning is different, so detailed description of the devices will be omitted here. Fig. 12 is a conceptual diagram showing the learning processing of a generative model in the second embodiment. Fig. 12 shows only the learning unit 22 in Fig. 4, and omits the information acquisition unit 21 and derivation unit 23.

[0078] Here, it is desirable that the training virtual image TV0 output by the generator 31 completely matches the second training image TR0, but since it has been processed by the generator 31, a difference occurs between the training virtual image TV0 and the second training image TR0. As a result, a difference occurs between the transformed training image TRk and the transformed training virtual image TVk, which have been subjected to the same type of scale conversion processing. FIG. 13 is a diagram for explaining the difference between the transformed training image TRk and the transformed training virtual image TVk, which have been subjected to the scale conversion processing. As shown in FIG. 13, it is assumed that the transformed training virtual images TV1 and TV2 are derived from the training virtual image TV0 by two types of scale conversion processing. It is also assumed that the transformed training images TR1 and TR2 are derived from the second training image TR0 by the same two types of scale conversion processing.

[0079] In the second embodiment, the learning unit 22 includes a difference derivation unit 22B, which derives the difference between the transformation learning image TRk and the transformation learning virtual image TVk that have been subjected to the same type of scale conversion processing as the loss L1. That is, the difference between the transformation learning virtual image TV1 and the transformation learning image TR1 shown in FIG. 13, and the difference between the transformation learning virtual image TV2 and the transformation learning image TR2 are each derived as the loss L1. Note that the difference derivation unit 22B derives the sum of the absolute values ​​of the differences between corresponding pixels between the transformation learning image TRk and the transformation learning virtual image TVk that have been subjected to the same type of scale conversion processing as the loss L1, but is not limited thereto. The difference derivation unit 22B may derive the sum of the squared sums of the differences between corresponding pixels as the loss L1.

[0080] In the second embodiment, the learning unit 22 learns the generator 31 and the discriminator 32 based on the loss L0 based on the discrimination result RF0 output by the discriminator 32 and the loss L1 derived by the difference derivation unit 22B. That is, in the second embodiment, the learning unit 22 learns the discriminator 32 so that the discrimination result RF0 of whether the input image is a real image or a virtual image generated by the generator 31 is correct. That is, the learning unit 22 learns the discriminator 32 so as to maximize the loss L0. In addition, the learning unit 22 derives a high-resolution virtual image resembling the real image from the input image, and learns the generator 31 so that the discriminator 32 determines the discrimination result RF0 to be incorrect and the loss L1 is minimized. Learning to minimize the loss L1 preferably means learning to make the loss L1 0 or less than a predetermined threshold value.

[0081] In the second embodiment, losses L0 and L1 corresponding to the number of set viewing conditions are derived. In learning, the generator 31 and the discriminator 32 may be trained by using the losses L0 and L1 derived based on the discrimination result RF0 for each viewing condition. In addition, the sum of the losses L0 and the sum of the losses L1 derived based on the discrimination result RF0 may be derived, and the generator 31 and the discriminator 32 may be trained by using the sum of the losses L0 and the sum of the losses L1.

[0082] Next, the process performed in the second embodiment will be described. FIG. 14 is a flowchart showing the process performed in the second embodiment. In the flowchart shown in FIG. 14, the process from step ST21 to step ST26 is the same as the process from step ST1 to step ST6 shown in FIG. 10, so detailed description will be omitted here. In the second embodiment, following step ST26, the learning unit 22 derives a loss L0 based on the discrimination result RF0, and derives the difference between the transformation learning image TRk and the transformation learning virtual image TVk, which have been subjected to the same type of scale conversion process, as a loss L1 (step ST27). Furthermore, the learning unit 22 learns the generator 31 and the discriminator 32 based on the losses L0 and L1 (step ST28). Then, the process returns to step ST21, and the next learning image is obtained from the storage 13, and the process from step ST21 to step ST28 is repeated. As a result, the learned generative model 30 is constructed.

[0083] Next, a third embodiment of the present disclosure will be described. The configurations of the learning device and image generating device according to the third embodiment are the same as those of the learning device and image generating device according to the first embodiment, and only the processing performed during learning is different, so detailed description of the devices will be omitted here. Figure 15 is a conceptual diagram showing the learning processing of a generative model in the third embodiment. Note that in Figure 15, only the learning unit 22 in Figure 4 is shown, and the information acquisition unit 21 and the derivation unit 23 are omitted.

[0084] As shown in FIG. 15, the third embodiment differs from the first embodiment in that the discriminator 32 is omitted, the learning unit 22 includes a difference derivation unit 22B, and the generator 31 is trained based only on the loss L1, which is the difference between a transformation training image TRk and a transformation training virtual image TVk that have been subjected to the same type of scale transformation processing.

[0085] In the third embodiment, the learning unit 22 learns the generator 31 based only on the loss L1. That is, in the third embodiment, the learning unit 22 derives a high-resolution virtual image that resembles the actual image from the input image, and learns the generator 31 so that the loss L1 is minimized. Learning to minimize the loss L1 preferably means learning so that the loss L1 becomes 0 or is equal to or less than a predetermined threshold value.

[0086] In the third embodiment, the loss L1 corresponding to the number of set observation conditions is derived. During learning, the generator 31 may be trained by using the loss L1 derived for each observation condition. Alternatively, the sum of the losses L1 derived based on each observation condition may be derived, and the generator 31 may be trained by using the sum of the losses L1.

[0087] Next, the process performed in the third embodiment will be described. FIG. 16 is a flowchart showing the process performed in the third embodiment. In the flowchart shown in FIG. 16, the process from step ST31 to step ST36 is the same as the process from step ST1 to step ST6 shown in FIG. 10, so detailed description will be omitted here. In the third embodiment, following step ST36, the learning unit 22 derives the difference between the transformation learning image TRk and the transformation learning virtual image TVk, which have been subjected to the same type of scale conversion process, as a loss L1 (step ST37). Furthermore, the learning unit 22 learns the generator 31 based on the loss L1 (step ST38). Then, the process returns to step ST31, the next learning image is obtained from the storage 13, and the process from step ST31 to step ST38 is repeated. As a result, the learned generation model 30 is constructed.

[0088] In the first embodiment, as shown in FIG. 17, the region information B0 included in the learning image TR0 may be input to the discriminator 32. Also in the second embodiment, the region information B0 included in the learning image TR0 may be input to the discriminator 32. Here, the region information B0 is information representing the region of the subject included in the learning image TR0. The region information includes, for example, the head, chest, abdomen, and legs. Here, the multiple types of observation conditions WW and WL are set according to the region included in the learning image TR0, i.e., the CT image, as shown in FIG. 7. Therefore, by inputting the region information B0 to the discriminator 32, the discriminator 32 can be trained to output an appropriate discrimination result RF0 according to the region included in the image.

[0089] Next, a fourth embodiment of the present disclosure will be described. In the above first and second embodiments, a discrimination result RF0 for the transformation learning virtual image TVk and the transformation learning image TRk according to the number of set observation conditions is derived by using one discriminator 32. The fourth embodiment differs from the first and second embodiments in that the generative model 30 is provided with a number of discriminators according to the number of set observation conditions.

[0090] FIG. 18 is a conceptual diagram showing the learning process of the generative model in the fourth embodiment. Note that FIG. 18 shows only the generative model in FIG. 4. As shown in FIG. 18, the generative model 30 in the fourth embodiment includes first to third discriminators 32A to 32C, the number of which corresponds to the number of set observation conditions (three in this case). The first discriminator 32A receives a transformation learning virtual image TV1 and a transformation learning image TR1 that have been subjected to the same scale conversion process, and outputs a discrimination result RF1. The second discriminator 32B receives a transformation learning virtual image TV2 and a transformation learning image TR2 that have been subjected to the same scale conversion process, and outputs a discrimination result RF2. The third discriminator 32C receives a transformation learning virtual image TV3 and a transformation learning image TR3 that have been subjected to the same scale conversion process, and outputs a discrimination result RF3.

[0091] In the fourth embodiment, the learning unit 22 derives losses L0-1, L0-2, and L0-3 based on the discrimination results RF1 to RF3, respectively. Then, the learning unit 22 uses the losses L0-1, L0-2, and L0-3, respectively, to train the generator 31 and the discriminator 32. Note that the sum of the losses L0-1, L0-2, and L0-3 may be derived, and the generator 31 and the discriminator 32 may be trained by using the sum of the losses L0-1, L0-2, and L0-3.

[0092] It should be noted that the second embodiment may also be provided with a plurality of discriminators 32A to 32C, as in the fourth embodiment.

[0093] In the first and second embodiments, the generative model 30 is trained using the first training image TL0 and the second training image TR0 having the same image content, but the present invention is not limited to this. The first training image and the second training image having different image content may be used.

[0094] In this case, the learning unit 22 may use, for example, the CycleGAN method described in "Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks", Jun-Yan Zhu, arxiv:1703.10593. When the CycleGAN method is applied to this embodiment, the generation model 30 includes a generator 31 that increases the resolution of an input image and a discriminator 32 that determines whether the input high-resolution image is true or false, as well as another generator that reduces the resolution of the input image and another discriminator that determines whether the input low-resolution image is true or false. When learning, any learning images with different resolutions are prepared, and a low-resolution image is input to the generator 31 to generate a high-resolution virtual image. Also, a high-resolution image is input to another generator to generate a low-resolution virtual image. Then, the high-resolution image and the high-resolution virtual image are input to the discriminator 32 to output a discrimination result. Also, the low-resolution image and the low-resolution virtual image are input to another discriminator to output a discrimination result. Then, based on the discrimination results of both, a combination of a generator 31 and a discriminator 32 and a combination of another generator and another discriminator are learned, thereby constructing a learned generative model 30.

[0095] In addition, in each of the above embodiments, the generator 31 performs a task of increasing the resolution of the input image to generate a virtual image, but is not limited to this. For example, the generator may perform a task of emphasizing or attenuating a specific signal component of the input image, as described in "Review of Convolutional Neural Networks for Inverse Problems in Imaging, Michael T. McCann et al., arXiv:1710.04011v1 [eess.IV] 11 Oct 2017." In this case, the difference between the first feature and the second feature is the difference in the degree of intensity of a specific signal component included in the image.

[0096] Alternatively, the task of removing artifact-like noise from an input image may be performed as described in "Sharpness-aware Low dose CT denoising using conditional generative adversarial network, Xin Yi et al., arXiv:1708.06453, 22 Aug 2017." In this case, the first feature is that the input image contains artifact-like noise, and the second feature is that the input image does not contain noise.

[0097] Alternatively, the task may be to suppress a specific structure included in an input image, as described in "Learning Bone Suppression from Dual Energy Chest X-rays using Adversarial Networks, Dong Yul Oh et al., arXiv:1811.02628, 5 Nov 2018." In this case, the first feature is that a specific structure is included, and the second feature is that the specific structure is suppressed.

[0098] Alternatively, the GAN may perform a task of embedding pseudo-lesions in an input image, as described in "Synthesizing Diverse Lung Nodules Wherever Massively: 3D Multi-Conditional GAN-based CT Image Augmentation for Object Detection, Changhee Han et al., arXiv:1906.04962, 12 Jun 2019." In this case, the first feature is the absence of a lesion, and the second feature is the inclusion of a lesion. Thus, the difference between the first feature and the second feature is the difference between the presence or absence of a lesion.

[0099] Furthermore, the task may be a task of converting the representation format of an input image into a different representation format, as described in "Deep MR to CT Synthesis using Unpaired Data, Jelmer M. Wolterink et al., Simulation and Synthesis in Medical Imaging pp 14-23, 26 September 2017." An example of such a task is a task of converting the representation format of an MRI image into the representation format of a CT image. In this case, the difference between the first feature and the second feature is the difference in the representation format.

[0100] In addition, in each of the above embodiments, the image input to the generator 31 is a CT image, but is not limited thereto. In addition to three-dimensional images such as MRI images and PET images, ultrasound images and radiographic images acquired by simple radiography can be used as input images. In this case, observation conditions according to the modality of the input image are prepared, and the learning image is subjected to a scale conversion process and is used for learning the generator 31 and the discriminator 32. As a result, the learning unit 22 learns the generator 31 and the discriminator 32 so as to adapt to the modality of the input image.

[0101] In addition, in each of the above embodiments, a medical image is used as the image input to the generator 31, but the present invention is not limited to this. For example, a photographic image can also be used as the input image. In this case, a plurality of types of observation conditions are prepared according to the desired density and contrast of the input photographic image, and the scale conversion process of the learning image is performed, and the image is used for learning of the generator 31 and the discriminator 32.

[0102] In addition, in each of the above embodiments, the image input to the generator 31 is normalized, but the present invention is not limited to this. The input image may be input to the generator 31 without being normalized. In this case, the target image G0 can be input to the trained generative model 30 without being normalized.

[0103] In each of the above embodiments, a virtual image derived from an input image may be transmitted to the image storage server 3 and stored therein. Also, the virtual image may be recorded on a recording medium such as a CD-ROM, a DVD, or a USB memory. The virtual image recorded in this manner can be used for machine learning of a learning model that executes a task different from that of the generation model 30 of this embodiment. For example, as shown in a learning device 60 according to another embodiment shown in FIG. 19, the virtual image GV0 generated by this embodiment may be used as teacher data for learning a learning model 61.

[0104] An example of such a learning model 61 is a learning model for executing a task of estimating the position of a lesion included in a medical image. When learning such a learning model 61, a sufficient number of high-resolution teacher data may not be available. In such a case, by using the virtual image derived by the generation model 30 according to this embodiment as teacher data, the shortage of teacher data can be eliminated and the robustness of the learning model 61 can be improved.

[0105] Further, the learning model 61 may be a learning model for executing a task of identifying regions of structures such as organs included in a CT image. When learning such a learning model 61, a correct answer label for identifying regions of structures such as organs included in a CT image is required as training data. However, there are cases where training data for CT images cannot be sufficiently prepared. In such a case, if the learning device of each of the above embodiments is used to construct a generation model 30 that executes a task of converting the representation format of an input image from MRI to CT, a virtual CT image having a correct answer label of a structure region such as an organ can be derived by the generation model 30 from an MRI image having a correct answer label of a structure region. Therefore, even in such a case, the shortage of training data can be eliminated, and the robustness of the learning model 61 can be improved.

[0106] In each of the above embodiments, the following various processors can be used as the hardware structure of the processing unit that executes various processes, such as the information acquisition unit 21, the learning unit 22, the derivation unit 23, the generation unit 24, and the display control unit 25. As described above, the above various processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, as well as a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), and a dedicated electric circuit, which is a processor having a circuit configuration designed specifically for executing specific processes, such as an ASIC (Application Specific Integrated Circuit).

[0107] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor.

[0108] As an example of configuring multiple processing units with one processor, first, there is a form in which one processor is configured with a combination of one or more CPUs and software, as typified by computers such as client and server, and this processor functions as multiple processing units. Second, there is a form in which a processor is used that realizes the functions of the entire system including multiple processing units with one IC (Integrated Circuit) chip, as typified by System On Chip (SoC), etc. In this way, the various processing units are configured using one or more of the above various processors as a hardware structure.

[0109] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements. [Explanation of symbols]

[0110] 1 computer 2. Imaging Device 3. Image storage server 4. Network 11 CPU 12A Study Program 12B Image Generation Program 13. Storage 14 Display 15 Input device 16 Memory 17 Network Interface 20 Image Generation System 21 Information Acquisition Department 21A Low resolution section 22 Learning Department 22A Normalization section 22B Difference derivation part 23 Derivation part 23A Denormalization section 24 Generation part 25 Display control section 30 Generative Model 31 Generator 32,32A~32C Discriminator 33 Encoder 34 Decoder 40 images 41 Virtual Images 50 display screen 60 Learning Device 61 Learning Model G0 Target image GV0 Virtual Image TR0 Learning Images TRk transformation learning images TV0 Virtual image for learning Virtual images for TVk transformation training RF0~RF3 discrimination results L0,L1 loss

Claims

1. A learning device that learns a generative model for generating a virtual image having a second feature different from an input image having a first feature, the learning device comprising: At least one processor; The processor, acquiring a first learning image having the first feature and a second learning image having the second feature; A first learning image having the first feature is input to a generator that is the generative model, thereby generating a learning virtual image having the second feature; performing a plurality of types of observation condition conversion processing on the second learning image to derive a plurality of types of converted learning images having different observation conditions; deriving a plurality of types of transformed learning virtual images having different viewing conditions by performing the plurality of types of viewing condition transformation processes on the learning virtual images; A learning device that learns the generative model using evaluation results regarding the multiple types of transformation learning images and the multiple types of transformation learning virtual images.

2. The generative model has a discriminator that outputs a discrimination result as to whether each of the plurality of types of transformation learning images and the plurality of types of transformation learning virtual images is a real image or a virtual image generated by the generator, The learning device according to claim 1 , wherein the processor learns the generative model by using a discrimination result by the discriminator as the evaluation result.

3. The generative model has a discriminator that outputs a discrimination result as to whether each of the plurality of types of transformation learning images and the plurality of types of transformation learning virtual images is a real image or a virtual image generated by the generator, The processor derives a difference between the transformation learning image and the transformation learning virtual image that have been subjected to the same type of viewing condition processing, The learning device according to claim 1 , wherein the generative model is learned by using the discrimination result by the discriminator and the difference as the evaluation result.

4. When the first learning image and the second learning image are medical images, The processor further inputs region information representing regions of the subject included in the first learning image and the second learning image to the discriminator; The learning device according to claim 2 or 3, wherein the discriminator outputs the discrimination result that reflects the part information.

5. The processor derives a difference between the transformation learning image and the transformation learning virtual image that have been subjected to the same type of viewing condition processing, The learning device according to claim 1 , wherein the difference is used as the evaluation result to learn the generative model.

6. The processor normalizes the first training image; generating the normalized virtual training image by inputting the normalized first training image to the generator; As the plurality of types of viewing condition conversion processes, a plurality of types of scale conversion processes are performed on the second learning image to derive the plurality of types of converted learning images; The learning device according to any one of claims 1 to 5, wherein the learning virtual image is inverse-normalized, and the multiple types of scale conversion processing are applied to the inverse-normalized learning virtual image to derive the multiple types of converted learning virtual images.

7. The learning device according to claim 1 , wherein the first learning image and the second learning image include the same part of the same subject and include the same body position.

8. At least one processor; A generation model trained by the learning device according to any one of claims 1 to 7, The processor is an image generation device that generates a virtual image having a second feature by inputting an image having a first feature into the generative model.

9. At least one processor; The processor, Obtaining a virtual image generated by the image generating device according to claim 8; A learning device that constructs a trained model by performing machine learning using the virtual image as training data.

10. 1. A learning method for learning a generative model for generating a virtual image having a second feature different from an input image having a first feature, the method comprising: acquiring a first learning image having the first feature and a second learning image having the second feature; A first learning image having the first feature is input to a generator that is the generative model, thereby generating a learning virtual image having the second feature; performing a plurality of types of observation condition conversion processing on the second learning image to derive a plurality of types of converted learning images having different observation conditions; deriving a plurality of types of transformed learning virtual images having different viewing conditions by performing the plurality of types of viewing condition transformation processes on the learning virtual images; A learning method for learning the generative model using evaluation results for the multiple types of transformation learning images and the multiple types of transformation learning virtual images.

11. An image generation method for generating a virtual image having a second feature by inputting an image having a first feature into a generative model trained by a learning device described in any one of claims 1 to 7.

12. Obtaining a virtual image generated by the image generating device according to claim 8; A learning method for constructing a trained model by performing machine learning using the virtual image as training data.

13. A learning program for causing a computer to execute a learning method for learning a generative model for generating a virtual image having a second feature different from an input image having a first feature, the method comprising: acquiring a first learning image having the first feature and a second learning image having the second feature; A step of generating a virtual learning image having the second feature by inputting a first learning image having the first feature to a generator that is the generative model; a step of performing a plurality of types of observation condition conversion processing on the second learning image to derive a plurality of types of converted learning images having different observation conditions; a step of performing the plurality of types of viewing condition conversion processes on the learning virtual image to derive a plurality of types of converted learning virtual images having different viewing conditions; and a step of learning the generative model using evaluation results regarding the plurality of types of transformation learning images and the plurality of types of transformation learning virtual images.

14. An image generation program that causes a computer to execute a procedure for generating a virtual image having a second characteristic by inputting an image having a first characteristic into a generative model learned by a learning device described in any one of claims 1 to 7.

15. acquiring a virtual image generated by the image generating device according to claim 8; and a step of constructing a trained model by performing machine learning using the virtual image as training data.

Images