Image translation device, image diagnostic system, image translation method, control program, recording medium

Abstract

The present invention achieves an image translation device, for example, that can effectively utilize an existing diagnostic imaging model. An image translation device (1) comprises an image translation unit (22) that includes a first generator (221) and a second generator (222), the image translation unit (22) having learned the relationship between a color-related feature of a first image group obtained by capturing images of a tissue on which a first process including an embedding process and slicing has been performed, and a color-related feature of a second image group obtained by capturing images of the tissue on which a second process that does not include the embedding process and slicing has been performed. A first object image belonging to the first image group or a second object image belonging to the second image group is input to the image translation unit (22), which then outputs a first translation image generated from the first object image or a second translation image generated from the second object image.

Description

Technical Field

[0006] , , , , , , , , , , ,

[0005] ,

[0001] The present invention relates to an image translation device that converts an image of an imaged tissue, an image translation method, an image diagnosis system including the image translation device, and the like.

Background Art

[0002] A technique for generating different images from acquired medical images using artificial intelligence is known.

[0003] Patent Document 1 discloses a medical image processing device that generates a high-quality image from an acquired medical image using a high-quality engine (artificial intelligence).

[0004] Non-Patent Document 1 discloses a technique for generating a virtual HE-stained image from an image of non-stained lung tissue by conditional GAN (CGAN). Non-Patent Document 2 discloses a technique for generating a virtual tissue image obtained when a staining method different from HE staining is hypothetically applied from an HE-stained image by deep learning. The staining methods specifically described in Non-Patent Document 2 as staining methods different from HE staining are Masson's trichrome staining method, PAS (periodic acid-Schiff) staining method, and Jones silver staining method.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Non-Patent Documents

Non-Patent Document 1

[0007] There are many types of embedding and staining treatments that can be applied to tissue in order to acquire images of that tissue. For example, the color tone (color characteristics) of the imaged tissue will differ depending on the staining treatment used. Furthermore, techniques have been developed to observe and image tissue by irradiating it with light of wavelengths not previously used or by applying staining methods not previously used.

[0008] When tissue images are acquired using new processing methods and technologies, existing image analysis models cannot be applied. Therefore, it is necessary to create new image analysis models to analyze the newly acquired images. For example, existing image diagnostic models applicable to pathological images are created by learning from images of tissues associated with various diseases (including diseases with few cases) that have been collected (accumulated) over many years. Such existing image diagnostic models are applicable to images of tissues that have undergone conventional typical embedding and staining processes, but they cannot be applied to images acquired using new processing methods and technologies.

[0009] The limited range of images to which existing diagnostic imaging models can be applied is a common problem across various fields that employ diagnostic imaging, and is not limited to medicine and pathology. Existing diagnostic imaging technologies are based on a vast accumulation of past knowledge, and there is a widespread need for technologies to effectively utilize this knowledge. [Means for solving the problem]

[0010] Furthermore, an image translation device according to embodiment 1 of the present invention includes an image translation unit equipped with a first generator and a second generator that have learned the relationship between the color characteristics of a first image group of images of tissue that has undergone a first process including embedding and sectioning of tissue in a predetermined embedding agent, and the color characteristics of a second image group of images of tissue that has undergone a second process that does not include the embedding and sectioning; a first input unit that inputs either a first target image belonging to the first image group or a second target image belonging to the second image group to the image translation unit; and a translation image output unit that outputs a first translation image generated from the first target image or a second translation image generated from the second target image, wherein the first generator generates the first translation image having the color characteristics of the second image group from the first target image, and the second generator generates the second translation image having the color characteristics of the first image group from the second target image.

[0011] An image diagnostic system according to one aspect of the present invention is an image diagnostic system comprising an image translation device and an image diagnostic device according to aspect 1, wherein the image diagnostic device comprises a diagnostic unit comprising at least one of the following: a first training image group in which the first processing has been captured and a first neural network that has learned the correspondence between the state of the tissue depicted in each of the first training image group, or a second training image group in which the second training image group in which the second training image group has learned the correspondence between the state of the tissue depicted in each of the second training image group; a second input unit that inputs the first translated image or the second translated image to the diagnostic unit; and a diagnostic information output unit that outputs diagnostic information relating to the state of the tissue shown in the first translated image or the second translated image output from the diagnostic unit.

[0012] An image translation method according to one aspect of the present invention includes an input step in which either a first target image belonging to the first image group or a second target image belonging to the second image group is input to a neural network comprising a first generator and a second generator that has learned the relationship between color features of a first image group of images of tissue that has undergone a first process including embedding or sectioning of tissue in a predetermined embedding agent and color features of a second image group of images of tissue that has undergone a second process not including the embedding and sectioning; and a translation image output step in which either a first translation image generated from the first target image or a second translation image generated from the second target image is output, wherein the first generator generates the first translation image having the color features of the second image group from the first target image, and the second generator generates the second translation image having the color features of the first image group from the second target image.

[0013] Each aspect of the present invention may be implemented by a computer, in which case a control program for the image translation device that enables the computer to implement the image translation device by operating the computer as each part (software element) of the image translation device, and a computer-readable recording medium on which the program is recorded, also fall within the scope of the present invention.

Advantages of the Invention

[0014] According to one aspect of the present invention, an existing image diagnosis model can be effectively utilized.

Brief Description of the Drawings

[0015] [Figure 1] It is a block diagram showing a configuration example of an image translation device according to Embodiment 1 of the present invention. [Figure 2] It is a functional block diagram showing an example of the configuration of an image translation device. [Figure 3] It is a flowchart showing an example of the processing performed by an image translation device. [Figure 4] It is a diagram for explaining an example of the flow of the first process for acquiring an image of a living body tissue. [Figure 5] It is a diagram for explaining an example of the flow of the second process for acquiring an image of a living body tissue. [Figure 6] It is a diagram for explaining another example of the flow of the second process for acquiring an image of a living body tissue. [Figure 7] It is a diagram showing an example of image translation by an image translation device. [Figure 8] It is a functional block diagram showing another configuration example of an image translation device. [Figure 9] It is an explanatory diagram for explaining the functions of the first discriminator and the second discriminator. [Figure 10] It is a block diagram showing a configuration example of an image diagnosis system according to Embodiment 2 of the present invention. [Figure 11] It is a functional block diagram showing a configuration example of an image diagnosis system. [Figure 12] It is a functional block diagram showing a configuration example of an image diagnosis system.

Modes for Carrying Out the Invention

[0016] 〔Embodiment 1〕 Hereinafter, an embodiment of the present invention will be described in detail.

[0017] (Outline of Image Translation Device 1) An image translation device 1 according to one embodiment of the present invention creates a translated image (first translated image) that resembles an image taken after a second process different from the first process has been applied to the tissue, from an image actually taken of the tissue that has undergone the first process. The image translation device 1 also creates a translated image (second translated image) that resembles an image taken after the first process has been applied to the tissue, from an image actually taken of the tissue that has undergone the second process. Here, the first process is a process that includes embedding the tissue in a predetermined embedding medium and sectioning, and the second process is a process that does not include embedding and sectioning.

[0018] In this specification, tissue may include structures composed of any of cells, fungi, and bacteria. That is, tissue may be organs of a living body, colonies of cultured cells, fungal aggregates, and bacterial colonies and microbiota.

[0019] (Configuration of Image Translation Device 1) First, the configuration of the image translation device 1 will be explained using Figures 1 and 2. Figure 1 is a block diagram showing an example of the configuration of the image translation device 1 according to Embodiment 1 of the present invention. Figure 2 is a functional block diagram showing an example of the configuration of the image translation device 1.

[0020] The image translation device 1 is, for example, a computer, and as shown in Figure 1, comprises a processor unit 2, a hard disk 3, memory 4, and a display unit 5.

[0021] The processor unit 2 reads and executes various programs from the hard disk 3. The processor unit 2 may be, for example, at least one of a CPU and a GPU.

[0022] The hard disk 3 stores various programs that the processor unit 2 executes. The hard disk 3 may also store various image data that the processor unit 2 uses to execute these programs.

[0023] Memory 4 stores various data and programs used in various processes being executed by the processor unit 2. For example, memory 4 functions as working memory that stores programs that implement neural network structures loaded from the hard disk 3. In this specification, "memory" may refer to main memory or the memory of the GPU.

[0024] The display unit 5 may be any display for displaying various images (e.g., target images) required for the execution of various processes performed by the processor unit 2, and various images (e.g., translated images) generated by the various processes performed by the processor unit 2. Note that the display unit 5 is not an essential component of the image translation device 1. For example, the image translation device 1 may be configured to transmit various data to an external display device (not shown) that is communicably connected to the image translation device 1, and to display the data on the display device.

[0025] As shown in Figure 2, the image translation device 1 includes a control unit 20 corresponding to the processor unit 2 and memory 4 shown in Figure 1, a storage unit 30 corresponding to the hard disk 3 shown in Figure 1, and a display unit 5.

[0026] The control unit 20 includes a first input unit 21, an image translation unit 22, and a translated image output unit 23.

[0027] The first input unit 21 inputs either a first target image belonging to a first image group, which is an image of tissue that has undergone a first process, or a second target image belonging to a second image group, which is an image of tissue that has undergone a second process different from the first process, to the image translation unit 22, which will be described later.

[0028] The image translation unit 22 includes a first generator 221 and a second generator 222 that have learned the relationship between the color features of the first image group and the color features of the second image group. The first generator 221 and the second generator 222 are neural networks (generative models) that extract features from an input image and generate a new image having the extracted features. Known deep learning algorithms such as generative adversarial networks (GANs) may be applied to the training of the first generator 221 and the second generator 222. The training of the first generator 221 and the second generator 222 is not limited to training using a generative adversarial network. For example, training may be performed using either the first image group or the second image group as input data, and training may be performed using images generated by transforming either the first image group or the second image group by an AI capable of generating images (e.g., Dalle2) as training data. The learning process for the first generator 221 and the second generator 222 may be performed using a computer different from the image translation device 1. In this case, the computer can be made to function as the image translation device 1 by installing the learned first generator 221 and the second generator 222, along with a predetermined optional program, on any computer.

[0029] The first generator 221 generates a first translated image from the first target image that has the color characteristics of the second image group without significantly altering the structure of the tissue depicted in the first target image. The second generator 222 generates a second translated image from the second target image that has the color characteristics of the first image group without significantly altering the structure of the tissue depicted in the second target image.

[0030] The translated image output unit 23 acquires and outputs the first translated image or the second translated image generated by the image translation unit 22. For example, the translated image output unit 23 may output the first translated image or the second translated image to the display unit 5.

[0031] The memory unit 30 may store the target image 31 and the translated image 32. In this case, the target image 31 may store the first target image belonging to the first image group and the second target image belonging to the second image group. The translated image 32 may store the translated image generated by the image translation unit 22.

[0032] (Processing flow performed by image translation device 1) Next, we will explain the processing flow performed by the image translation device 1 using Figure 3. Figure 3 is a flowchart showing an example of the processing performed by the image translation device 1.

[0033] First, the first input unit 21 inputs either the first target image belonging to the first image group or the second target image belonging to the second image group to the neural network (first generator 221 or second generator 222) (step S1: input step).

[0034] Next, the translated image output unit 23 outputs a first translated image generated from the first target image, or a second translated image generated from the second target image, which was generated by the image translation unit 22 (Step S2: Translated Image Output Step).

[0035] Thus, the image translation device 1 can perform image translation from an image of tissue that has undergone the first processing (first target image) to a translated image that appears as if it were an image of tissue that has undergone the second processing. Furthermore, the image translation device 1 can perform image translation from an image of tissue that has undergone the second processing (second target image) to a translated image that appears as if it were an image of tissue that has undergone the first processing.

[0036] (First process and second process) Here, the first and second processes will be explained using Figures 4 and 5, with the example of imaging a lesion site (tissue) in a living body. Figure 4 is a diagram illustrating an example of the flow of the first process for acquiring an image of tissue in a living body. Figures 5 and 6 are diagrams illustrating an example of the flow of the second process for acquiring an image of tissue.

[0037] The first procedure shown in Figure 4 is a conventional method for preparing and imaging pathological specimens. In the first procedure, tissue is first collected from the living organism (step S11). Next, the collected tissue is fixed with a fixative such as formalin, and then embedded using an embedding medium such as paraffin and resin (step S12). Then, the embedded tissue is sectioned (step S13), and the sectioned tissue is stained using a predetermined staining method (step S14). Here, sectioning is a process performed using a microtome, and by sectioning, the tissue is typically sliced ​​to a thickness of several μm to 10 μm. An example of a predetermined staining method is HE staining. HE staining is one of the methods used to stain collected tissue pieces, and it uses a combination of hematoxylin staining and eosin staining. Hematoxylin stains the chromatin of the cell nucleus and the ribosomes of the cytoplasm blue-violet. On the other hand, eosin stains the cytoplasmic components and the extracellular matrix red. Next, the stained tissue is imaged using a bright-field microscope or the like (Step S15).

[0038] The second procedure shown in Figure 5 does not include embedding or sectioning. In the second procedure shown in Figure 5, tissue is first collected from the living organism (step S21). Subsequently, the tissue or tissue sample is imaged (step S22). For imaging the tissue or tissue sample, a microscope capable of obtaining images of unstained tissue that are applicable to diagnostic imaging may be used, such as the following. • Fluorescence microscope • Raman microscope • Multiphoton microscopes (two-photon fluorescence microscopes, three-photon fluorescence microscopes, second-harmonic generation (SHG) microscopes, third-harmonic generation (THG) microscopes, stimulated Raman scattering (SRS) microscopes, coherent anti-Stokes Raman scattering (CARS) microscopes, etc.).

[0039] The collected tissue may be prepared into tissue sections. Tissue preparation is a process of slicing the tissue to a thickness of typically 1 mm to several mm, and is different from thin sectioning. When preparing tissue sections, the surface of the tissue may be temporarily fixed by blowing a low-temperature gas onto it to prevent deformation or crushing of the tissue by the blade that penetrates it. When using a multiphoton microscope or a confocal optical microscope, tissue preparation in step S22 is not mandatory. Also, when using these microscopes like an endoscope for imaging, tissue collection in step S21 and tissue preparation in step S22 are not mandatory.

[0040] Furthermore, the second process shown in Figure 6 does not include embedding or sectioning. In the second process shown in Figure 6, tissue is first collected from the living organism (step S31). Subsequently, the collected tissue is stained using a predetermined staining method (step S32). Then, the stained tissue is imaged (step S33). For imaging the tissue, a deep ultraviolet excitation fluorescence microscope or similar device may be used to obtain images of unembedded tissue that are suitable for diagnostic imaging. Note that when using a deep ultraviolet excitation fluorescence microscope, staining in step S32 is not mandatory.

[0041] Image translation device 1 can, for example, translate images from images of tissue processed using a newly developed processing method into translated images that resemble images of lesion specimens processed using a conventional processing method. Furthermore, image translation device 1 can, for example, translate images from images of lesion specimens processed using a conventional processing method into translated images that resemble images of tissue processed using a newly developed processing method.

[0042] (Example of image translation using an image translation device) Figure 7 shows an example of image translation by the image translation device 1. In Figure 7, a virtual deep ultraviolet excited fluorescence microscope image (first translated image) translated from an HE-stained image of a tissue section actually observed after HE staining (first target image) and a virtual HE-stained image (second translated image) translated from an image of a tissue section actually observed with a deep ultraviolet excited fluorescence microscope (second target image) (no sectioning, stained).

[0043] As shown in Figure 7, the image translation device 1 can convert HE-stained images of tissue sections actually observed after HE staining to virtual deep ultraviolet-excited fluorescence microscope images, and can also convert deep ultraviolet-excited fluorescence microscope images to HE-stained images. Furthermore, an image diagnostic model that has been previously trained with HE images of cancer cell tissue and normal cell tissue can classify cancer cell tissue and normal cell tissue with high accuracy (for example, an AUC (Area Under the Curve, an index of accuracy) of 0.9 or higher).

[0044] As an example, when classifying tissue containing cancer from tissue without cancer using an image diagnostic model trained on a first target image and a second target image actually captured, the classification is performed with an accuracy of 66.4%. In contrast, when the second target image actually captured is converted into a second translated image, and the same classification is performed on the second translated image using the image diagnostic model trained on the first target image, the classification is performed with an accuracy of 84.6%.

[0045] Here, the image translation unit 22 may perform a negative-positive inversion process on the input image as a preprocessing step in order to generate the translated image. By performing a negative-positive inversion process, the quality of the translated image generated by the image translation unit 22 can be improved. This will be explained below.

[0046] For example, in deep ultraviolet-excited fluorescence images, the brightness of background areas where no tissue is visible is low, while in HE-stained images (bright-field images), the brightness of background areas where no tissue is visible is high (see Figure 7). If negative-positive inversion is performed on a deep ultraviolet-excited fluorescence image, the brightness of the background areas in the inverted image will be close to the brightness of the background areas in the generated virtual HE-stained image. If negative-positive inversion is performed on an HE-stained image, the brightness of the background areas in the inverted image will be close to the brightness of the background areas in the generated virtual deep ultraviolet-excited fluorescence image. Such domain-adaptive preprocessing can contribute to improving the learning efficiency of image translation models, and as a result, the completeness of translated images can be improved.

[0047] (Usefulness of translated images generated by image translation device 1) In recent years, medical imaging technology has been used in various fields. In medical imaging technology, a medical imaging model is sometimes created that outputs diagnostic information (estimated results) based on images of tissue that has undergone a predetermined process. For example, a medical imaging model created using a first set of images of tissue that has undergone a first process as training data can output highly reliable diagnostic information when images with the color characteristics of the first set of images are input. However, if images with the color characteristics of a second set of images of tissue that has undergone a second process are input to such a medical imaging model, it may not be possible to obtain correct diagnostic information. This is because the color characteristics of the first set of images and the color characteristics of the second set of images are different.

[0048] The image translation device 1 can generate a first translated image possessing the color characteristics of a second image group from a first target image possessing the color characteristics of a first image group, without significantly altering the structure depicted in the target image. The generated first translated image is applicable to an existing image diagnostic model created using the second image group as training data, and similarly, the generated second translated image is applicable to an existing image diagnostic model created using the first image group as training data. In other words, by employing the image translation device 1, it is possible to generate images to which an existing image diagnostic model can be applied, even from images to which the existing image diagnostic model cannot be applied. Therefore, it becomes unnecessary to create separate image analysis models depending on whether or not the image is of tissue that has undergone a predetermined process.

[0049] For example, in the first processing step, embedding and sectioning are time-consuming and labor-intensive processes. Therefore, obtaining the second target image is simpler than obtaining the first target image. However, because the second processing step is a relatively new method, there may be few or incomplete image diagnostic models created based on the second target image. In such cases, an image translation device 1 can be used to generate a translated image from the second target image, and this translated image can be applied to the image diagnostic model created based on the first target image.

[0050] To realize image diagnosis targeting images of tissues that have undergone new processing, or images that employ new imaging techniques, it is necessary to accumulate new images of tissues that have undergone new processing and create a new image diagnosis model. Image translation device 1 can facilitate the creation of such image diagnosis models. Image translation device 1 can translate images of tissues processed using existing methods into images of the same tissues processed using the newly developed method. Using such translated images, it is possible to efficiently create a new image diagnosis model that outputs diagnostic information based on images of tissues processed using the newly developed method. Furthermore, the image diagnosis model created in this way can diagnose tissues processed using existing methods. Alternatively, image translation device 1 may translate images of tissues processed using the newly developed method into images of tissues processed using existing methods. Using such translated images, it is possible to efficiently create a new image diagnosis model that outputs diagnostic information based on images of tissues processed using existing methods. Furthermore, the image diagnosis model created in this way can diagnose tissues processed using the newly developed method.

[0051] For conditions with low incidence (e.g., diseases with few cases), it has been particularly difficult to realize early image diagnosis targeting images of tissues that have undergone new processing or images using new imaging techniques. This is because, due to the low incidence, there are few images of tissues that have undergone new processing. Images of tissues with low incidence are often accumulated as images of tissues processed using existing methods. Therefore, the image translation device 1 can also image translate images of tissues with low incidence that have been processed using existing methods into images of the same tissue processed using a newly developed method. Using such translated images, it becomes possible to efficiently create a new image diagnosis model that outputs diagnostic information based on images of tissues processed using a newly developed method, even for tissues with low incidence. Furthermore, the image diagnosis model created in this way can diagnose tissues with low incidence that have been processed using existing methods. Alternatively, the image translation device 1 may image translate images of tissues processed using a newly developed method into images of tissues with low incidence that have been processed using existing methods. Using such translated images, it becomes possible to efficiently create novel image diagnostic models that output diagnostic information based on images of tissues in low-frequency states processed using existing methods. Furthermore, these image diagnostic models can diagnose tissues in low-frequency states that have been processed using newly developed methods.

[0052] Although translated images are not images obtained by actually observing tissue, they do not significantly alter the structure of the tissue. Therefore, translated images can be treated as images of tissue, just as images obtained by actually observing tissue. For example, translated images generated by the image translation device 1 can be used to train an image diagnostic model. For instance, if a translated image is generated from a first target image using the image translation device 1, this translated image can be used to train an image diagnostic model based on a second target image.

[0053] [Embodiment 2] Other embodiments of the present invention are described below. For the sake of clarity, components having the same function as those described in the above embodiments will be denoted by the same reference numerals, and their descriptions will not be repeated.

[0054] (Configuration of image translation device 1a) The image translation unit 22 of the image translation device 1 only needs to include a first generator 221 and a second generator 222 that extract features from an input image and generate a new image having the extracted features, and is not limited to the configuration shown in Figure 2. For example, CycleGAN may be applied to realize the image translation unit 22.

[0055] The configuration of the image translation device 1a, which includes an image translation unit 22a to which CycleGAN is applied, will be explained with reference to Figure 8. Figure 8 is a functional block diagram showing another example configuration of the image translation device 1a.

[0056] As shown in Figure 8, the image translation unit 22a may further include a first classifier 223 and a second classifier 224.

[0057] The first discriminator 223 distinguishes between images included in the first image group and translated images generated by the second generator, based on a first error between the color features of the first image group and the color features of the translated images generated by the second generator.

[0058] The second discriminator 224 distinguishes between images in the second image group and the translated image generated by the first generator based on a second error between the color features of the second image group and the color features of the translated image generated by the first generator.

[0059] <Processing of image translation unit 22a> Figure 9 illustrates an example of processing performed by the image translation unit 22a, which includes a first classifier 223 and a second classifier 224.

[0060] The first input unit 21 also inputs the images from the first image group that were input to the first generator 221 to the first classifier 223. The first generator 221 generates a translated image from the input images.

[0061] The second generator 222 generates further translated images from the translated images generated by the first generator 221. The first discriminator 223 calculates a first error between the color features of the translated image generated by the second generator 222 and the color features of the first group of images from which the translated image was derived.

[0062] The first input unit 21 also inputs the images from the second image group, which were input to the second generator 222, to the second discriminator 224. The second generator 222 generates a translated image from the input images.

[0063] The first generator 221 generates further translated images from the translated images generated by the second generator 222. The second discriminator 224 calculates a second error between the color features of the translated image generated by the first generator 221 and the color features of the second group of images from which the translated image was derived.

[0064] The image translation unit 22a generates a translated image by repeating the above process.

[0065] Referring to Figure 8, the subsequent processing is explained as follows: The image translation unit 22a outputs a translated image in which the first error and the second error are below a predetermined level as the first translated image or the second translated image. The image translation unit 22a may also calculate the cycle consistency loss based on the first error and the second error, and output a translated image in which the cycle consistency loss is below a predetermined value as the first translated image or the second translated image.

[0066] An image translation device 1a with such a configuration can generate and output highly accurate translated images.

[0067] For example, it is impossible to apply a second processing step to the same location within the same tissue that has already undergone a first processing step. Thus, it is difficult (or impossible) to prepare pairs of images taken with different processing steps for the same location within the same tissue. However, because CycleGAN can learn the relationship between the color features of the first image group and the color features of the second image group, it does not require preparing pairs of images taken with different processing steps for the same location within the same tissue.

[0068] [Embodiment 3] Other embodiments of the present invention are described below. For the sake of clarity, components having the same function as those described in the above embodiments will be denoted by the same reference numerals, and their descriptions will not be repeated.

[0069] (Outline configuration of the medical imaging system 100) Figure 10 is a block diagram showing an example configuration of the image diagnostic system 100 according to Embodiment 3 of the present invention.

[0070] The medical imaging system 100 includes image translation devices 1, 1a, and a medical imaging device 7.

[0071] The medical imaging device 7 is, for example, a computer that is communicatively connected to the image translation devices 1 and 1a. As shown in Figure 10, the medical imaging device 7 includes a processor unit 71, a hard disk 73, a memory 72, and a display unit 74.

[0072] The processor unit 71 reads and executes various programs from the hard disk 73. The processor unit 71 may be, for example, at least one of a CPU and a GPU.

[0073] The hard disk 73 stores various programs that the processor unit 71 executes. The hard disk 73 may also store various image data that the processor unit 71 uses to execute these programs.

[0074] Memory 72 stores various data and programs used in various processes being executed by the processor unit 71. For example, memory 72 functions as working memory that stores programs that implement neural network structures loaded from the hard disk 73.

[0075] The display unit 74 may be any display for displaying images necessary for executing various processes performed by the processor unit 71, and diagnostic information output by the processor unit 71. Note that the display unit 74 is not an essential component of the medical imaging device 7. For example, the medical imaging device 7 may be configured to transmit diagnostic information to an external display device (not shown) or image translation device 1, 1a that is connected to the medical imaging device 7 in a communicative manner.

[0076] The following describes the medical imaging system 100, which includes a medical imaging device 7. The medical imaging device 7 is equipped with a medical imaging model (the first neural network 7121, described later) that estimates the state of tissue based on images of tissue that has undergone the first processing shown in Figure 4.

[0077] Figure 11 is a functional block diagram showing an example configuration of the medical imaging system 100. As shown in Figure 11, the medical imaging device 7 includes a control unit 710 corresponding to the processor unit 71 and memory 72 shown in Figure 10, a storage unit corresponding to the hard disk 73 shown in Figure 10, and a display unit 5. Note that the storage unit is not shown in the diagram for the sake of simplicity.

[0078] The control unit 710 includes a second input unit 711, a diagnostic unit 712, and a diagnostic information output unit 713.

[0079] The second input unit 711 inputs the second translated image, which has the color characteristics of the first image group acquired from the image translation devices 1 and 1a, to the diagnostic unit 712.

[0080] The diagnostic unit 712 includes a first neural network 7121. The first neural network 7121 is a neural network (inference model) that has learned the correspondence between a first training image set, which is an image of tissue that has undergone the first processing, and the state of the tissue depicted in each of the first training images. A known supervised machine learning algorithm may be applied to the training of the first neural network 7121. The training process of the first neural network 7121 may be performed using a computer different from the image diagnostic device 7. In this case, by installing the trained first neural network 7121 on any computer, it is possible to make that computer function as the image diagnostic device 7.

[0081] The diagnostic information output unit 713 acquires and outputs the diagnostic information output from the diagnostic unit 712. For example, the diagnostic information output unit 713 may output the diagnostic information to the display unit 74.

[0082] In the image diagnostic system 100, the image translation devices 1 and 1a generate a second translated image from an image of tissue that has undergone new processing, or from an image using a new imaging technique. By inputting the second translated image into the first neural network 7121, which is an existing inference model, diagnostic information (estimated results) based on the existing inference model can be obtained. In this way, the image diagnostic system 100 can, for example, generate a translated image to which existing disease pathology criteria can be applied from an image to which existing disease pathology criteria cannot be applied, and output diagnostic information based on the translated image. In the image diagnostic system 100, the image translation devices 1 and 1a may also have the configuration of the image diagnostic device 7. For example, the image translation devices 1 and 1a may include a diagnostic unit 712 having the first neural network 7121, and a diagnostic information output unit 713. As a result, the image translation devices 1 and 1a generate a second translated image from an image of tissue that has undergone new processing, or from an image using a new imaging technique, and perform estimation using the first neural network 7121 based on the second translated image. By performing estimation in this manner, the image translation devices 1 and 1a can output diagnostic information (estimation results) based on an existing inference model.

[0083] Furthermore, the second translated image generated by the image translation devices 1 and 1a can be used as at least part of the first set of training images for creating the first neural network 7121.

[0084] (modified version) A diagnostic imaging system 100a equipped with a diagnostic imaging device 7a will be described. The diagnostic imaging device 7a is equipped with a diagnostic imaging model (second neural network 7122, described later) that estimates the state of tissue based on images of tissue that has undergone the second processing shown in Figure 5.

[0085] Figure 12 is a functional block diagram showing an example configuration of the image diagnostic system 100a. The image diagnostic device 7a shown in Figure 12 is equipped with a second neural network 7122 in the diagnostic unit 712. The second neural network 7122 is a neural network (inference model) that has learned the correspondence between a second training image set, which is an image of tissue that has undergone a second processing, and the state of the tissue depicted in each of the second training images. A known supervised machine learning algorithm can be applied to train the second neural network 7122.

[0086] Images of tissue captured using a new method may contain more readable features than images of tissue captured using an existing method. Therefore, the new inference model, the second neural network 7122, may be able to output more accurate diagnostic information than existing inference models. In the image diagnostic system 100a, the image translation devices 1 and 1a generate a first translated image from images of tissue captured in the past (e.g., pathological images). By inputting the first translated image into the second neural network 7122, a new inference model that outputs diagnostic information based on images of tissue that has undergone new processing or images that employ a new imaging technique, diagnostic information (estimated results) based on the new inference model can be obtained.

[0087] Furthermore, the first translated images generated by the image translation devices 1 and 1a can be used as at least part of the second set of training images for creating the second neural network 7122.

[0088] The image diagnostic device 7a may also be configured to include diagnostic units 712 and 712. In this case, the configuration should allow switching the neural network used depending on whether the translated image acquired from the image translation devices 1 and 1a is a first translated image or a second translated image.

[0089] [Examples of implementation using software] The functions of the image translation devices 1 and 1a (hereinafter referred to as "devices") can be realized by programs that cause a computer to function as the device, and these programs cause a computer to function as each control block of the device (particularly each part included in the control units 20 and 20a).

[0090] In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., memory) as hardware for executing the program. By executing the program using this control device and storage device, the functions described in each of the embodiments are realized.

[0091] The above program may be recorded on one or more computer-readable recording media, not temporary ones. These recording media may or may not be provided by the above device. In the latter case, the program may be supplied to the above device via any wired or wireless transmission medium.

[0092] Furthermore, some or all of the functions of each of the above control blocks can also be realized by logic circuits. For example, an integrated circuit in which logic circuits functioning as each of the above control blocks are formed is also included in the scope of the present invention. In addition, it is also possible to realize the functions of each of the above control blocks by, for example, a quantum computer.

[0093] 〔summary〕 An image translation device according to embodiment 1 of the present invention comprises: an image translation unit equipped with a first generator and a second generator that have learned the relationship between the color characteristics of a first image group of images of tissue that has undergone a first process including embedding and sectioning of tissue in a predetermined embedding medium, and the color characteristics of a second image group of images of tissue that has undergone a second process not including the embedding and sectioning; a first input unit that inputs either a first target image belonging to the first image group or a second target image belonging to the second image group to the image translation unit; and a translation image output unit that outputs a first translation image generated from the first target image or a second translation image generated from the second target image, wherein the first generator generates the first translation image having the color characteristics of the second image group from the first target image, and the second generator generates the second translation image having the color characteristics of the first image group from the second target image.

[0094] The image translation apparatus according to Embodiment 2 of the present disclosure, in Embodiment 1, comprises: a first discriminator that distinguishes between an image included in the first image group and a translated image generated by the second generator based on a first error between the color characteristics of the first image group and the color characteristics of the translated image generated by the second generator; and a second discriminator that distinguishes between an image included in the second image group and a translated image generated by the first generator based on a second error between the color characteristics of the second image group and the color characteristics of the translated image generated by the first generator, and the translated image in which the first and second errors are below a predetermined level may be output as the first translated image or the second translated image.

[0095] In the image translation apparatus according to aspect 3 of the present disclosure, in aspect 1 or 2, the tissue may have a structure composed of an aggregate of cells, fungi, and bacteria.

[0096] In the image translation apparatus according to embodiment 4 of this disclosure, in any of embodiments 1 to 3, the second target image may be acquired using a deep ultraviolet excitation fluorescence microscope.

[0097] An image diagnostic system according to aspect 5 of the present disclosure is an image diagnostic system comprising an image translation device and an image diagnostic device according to any one of aspects 1 to 4, wherein the image diagnostic device comprises a diagnostic unit comprising at least one of the following: a first training image group in which the first processing has been captured and a first neural network that has learned the correspondence between the state of the tissue depicted in each of the first training image group, or a second training image group in which the second training image group in which the second training image group has learned the correspondence between the state of the tissue depicted in each of the second training image group; a second input unit that inputs the first translated image or the second translated image to the diagnostic unit; and a diagnostic information output unit that outputs diagnostic information relating to the state of the tissue shown in the first translated image or the second translated image output from the diagnostic unit.

[0098] An image translation method according to aspect 6 of the present disclosure includes an input step of inputting either a first target image belonging to the first image group or a second target image belonging to the second image group to a neural network comprising a first generator and a second generator that has learned the relationship between color features of a first image group of images of tissue that has undergone a first process including embedding the tissue in a predetermined embedding medium and sectioning, and color features of a second image group of images of tissue that has undergone a second process not including the embedding process and sectioning; and a translation image output step of outputting a first translation image generated from the first target image or a second translation image generated from the second target image, wherein the first generator generates the first translation image having the color features of the second image group from the first target image, and the second generator generates the second translation image having the color features of the first image group from the second target image.

[0099] A control program according to aspect 7 of this disclosure is a control program for causing a computer to function as an image translation device according to any one of aspects 1 to 4, wherein the computer functions as the image translation unit, the first input unit, and the translated image output unit.

[0100] The recording medium according to aspect 8 of this disclosure is a computer-readable recording medium on which the control program described in aspect 7 is recorded.

[0101] The present invention is not limited to the embodiments described above, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. [Explanation of Symbols]

[0102] 1, 1a Image translation device 7, 7a Diagnostic imaging equipment 21 First Input Section 22, 22a Image Translation Department 23 Translation Image Output Unit 100, 100a Medical Imaging System 221 1st generator 222 2nd generator 223 First Identifier 224 Second Identifier 7121 First Neural Network 7122 Second Neural Network S1 Input Step S2 Translated Image Output Step

Claims

1. An image translation unit comprising a first generator and a second generator that learns the relationship between the color characteristics of a first group of images of tissue that has undergone a first process including embedding in a predetermined embedding medium and sectioning, and the color characteristics of a second group of images of tissue that has undergone a second process that does not include the embedding process and sectioning, A first input unit inputs either a first target image belonging to the first image group or a second target image belonging to the second image group to the image translation unit, The system includes a translation image output unit that outputs a first translation image generated from the first target image, or a second translation image generated from the second target image, The first generator generates the first translated image having the color characteristics of the second image group from the first target image, The second generator generates a second translated image from the second target image that has color characteristics of the first image group. Image translation device.

2. The aforementioned image translation unit, A first discriminator that distinguishes between an image in the first image group and a translated image generated by the second generator based on a first error between the color features of the first image group and the color features of the translated image generated by the second generator, The system includes a second discriminator that distinguishes between an image in the second image group and a translated image generated by the first generator based on a second error between the color characteristics of the second image group and the color characteristics of the translated image generated by the first generator, A translated image in which the first error and the second error are below a predetermined level is output as the first translated image or the second translated image. The image translation device according to claim 1.

3. The image translation device according to claim 1, wherein the tissue has a structure composed of an aggregate of cells, fungi, and bacteria.

4. The second target image is acquired using a deep ultraviolet excitation fluorescence microscope, a second-harmonic generation (SHG) microscope, a stimulated Raman scattering (SRS) microscope, a coherent anti-Stokes Raman scattering (CARS) microscope, or a fluorescence microscope. The image translation device according to claim 1.

5. An image diagnostic system comprising an image translation device and an image diagnostic device according to claim 1, The aforementioned diagnostic imaging device is A diagnostic unit comprising at least one of the following: a first training image set obtained by imaging tissue subjected to the first processing, and a first neural network that has learned the correspondence between the state of the tissue depicted in each of the first training image sets; or a second training image set obtained by imaging tissue subjected to the second processing, and a second neural network that has learned the correspondence between the state of the tissue depicted in each of the second training image sets. A second input unit inputs the first translated image or the second translated image to the diagnostic unit, The system includes a diagnostic information output unit that outputs diagnostic information relating to the state of the tissue shown in the first translated image or the second translated image output from the diagnostic unit. Medical imaging system.

6. An image translation method in an image translation device, The image translation device comprises an image translation unit equipped with a first generator and a second generator that learn the relationship between the color characteristics of a first group of images of tissue that has undergone a first process including embedding and sectioning of tissue in a predetermined embedding medium, and the color characteristics of a second group of images of tissue that has undergone a second process that does not include the embedding and sectioning; a first input unit; and a translated image output unit. The first input unit inputs either a first target image belonging to the first image group or a second target image belonging to the second image group to the image translation unit in an input step, The translated image output unit includes a translated image output step of outputting a first translated image generated from the first target image, or a second translated image generated from the second target image, The first generator generates the first translated image having the color characteristics of the second image group from the first target image, The second generator generates a second translated image from the second target image that has color characteristics of the first image group. Image translation methods.

7. A control program for causing a computer to function as an image translation device according to claim 1, wherein the computer functions as the image translation unit, the first input unit, and the translated image output unit.

8. A computer-readable recording medium having the control program described in claim 7 recorded on it.

Images