Image translation device, image diagnostic system, image translation method, control program, and recording medium

Abstract

An image translation device and the like capable of effectively utilizing an existing image diagnosis model are realized. The image translation device (1) is provided with an image translation unit (22). This image translation unit (22) is provided with a first generator (221) that learns the relationship between color-related features of a first group of images captured by tissue subjected to first processing including embedding processing and thinning processing and color-related features of a second group of images captured by tissue subjected to second processing not including embedding processing and thinning processing, and a second generator (221) that learns the relationship between color-related features of a first group of images captured by tissue subjected to second processing including embedding processing and thinning processing. And a second generator (222) that inputs, to the image translation unit (22), either a first target image belonging to the first image group or a second target image belonging to the second image group, and outputs a first translated image generated from the first target image or a second translated image generated from the second target image.

Description

Technical Field

[0001] The present invention relates to an image translation device for converting an image taken of a tissue, an image translation method, and an image diagnosis system having the image translation device. Background Art

[0002] Techniques for generating different images from acquired medical images using artificial intelligence are known.

[0003] Patent Document 1 discloses a medical image processing device that generates a high-quality image from an acquired medical image using an image quality enhancement engine (artificial intelligence).

[0004] Non-patent document 1 discloses a technique for generating a virtual HE-stained image from an unstained lung tissue image using a conditional generative adversarial network (Conditional GAN, CGAN). Non-patent document 2 discloses a technique for generating a virtual tissue image obtained by assuming that a staining method different from HE staining is applied from an HE-stained image by deep learning. As staining methods different from HE staining, the staining methods specifically described in non-patent document 2 are Masson's trichrome staining, periodic acid-Schiff (PAS) staining, and Jones silver staining.

[0005] Prior art literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Publication No. 2020-166813

[0008] Non-patent document 1: Bayramoglu N et al., "Towards Virtual H&E Staining ofHyperspectral Lung Histology Images Using Conditional Generative AdversarialNetworks", IEEE Conference Proceedings, Vol.2017, P.64-71, 2017.

[0009] Non-patent literature 2: de Haan K et al., "Deep learning-based transformation of H&E stained tissues into special stains", Nature Communications 12: 4884, doi.org / 10.1038 / s41467-021-25221-2, 2021. Summary of the invention

[0010] Problem that the invention aims to solve

[0011] In order to obtain an image of a tissue, there are many types of embedding and staining treatments that can be applied to the tissue. For example, the hue (characteristics related to color) of the captured tissue image varies depending on the staining treatment used. In addition, technologies are being developed to observe and photograph tissues by irradiating light of wavelengths that have not been used before or by applying staining methods that have not been used before.

[0012] When images of tissues are acquired using new processing methods and new technologies, existing image analysis models cannot be applied. Therefore, it is necessary to build new image analysis models to analyze newly acquired images. For example, existing image diagnosis models that can be applied to pathological images are created by learning images of tissues related to various diseases (including diseases with fewer cases) that have been collected (accumulated) over the years. Such existing image diagnosis models can be applied to images of tissues that have undergone traditional representative embedding and staining processes, but cannot be applied to images acquired using new processing methods and new technologies.

[0013] The applicable images of existing image diagnosis models are limited. This is a common problem in various fields that use image diagnosis methods, not just in the fields of medicine and pathology. Existing image diagnosis technology is created based on the vast accumulation of past knowledge, and there is a wide demand for technology that can effectively utilize it.

[0014] Solutions to the problem

[0015] In addition, the image translation device of the first embodiment of the present invention comprises: an image translation unit, which comprises a first generator and a second generator that learn the relationship between the color-related features of a first image group captured by a tissue that has undergone a first treatment and the color-related features of a second image group captured by a tissue that has undergone a second treatment, wherein the first treatment includes embedding treatment of embedding the tissue in a given embedding agent and thinning, and the second treatment does not include the embedding treatment and the thinning; a first input unit that inputs any one of a first target image belonging to the first image group and a second target image belonging to the second image group into the image translation unit; and a translation image output unit that outputs a first translation image generated from the first target image or a second translation image generated from the second target image, wherein the first generator generates the first translation image having the color-related features of the second image group from the first target image, and the second generator generates the second translation image having the color-related features of the first image group from the second target image.

[0016] An image diagnostic system according to one embodiment of the present invention is an image diagnostic system comprising an image translation device according to embodiment 1 and an image diagnostic device, wherein the image diagnostic device comprises: a diagnostic unit comprising at least one of a first neural network or a second neural network, wherein the first neural network has learned the correspondence between a first training image group taken of a tissue that has undergone the first treatment and the state of the tissue displayed in each of the first training image groups, and wherein the second neural network has learned the correspondence between a second training image group taken of a tissue that has undergone the second treatment and the state of the tissue displayed in each of the second training image groups; a second input unit that inputs the first translated image or the second translated image into the diagnostic unit; and a diagnostic information output unit that outputs diagnostic information related to the state of the tissue shown in the first translated image or the second translated image output from the diagnostic unit.

[0017] An image translation method according to one embodiment of the present invention includes: an input step of inputting any one of a first target image belonging to the first image group and a second target image belonging to the second image group into a neural network having a first generator and a second generator that have learned the relationship between color-related features of a first image group photographed for a tissue that has undergone a first treatment and color-related features of a second image group photographed for a tissue that has undergone a second treatment, wherein the first treatment includes embedding treatment of embedding the tissue in a given embedding agent and thinning, and the second treatment does not include the embedding treatment and the thinning; and a translation image output step of outputting a first translation image generated from the first target image or a second translation image generated from the second target image, wherein the first generator generates the first translation image having the color-related features of the second image group from the first target image, and the second generator generates the second translation image having the color-related features of the first image group from the second target image.

[0018] The image translation device of each mode of the present invention can be implemented by a computer. In this case, the control program of the image translation device that implements the image translation device by making the computer run as the various components (software elements) possessed by the image translation device and the computer-readable recording medium recording the control program also fall within the scope of the present invention.

[0019] Effects of the Invention

[0020] According to one embodiment of the present invention, an existing image diagnosis model can be effectively utilized. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] [ Figure 1 ] is a block diagram showing a structural example of an image translation device according to embodiment 1 of the present invention.

[0022] [ Figure 2 ] is a functional block diagram showing an example of the structure of an image translation device.

[0023] [ Figure 3 ] is a flowchart showing an example of processing performed by the image translation device.

[0024] [ Figure 4 ] is a diagram illustrating an example of a first processing flow for acquiring an image captured of living tissue.

[0025] [ Figure 5 ] is a diagram illustrating an example of a second processing flow for acquiring an image captured of living tissue.

[0026] [ Figure 6 ] is a diagram illustrating another example of the second processing flow for acquiring images captured of living tissue.

[0027] [ Figure 7 ] is a diagram showing an example of image translation performed by an image translation device.

[0028] [ Figure 8 ] is a functional block diagram showing another structural example of an image translation device.

[0029] [ Fig. 9 ] is an explanatory diagram for explaining the functions of the first identifier and the second identifier.

[0030] [ Fig.10 ] is a block diagram showing a structural example of an image diagnosis system according to a second embodiment of the present invention.

[0031] [ Fig.11 ] is a functional block diagram showing a structural example of an image diagnosis system.

[0032] [ Fig.12 ] is a functional block diagram showing a structural example of an image diagnosis system. DETAILED DESCRIPTION

[0033] [Implementation Method 1]

[0034] An embodiment of the present invention is described in detail below.

[0035] (Overview of Image Translation Device 1)

[0036] An image translation device 1 according to an embodiment of the present invention creates a translation image (first translation image) that is the same as an image captured by performing a second treatment different from the first treatment on the tissue from an image actually captured on the tissue after the first treatment. In addition, the image translation device 1 creates a translation image (second translation image) that is the same as an image captured by performing the first treatment on the tissue from an image actually captured on the tissue after the second treatment. Here, the first treatment is an embedding treatment that includes embedding the tissue in a given embedding medium and thinning, and the second treatment is a treatment that does not include embedding and thinning.

[0037] In the present specification, tissue may include a structure formed by any of cells, fungi and bacteria. That is, tissue may be a living body organ, a colony of cultured cells, an aggregate of fungi, a colony or a flora of bacteria, and the like.

[0038] (Structure of Image Translation Device 1)

[0039] First, use Figure 1 and Figure 2 The structure of the image translation device 1 will be described. Figure 1 This is a block diagram showing a configuration example of the image translation device 1 according to the first embodiment of the present invention. Figure 21 is a functional block diagram showing an example of the structure of the image translation device 1 .

[0040] The image translation device 1 is, for example, a computer. Figure 1 As shown, the computer readable medium includes a processor unit 2, a hard disk 3, a memory 4, and a display unit 5.

[0041] The processor unit 2 reads and executes various programs from the hard disk 3. The processor unit 2 may be at least one of a CPU and a GPU, for example.

[0042] The hard disk 3 stores various programs executed by the processor unit 2. In addition, the hard disk 3 may store various image data used by the processor unit 2 to execute the various programs.

[0043] The memory 4 stores various data and various programs used in various processes executed by the processor unit 2. For example, the memory 4 functions as a working memory for storing a program that implements a neural network structure loaded from the hard disk 3. In addition, in this specification, "memory" can refer to both a main memory and a memory of a GPU.

[0044] The display unit 5 may be any display for displaying various images (e.g., target images) required for executing various processes executed by the processor unit 2, and various images (e.g., translated images) generated by the various processes executed by the processor unit 2. The display unit 5 is not an essential structure of the image translation device 1. For example, the image translation device 1 may be configured to transmit various data to an external display device (not shown) communicatively connected to the image translation device 1 and display the data on the display device.

[0045] like Figure 2 As shown, the image translation device 1 includes Figure 1 The processor unit 2 and the memory 4 shown in the figure correspond to the control unit 20, and Figure 1 The hard disk 3 shown corresponds to the storage unit 30 and the display unit 5 .

[0046] The control unit 20 includes a first input unit 21 , an image translation unit 22 , and a translated image output unit 23 .

[0047] The first input unit 21 inputs any one of a first target image belonging to a first image group photographed on tissue subjected to a first treatment and a second target image belonging to a second image group photographed on tissue subjected to a second treatment different from the first treatment to an image translation unit 22 described later.

[0048] The image translation unit 22 includes a first generator 221 and a second generator 222 that learn the relationship between the color-related features of the first image group and the color-related features of the second image group. The first generator 221 and the second generator 222 are neural networks (generative models) that extract features of the input image and generate new images having the extracted features. In addition, in the learning of the first generator 221 and the second generator 222, well-known deep learning algorithms such as generative adversarial networks (GAN) can be applied. The learning of the first generator 221 and the second generator 222 is not limited to the learning of the generative adversarial network. For example, any one of the first image group and the second image group can be used as input data, and an image generated by converting any one of the first image group and the second image group using AI (e.g., Dalle2) that can generate images can be used as training data for learning. In addition, the learning process of the first generator 221 and the second generator 222 can be performed using a computer different from the image translation device 1. In this case, by installing the trained first generator 221 and second generator 222 and a given arbitrary program on an arbitrary computer, the computer can be made to function as the image translation device 1 .

[0049] The first generator 221 generates a first translation image having color-related features of the second image group from the first target image without significantly changing the structure of the tissue shown in the first target image. The second generator 222 generates a second translation image having color-related features of the first image group from the second target image without significantly changing the structure of the tissue shown in the second target image.

[0050] The translated image output unit 23 acquires and outputs the first translated image or the second translated image generated by the image translation unit 22. For example, the translated image output unit 23 may output the first translated image or the second translated image to the display unit 5.

[0051] The target image 31 and the translated image 32 may be stored in the storage unit 30. In this case, the first target image belonging to the first image group and the second target image belonging to the second image group may be stored in the target image 31. The translated image generated by the image translation unit 22 may be stored in the translated image 32.

[0052] (Processing Flow Performed by Image Translation Device 1)

[0053] Next, use Figure 3 The flow of processing performed by the image translation device 1 will be described. Figure 3 This is a flowchart showing an example of processing performed by the image translation device 1 .

[0054] First, the first input unit 21 inputs any one of the first target image belonging to the first image group and the second target image belonging to the second image group to the neural network (the first generator 221 or the second generator 222) (step S1: input step).

[0055] Next, the translated image output unit 23 outputs the first translated image generated from the first target image or the second translated image generated from the second target image generated by the image translation unit 22 (step S2: translated image output step).

[0056] In this manner, the image translation apparatus 1 can perform image translation from an image actually captured of the tissue that has undergone the first treatment (first target image) to a translation image as if the tissue that has undergone the second treatment has been captured. In addition, the image translation apparatus 1 can perform image translation from an image actually captured of the tissue that has undergone the second treatment (second target image) to a translation image as if the tissue that has undergone the first treatment has been captured.

[0057] (1st process and 2nd process)

[0058] Here, we will take the case of photographing a diseased part (tissue) in a living body as an example, using Figure 4 and Figure 5 The first process and the second process will be described separately. Figure 4 FIG. 1 is a diagram for explaining an example of a first processing flow for acquiring an image of a living tissue. Figure 5 and Figure 6 This is a diagram for explaining an example of the second processing flow for acquiring an image of a tissue.

[0059] Figure 4 The first treatment shown is a method for preparing a traditional pathological specimen and photographing it. In the first treatment, tissue is first collected from a living body (step S11). Next, after fixing the collected tissue with a fixative such as formalin, it is embedded using an embedding agent such as paraffin and resin (step S12). Next, the embedded tissue is thinned (step S13), and the thinned tissue is stained using a given staining method (step S14). Here, thinning is a process performed using a microtome, etc., and by thinning, the tissue is usually sliced ​​into a thickness of about several μm to 10 μm. As an example of a given staining method, HE staining can be cited. HE staining is one of the methods for staining the collected tissue slices, which combines hematoxylin staining and eosin staining. Hematoxylin stains the chromatin in the cell nucleus and the ribosomes in the cytoplasm into blue-purple. On the other hand, eosin stains the components of the cytoplasm and the extracellular matrix into red. Next, the stained tissue is photographed using a bright field microscope, etc. (step S15).

[0060] Figure 5 The second treatment shown does not include embedding and thinning. Figure 5 In the second process shown, tissue is first collected from a living body (step S21). Then, the tissue or tissue slice is photographed (step S22). The following microscope can be used to photograph the tissue or tissue slice, which can obtain an image of an unstained tissue and can be used for image diagnosis, etc.

[0061] Fluorescence microscopy

[0062] Raman microscope

[0063] Multiphoton microscopy (two-photon fluorescence microscopy, three-photon fluorescence microscopy, second-harmonic generation (SHG) microscopy, third-harmonic generation (THG) microscopy, stimulated Raman scattering (SRS) microscopy, coherent anti-Stokes Raman scattering (CARS) microscopy, etc.).

[0064] In addition, the collected tissue can also be processed into tissue slices. Tissue slices are a method of cutting tissue into a thickness of usually 1 mm to several mm, which is different from thinning. When performing tissue slices, low-temperature gas can be sprayed onto the surface of the tissue for temporary fixation to prevent the tissue from being deformed or crushed by the blade invading the tissue. When using a multiphoton microscope or a confocal optical microscope, the tissue slices in step S22 are not necessary. In addition, when these microscopes are used for imaging like an endoscope, the collection of tissue in step S21 and the tissue slices in step S22 are not necessary.

[0065] in addition, Figure 6 The second treatment shown also does not include embedding and thinning. Figure 6 In the second process shown, tissue is first collected from a living body (step S31). Next, the collected tissue is stained using a given staining method (step S32). The stained tissue is then photographed (step S33). In photographing the tissue, a deep ultraviolet excitation fluorescence microscope or the like can be used, which can obtain an image of the tissue that has not been stained and can be used for image diagnosis, etc. In addition, when a deep ultraviolet excitation fluorescence microscope is used, the staining in step S32 is not necessary.

[0066] The image translation device 1 can, for example, perform image translation from an image captured of a tissue processed using a newly developed processing method to a translation image that is the same as an image captured of a specimen of a lesion portion processed using a conventional processing method. In addition, the image translation device 1 can, for example, perform image translation from an image captured of a specimen of a lesion portion processed using a conventional processing method to a translation image that is the same as an image captured of a tissue processed using a newly developed processing method.

[0067] (Example of image translation by image translation device)

[0068] Figure 7 FIG. 1 is a diagram showing an example of image translation by the image translation device 1. Figure 7 , a virtual deep ultraviolet excitation fluorescence microscope image (first translated image) obtained by image translation from an HE staining image (first target image) of a tissue section actually observed after HE staining, and a virtual HE staining image (second translated image) obtained by image translation from an image (second target image) of a tissue section actually observed using a deep ultraviolet excitation fluorescence microscope (unthinned, stained) are shown.

[0069] like Figure 7 As shown, the image translation device 1 is capable of converting the HE-stained image of the tissue section actually observed after HE staining into a virtual deep ultraviolet excitation fluorescence microscope image, and converting the deep ultraviolet excitation fluorescence microscope image into the HE-stained image. In addition, the image diagnosis model that has previously learned the HE images of cancer cell tissue and normal cell tissue can classify cancer cell tissue and normal cell tissue with high accuracy (for example, AUC (Area Under the Curve, accuracy index) is 0.9 or more).

[0070] For example, when the image diagnosis model that learned the first target image and the second target image that was actually taken were used to classify tissues containing cancer and tissues not containing cancer, the classification was performed with an accuracy of 66.4%. In contrast, when the second target image that was actually taken was converted into a second translation image and the image diagnosis model that learned the first target image was used to perform the same classification on the second translation image, the classification was performed with an accuracy of 84.6%.

[0071] Here, the image translation unit 22 may perform negative-positive inversion processing on the input image as a pre-processing in order to generate the translated image. By performing the negative-positive inversion processing, the completeness of the translated image generated by the image translation unit 22 can be improved. This point is described below.

[0072] For example, in deep ultraviolet fluorescence images, the brightness of the background area where the tissue is not photographed is low, while in HE staining images (bright field images), the brightness of the background area where the tissue is not photographed is high (refer to Figure 7 ). If the deep ultraviolet excited fluorescence image is subjected to negative-to-positive inversion processing, the brightness of the background area of ​​the inverted image is close to the brightness of the background area of ​​the generated virtual HE-stained image. If the HE-stained image is subjected to negative-to-positive inversion processing, the brightness of the background area of ​​the inverted image is close to the brightness of the background area of ​​the generated virtual deep ultraviolet excited fluorescence image. This preprocessing using domain adaptation helps to improve the learning efficiency of the image translation model, thereby improving the completeness of the translated image.

[0073] (Usefulness of the translated image generated by the image translation device 1)

[0074] In recent years, image diagnosis technology has been applied to various fields. In image diagnosis technology, an image diagnosis model is sometimes created that outputs diagnostic information (inferred results) based on images taken of tissues that have undergone a given treatment. For example, when an image having color-related features of the first image group is input, an image diagnosis model created using the first image group taken of the tissue that has undergone the first treatment as training data can output highly reliable diagnostic information. However, even if an image having color-related features of the second image group taken of the tissue that has undergone the second treatment is input into such an image diagnosis model, correct diagnostic information may not be obtained. This is because the color-related features of the first image group are different from the color-related features of the second image group.

[0075] The image translation device 1 can generate a first translation image having color-related features of a second image group from a first target image having color-related features of a first image group without significantly changing the structure shown in the target image. The generated first translation image can be applied to an existing image diagnosis model created using the second image group as training data, and similarly, the generated second translation image can be applied to an existing image diagnosis model created using the first image group as training data. That is, if the image translation device 1 is used, an image to which the existing image diagnosis model can be applied can be generated from an image to which the existing image diagnosis model cannot be applied. Therefore, there is no need to create an image analysis model separately depending on whether the image is taken of a tissue that has undergone a given treatment.

[0076] For example, in the first process, embedding and thinning are processes that require time and effort. Therefore, it is easier to obtain the second target image than to obtain the first target image. However, since the second process is a process method with a short history, the image diagnosis model created based on the second target image may be few or unfinished. In this case, the image translation device 1 is used to generate a translation image from the second target image, and the translation image is applied to the image diagnosis model created based on the first target image.

[0077] In order to realize image diagnosis for images taken of newly processed tissues or images using new imaging techniques, it is necessary to newly accumulate images taken of newly processed tissues and create new image diagnosis models. The image translation device 1 can more easily create such image diagnosis models. The image translation device 1 can perform image translation from images taken of tissues processed using existing methods to images of the tissues processed using the newly developed method. If such translation images are used, a new image diagnosis model based on the output of diagnostic information from images taken of tissues processed using the newly developed method can be efficiently created. In addition, the image diagnosis model thus created can diagnose tissues processed using existing methods. In addition, the image translation device 1 can perform image translation from images taken of tissues processed using the newly developed method to images taken of tissues processed using the existing method. If such translation images are used, a new image diagnosis model based on the output of diagnostic information from images taken of tissues processed using the existing method can be efficiently created. In addition, the image diagnosis model thus created can diagnose tissues processed using the newly developed method.

[0078] For states with low frequency of occurrence (e.g., diseases with few cases), it is particularly difficult to implement image diagnosis for images taken of newly processed tissues or images using new imaging technologies in the early stage. This is because, due to the low frequency of occurrence, there are few images taken of newly processed tissues themselves. In many cases, images taken of tissues in states with low frequency of occurrence are accumulated as images taken of tissues processed using existing methods. Therefore, the image translation device 1 can perform image translation from images taken of tissues in states with low frequency of occurrence processed using existing methods to images of the tissues processed using newly developed methods. If such translated images are used, even for tissues in states with low frequency of occurrence, a new image diagnosis model based on the output of diagnostic information from images taken of tissues processed using newly developed methods can be efficiently created. In addition, the image diagnosis model created in this way can diagnose tissues in states with low frequency of occurrence processed using existing methods. In addition, the image translation device 1 can perform image translation from images taken of tissues processed using newly developed methods to images taken of tissues in states with low frequency of occurrence processed using existing methods. By using such a translated image, a new image diagnosis model can be efficiently created that outputs diagnostic information based on images taken of tissues in a state that occurs less frequently and is processed using an existing method. In addition, the image diagnosis model thus created can diagnose tissues in a state that occurs less frequently and is processed using a newly developed method.

[0079] Although the translated image is not an image obtained by actually observing the tissue, the structure of the tissue is not significantly changed. Therefore, like the image obtained by actually observing the tissue, the translated image can be regarded as an image taken of the tissue. For example, the translated image generated by the image translation device 1 can be used for learning an image diagnosis model. For example, if the image translation device 1 is used to generate a translated image from the first target image, the translated image can be used for learning to create an image diagnosis model based on the second target image.

[0080] [Implementation Method 2]

[0081] In the following, other embodiments of the present invention will be described. In addition, for the sake of convenience of description, the same reference numerals are given to components having the same functions as those described in the above embodiments, and their description will not be repeated.

[0082] (Structure of Image Translation Device 1a)

[0083] The image translation unit 22 of the image translation device 1 only needs to include a first generator 221 and a second generator 222 for extracting features of an input image and generating a new image having the extracted features. Figure 2For example, a cycle generative adversarial network (CycleGAN) may be applied to implement the image translation unit 22 .

[0084] use Figure 8 The structure of the image translation device 1a including the image translation unit 22a to which CycleGAN is applied will be described. Figure 8 2 is a functional block diagram showing another configuration example of the image translation device 1a.

[0085] like Figure 8 As shown, the image translation unit 22 a may further include a first recognizer 223 and a second recognizer 224 .

[0086] The first identifier 223 identifies the images included in the first image group and the translated image generated by the second generator based on a first error between the color-related features of the first image group and the color-related features of the translated image generated by the second generator.

[0087] The second identifier 224 identifies the images included in the second image group and the translated image generated by the first generator based on a second error between the color-related features of the second image group and the color-related features of the translated image generated by the first generator.

[0088] <Processing by the Image Translation Unit 22a>

[0089] Fig. 9 1 is a diagram for explaining an example of processing executed by the image translation unit 22 a including the first recognizer 223 and the second recognizer 224 .

[0090] The first input unit 21 also inputs the images of the first image group input to the first generator 221 to the first recognizer 223. The first generator 221 generates a translated image based on the input images.

[0091] The second generator 222 further generates a translation image from the translation image generated by the first generator 221. The first identifier 223 calculates a first error between the color-related features of the translation image generated by the second generator 222 and the color-related features of the images of the first image group as the source of the translation image.

[0092] The first input unit 21 also inputs the images of the second image group input to the second generator 222 to the second recognizer 224. The second generator 222 generates a translated image based on the input images.

[0093] The first generator 221 further generates a translation image based on the translation image generated by the second generator 222. The second identifier 224 calculates a second error between the color-related features of the translation image generated by the first generator 221 and the color-related features of the image of the second image group as the source of the translation image.

[0094] The image translation unit 22a generates a translated image by repeating the above-mentioned processing.

[0095] If reference Figure 8 To explain the subsequent processing, the image translation unit 22a outputs the translation image with the first error and the second error being less than a given level as the first translation image or the second translation image. The image translation unit 22a may calculate the cycle consistency loss based on the first error and the second error, and output the translation image with the cycle consistency loss being less than a given value as the first translation image or the second translation image.

[0096] The image translation device 1a having such a configuration can generate and output a highly accurate translated image.

[0097] For example, it is impossible to re-apply the second treatment to the same position of the same tissue as the tissue that was previously subjected to the first treatment. Thus, it is difficult (or impossible) to prepare paired images of the same position of the same tissue subjected to different treatments. However, CycleGAN can learn the relationship between the color-related features of the first image group and the color-related features of the second image group, so there is no need to prepare paired images of the same position of the same tissue subjected to different treatments.

[0098] [Implementation method 3]

[0099] In the following, other embodiments of the present invention will be described. In addition, for the sake of convenience of description, the same reference numerals are given to components having the same functions as those described in the above embodiments, and their description will not be repeated.

[0100] (Schematic Structure of Image Diagnosis System 100)

[0101] Fig.10 This is a block diagram showing a configuration example of an image diagnosis system 100 according to Embodiment 3 of the present invention.

[0102] The image diagnosis system 100 includes an image interpretation device 1 , 1 a and an image diagnosis device 7 .

[0103] The image diagnosis device 7 is, for example, a computer connected to the image translation devices 1 and 1a so as to be communicable. Fig.10 As shown, the image diagnosis device 7 includes a processor unit 71 , a hard disk 73 , a memory 72 , and a display unit 74 .

[0104] The processor unit 71 reads and executes various programs from the hard disk 73. The processor unit 71 may be at least one of a CPU and a GPU, for example.

[0105] The hard disk 73 stores various programs executed by the processor unit 71. In addition, the hard disk 73 may store various image data used by the processor unit 71 to execute the various programs.

[0106] The memory 72 stores various data and various programs used in various processes executed by the processor unit 71. For example, the memory 72 functions as a working memory for storing a program that realizes a neural network structure loaded from the hard disk 73.

[0107] The display unit 74 may be any display for displaying images required for executing various processes executed by the processor unit 71 and diagnostic information output by the processor unit 71. In addition, the display unit 74 is not an essential structure of the image translation device 7. For example, the image translation device 7 may be configured to transmit diagnostic information to an external display device (not shown) or the image translation device 1, 1a that is communicably connected to the image translation device 7.

[0108] Next, an image diagnosis system 100 including an image diagnosis device 7 is described. The image diagnosis device 7 includes an image diagnosis model (a first neural network 7121 described later) that is configured to generate a neural network based on the image diagnosis data. Figure 4 The state of the tissue is estimated based on the image captured by the first processed tissue shown.

[0109] Fig.11 1 is a functional block diagram showing a configuration example of the image diagnosis system 100. Fig.11 As shown, the image diagnosis device 7 has Fig.10 The processor unit 71 and the memory 72 shown in the figure correspond to the control unit 710 and the Fig.10 The hard disk 73 shown corresponds to the storage unit and the display unit 5. In order to simplify the description, the storage unit is omitted from the illustration.

[0110] The control unit 710 includes a second input unit 711 , a diagnosis unit 712 , and a diagnosis information output unit 713 .

[0111] The second input unit 711 inputs the second translated image having the color feature of the first image group acquired from the image translation apparatus 1 or 1 a to the diagnosis unit 712 .

[0112] The diagnosis unit 712 includes a first neural network 7121. The first neural network 7121 is a neural network (inference model) that has learned the correspondence between the first training image group taken of the tissue that has undergone the first treatment and the state of the tissue shown in each of the first training image groups. In addition, in the learning of the first neural network 7121, a well-known supervised machine learning algorithm can be applied. In addition, the learning process of the first neural network 7121 can also be performed using a computer different from the image diagnosis device 7. In this case, by installing the trained first neural network 7121 in an arbitrary computer, the computer can function as the image diagnosis device 7.

[0113] The diagnosis information output unit 713 acquires and outputs the diagnosis information output from the diagnosis unit 712. For example, the diagnosis information output unit 713 may output the diagnosis information to the display unit 74.

[0114] In the image diagnosis system 100, the image translation device 1, 1a generates a second translation image from an image taken of a newly processed tissue or an image using a new imaging technology. If the second translation image is input to the first neural network 7121 as an existing inference model, diagnostic information (inference result) based on the existing inference model can be obtained. In this way, the image diagnosis system 100 can, for example, generate a translation image to which the pathological determination standard of the existing disease can be applied from an image to which the pathological determination standard of the existing disease cannot be applied, and output diagnostic information based on the translation image. In addition, in the image diagnosis system 100, the image translation device 1, 1a can have the structure of the image diagnosis device 7. For example, the image translation device 1, 1a can include a diagnosis unit 712 having a first neural network 7121 and a diagnosis information output unit 713. Thus, the image translation device 1, 1a generates a second translation image from an image taken of a newly processed tissue or an image using a new imaging technology, and performs inference based on the second translation image using the first neural network 7121. By performing inference in this way, the image translation device 1, 1a can output diagnostic information (inference result) based on the existing inference model.

[0115] Furthermore, the second translated image generated by the image translation apparatus 1 or 1 a may be used as at least a part of the first image group for training to create the first neural network 7121 .

[0116] (Variation Example)

[0117] The following describes an image diagnosis system 100a including an image diagnosis device 7a. The image diagnosis device 7a includes an image diagnosis model (a second neural network 7122 described later) that is configured to generate a plurality of image diagnosis models based on the image diagnosis data. Figure 5 The state of the tissue is estimated based on the image captured by the second processed tissue shown.

[0118] Fig.12 2 is a functional block diagram showing a configuration example of the image diagnosis system 100 a . Fig.12 The image diagnosis device 7a shown in the figure has a second neural network 7122 in the diagnosis unit 712. The second neural network 7122 is a neural network (inference model) that has learned the correspondence between the second training image group taken of the tissue that has undergone the second treatment and the state of the tissue shown in each of the second training image groups. In addition, in the learning of the second neural network 7122, a known supervised machine learning algorithm can be applied.

[0119] Tissue images taken using the new method may have more readable features than tissue images taken using the existing method. Therefore, the second neural network 7122, which is a newer reasoning model, may be able to output diagnostic information with higher accuracy than the existing reasoning model. In the image diagnosis system 100a, the image translation device 1, 1a generates a first translated image from an image of tissue taken in the past (for example, a pathological image). If the first translated image is input to the second neural network 7122, which is a new reasoning model that outputs diagnostic information based on images taken of newly processed tissues or images using new imaging technology, diagnostic information (inference results) based on the new reasoning model can be obtained.

[0120] Furthermore, the first translated image generated by the image translation apparatus 1 or 1 a may be used as at least a part of the second image group for training to create the second neural network 7122 .

[0121] The image diagnosis device 7a may also be configured to have the functions of the diagnosis units 712 and 712. In this case, the configuration of the neural network to be used may be switched depending on whether the translated image acquired from the image translation device 1 or 1a is the first translated image or the second translated image.

[0122] (Example implemented by software)

[0123] The function of the image translation device 1, 1a (hereinafter referred to as "device") is a program for causing a computer to function as the device, and can be realized by a program for causing each control block of the computer to function as the device (especially each part included in the control unit 20, 20a).

[0124] In this case, the device has a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. By executing the program using the control device and the storage device, the functions described in the above embodiments are realized.

[0125] The above program may be recorded on one or more computer-readable recording media, rather than being temporarily recorded. The recording medium may or may not have the above device. In the latter case, the above program may be provided to the above device via any transmission medium, whether wired or wireless.

[0126] In addition, some or all of the functions of the above-mentioned control blocks can also be realized by logic circuits. For example, an integrated circuit having a logic circuit that functions as the above-mentioned control blocks is also included in the scope of the present invention. In addition, the functions of the above-mentioned control blocks can also be realized by, for example, a quantum computer.

[0127] 〔Summarize〕

[0128] An image translation device according to a first aspect of the present invention includes: an image translation unit having a first generator and a second generator which learn the relationship between color-related features of a first image group taken of a tissue that has undergone a first treatment and color-related features of a second image group taken of a tissue that has undergone a second treatment, wherein the first treatment includes embedding treatment of embedding the tissue in a given embedding agent and thinning, and the second treatment does not include the embedding treatment and the thinning; a first input unit which inputs any one of a first target image belonging to the first image group and a second target image belonging to the second image group into the image translation unit; and a translation image output unit which outputs a first translation image generated from the first target image or a second translation image generated from the second target image, wherein the first generator generates the first translation image having the color-related features of the second image group from the first target image, and the second generator generates the second translation image having the color-related features of the first image group from the second target image.

[0129] The image translation device of mode 2 of the present invention, in mode 1, the image translation unit comprises: a first identifier, which identifies the image included in the first image group and the translation image generated by the second generator based on a first error between the color-related features of the first image group and the color-related features of the translation image generated by the second generator; a second identifier, which identifies the image included in the second image group and the translation image generated by the first generator based on a second error between the color-related features of the second image group and the color-related features of the translation image generated by the first generator, and can output the translation image in which the first error and the second error are below a given level as the first translation image or the second translation image.

[0130] In the image translation device according to aspect 3 of the present invention, in aspect 1 or 2, the tissue may have a structure composed of any one of cells, fungi and bacteria.

[0131] According to the image translation device of aspect 4 of the present invention, in any one of aspects 1 to 3, the second target image can be photographed using a deep ultraviolet excitation fluorescence microscope.

[0132] The image diagnosis system of mode 5 of the present invention includes the image translation device and the image diagnosis device described in any one of modes 1 to 4, wherein the image diagnosis device comprises: a diagnosis unit, which comprises at least one of a first neural network or a second neural network, wherein the first neural network has learned the correspondence between a first training image group taken of the tissue that has undergone the first treatment and the state of the tissue displayed in each of the first training image groups, and the second neural network has learned the correspondence between a second training image group taken of the tissue that has undergone the second treatment and the state of the tissue displayed in each of the second training image groups; a second input unit, which inputs the first translated image or the second translated image into the diagnosis unit; and a diagnosis information output unit, which outputs diagnosis information related to the state of the tissue shown in the first translated image or the second translated image output from the diagnosis unit.

[0133] The image translation method of mode 6 of the present invention includes: an input step of inputting any one of a first target image belonging to the first image group and a second target image belonging to the second image group into a neural network having a first generator and a second generator that have learned the relationship between color-related features of a first image group taken of a tissue that has undergone a first treatment and color-related features of a second image group taken of a tissue that has undergone a second treatment, wherein the first treatment includes embedding treatment of embedding the tissue in a given embedding agent and thinning, and the second treatment does not include the embedding treatment and the thinning; and a translation image output step of outputting a first translation image generated from the first target image or a second translation image generated from the second target image, wherein the first generator generates the first translation image having the color-related features of the second image group from the first target image, and the second generator generates the second translation image having the color-related features of the first image group from the second target image.

[0134] The control program of mode 7 of the present invention is a control program for causing a computer to function as the image translation device described in any one of modes 1 to 4, and is used to cause a computer to function as the image translation unit, the first input unit, and the translated image output unit.

[0135] A recording medium according to an eighth aspect of the present invention is a computer-readable recording medium on which the control program according to the seventh aspect is recorded.

[0136] The present invention is not limited to the above-mentioned embodiments, and various changes can be made within the scope of the claims. Embodiments obtained by appropriately combining technical means respectively disclosed in different embodiments are also included in the technical scope of the present invention.

[0137] Description of Reference Numerals

[0138] 1.1a Image translation device

[0139] 7.7a Image diagnostic device

[0140] 21 1st input section

[0141] 22.22a Image Translation Department

[0142] 23 Translation image output unit

[0143] 100, 100a Image Diagnosis System

[0144] 221 Generator 1

[0145] 222 Generator 2

[0146] 223 1st Identifier

[0147] 224 Second Identifier

[0148] 7121 No. 1 Neural Network

[0149] 7122 2nd Neural Network

[0150] S1 Input Steps

[0151] S2 Translation Image Output Step

Claims

1. An image translation device having: an image translation unit including a first generator and a second generator that learn the relationship between color-related features of a first image group captured of a tissue that has undergone a first treatment and color-related features of a second image group captured of a tissue that has undergone a second treatment, wherein the first treatment includes embedding treatment of embedding the tissue in a given embedding medium and thinning, and the second treatment does not include the embedding treatment and the thinning; a first input unit for inputting any one of a first target image belonging to the first image group and a second target image belonging to the second image group to the image translation unit; a translation image output unit configured to output a first translation image generated from the first target image or a second translation image generated from the second target image, The first generator generates the first translated image having color-related features of the second image group from the first target image. The second generator generates a second translated image having color-related features of the first image group from the second target image.

2. The image translation device according to claim 1, wherein the image translation unit comprises: a first identifier for identifying images included in the first image group and the translated image generated by the second generator based on a first error between a color-related feature of the first image group and a color-related feature of the translated image generated by the second generator; a second identifier for identifying the images included in the second image group and the translated image generated by the first generator based on a second error between the color-related features of the second image group and the color-related features of the translated image generated by the first generator, A translation image in which the first error and the second error are equal to or less than a predetermined level is output as the first translation image or the second translation image.

3. The image translation device according to claim 1, wherein the tissue has a structure composed of any one of cells, fungi and bacteria.

4. The image translation device according to claim 1, wherein the second target image is captured using a deep ultraviolet excitation fluorescence microscope, a second harmonic generation (SHG: Second-Harmonic-Generation) microscope, a stimulated Raman scattering (SRS: Stimulated Raman scattering) microscope, a coherent anti-Stokes Raman scattering (CARS: Coherent anti-Stokes Raman scattering) microscope or a fluorescence microscope.

5. An image diagnosis system comprising the image translation device according to claim 1 and an image diagnosis device, The image diagnosis device comprises: a diagnosis unit including at least one of a first neural network or a second neural network, wherein the first neural network has learned a correspondence between a first training image group taken of the tissue that has undergone the first treatment and a state of the tissue shown in each of the first training image groups, and the second neural network has learned a correspondence between a second training image group taken of the tissue that has undergone the second treatment and a state of the tissue shown in each of the second training image groups; a second input unit that inputs the first translation image or the second translation image into the diagnosis unit; The diagnostic information output unit outputs the diagnostic information related to the state of the tissue indicated by the first translation image or the second translation image output by the diagnostic unit.

6. Image translation methods, including: An input step of inputting any one of a first target image belonging to the first image group and a second target image belonging to the second image group into a neural network having a first generator and a second generator that have learned a relationship between color-related features of a first image group taken of a tissue that has undergone a first treatment and color-related features of a second image group taken of a tissue that has undergone a second treatment, wherein the first treatment includes embedding treatment of embedding the tissue in a given embedding medium and thinning, and the second treatment does not include the embedding treatment and the thinning; a translation image outputting step of outputting a first translation image generated from the first target image or a second translation image generated from the second target image, The first generator generates the first translated image having color-related features of the second image group from the first target image. The second generator generates a second translated image having color-related features of the first image group from the second target image.

7. A control program for causing a computer to function as the image translation device according to claim 1, wherein the control program is for causing a computer to function as the image translation unit, the first input unit, and the translated image output unit.

8. A computer-readable recording medium having recorded thereon the control program according to claim 7.

Images