Image-level tumor detection system using a generative adversarial network, and method for detecting tumors.

Abstract

To provide a tumor detection system using GAN, configured to reduce the burden for preparing learning data.SOLUTION: In a tumor detection system, a machine learning model of Cycle GAN is constructed, including a first generator, a discriminator, a second generator, and a second discriminator. During learning, normal image data is input to the first generator to generate abnormal image data, and the generated abnormal image data is input to the second generator to generate normal image data. The normal image data generated in the second generator is compared with the initial normal image data to perform learning in the first and second generators and the first and second discriminators. During detection, a processing circuit inputs image data to the first generator, inputs image data generated by the input to the second generator, and determines an error between image data generated by the input and the original image data input to the first generator. When the error is not smaller than a predetermined threshold, the original image data is determined to be abnormal image data.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present disclosure relates to a main detection system and method, particularly to a tumor generation system and method at the image level using an adversarial generation network.

Background Art

[0002] It is required to image the human body using imaging techniques such as contrast CT (Computer Tomography), MRI (Magnetic Resonance Image), and X-ray, and accurately detect abnormalities hidden in the human body.

[0003] Among these, contrast CT is an imaging technique that repeatedly images a part of the human body where a contrast agent has been rapidly intravenously injected at each time point. As contrast CT, for example, there are three types: ART (imaging after 30 seconds) of hepatic arterial phase CT images, PV (imaging after 70 seconds) of portal venous phase, and NC (without a drug to be intravenously injected) of equilibrium phase.

[0004] In this specification, mainly ART is taken as an example.

[0005] Tumor detection methods that use image data taken by contrast CT and AI technologies such as deep learning have been developed in various ways. For example, a tumor classification method by classification in pixel units that requires annotation at the pixel level and a tumor classification method by object detection that requires annotation at the lesion level (bounding box level) have been proposed. However, the above two classification methods perform supervised learning. One requires the user to create training data by annotation at the pixel level, and the other requires the user to create training data by annotation at the bounding box level. For medical workers including doctors, preparing such training data is a job that takes a lot of time and effort, and is actually an impossible task.

Prior Art Documents

Patent Documents

[0006] [Patent Document 1] Special Publication No. 2019-536132 [Overview of the project] [Problems that the invention aims to solve]

[0007] There is a widespread demand for image-level tumor detection systems and methods that utilize AI with a low burden of preparing training data, such as generative adversarial networks (GANs). [Means for solving the problem]

[0008] The tumor detection system of this disclosure comprises a computer device, the computer device including an interface device and a processing circuit. The computer device has a trained CycleGAN machine learning model built on it, which includes a first generator, a first discriminator, a second generator, and a second discriminator. Here, the first generator is trained to output fake tumor image data from real normal image data input, and the first discriminator is trained to determine that the fake tumor image data generated by the first generator is fake and that real tumor image data is real. Furthermore, the second generator is trained to output fake normal image data from real tumor image data input, and the second discriminator is trained to determine that the fake generated image data generated by the second generator is fake and that real normal image data is real. Furthermore, in a trained CycleGAN machine learning model, during training, normal image data is input to the first generator to generate abnormal image data, and this generated abnormal image data is input to the second generator to generate normal image data. By comparing the normal image data generated by the second generator with the initial normal image data, training is performed in the first and second generators, as well as in the first and second classifiers. For detection, The processing circuit inputs the image data acquired via the interface device to the first generator. The processing circuit, upon input to the first generator, inputs the image data generated by the first generator to the second generator. The processing circuit calculates the error between the image data generated by the second generator based on the input to the second generator and the original image data input to the first generator. The processing circuit determines that the image data input to the first generator is normal image data if the calculated error is smaller than a predetermined threshold, and determines that the image data input to the first generator is abnormal image data if the calculated error is not smaller than the predetermined threshold. [Effects of the Invention]

[0009] The tumor detection system and method for detecting tumors described herein require very little preparation of training data for the user, while being able to detect with high accuracy whether or not image data contains abnormal areas. [Brief explanation of the drawing]

[0010] [Figure 1] Figure 1 is a system configuration diagram of the tumor detection system according to Embodiment 1. [Figure 2] Figure 2 is a schematic diagram of the CycleGAN used in the tumor detection system according to Embodiment 1. [Figure 3] Figure 3 is a schematic diagram of the U-Net configuration, including skip connections. [Figure 4]Figure 4 is a flowchart showing the operation of tumor detection in the tumor detection system according to Embodiment 1. [Figure 5A] Figure 5A shows examples of (5A-1) input CT image, (5A-2) output CT image, (5A-3) input liver image, (5A-4) output liver image, and (5A-5) detection result image related to a METS (metastatic cancer) liver tumor detected by the tumor detection system according to Embodiment 1, and is an example of an image that does not actually contain a tumor. [Figure 5B] Figure 5B shows examples of (5B-1) input CT image, (5B-2) output CT image, (5B-3) input liver image, (5B-4) output liver image, and (5B-5) detection result image related to an HCC (hepatocellular carcinoma) liver tumor detected by the tumor detection system according to Embodiment 1, and is an example of an image that does not actually contain a tumor. [Figure 5C] Figure 5C shows examples of (5C-1) input CT image, (5C-2) output CT image, (5C-3) input liver image, (5C-4) output liver image, and (5C-5) detection result image related to a CCC (intrahepatic bile duct cancer) liver tumor detected by the tumor detection system according to Embodiment 1, and is an example of an image that does not actually contain a tumor. [Figure 6A] Figure 6A shows examples of (6A-1) input CT image, (6A-2) output CT image, (6A-3) input liver image, (6A-4) output liver image, and (6A-5) detection result image related to a METS (metastatic cancer) liver tumor detected by the tumor detection system according to Embodiment 1, and is an example of an image that actually contains a tumor. [Figure 6B] Figure 6B shows examples of (6B-1) input CT image, (6B-2) output CT image, (6B-3) input liver image, (6B-4) output liver image, and (6B-5) detection result image related to an HCC (hepatocellular carcinoma) liver tumor detected by the tumor detection system according to Embodiment 1, and is an example of an image that actually contains a tumor. [Figure 6C]Figure 6C shows an example of (6C-1) an input CT image, (6C-2) an output CT image, (6C-3) an input liver image, (6C-4) an output liver image, and (6C-5) a detection result image for a CCC (intrahepatic cancer) liver tumor by the tumor detection system according to Embodiment 1, and it is an example of an image actually containing a tumor. [Figure 7A] Figure 7A shows an example of the detection result of METS (metastatic cancer) by the tumor detection system according to Embodiment 1, and each graph shows the mean squared error (MSE) between the input image and the output image. [Figure 7B] Figure 7B shows an example of the detection result of HCC (hepatocellular carcinoma) by the tumor detection system according to Embodiment 1, and each graph shows the mean squared error (MSE) between the input image and the output image. [Figure 7C] Figure 7C shows an example of the detection result of CCC (intrahepatic cancer) by the tumor detection system according to Embodiment 1, and each graph shows the mean squared error (MSE) between the input image and the output image. [Figure 8] Figures 8(8-1)(8-2)(8-3) show examples of contrast CT images of the liver actually containing a tumor (abnormality), and Figures 8(8-4)(8-5)(8-6) show examples of pseudo-abnormality images (tumor images) used in the tumor detection system according to Embodiment 2. [Figure 9A] Figure 9A is a schematic configuration diagram of CycleGAN used in the tumor detection system according to Embodiment 3. [Figure 9B] Figure 9B is a schematic diagram showing the state of abnormal (tumor) detection by CycleGAN used in the tumor detection system according to Embodiment 3. [Figure 10] Figure 10 is a flowchart showing the operation of tumor detection in the tumor detection system according to Embodiment 3. [Figure 11] Figure 11 is a schematic configuration diagram of pix2pix used in the tumor detection method of the existing method.

Embodiments for Carrying Out the Invention

[0011] The embodiments will be described in detail below, with reference to the drawings as appropriate. However, unnecessarily detailed explanations may be omitted. For example, detailed explanations of already well-known matters and redundant explanations of substantially identical configurations may be omitted. This is to avoid the following explanation becoming unnecessarily verbose and to facilitate understanding for those skilled in the art.

[0012] The inventors provide the accompanying drawings and the following description so that those skilled in the art can fully understand the disclosure, and not to limit the subject matter described in the claims.

[0013] 1. [Background leading to this disclosure] There is a need for detection systems that can accurately detect abnormalities hidden in the human body using imaging technologies such as contrast-enhanced CT, MRI, and X-ray, for example, AI-based systems. Similarly, there is a widespread demand for AI-based tumor detection systems that require less preparation of training data for medical professionals, such as image-level tumor detection systems using Generative Adversarial Networks (GANs).

[0014] One existing image-level tumor detection method using Generative Adversarial Networks (GANs) employs pix2pix. Figure 11 shows a schematic diagram of pix2pix used in such an existing tumor detection method. Here, the generator (G) 102 outputs a fake generated image based on a real image (i.e., it maintains the content of the input image). The discriminator (D) 104 is trained to determine that the real image is genuine and the fake generated image is fake. The training of the generator (G) 102 and the training of the discriminator (D) 104 are in an adversarial relationship.

[0015] In the existing method shown in Figure 11, only normal data is used as input data during training, and the corresponding output data during training uses the same image data as the input data. Therefore, it does not require supervised data, and effectively performs unsupervised learning.

[0016] In other words, during training, the generator (G) 102 is trained using only normal data. Therefore, during testing, when normal data is input, it can generate (output) an image with high accuracy. On the other hand, since input and output related to tumor data (abnormal data) have not been trained, it is expected that even if tumor data (abnormal data) is input, tumor regions cannot be generated (output). Therefore, when tumor data (abnormal data) is input, the tumor region will be detected by checking the error between the output data in which no tumor region is generated and the input data that originally contained the tumor region.

[0017] In other words, during testing, the reconstruction error is small for normal data, while the reconstruction error is large for tumor data in the tumor region.

[0018] Typically, pix2pix learns using pairs of images with perfectly matched contours. In other words, pix2pix is ​​essentially a supervised learning method. On the other hand, the existing method described above learns using the same image as the input data as the output data, so it is effectively an unsupervised learning method. This significantly reduces the burden of preparing training data for healthcare professionals.

[0019] However, existing methods for detecting tumor regions are based on the assumption that "even if tumor data is input, no tumor region will be output." This limits the accuracy of anomaly detection. A tumor detection system capable of detecting tumor regions with greater accuracy is needed.

[0020] 2. [Embodiment 1] A preferred embodiment 1 of this disclosure will be described below with reference to the attached drawings.

[0021] 2.1. [System Configuration] The tumor detection system and tumor detection method according to Embodiment 1 are a system and method for automatically detecting the presence and location of abnormalities (tumors) in the human body from human body image data. Figure 1 is a system configuration diagram of the tumor detection system 2 according to Embodiment 1.

[0022] The tumor detection system 2 comprises a computer device 4 and a storage device 12. The computer device 4 and the storage device 12 are connected by a wired or wireless communication line and can send and receive data from each other. The tumor detection system 2 is further connected to an external network 18 and exchanges data with other computer systems connected to the external network 18. The tumor detection system 2 is also connected to an external terminal 16. Furthermore, it is desirable for the tumor detection system 2 to be connected to a learning server 14.

[0023] Computer device 4 is a server machine or workstation computer equipped with one or more processors.

[0024] The storage device 12 is a storage device located outside the computer device 4, such as a disk drive or flash memory, and stores various databases, various data sets, and various computer programs used by the computer device 4. For example, image data transmitted from the external terminal 16, which will be described later, is recorded in the storage device 12.

[0025] The external terminal 16 is a personal computer or tablet device, and is preferably connected to the CT scanner or CT operation console. The external terminal 16 transmits and receives contrast-enhanced CT image data acquired from the CT scanner or CT operation console to and from the computer device 4 and the learning server 14.

[0026] The learning server 14 uses the image data and other data recorded in the storage device 12 to train a machine learning model, which will be explained later.

[0027] The external network 18 is, for example, the internet, and is connected to the computer device 4 via an interface device 6 such as a network terminal.

[0028] Furthermore, the computer device 4 includes an interface device 6, a processing circuit 8, and a memory 10.

[0029] The interface device 6 is an interface unit that can exchange data with the outside world, including a network terminal, video input terminal, USB terminal, keyboard, mouse, etc. The data here is, for example, ART image data obtained by contrast-enhanced CT, which will be explained later. After acquisition, this data can be recorded in the storage device 12. The data recorded in the storage device 12 can be accessed into the computer device 4 via the interface device 6 as appropriate.

[0030] The processing circuit 8 is comprised of a processor. Here, the processor encompasses a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The various processes of the tumor detection system 2 according to Embodiment 1 are realized by the execution of various programs by the processing circuit 8. These various processes may be realized by an ASIC (Application Specific Integrated Circuit) or a combination thereof.

[0031] The processing circuit 8 in this disclosure may consist of multiple signal processing circuits. Each signal processing circuit may be, for example, a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) and may be called a "processor". In the tumor detection system 2 according to this embodiment, one processor (e.g., a GPU) may perform some of the various processes, and another processor (e.g., a CPU) may perform some of the other processes.

[0032] Memory 10 is a data-rewritable storage unit within the computer device 4, and is composed of, for example, a RAM (Random Access Memory) containing a large number of semiconductor memory elements. Memory 10 temporarily stores specific computer programs, variable values, parameter values, etc., when the processing circuit 8 performs various processes. Memory 10 may also include so-called ROM (Read Only Memory). The ROM pre-stores a computer program that implements the processing of the tumor detection system 2 described below. The processing circuit 8 reads the computer program from the ROM and loads it into the RAM, making the processing circuit 8 able to execute the computer program.

[0033] The various processing programs for the computer device 4 in the tumor detection system 2 according to Embodiment 1 are constructed using a computer language such as Python. The computer language that can be used to construct the tumor detection system 2 according to this disclosure is not limited to this, and of course, other computer languages ​​may be used.

[0034] Furthermore, in the computer device 4 of the tumor detection system 2 according to this embodiment, a trained machine learning model is constructed. As will be explained later, the machine learning model according to this embodiment is constructed using a network structure such as CycleGAN, which includes skip connections (see Figure 3).

[0035] 2.1.1.[Machine Learning Model Configuration] Figure 2 is a schematic diagram of the machine learning model used in the tumor detection system 2 according to Embodiment 1. The machine learning model shown in Figure 2 has the configuration of CycleGAN.

[0036] The machine learning model with the configuration of CycleGAN shown in Figure 2 includes a first generator (G1) 22, a first discriminator (D1) 28, a second generator (G2) 24, and a second discriminator (D2) 26. The first generator (G1) 22 generates fake tumor image data from real normal image data obtained by contrast-enhanced CT ART. First, the first generator (G1) 22 is trained to output fake tumor image data that deceives the first discriminator (D1) 28 from the input of real normal image data. In contrast, the first discriminator (D1) 28 is trained to determine that the fake tumor image data generated by the first generator (G1) 22 is fake, and that the real tumor image data is real. Thus, the training of the first generator (G1) 22 and the training of the first discriminator (D1) 28 are in an adversarial relationship.

[0037] The second generator (G2) 24 generates fake normal image data from real tumor image data. First, the second generator (G2) 24 is trained through learning to output fake normal image data that deceives the second discriminator (D2) 26, based on the input of real tumor image data. In response, the second discriminator (D2) 26 is trained through learning to determine that the fake normal image data generated by the second generator (G2) 24 is fake, and that the real normal image data is real. Thus, the training of the second generator (G2) 24 and the training of the second discriminator (D2) 26 are in an adversarial relationship.

[0038] Furthermore, in the machine learning model with the CycleGAN configuration shown in Figure 2, when normal image data is used during training, the normal image data is input to the first generator (G1) 22 to first generate abnormal image data. The generated abnormal image data is then input to the second generator (G2) 24 to subsequently generate normal image data. The first generator (G1) 22 and the second generator (G2) 24, as well as the first discriminator (D1) 28 and the second discriminator (D2) 26, are trained by comparing the normal image data generated by the second generator (G2) 24 with the initial normal image data.

[0039] The first generator (G1) 22 and the second generator (G2) 24 preferably have an autoencoder configuration consisting of an encoder and a decoder. Furthermore, the first generator (G1) 22 and the second generator (G2) 24 are generative networks that generate images while maintaining the features of the input image, and it is preferable that each layer of the encoder and each layer of the decoder are skip-connected. In other words, it is preferable that the feature quantities obtained at each layer can be transmitted from the encoder to the decoder. Figure 3 is a schematic configuration diagram of a U-Net including skip connections, and shows the network structures that the first generator (G1) 22 and the second generator (G2) 24 can adopt.

[0040] While the aforementioned pix2pix learns using pairs of images with perfectly matched contours, the above-described machine learning model, which has the configuration of CycleGAN, does not require that the target variable, anomalous image data, of the first generator (G1) 22 be paired with the explanatory variable, normal image data, during training. In other words, the anomalous image data only needs to be labeled with a domain relationship indicating the presence of an anomaly, and does not need to be labeled with the location or shape of the anomaly. Since the input image data and output image data during training only need to be images of Ampere and satisfy a predetermined domain relationship (i.e., the output image data only needs to have a label indicating the presence of some kind of anomaly), this type of training is referred to as "weakly supervised learning" in this specification. The same applies to training using the target variable, normal image data, and the explanatory variable, anomalous image data, of the second generator (G2) 24.

[0041] Users of the tumor detection system 2 according to Embodiment 1, i.e., healthcare professionals, only need to randomly prepare, for example, normal ART image data and abnormal ART image data from contrast-enhanced CT in order to perform the aforementioned "weakly supervised learning." The normal and abnormal image data do not need to be paired images. This significantly reduces the burden of preparing training data for healthcare professionals.

[0042] In training a machine learning model with the CycleGAN configuration described above, the following loss function is considered, which includes the adversarial loss and the cycle consistency loss.

[0043] [Hostile Loss] The adversarial loss (GAN Loss) is given by the following formula.

number

number

number

[0044] [Cycle consistency loss] Cycle consistency loss represents the function that occurs when the image generated by the generator is returned to the input image. The cycle consistency loss is given by the following formula:

number

[0045] This cycle consistency loss is calculated by comparing the "image reconstructed from the converted image" with the "original input data."

[0046] 2.2. [System Operation] When detecting a tumor in ART image data from contrast-enhanced CT using the tumor detection system 2 according to Embodiment 1, a test image is input to the first generator (G1) 22, and the generated image is further input to the second generator (G2) 24. The error between the image generated by the second generator (G2) 24 and the input image is calculated, and if the error is small, it is determined to be normal, and if the error is large, it is determined that a tumor is present.

[0047] Figure 4 is a flowchart showing the operation of tumor detection in the tumor detection system 2 according to Embodiment 1. After the start (step S02), the processing circuit 8 inputs the image data acquired via the interface device 6 to the first generator (G1) 22 (step S04).

[0048] The processing circuit 8 inputs the image data generated by the first generator (G1) 22 to the second generator (G2) 24 (step S06).

[0049] The processing circuit 8 calculates the error between the image data generated by the second generator (G2) 24 and the original image data (step S08), and determines whether the calculated error is smaller than a predetermined threshold (step S10).

[0050] If the error is smaller than a predetermined threshold (Step S10 - YES), the processing circuit 8 determines that the image data is normal (Step S12) and terminates (Step S16). If the error is not smaller than a predetermined threshold (Step S10 - NO), the processing circuit 8 determines that the image data is abnormal, i.e., tumor image data (Step S14) and terminates (Step S16).

[0051] Figure 5A shows examples of (5A-1) input CT image, (5A-2) output CT image, (5A-3) input liver image, (5A-4) output liver image, and (5A-5) detection result image for a METS (metastatic cancer) liver tumor detected by the tumor detection system 2 according to Embodiment 1.

[0052] (5A-3) The input liver image is an extraction of only the liver portion from the input CT image (5A-1), and the output liver image (5A-4) is an extraction of only the liver portion from the output CT image (5A-2). (5A-5) The detection result image is the difference (error) between the output liver image (5A-4) and the input liver image (5A-3). The same applies to Figures 5B to 6C below.

[0053] In Figure 5A (5A-5), the detection results show virtually nothing, and therefore, the input CT image (5A-1) here is determined to be normal image data.

[0054] Figure 5B shows examples of (5B-1) input CT image, (5B-2) output CT image, (5B-3) input liver image, (5B-4) output liver image, and (5B-5) detection result image for an HCC (hepatocellular carcinoma) liver tumor using the tumor detection system 2 according to Embodiment 1. In Figure 5B (5B-5) detection result, virtually nothing is shown, and therefore, the input CT image (5B-1) here is also determined to be normal image data.

[0055] Figure 5C shows an example of (5C-1) input CT image, (5C-2) output CT image, (5C-3) input liver image, (5C-4) output liver image, and (5C-5) detection result image for a CCC (intrahepatic bile duct cancer) liver tumor using the tumor detection system 2 according to Embodiment 1. In Figure 5C (5C-5) detection result, virtually nothing is shown, and therefore, the (5C-1) input CT image here is also determined to be normal image data.

[0056] In relation to the above, Figure 6A shows examples of (6A-1) input CT image, (6A-2) output CT image, (6A-3) input liver image, (6A-4) output liver image, and (6A-5) detection result image related to a METS (metastatic cancer) liver tumor detected by the tumor detection system 2 according to Embodiment 1, and these are examples of images that actually contain a tumor. In other words, in the detection result of Figure 6A (6A-5), the difference (error) is mainly represented in the area enclosed by the white line box. In this case, the input CT image (6A-1) is determined to be abnormal image data. Note that the "white line box" in the image examples of Figures 6A to 6C was added additionally.

[0057] Figure 6B shows examples of (6B-1) input CT image, (6B-2) output CT image, (6B-3) input liver image, (6B-4) output liver image, and (6B-5) detection result image related to an HCC (hepatocellular carcinoma) liver tumor detected by the tumor detection system 2 according to Embodiment 1. These are also examples of images that actually contain tumors. In the detection result of Figure 6B (6B-5), the difference (error) is mainly shown in the area enclosed by the white box. The input CT image (6B-1) in this case is also determined to be abnormal image data.

[0058] Figure 6C shows examples of (6C-1) input CT image, (6C-2) output CT image, (6C-3) input liver image, (6C-4) output liver image, and (6C-5) detection result image related to a CCC (intrahepatic bile duct cancer) liver tumor detected by the tumor detection system 2 according to Embodiment 1. These are also examples of images that actually contain tumors. In Figure 6B (6B-5) detection results, the difference (error) image is mainly shown in the area enclosed by the white box. In this case, the (6C-1) input CT image is also determined to be abnormal image data.

[0059] Figure 7A shows a numerical example of the detection results of METS (metastatic cancer) by the tumor detection system 2 according to Embodiment 1, where each graph shows the mean squared error (MSE) between the input image and the output image. White graphs below the threshold show values ​​for normal image data, while gray graphs above the threshold show values ​​for abnormal image data.

[0060] Figure 7B shows a numerical example of the detection results of HCC (hepatocellular carcinoma) by the tumor detection system 2 according to Embodiment 1. Here, each graph represents the mean squared error (MSE) between the input and output images. White graphs below the threshold show values ​​for normal image data, while gray graphs above the threshold show values ​​for abnormal image data.

[0061] Figure 7C shows a numerical example of the detection results of CCC (intrahepatic bile duct cancer) by the tumor detection system 2 according to Embodiment 1. Here, each graph represents the mean squared error (MSE) between the input and output images. White graphs below the threshold show values ​​for normal image data, while gray graphs above the threshold show values ​​for abnormal image data.

[0062] 2.2.1. [Quantitative comparison with conventional methods] The inventors have performed a quantitative comparison between the tumor detection method using the tumor detection system 2 according to Embodiment 1 and a conventional method. The conventional method here refers to the existing method shown in Figure 11, for example. When using the conventional method, it has been confirmed that the AUC is values ​​such as "0.598", "0.631", "0.610", and "0.625". In contrast, it has been confirmed that the tumor detection method according to Embodiment 1 has high values ​​such as an MSE (Mean Squared Error) AUC of "0.682" and a GMS (Gradient Magnitude Similarity) AUC of "0.740".

[0063] 2.3. [Summary of Embodiment 1] The tumor detection system 2 according to this embodiment 1 has a computer device 4, which includes an interface device 6 and a processing circuit 8. The computer device 4 has a trained CycleGAN machine learning model built on it, which includes a first generator (G1) 22, a first discriminator (D1) 28, a second generator (G2) 24, and a second discriminator (D2) 26. Here, the first generator (G1) 22 is trained to output fake tumor image data from real normal image data input. The first discriminator (D1) 28 is trained to determine that the fake tumor image data generated by the first generator (G1) 22 is fake, and that real tumor image data is real. Furthermore, the second generator (G2) 24 is trained to output fake normal image data from real tumor image data input. The second classifier (D2) 26 is trained through learning to determine that fake generated image data produced by the second generator (G2) 24 is fake, and that real normal image data is real. Furthermore, in the trained CycleGAN machine learning model, during training, normal image data is input to the first generator (G1) 22 to generate abnormal image data, the generated abnormal image data is input to the second generator (G2) 24 to generate normal image data, and the first generator (G1) 22 and the second generator (G2) 24, as well as the first classifier (D1) 28 and the second classifier (D2) 26, learn by comparing the normal image data generated by the second generator (G2) 24 with the initial normal image data. For detection, the processing circuit 8 inputs the image data acquired via the interface device 6 to the first generator (G1) 22. The processing circuit 8 inputs the image data generated by the first generator (G1) 22 to the second generator (G2) 24 based on the input to the first generator (G1) 22. The processing circuit 8 calculates the error between the image data generated by the second generator (G2) 24 based on the input to the second generator (G2) 24 and the original image data input to the first generator (G1) 22.The processing circuit determines that the image data input to the first generator (G1) 22 is normal image data if the calculated error is smaller than a predetermined threshold, and determines that the image data input to the first generator (G1) 22 is abnormal image data if the calculated error is not smaller than a predetermined threshold.

[0064] The tumor detection system 2 described above requires very little preparation of training data from the user and can detect with high accuracy whether or not image data contains abnormal areas. It is possible.

[0065] 3. [Embodiment 2] The tumor detection system 2 according to Embodiment 2 has the same configuration as the tumor detection system 2 according to Embodiment 1. Furthermore, the tumor detection system 2 according to Embodiment 2 operates in the same manner as the tumor detection system 2 according to Embodiment 1.

[0066] The tumor detection system 2 according to Embodiment 2 has a distinctive feature in the tumor image data used during training. In the tumor detection system 2 according to Embodiment 1, actual tumor image data is used as tumor image data during training. In contrast, in the tumor detection system 2 according to Embodiment 2, pseudo-tumor image data, which is semi-automatically and artificially created, is used as tumor image data during training.

[0067] Figures 8(8-1), (8-2), and (8-3) are examples of contrast-enhanced CT images of the liver that actually contain a tumor (abnormality). In contrast, Figures 8(8-4), (8-5), and (8-6) are examples of pseudo-abnormality image data (tumor image data) used in the tumor detection system 2 according to Embodiment 2.

[0068] The pseudo-abnormal liver images shown in Figures 8(8-4), (8-5), and (8-6) are created by embedding a reduced-size image of a normal liver as a "pseudo" tumor (image) in an arbitrary location within a normal liver image. This pseudo-abnormal image data is then applied to the training of a machine learning model with the CycleGAN configuration shown in Figure 2. In other words, the abnormal image data applied to the training of the machine learning model is created by embedding a reduced-size image of the object (i.e., the liver) from another normal image as a pseudo-abnormal portion within one normal image.

[0069] In the training of the machine learning model in the tumor detection system 2 according to Embodiment 2, the preparation of actual abnormal image data is substantially unnecessary. As a result, a machine learning model with the configuration of CycleGAN shown in Figure 2 is trained through unsupervised learning.

[0070] Unsupervised learning significantly reduces the burden on healthcare professionals in preparing training data.

[0071] 3.1. [Quantitative comparison with conventional methods] The inventors have conducted a quantitative comparison between the tumor detection method using the tumor detection system 2 according to Embodiment 2, the conventional method, and the tumor detection method using the tumor detection system 2 according to Embodiment 1. The conventional method here refers to the existing method shown in Figure 11, for example. As previously mentioned, it has been confirmed that when using the conventional method, the AUC values ​​are "0.598", "0.631", "0.610", "0.625", etc.

[0072] Furthermore, as previously mentioned, in the tumor detection method according to Embodiment 1, the AUC of MSE (Mean Squared Error) is "0.682" and the AUC of GMS (Gradient Magnitude Similarity) is "0.740". In contrast to these, it has been confirmed that the tumor detection method according to Embodiment 1 also shows high values, such as the AUC of MSE (Mean Squared Error) being "0.644" and the AUC of GMS (Gradient Magnitude Similarity) being "0.810".

[0073] 3.2. [Summary of Embodiment 2] In the tumor detection system 2 according to this embodiment 2, abnormal image data applied in the training of the machine learning model is created by embedding a reduced image portion of the object captured in another normal image data as a pseudo-abnormal portion into one normal image data.

[0074] The tumor detection system 2 described above requires very little preparation of training data from the user and can detect with high accuracy whether or not image data contains abnormal areas.

[0075] 4. [Embodiment 3] The tumor detection system 2 according to Embodiment 3 has the same configuration as the tumor detection system 2 according to Embodiments 1 and 2. Furthermore, the tumor detection system 2 according to Embodiment 3 operates substantially the same as the tumor detection system 2 according to Embodiments 1 and 2.

[0076] 4.1. [Machine Learning Model Configuration] Figure 9A is a schematic diagram of the CycleGAN used in the tumor detection system 2 according to Embodiment 3. The configuration of CycleGAN, the machine learning model used in the tumor detection system 2 according to Embodiment 3, is similar to that of the CycleGAN machine learning model used in the tumor detection system 2 according to Embodiments 1 and 2 shown in Figure 2, although the notation is slightly different.

[0077] The machine learning model with the CycleGAN configuration shown in Figure 9A is similar to that shown in Figure 2, including a first generator (G1) 22, a first discriminator (D1) 28, a second generator (G2) 24, and a second discriminator (D2) 26. In the machine learning model shown in Figure 9A, as in Figure 2, the training of the first generator (G1) 22 and the training of the first discriminator (D1) 28 are in an adversarial relationship.

[0078] Furthermore, in the machine learning model shown in Figure 9A, just as in Figure 2, the training of the second generator (G2) 24 and the training of the second discriminator (D2) 26 are in an adversarial relationship.

[0079] Furthermore, in a machine learning model with the CycleGAN configuration shown in Figure 9A, when using normal image data during training, the normal image data is input to the first generator (G1) 22 to first generate abnormal image data. The generated abnormal image data is then input to the second generator (G2) 24 to subsequently generate normal image data. The first generator (G1) 22 and the second generator (G2) 24, as well as the first discriminator (D1) 28 and the second discriminator (D2) 26, are trained by comparing the normal image data generated by the second generator (G2) 24 with the initial normal image data. The characteristic training process in a machine learning model with such a CycleGAN configuration is the same for both Figure 9A and Figure 2.

[0080] In the tumor detection system 2 according to Embodiment 3 shown in Figure 9A, it is preferable that the first generator (G1) 22 and the second generator (G2) 24 are autoencoders equipped with a U-net configuration including skip connections.

[0081] Furthermore, the tumor detection system 2 according to Embodiment 3 performs either "weakly supervised learning" similar to Embodiment 1 or "unsupervised learning" similar to Embodiment 2 during the learning process. Therefore, the burden of preparing learning data for healthcare professionals is very small.

[0082] Furthermore, in CycleGAN, the machine learning model used in the tumor detection system 2 according to Embodiment 3, the loss function including the aforementioned [adversarial loss] and [cycle consistency loss] is also considered during training.

[0083] 4.2. [System Operation] Figure 9B is a schematic diagram showing the abnormality (tumor) detection process using a machine learning model with the configuration of CycleGAN used in the tumor detection system 2 according to Embodiment 3. When detecting tumors in ART image data of contrast-enhanced CT using the tumor detection system 2 according to Embodiment 3, the test image is directly input to the second generator (G2) 24, as shown in Figures 9B(9B-1) and (9B-2). That is, Embodiments 1 and 2 Unlike the tumor detection system 2 described above, no input is given to the first generator (G1) 22. The tumor detection system 2 according to Embodiment 3 calculates the error between the image generated by the second generator (G2) 24 and the input image. If the error is small, it is considered normal, and if the error is large, it is determined that there is a tumor.

[0084] Figure 10 is a flowchart showing the operation of tumor detection in the tumor detection system 2 according to Embodiment 3. After the start (step S22), the processing circuit 8 inputs the image data acquired via the interface device to the second generator (G2) 24 (step S24).

[0085] The processing circuit 8 calculates the error between the image data generated by the second generator (G2) 24 and the original image data (step S26), and determines whether the calculated error is smaller than a predetermined threshold (step S28).

[0086] If the error is smaller than a predetermined threshold (Step S28 - YES), the processing circuit 8 determines that the image data is normal (Step S30) and terminates (Step S34). If the error is not smaller than a predetermined threshold (Step S28 - NO), the processing circuit 8 determines that the image data is abnormal, i.e., tumor image data (Step S32) and terminates (Step S34).

[0087] 3.3. [Summary of Embodiment 3] The tumor detection system 2 according to this third embodiment includes a computer device 4, which includes an interface device 6 and a processing circuit 8. The computer device 4 has a trained CycleGAN machine learning model built on it, which includes a first generator (G1) 22, a first discriminator (D1) 28, a second generator (G2) 24, and a second discriminator (D2) 26. Here, the first generator (G1) 22 is trained to output fake tumor image data from real normal image data input. The first discriminator (D1) 28 is trained to determine that the fake tumor image data generated by the first generator (G1) 22 is fake, and that real tumor image data is real. Furthermore, the second generator (G2) 24 is trained to output fake normal image data from real tumor image data input. The second classifier (D2) 26 is trained through learning to determine that fake generated image data produced by the second generator (G2) 24 is fake, and that real normal image data is real. Furthermore, in the trained CycleGAN machine learning model, during training, normal image data is input to the first generator (G1) 22 to generate abnormal image data, the generated abnormal image data is input to the second generator (G2) 24 to generate normal image data, and the first generator (G1) 22 and the second generator (G2) 24, as well as the first classifier (D1) 28 and the second classifier (D2) 26, learn by comparing the normal image data generated by the second generator (G2) 24 with the initial normal image data. For detection, the processing circuit 8 inputs the image data acquired via the interface device 6 to the second generator (G2) 24. The processing circuit 8 receives input to the second generator (G2) 24 and the image data generated by the second generator (G2) 24, Second generator (G2) 24 The error between the input image data and the original image data is calculated. If the calculated error is smaller than a predetermined threshold, the processing circuit proceeds. Second generator (G2) 24 If the input image data is determined to be normal image data, and the calculated error is not smaller than a predetermined threshold, Second generator (G2) 24The system determines that the input image data is abnormal image data.

[0088] The tumor detection system 2 described above requires very little preparation of training data from the user and can detect with high accuracy whether or not image data contains abnormal areas.

[0089] Furthermore, the tumor detection system 2 according to Embodiment 3 may also apply the tumor image data used by the tumor detection system 2 according to Embodiment 2 during training during training.

[0090] 3. [Other Embodiments] As described above, embodiments have been explained as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited thereto and can be applied to embodiments that have been modified, replaced, added, or omitted as appropriate.

[0091] The tumor detection system 2 according to Embodiments 1 to 3 has been described as primarily for detecting abnormalities in the liver. However, by appropriately changing the training data during learning, the tumor detection system 2 according to Embodiments 1 to 3 can accurately detect abnormalities in other organs.

[0092] Furthermore, while the tumor detection system 2 according to embodiments 1 to 3 uses contrast-enhanced CT image data as input data, the input data is not limited to this. By utilizing the technology of this disclosure, it is also possible to construct a tumor detection system 2 that uses MRI or X-ray image data as input data.

[0093] Furthermore, the detection systems according to Embodiments 1 to 3 are systems that detect abnormalities inside the human body using image data of the inside of the human body. For example, if image data of a building wall, with or without cracks or fissures, is used as training data during learning, the detection system according to the embodiments of this disclosure can operate as a system that detects abnormalities in the building wall by inputting image data of the building wall. Similarly, the detection system according to the embodiments of this disclosure can be used to detect abnormalities in various objects.

[0094] Furthermore, attached drawings and a detailed description are provided to illustrate the embodiments. Therefore, among the components described in the attached drawings and detailed description, there may be not only components essential for solving the problem, but also components that are not essential for solving the problem, provided that they illustrate the above technology. For this reason, the mere presence of such non-essential components in the attached drawings and detailed description should not be immediately assumed to mean that those non-essential components are essential.

[0095] Furthermore, since the embodiments described above are for illustrative purposes of the technology described herein, various modifications, substitutions, additions, omissions, etc., can be made within the claims or their equivalents. [Explanation of Symbols]

[0096] 2... Tumor detection system, 4... Computer device, 6... Interface device, 8... Processing circuit, 10... Memory, 12... Storage device, 14... Learning server, 16... External terminal, 22... First generator (G1), 24... Second generator (G2), 26... Second discriminator (D2), 28... First discriminator (D1), 102... Generator (G), 104... Discriminator (D).

Claims

1. In an anomaly detection system, The anomaly detection system includes a computer device, and the computer device includes an interface device and a processing circuit. The computer device has a trained CycleGAN machine learning model built on it, which includes a first generator, a first discriminator, a second generator, and a second discriminator. Here, the first generator is trained to output fake abnormal image data from real normal image data input, and the first discriminator is trained to determine that the fake abnormal image data generated by the first generator is fake, and that the real abnormal image data is real. Furthermore, the second generator is trained to output fake normal image data from real abnormal image data input, and the second discriminator is trained to determine that the fake normal image data generated by the second generator is fake, and that the real normal image data is real. Furthermore, in the trained CycleGAN machine learning model, during training, normal image data is input to the first generator to generate abnormal image data, the generated abnormal image data is input to the second generator to generate normal image data, and the first and second generators, as well as the first and second classifiers, are trained by comparing the normal image data generated by the second generator with the initial normal image data. For detection, The processing circuit inputs the image data acquired via the interface device to the first generator. The processing circuit, upon input to the first generator, inputs the image data generated by the first generator to the second generator. The processing circuit calculates the error between the image data generated by the second generator based on the input to the second generator and the original image data input to the first generator. The processing circuit determines that the image data input to the first generator is normal image data if the calculated error is smaller than a predetermined threshold, and determines that the image data input to the first generator is abnormal image data if the calculated error is not smaller than a predetermined threshold. Anomaly detection system.

2. In an anomaly detection system, The anomaly detection system includes a computer device, and the computer device includes an interface device and a processing circuit. The computer device has a trained CycleGAN machine learning model built on it, which includes a first generator, a first discriminator, a second generator, and a second discriminator. Here, the first generator is trained to output fake abnormal image data from real normal image data input, and the first discriminator is trained to determine that the fake abnormal image data generated by the first generator is fake, and that the real abnormal image data is real. Furthermore, the second generator is trained to output fake normal image data from real abnormal image data input, and the second discriminator is trained to determine that the fake normal image data generated by the second generator is fake, and that the real normal image data is real. Furthermore, in the trained CycleGAN machine learning model, during training, normal image data is input to the first generator to generate abnormal image data, the generated abnormal image data is input to the second generator to generate normal image data, and the first and second generators, as well as the first and second classifiers, are trained by comparing the normal image data generated by the second generator with the initial normal image data. For detection, The processing circuit inputs the image data acquired via the interface device to the second generator. The processing circuit calculates the error between the image data generated by the second generator based on the input to the second generator and the original image data input to the second generator. The processing circuit determines that the image data input to the second generator is normal image data if the calculated error is smaller than a predetermined threshold, and determines that the image data input to the second generator is abnormal image data if the calculated error is not smaller than a predetermined threshold. Anomaly detection system.

3. The abnormal image data applied in the training of the aforementioned machine learning model is created by embedding a reduced version of the image portion of an object from another normal image data as a pseudo-abnormal portion into one normal image data. An anomaly detection system according to claim 1 or 2.

4. The aforementioned abnormality detection system is a system that detects tumors as abnormalities, The aforementioned normal image data is image data that does not include tumor images. The aforementioned abnormal image data is image data including tumor images. The anomaly detection system according to claim 3.

5. A step of constructing a trained CycleGAN machine learning model, which includes a first generator, a first discriminator, a second generator, and a second discriminator, using processing circuits that constitute a computer device, Here, the first generator is trained to output fake abnormal image data from real normal image data input, and the first discriminator is trained to determine that the fake abnormal image data generated by the first generator is fake, and that the real abnormal image data is real. Furthermore, the second generator is trained to output fake normal image data from real abnormal image data input, and the second discriminator is trained to determine that the fake normal image data generated by the second generator is fake, and that the real normal image data is real. Furthermore, in the trained CycleGAN machine learning model, during training, normal image data is input to the first generator to generate abnormal image data, the generated abnormal image data is input to the second generator to generate normal image data, and the first and second generators, as well as the first and second classifiers, perform training by comparing the normal image data generated by the second generator with the initial normal image data. Steps to build a pre-trained CycleGAN machine learning model, The processing circuit inputs image data acquired via an interface device constituting the computer device to the first generator, The processing circuit includes the step of inputting the image data generated by the first generator to the second generator based on the input to the first generator, The processing circuit performs the steps of calculating the error between the image data generated by the second generator based on the input to the second generator and the original image data input to the first generator, If the calculated error is smaller than a predetermined threshold, the processing circuit determines that the image data input to the first generator is normal image data. If the calculated error is not smaller than a predetermined threshold, the processing circuit determines that the image data input to the first generator is abnormal image data. A method for detecting anomalies from image data, including [specific data / features].

6. A step of constructing a trained CycleGAN machine learning model, which includes a first generator, a first discriminator, a second generator, and a second discriminator, using processing circuits that constitute a computer device, Here, the first generator is trained to output fake abnormal image data from real normal image data input, and the first discriminator is trained to determine that the fake abnormal image data generated by the first generator is fake, and that the real abnormal image data is real. Furthermore, the second generator is trained to output fake normal image data from real abnormal image data input, and the second discriminator is trained to determine that the fake normal image data generated by the second generator is fake, and that the real normal image data is real. Furthermore, in the trained CycleGAN machine learning model, during training, normal image data is input to the first generator to generate abnormal image data, the generated abnormal image data is input to the second generator to generate normal image data, and the first and second generators, as well as the first and second classifiers, perform training by comparing the normal image data generated by the second generator with the initial normal image data. Steps to build a pre-trained CycleGAN machine learning model, The processing circuit inputs image data acquired via an interface device constituting the computer device to the second generator, The processing circuit performs the steps of calculating the error between the image data generated by the second generator based on the input to the second generator and the original image data input to the second generator, If the calculated error is smaller than a predetermined threshold, the processing circuit determines that the image data input to the second generator is normal image data. If the calculated error is not smaller than a predetermined threshold, the processing circuit determines that the image data input to the second generator is abnormal image data. A method for detecting anomalies from image data, including [specific data / features].

7. The abnormal image data applied in the training of the aforementioned machine learning model is created by embedding a reduced-size image portion of an object from another normal image data as a pseudo-abnormal portion into one normal image data. The method according to claim 5 or 6.

8. The aforementioned normal image data is image data that does not include tumor images. The aforementioned abnormal image data is image data including tumor images. The method according to claim 7.

Images