Acquisition method for virtually cleaned intestinal tract image
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Filing Date
- 2024-02-14
- Publication Date
- 2025-02-27
Abstract
Description
Method for acquiring virtual bowel cleaning images
[0001] This method relates to a method for obtaining a virtual intestinal cleansing image, and more specifically, to a method for performing image processing to remove residues from a CT image of a subject's large intestine, and obtaining an image of the large intestine as if virtually undergoing intestinal cleansing.
[0002] Japanese Patent Publication No. 7304594 describes a system that uses deep learning to determine whether tissue contained in medical images indicates normal or abnormal disease. However, when attempting to diagnose colonic diseases (e.g., the presence or absence of colonic polyps or colon cancer), it can be difficult to make a diagnosis using medical images alone due to intestinal residue (e.g., feces). Because of this residue, it is difficult to grasp the intestinal condition. For this reason, a laxative is administered to remove the intestinal residue, and then the intestinal environment is diagnosed using an endoscope.
[0003] Patent No. 7304594
[0004] One object of the present invention is to provide a method for acquiring virtual bowel cleaning images that can remove residue from images of the large intestine captured without using laxatives. Another object of the present invention is to provide a method for acquiring virtual bowel cleaning images that can prevent situations that interfere with the diagnosis of large intestine-related diseases. Another object of the present invention is to provide a method for detecting large intestine-related diseases (particularly colon polyps and colon cancer) from images of the large intestine. Another object of the present invention is to provide a generative AI model for virtual bowel cleaning images and a generative AI model for detecting large intestine-related diseases.
[0005] The present invention aims to achieve any of the above. This invention is based on the finding that a virtual bowel cleansing image, which is an image of a virtually cleaned bowel, can be obtained by performing image processing to remove residue from an image of a portion of a patient's large intestine. This invention is further based on the finding that by using machine learning, a method can be provided for acquiring a virtual bowel cleansing image that can prevent the occurrence of situations that interfere with the diagnosis of large intestine-related diseases, even without administering a laxative to the patient.
[0006] This specification describes a method for acquiring a virtual bowel cleaning image, which is a computer- or processor-implemented method, and includes an image input step and a virtual bowel cleaning step.
[0007] The image input step is a step of inputting a photographed image, which is an image of a part of a patient's large intestine, into the system for acquiring a virtual bowel cleaning image. An example of the photographed image is an image included in a three-dimensional photographed image of the patient's large intestine.
[0008] The virtual bowel cleansing process is a process in which the system obtains a virtual bowel cleansing image. The virtual bowel cleansing image is an image of a portion of a patient's large intestine from which the residue portion in the large intestine contained in the photographed image has been removed. An example of the virtual bowel cleansing process is a process in which the photographed image is directly converted into a virtual bowel cleansing image using a learning model. An example of the learning model is a model obtained by using, as training data, a residue-containing image, which is an image of a portion of the large intestine containing residue, and a residue-removed image, which is an image of a portion of the large intestine after the residue has been removed, and obtaining image processing parameters for obtaining the residue-removed image from the residue-containing image.
[0009] This specification also describes a program for causing a computer to execute the method for acquiring the virtual intestinal cleansing image described above, and a computer-readable non-transitory information recording medium storing such a program.
[0010] This specification also provides a method for detecting polyps from photographed images using a computer. This method includes a colon extraction step for obtaining a colon image, which is a partial image obtained by extracting only the colon portion included in the photographed image, which is an image of a portion of a patient's colon, and a polyp extraction step for obtaining a polyp image, which is an image obtained by extracting polyp portions included in the colon image. This method may further include a false positive removal step for removing false positives included in the polyp image. The polyp extraction step may be a cancer (colon cancer) extraction step. The false positive removal step may be a step for removing portions included in the polyp image that are determined to be cancerous even though they are not cancerous.
[0011] An example of the large intestine extraction process is a process of directly converting a photographed image into a large intestine image using a learning model for large intestine extraction. The learning model for large intestine extraction is a model obtained by using, as training data, a large intestine correct region image, which is an image obtained by removing organs other than the large intestine from a photographed image and extracting only the large intestine. The large intestine correct region image may be generated manually (by a person inputting information into a computer and processing it, etc.) or automatically. The photographed image is a photographed image prepared in advance to build a learning model. Parameters for the learning model may be obtained from the training data to obtain a trained model for large intestine extraction.
[0012] An example of the polyp extraction process is a process of directly converting a colon image into a polyp image using a learning model. An example of the learning model used in the polyp extraction process is a model obtained by using, as training data, a certain colon image and a polyp correct region image, which is an image obtained by extracting only the polyp region portion from the certain colon image. The certain captured image is a captured image prepared in advance for constructing the learning model. Parameters for the learning model may be obtained from the training data to obtain a trained model for polyp extraction.
[0013] This specification also describes a program for causing a computer to execute the method for acquiring the virtual intestinal cleansing image described above, and a computer-readable non-transitory information recording medium storing such a program.
[0014] The present invention can provide a method for acquiring a virtual bowel cleaning image that can remove residue from an image of the large intestine captured without using a laxative. The present invention can provide a method for acquiring a virtual bowel cleaning image that can reduce the occurrence of situations that interfere with the diagnosis of large intestine-related diseases. According to the present invention, a laxative-free colon CT examination can be performed using image processing by virtual bowel cleaning using machine learning. The present invention can provide a method for detecting large intestine-related diseases (particularly colon polyps and colon cancer) from images of the large intestine. The present invention can provide a generative AI model that generates a virtual bowel cleaning image and a generative AI model that detects large intestine-related diseases.
[0015] FIG. 1 is a process diagram illustrating a method for acquiring a virtual intestinal cleansing image. FIG. 2 is a block diagram illustrating a system for acquiring a virtual intestinal cleansing image. FIG. 3 is a flowchart illustrating an example of the virtual intestinal cleansing process. FIG. 4 shows images obtained when virtual intestinal cleansing is actually performed using the system. FIG. 5 is a process diagram illustrating a method for detecting colon polyps. FIG. 6 is a block diagram illustrating a system for detecting colon polyps. FIG. 7 is a flowchart illustrating an example of the large intestine extraction process. FIG. 8 is a flowchart illustrating an example of the polyp extraction process. FIG. 9 is a flowchart illustrating an example of the false positive removal process. FIG. 10 shows images obtained when large intestine extraction is actually performed using the system. FIG. 11 shows examples of virtual endoscopic image displays and cross-sectional image displays of a polyp correct region image used in the training and test phases of a learning model for polyp extraction. FIG. 12 shows examples of virtual endoscopic image displays and cross-sectional image displays of a polyp image output from a different learned model for polyp extraction. FIG. 13 shows images obtained when false positive removal is actually performed using the system.
[0016] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The present invention is not limited to the embodiments described below, but also includes appropriate modifications of the embodiments below within the scope obvious to those skilled in the art.
[0017] FIG. 1 is a process diagram illustrating a method for acquiring a virtual bowel cleansing image. As shown in FIG. 1, this method includes an image input step (S101) and a virtual bowel cleansing step (S102). S indicates a step. This method is implemented by a computer or processor. This method can also be considered a laxative-free colon CT examination using image processing based on virtual bowel cleansing using machine learning.
[0018] FIG. 2 is a block diagram showing a system for acquiring a virtual intestinal cleaning image. As shown in FIG. 2, the system 1 for acquiring a virtual intestinal cleaning image includes an image input unit 3 and a virtual intestinal cleaning unit 5. This system is implemented by a computer or a processor. The system 1 includes a processor and a memory storing a program. The program may be configured to cause the processor to execute each step of the method for acquiring a virtual intestinal cleaning image. As shown in FIG. 2, this system may further include one or more elements of a three-dimensional model creation unit 7, a machine learning unit 9, and a learning model storage unit 11.
[0019] The computer has an input unit, an output unit, a control unit, a calculation unit, and a memory unit, and each element is connected by a bus or the like to enable information exchange. For example, a control program or various information may be stored in the memory unit. When predetermined information is input from the input unit, the control unit reads the control program stored in the memory unit. Then, the control unit reads the information stored in the memory unit as appropriate and transmits it to the calculation unit. The control unit also transmits the input information to the calculation unit as appropriate. The calculation unit performs calculation processing using the received various information and stores it in the memory unit. The control unit reads the calculation results stored in the memory unit and outputs them from the output unit. In this manner, various processes and steps are performed. The various units and means execute these various processes. The computer may have a processor, and the processor may realize various functions and steps. The computer may be standalone. Some of the functions of the computer may be distributed between a server and a terminal. In this case, it is preferable that the server and the terminal can exchange information via a network such as the Internet or an intranet. The computer may include a processor and a memory coupled to the processor. The memory may store instructions that, when executed by the processor, cause the computer to perform various processes or function as various elements. The computer may be provided with various training data to construct a learning model and perform various calculations through machine learning. In this case, the computer may perform various analyses using the learning model created through machine learning and deep learning of AI (artificial intelligence).
[0020] The image input step (S101) is a step in which a photographed image, which is an image of a portion of a patient's large intestine, is input into a system for acquiring a virtual bowel cleansing image. An example of the photographed image is an image included in a three-dimensional photographed image of the patient's large intestine. The photographed image may also be a medical image of the patient's abdomen. Using this system, an image of a portion of a patient's large intestine from which residual portions have been removed can be obtained. Therefore, the patient does not need to take a laxative to remove residual portions (feces) from the large intestine. Therefore, an example of the photographed image is a medical image of the patient in their normal state (when no laxative has been taken or administered). In this way, the photographed image used in the present invention may be a medical image from which residual portions have not been physically removed. This can significantly reduce the burden on the patient. A three-dimensional photographed image of the large intestine is formed by the existence of multiple cross-sectional images (photographs) of the large intestine. An example of such a three-dimensional photographed image is a CT scan image. An example of the photographed image is one or more of such cross-sectional images of the large intestine.
[0021] For example, system 1 is connected to a CT scanning system. When a CT scan of a patient's colon is taken, the CT scan image of the patient's colon is stored in the CT scanning system. The CT scan image stored in the CT scanning system is output from an output unit of the CT scanning system to system 1. The image input unit 3 of system 1 receives the CT scan image. In this manner, a captured image (an image of a portion of the patient's colon) is input into system 1. As imaging by the CT scanning system progresses, multiple captured images may be input into system 1. The captured images input into system 1 may be stored in a memory unit of system 1 as appropriate. In addition to the captured images, information about the patient may also be input into system 1. Examples of patient information include one or more of the patient's identification number (ID), the patient's name, the patient's gender, the patient's age, past medical conditions, information about the patient's drinking, and information about colorectal cancer in the patient's relatives.
[0022] FIG. 3 is a flowchart showing an example of a virtual intestinal cleansing process. The virtual intestinal cleansing process (S102) is a process in which the system 1 obtains a virtual intestinal cleansing image. The virtual intestinal cleansing image is an image of a portion of a patient's large intestine from which the residue portion in the large intestine contained in the photographed image has been removed. An example of the virtual intestinal cleansing process is a process in which a learning model is used to directly convert a photographed image into a virtual intestinal cleansing image. An example of the learning model is a trained model obtained by using, as training data, a residue-containing image, which is an image of a portion of the large intestine containing residue, and a residue-removed image, which is an image of a portion of the large intestine after the residue has been removed, and obtaining image processing parameters for obtaining the residue-removed image from the residue-containing image. This model can be continuously updated by adding training data. As shown in the examples, there is a correlation between the residue-containing image and the residue-removed image, so a learning model can be constructed by using the residue-containing image and the residue-removed image as training data. For example, the virtual intestinal cleansing unit 5 reads the captured image from the memory unit, and based on program instructions, the calculation unit performs image processing using a learning model to remove residue portions contained in the captured image, thereby obtaining a virtual intestinal cleansing image. The virtual intestinal cleansing step (S102) preferably directly converts a CT scan image volume of the large intestine into a corresponding virtual intestinal cleansing image volume using the learning model. The obtained virtual intestinal cleansing image may be stored in the memory unit as appropriate. As will be shown in the examples described below, this learning model can be used to obtain a virtual intestinal cleansing image. By enhancing machine learning, it is possible to obtain a virtual intestinal cleansing image by acquiring a captured image without administering a laxative or contrast agent to the patient. In this case, too, a trained model is constructed using CT scan images (captured images) of a patient without administering a laxative or contrast agent and captured images with administration of a laxative and contrast agent as training data, and the constructed trained model is used to obtain a virtual intestinal cleansing image from a CT scan image (captured image) of a patient without administering a laxative or contrast agent.
[0023] 3D Endoscope-Like Image Generation Step (S103) The 3D endoscope-like image generation step is a step for generating a 3D model image representing a patient's large intestine using multiple virtual intestinal cleansing images. In the virtual intestinal cleansing step (S102), image processing is performed on each captured image to generate a virtual intestinal cleansing image from which residue has been virtually removed. The 3D endoscope-like image generation step may be performed by the 3D model creation unit 7 of the system 1. The 3D model creation unit 7 can generate a model of a patient's large intestine using virtual intestinal cleansing images of a desired portion of the large intestine. Creating a 3D model of a large intestine using multiple CT scan images of the large intestine is well known. Therefore, the 3D model creation unit 7 can be implemented using a known program. The model of a patient's large intestine obtained in this manner is a model obtained by an endoscopic examination after a laxative treatment and physical removal of residue. Using this model, images of the patient's large intestine from various angles can be obtained, allowing the patient's large intestine to be visualized in 3D. For this reason, an image of a patient's colon created based on the 3D model is called a "3D endoscopic-like image." The memory unit of system 1 stores the obtained model of the patient's colon in the memory unit as needed. System 1 then reads the model of the colon from the memory unit and outputs the required cross-sectional view to an output unit such as a monitor.
[0024] Learning Model Construction Process (S001) The learning model construction process is a process of constructing and developing (training) a machine learning learning model used in virtual intestinal cleansing. This process is performed, for example, by the machine learning unit 9 of the virtual intestinal cleansing unit 5. The machine learning unit 9 may train the learning model using, for example, CT scan images of multiple patients undergoing intestinal cleansing (e.g., CT scan images after administration of a laxative) and CT scan images of those multiple patients under normal conditions as training data. By using such a learning model, the learning model can be applied to the patient's CT scan image under normal conditions to obtain a virtual intestinal cleansing image, which is a virtual image in which residue contained in the image has been removed, through image processing. The constructed and trained learning model (trained model) may be stored, for example, in the learning model storage unit 11 and may be updated as appropriate.
[0025] The learning model may be trained by administering an oral contrast agent to a patient, imaging the residue in the colon, acquiring CT images of the colon, and creating a residue-removed image using the learning model. The learning model may also be constructed and trained using a diffusion model, as described below.
[0026] FIG. 5 is a process diagram illustrating an example of a method for detecting colon polyps. As shown in FIG. 5, this example method includes an image input step (S101), a colon extraction step (S202), a polyp extraction step (S203), and a false-positive removal step (S204). S indicates a step. This method is implemented by a computer or processor. This method can also be described as a computer-aided diagnosis of a laxative-free colon CT scan using a colon polyp detection method based on image processing using a learning model. This method can be described as a method for detecting colon polyps by having a computer execute a colon extraction step, which is a step of obtaining a colon image, which is a partial image obtained by extracting only the colon portion contained in a captured image, and a polyp extraction step, which is a step of obtaining a polyp image, which is an image obtained by extracting a polyp portion contained in the colon image. This method may further include a false-positive removal step, which is a step of removing false-positives contained in the polyp image. The processor may include a memory unit, the memory unit may store instructions, and the processor may be configured to cause a computer to perform each of the above-described steps in response to the instructions.
[0027] FIG. 6 is a block diagram showing an example of a system for detecting colon polyps. As shown in FIG. 6, the system for detecting colon polyps includes an image input unit 21, a large intestine extraction unit 22, a polyp extraction unit 23, and a false positive removal unit 24. The large intestine extraction unit 22 includes a large intestine extraction machine learning unit 25 and a large intestine extraction learning model storage unit 26. The polyp extraction unit 23 includes a polyp extraction machine learning unit 27 and a polyp extraction learning model storage unit 28. The false positive removal unit 24 includes a false positive removal machine learning unit 29 and a false positive removal learning model storage unit 30. This system is implemented by a computer or a processor. The system includes a processor and a memory storing a program. The program may be configured to cause the processor to execute each step of the method for detecting colon polyps.
[0028] Image Input Step (S101) The image input step is the same as that described above.
[0029] FIG. 7 is a flowchart showing an example of a large intestine extraction process. The large intestine extraction process (S202) is a process by which the system obtains a large intestine image. The large intestine image is an image obtained by extracting only the large intestine portion included in a photographed image from the photographed image. Note that the large intestine image may also be an image obtained by extracting only the large intestine portion included in a virtual bowel cleansing image from the virtual bowel cleansing image. An example of the large intestine extraction process is a process of directly converting a photographed image (or a virtual bowel cleansing image) into a large intestine image using a learning model. An example of the learning model is a model obtained using, as training data, a pair of a photographed image (or a virtual bowel cleansing image) including the patient's large intestine and an image in which portions other than the large intestine included in the photographed image (or virtual bowel cleansing image) have been removed. This model may be a model obtained by obtaining image processing parameters for obtaining a large intestine image from a photographed image (or a virtual bowel cleansing image). The photographed images used in constructing this learning model are photographed images prepared for constructing the learning model. The photographed images may also be virtual bowel cleansing images. For example, the large intestine extraction unit 22 reads out the photographed image or the virtual intestinal cleansing image from the storage unit, and the large intestine extraction machine learning unit 25 performs image processing using a large intestine extraction learning model based on instructions from the program to extract only the large intestine region included in the photographed image or the virtual intestinal cleansing image, thereby obtaining a large intestine image. The large intestine extraction step (S202) preferably directly converts the CT scan image volume of the large intestine into a corresponding large intestine image volume using the learning model. The obtained large intestine image may be stored in the storage unit as appropriate.
[0030] Large Intestine Extraction Learning Model Construction Process (S302) The large intestine extraction learning model construction process is a process of constructing and developing (training) a machine learning learning model to be used in the large intestine extraction process. This process is performed, for example, by the machine learning unit 25 of the large intestine extraction unit 22. The machine learning unit 25 may train the learning model using, for example, CT scan images of multiple patients and large intestine correct region images, which are images obtained by removing parts other than the large intestine from those images and leaving only the large intestine region, as training data. By using such a learning model optimized for large intestine extraction, the learning model can be applied to CT scan images including the patient's large intestine, and large intestine images from the captured images can be obtained by image processing, with the large intestine extracted from the captured images. The constructed and trained learning model (trained model) can be stored, for example, in the learning model storage unit 26 and updated as appropriate. The large intestine extraction learning model may also be constructed using a diffusion model. Note that if polyps are included in the large intestine region, the polyps may also be included in the large intestine image.
[0031] Furthermore, the teacher data of the captured images used for training the learning model for large intestine extraction may be data to which abdominal images generated using the diffusion model described in the following examples have been added.
[0032] FIG. 8 is a flowchart showing an example of a polyp extraction process. The polyp extraction process (S203) is a process in which the system obtains a polyp image. The polyp image is an image in which only the polyp region contained in the large intestine image is extracted from the large intestine image. An example of the polyp extraction process is a process in which a large intestine image is directly converted into a polyp image using a learning model. An example of the learning model is a model obtained by using, as training data, a large intestine image, which is an image in which the large intestine of a patient is extracted, and a polyp correct region image, which is an image in which only the polyp region contained in the large intestine image is extracted. The polyp correct region image may be created manually or automatically. This model may be a model obtained by obtaining image processing parameters for obtaining a polyp image from a large intestine image. For example, the polyp extraction unit 23 reads out a colon image or a virtual intestinal cleansing image from the storage unit, and the polyp extraction machine learning unit 27 performs image processing using a polyp extraction learning model based on program instructions to extract only polyp regions included in the colon image or virtual intestinal cleansing image, thereby obtaining a polyp image. The polyp extraction step (S203) preferably directly converts a colon image (or colon image volume) into a corresponding polyp image (or polyp image volume) using the polyp extraction learning model. The obtained polyp image may be stored in the storage unit as appropriate. The polyp extraction step may also be a cancer (colon cancer) extraction step.
[0033] Polyp Extraction Learning Model Construction Step (S303) The polyp extraction learning model construction step is a step of constructing and developing (training) a machine learning learning model to be used in the polyp extraction step. This step is performed, for example, by the polyp extraction machine learning unit 27 of the polyp extraction unit 23. The machine learning unit 27 may train the polyp extraction learning model using, for example, colon images of multiple patients and polyp correct region images, which are images obtained by extracting only polyp regions from the colon images, as training data. The polyp correct region images may be created manually or automatically. By using such a learning model optimized for polyp extraction, the learning model can be applied to colon images to obtain polyp images in which polyps contained in the colon images are extracted by image processing. The constructed and trained learning model (trained model) may be stored, for example, in the polyp extraction learning model storage unit 28 and may be updated as appropriate. The polyp extraction learning model may be constructed using the diffusion model described in the following examples. Furthermore, the colon image training data used to train the polyp extraction learning model may be data to which colon images generated using the diffusion model described in the following examples have been added.
[0034] FIG. 9 is a flowchart showing an example of the false positive removal process. The false positive removal process (S204) is a process in which the system removes false positives to obtain an image containing only true polyps. A false positive may be an image portion that is not a polyp but is included in the polyp image, or a portion of the polyp image that is not cancer but is determined to be cancer. An example of the false positive removal process is a process in which an image containing false positives is removed from the polyp image using a false positive removal learning model. An example of the learning model is a model obtained by using a pair of polyp images and labels indicating whether the image contains a true polyp as training data, and obtaining image processing parameters for determining whether the polyp image contains a true polyp. For example, the false positive removal unit 24 reads a polyp image from the storage unit, and based on instructions from the program, the false positive removal machine learning unit 29 performs image processing using a false positive removal learning model to determine whether the polyp image contains a true polyp. If the polyp is not a true polyp, the polyp (false positive polyp portion, which is a polyp image region that is not a true polyp) is removed. The false positive removal learning model is stored in the false positive removal learning model storage unit 30, so it may be read and used as appropriate. Polyp portions that were not removed (true polyp portions) may be stored in the storage unit as appropriate. The false positive removal process may also be a process of removing portions included in a polyp image that are determined to be cancerous even though they are not cancerous.
[0035] False-Positive Removal Learning Model Construction Process (S304) The false-positive removal learning model construction process is a process of constructing and developing (training) a machine learning learning model used in the false-positive removal process. This process is performed, for example, by the false-positive removal machine learning unit 29 of the false-positive removal unit 24. The false-positive removal machine learning unit 29 may train the learning model using, for example, polyp images of multiple patients and identification information (labels) indicating whether the polyp images are true polyps as training data. By using such a learning model optimized for false-positive removal, it is possible to apply the learning model to polyp images and determine whether the polyp images contain true polyps. The constructed and trained learning model (trained model) may be stored, for example, in the learning model storage unit 30 and updated as appropriate. The false-positive removal learning model may be constructed using a diffusion model described in the following examples. The polyp image training data used to train the false-positive removal learning model may be data to which polyp images generated using the diffusion model described in the following examples are added.
[0036] The false positive removal model may use the size of a polyp region in a polyp image as a threshold. For example, the size of a polyp region (maximum 3D diameter of a polyp) may be used as a threshold. Examples of the threshold may be 6 mm or more and 10 mm or less, or 6.5 mm or more and 9.5 mm or less.
[0037] This specification also describes a program for causing a computer to execute the method for acquiring a virtual intestinal cleansing image described above, and a computer-readable non-transitory information recording medium storing such a program. Examples of non-transitory information recording media include CD-ROMs, DVDs, and USB memories.
[0038] Example 1: Development of a Learning Model (DDP Model) Below, we developed a basic denoising diffusion probabilistic model (DDP model) for generating synthetic high-quality image volumes. Training the DDP model involves minimizing the divergence between the target distribution and the distribution of the generated image, and maximizing the plausibility of the observed training data. The diffusion model is a latent variable model shown in the following equation: x １ , …, x Ｔ is data x ０ ~q(x ０ ) is a latent variable with the same dimension as the joint distribution is called the inverse process and is defined as a Markov chain with trained Gaussian transitions, which are given by The starting point is shown below.
[0039] Diffusion models differ from other latent variable models in their approximate posterior distribution, i.e., the forward or diffusion process. The approximate posterior distribution is expressed as follows: This approximate posterior distribution is anchored to a Markov chain that progressively adds Gaussian noise to the data. The forward process gives x at any time step t. ｔ can be sampled in a closed form. Next, we write Then, the approximate posterior distribution is expressed by the following formula: Training is performed by optimizing the usual variational lower bound against the negative log-likelihood. To implement the DDP model, the variances of the forward process are set as constants. These variances are expressed as β １ =10 －４ From β Ｔ = 0.02, where α ｔ = 1 - β ｔTo represent the inverse process, we used a globally group-normalized nD UNet as the backbone. Parameters were shared across time and specified in the network using Transformer sinusoidal location embedding.
[0040] Development of Learning Model (Diffusion-Based nD UNet Model) The nD UNet diffusion model can be trained for image translation problems by pairing input image volumes (e.g., MRI or CT scans) with their corresponding ground-truth images. By introducing noise into the input image volumes during training, the model learns to explore different variations and construct a diffusion embedding representation that captures the underlying image characteristics. This representation is then transformed into the corresponding ground-truth image. In this example, we used the Diff-UNet model, described below.
[0041] In the three-dimensional (3D) case, let V be the volume of the input image, and let V be expressed as: First, we one-hot encode the corresponding single-channel multi-label input map of size D × W × H into a multi-channel label map x ０ Convert into label map x ０ is expressed as follows: where N denotes the number of labels. Next, we consider the noisy label map x ｔ To obtain , we apply forward diffusion to add noise ε in t successive steps. To recover the original input map, we decompose V and x to obtain the multi-scale feature volume of the denoised nD UNet. ｔ is connected to the encoder in the channel direction. The multi-scale feature volume is given by the following equation: To introduce raw volumetric image features, we extracted multi-scale feature volumes from the intensity data via a feature encoder of the same size. The multi-scale feature volumes are given by: The multi-scale feature volume and the multi-scale feature volume can be summed to obtain a fused multi-scale feature that is fed to the decoder, thereby obtaining a prediction result, which is given by the following equation: The Diff-UNet model can be trained using a loss function, which is given by the following equation: Here, L Ｄｉｃｅ is the Dice loss, L ＢＣＥ is the binary cross entropy loss, L ＭＳＥ is the mean squared error loss.
[0042] In the test phase, the diffusion step was repeated t times using the denoising diffusion implicit (DDI) model. To improve the robustness of the segmentation, the final output is determined by the number of prediction steps and the associated uncertainty. This uncertainty is estimated similarly to the Monte Carlo (MC) dropout method. In this case, MC dropout activates a dropout layer and performs S forward passes to estimate the uncertainty map. In the test phase, random noise x ｔ By initializing , Diff-UNet already introduces randomness without activating dropout. The diffusion is done in t steps, and at each step S outputs are predicted and used to calculate the uncertainty. The uncertainty is calculated by the following formula: The fusion weight for combining the number of prediction steps and the uncertainties is calculated by the following formula: where σ is the sigmoid function, i is the current prediction step, and u is the uncertainty matrix. The prediction result of each step can be multiplied by a weight w to obtain the final fusion result. The final fusion result, which is the output, is given by the following equation:
[0043] Virtual Bowel Cleansing Figure 4 shows images obtained when virtual bowel cleansing was actually performed using the system. Figure 4(a) is a photograph in place of a drawing of a clinical CT scan image obtained using a contrast agent. In Figure 4(a), a polyp is submerged in a liquid filled with contrast agent. Figure 4(b) is an enlarged view of the polyp site. Figure 4(c) shows images obtained when virtual bowel cleansing was actually performed using the system. Figure 4(d) is a cross-sectional image of the large intestine obtained by creating a 3D model based on a CT scan image in which a polyp is submerged in a liquid filled with contrast agent. In this example, the polyp was not visible in the virtual endoscopic image because residual liquid obstructed the view. Figure 4(e) is a cross-sectional image of the large intestine obtained by creating a 3D model based on the virtual bowel cleansing image. In this example, the polyp can be seen in the area surrounded by the dotted line.
[0044] Quantitative evaluation of virtual bowel cleansing. To quantitatively evaluate the image quality of the virtual bowel cleansing images, a custom-made anthropomorphic colon phantom (Anthropomoorphic Colon Phantom, Phantom Laboratory, Salem, NY, USA) was injected with a simulated contrast residue consisting of 300 ml of saline and 10 g of cereal, mixed with 10 mg of 40 mg / ml non-ionic iodinated contrast agent (Omnipaque, GE Healthcare, Chiago, IL). The images were then scanned using a multi-row CT scanner (SOMATOM Definition Flash, Siemens Healthcare, Erlangen, Germany). A CT scan image volume of the colon phantom was obtained using a CT scanner (Germany) at a tube voltage of 120 kVp, a tube current of 236 mA, and a slice thickness of 0.6 mm. From this CT scan image volume, 100 volumetric regions of interest (VOIs) of 128 × 128 × 128 voxels each containing the intestine were extracted and directly converted into virtual bowel cleansing image volumes using the three learning models described below.
[0045] The first training model is the 3D UNet, a 3D extension of the conventional U-Net. The 3D UNet is a convolutional neural network with two symmetric paths: an encoder path consisting of convolutional network layers to capture the local context of the input image volume, and a decoder path consisting of transposed convolutional network layers to enable accurate localization. In 3D UNet, the corresponding encoding and decoding layers are connected through skip connections to preserve spatial information that may be lost due to the downsampling step in the encoder path. The second training model is the UNET Transformer (UNETR). Recently, self-attention-based transformers have emerged as an alternative to conventional convolutional neural networks. UNETR incorporates a transformer into the encoder path of the 3D U-Net to capture long-range dependencies within the input image volume. This enables learning of associations between all local patches within the input image volume, resulting in superior image processing performance compared to conventional convolutional networks. The third learning model is Diff-UNet, a variant of the previously described diffusion-based nD UNet generative AI model. Diff-UNet first introduces noise into the input image volume, and then generates a virtual bowel cleansing image volume through an iterative denoising process using a denoising diffusion probability volume rate model.
[0046] The difference between the virtual bowel cleansing image volume obtained by applying these three learning models to the above 100 VOIs and the CT scan image volume of the colon phantom before contrast residue injection was measured using the peak signal-to-noise ratio. The mean ± standard deviation values of the peak signal-to-noise ratio for 3D UNet, UNETR, and Diff-UNet were 28.70 ± 3.00, 29.88 ± 2.53, and 30.60 ± 1.81, respectively. These results demonstrate that any learning model can convert a CT scan image volume into a high-quality virtual bowel cleansing image volume. Furthermore, the use of the generative AI model, Diff-UNet, demonstrates that a virtual bowel cleansing image volume with superior image quality can be generated compared to the use of learning models other than generative AI.
[0047] [Example 2] Colon Extraction Figure 10 shows images obtained when colon extraction was actually performed using a system for detecting colon polyps. Figure 10(a) shows a CT scan image volume obtained clinically from a CT image of a patient's colon. Figure 10(a) displays the entire abdominal image volume including the colon. Figure 10(b) shows a 3D representation of the colon image obtained when colon extraction was actually performed using the system.
[0048] Example 3: Polyp Extraction In this example, we compared the performance of three types of polyp extraction learning models used to convert colon images into polyp images in the polyp extraction process. The polyp extraction learning models used were the previously described 3D UNet, UNETR, and the generative AI model Diff-UNet.
[0049] In this study, we used 150 colon CT scans from multiple clinical trials of colon cancer screening using colon CT to create training data for the polyp extraction model. These cases were orally administered iodine or barium contrast agents to visualize the residual material, and each case was scanned in one of two positions: supine, prone, or recumbent. Eleven CT scanners were used, with scan voltages of 120 kVp or 140 kVp, pixel spacing of 0.54-0.97 mm, and slice thicknesses of 0.75-5.00 mm. Experienced radiologists manually extracted polyp-specific regions from the colon CT scans using the colonoscopy and colon CT examination reports accompanying each case. To achieve this, we first interpolated the CT scan image into an image volume with an isotropic spatial resolution of 0.625 mm. Then, we extracted a 48 × 48 × 48 voxel volumetric region of interest (VOI) centered on a known polyp contained within the image volume. Polyp regions were then manually extracted using 3D Slicer (slicer.org). As a result, we obtained 946 pairs of VOIs and ground truth images, each containing a single polyp, as training data. Figure 11(a) shows an example of a VOI included in the training data (top panel) and the corresponding ground truth image (bottom panel) overlaid in white.
[0050] Each polyp extraction model was constructed using MONAI (monai.io) and PyTorch (pytorch.org). Training of the polyp extraction model was performed on a 32GB Tesla V100S GPU (NVIDIA, Santa Clara, CA). During the training process, the training data was first divided into 80% training data and 20% validation data. The training data was then augmented by adding random-intensity Gaussian noise, randomly flipping, 90-degree rotation, and scaling. The weights of the polyp extraction model were then initialized to random values. Training the model using the training data and measuring its polyp extraction performance using the validation data were repeated 1,000 times. Here, the polyp extraction performance was measured using the Dice score between the polyp correct region image paired with each VOI included in the validation data and the polyp image converted and output by the polyp extraction learning model using that VOI as input. For each polyp extraction learning model, the model that showed the highest Dice score after 1000 iterations was selected as the trained model to be used in the test phase described below.
[0051] The test data used in the test phase were a custom-made anthropomorphic colon phantom (Anthropom- ic colon phantom, Phantom Laboratory, Salem, NY, USA) containing simulated polyps. The phantom was scanned using a 16-row photon-counting CT scanner (OmniTom, NeuroLogica, MA, USA) with a tube voltage of 120 kVp, a tube current of 40 mA, a slice thickness of 0.71 mm, and an X-ray photon counting threshold of 50 keV. The resulting CT scan images were reconstructed using the filtered back projection method with a spatial resolution of 0.6 mm and a reconstruction interval of 0.6 mm. The resulting CT scan images were then interpolated into a CT scan image volume with an isotropic spatial resolution of 0.625 mm, and 17 VOIs (48 × 48 × 48 voxels) centered on the simulated polyps contained within. Polyp images were manually created using 3D Slicer, extracting the simulated polyp regions contained in these VOIs, and used as the polyp correct region images corresponding to the VOIs. Figure 11(b) shows an example of a VOI contained in the test data in the upper row, and the corresponding polyp correct region image superimposed in white in the lower row.
[0052] The polyp extraction performance of each polyp extraction trained model in the test phase was measured using the Dice score between the polyp image converted from each VOI included in the test data using the trained model and the ground truth polyp region image paired with that VOI. The mean ± standard deviation Dice score was 0.738 ± 0.14 for 3D UNet, 0.745 ± 0.14 for UNETR, and 0.751 ± 0.15 for Diff-UNet. Among these three polyp extraction trained models, Diff-UNet exhibited the highest mean Dice score, followed by UNETR. Furthermore, Diff-UNet achieved the highest Dice score (0.946) for all polyps included in the test data. Figure 12 shows examples of virtual endoscopic and cross-sectional images of polyp images obtained from each trained polyp extraction model. For all models, the polyp image of a pedunculated polyp, as shown in the figure, showed the best Dice score. Figure 12(a) shows an example of a polyp image obtained from 3D UNet. In this example, the Dice score was 0.88. Figure 12(b) shows an example of a polyp image output from UNETR. In this example, the Dice score was 0.88. Figure 12(c) shows an example of a polyp image output from Diff-UNet. In this example, the Dice score was 0.88. These results demonstrate that, regardless of which polyp extraction learning model is used, polyp images in which polyp regions are extracted from CT scan images of the large intestine can be obtained with high performance. In particular, the use of Diff-UNet, a generative AI model, demonstrates that polyp images can be generated with higher performance than when using polyp extraction learning models other than generative AI.
[0053] Example 4 Development of Learning Model (Improved Diffusion-Based nD UNet Model) A learning model for false positive removal was constructed by improving the previously described diffusion-based nD UNet model as follows. First, a diffusion-based nD UNet model was constructed and trained using the method described in Example 3. The encoder portion of the UNet in the resulting learning model was extracted, and a deep network was constructed by connecting a convolutional network for binary classification to its bottleneck layer. This deep network was trained using as training data pairs of VOIs included in the training data described in Example 3 and identification information (labels) indicating that the polyps included in the VOIs are true polyps, thereby constructing a learning model for false positive removal.
[0054] False positive removal Figure 13 shows an image obtained when false positive removal was actually performed using the above learning model. Figure 13(a) shows the polyp area obtained when polyp extraction was performed on the colon image shown in Figure 10(b), shown in white on a semi-transparent colon image. Figure 13(b) shows the polyp area remaining after false positive removal. Most of the polyps shown in Figure 13(a) have been removed, and only true polyps have been detected.
[0055] The present invention can be used in the field of medical devices.
[0056] REFERENCE SIGNS LIST 1 System 3 Image input unit 5 Virtual intestinal cleaning unit 7 Three-dimensional model creation unit 9 Machine learning unit 11 Learning model storage unit 21 Image input unit 22 Large intestine extraction unit 23 Polyp extraction unit 24 False positive removal unit 25 Large intestine extraction machine learning unit 26 Large intestine extraction learning model storage unit 27 Polyp extraction machine learning unit 28 Polyp extraction learning model storage unit 29 False positive removal machine learning unit 30 False positive removal learning model storage unit
Claims
1. A method for obtaining a virtual intestinal cleaning image, comprising: an image input step of inputting a photographed image of a part of a patient's large intestine; and a virtual intestinal cleaning step of obtaining a virtual intestinal cleaning image of a part of the patient's large intestine from the photographed image by removing residual parts of the large intestine contained in the photographed image.
2. The method of claim 1, wherein the captured image is an image contained in a three-dimensional captured image of the patient's colon.
3. A method according to claim 2, wherein the virtual intestinal cleaning step is a step of directly converting the captured image into the virtual intestinal cleaning image using a learning model.
4. A method according to claim 3, wherein the learning model is a model obtained by using a residue-containing image, which is an image of a portion of the large intestine containing residue, and a residue-removed image, which is an image of the portion of the large intestine after the residue has been removed, as training data, and obtaining image processing parameters for obtaining the residue-removed image from the residue-containing image.
5. A program for causing a computer to execute a method for obtaining a virtual intestinal cleaning image, the method including: an image input step of inputting a photographed image of a part of a patient's large intestine; and a virtual intestinal cleaning step of obtaining a virtual intestinal cleaning image of a part of the patient's large intestine from the photographed image by removing residual parts of the large intestine contained in the image.
6. A non-transitory information recording medium readable by a computer having stored thereon the program according to claim 5.
7. A method for detecting polyps from a captured image, comprising: a large intestine extraction step for obtaining a large intestine image, which is a partial image obtained by extracting only the large intestine portion contained in a captured image, the large intestine being captured, a polyp extraction step for obtaining a polyp image, which is an image obtained by extracting a polyp portion contained in the large intestine image, and a false positive removal step for removing false positives contained in the polyp image.
8. The method according to claim 7, wherein the large intestine extraction step is a step of directly converting the captured image into the large intestine image using a learning model for large intestine extraction.
9. A method according to claim 8, wherein the learning model for extracting the large intestine is a model obtained by using as training data an image of the correct region of the large intestine, which is an image obtained by removing organ parts other than the large intestine from a photographed image and extracting only the large intestine part.
10. The method according to claim 7, wherein the polyp extraction step is a step of directly converting the colon image into the polyp image using a learning model.
11. A method according to claim 10, wherein the learning model is a model obtained by using as training data a certain colon image and a polyp correct area image, which is an image obtained by extracting only the polyp area from the certain colon image.
12. A program for causing a computer to execute a method for detecting polyps from a photographed image, the program including: a large intestine extraction step for obtaining a large intestine image, which is a partial image obtained by extracting only the large intestine portion contained in a photographed image, the large intestine being captured; a polyp extraction step for obtaining a polyp image, which is an image obtained by extracting a polyp portion contained in the large intestine image; and a false positive removal step for removing false positives contained in the polyp image.
13. A non-transitory information recording medium readable by a computer having stored thereon the program according to claim 12.