IMAGE IDENTIFICATION PROGRAM, IMAGE IDENTIFICATION METHOD, IMAGE IDENTIFICATION DEVICE, AND INFORMATION PROCESSING SYSTEM

Abstract

To identify accurately, a state related to lesion from a medical image.SOLUTION: A processing part 1a generates three-dimensional volume data 2 on the basis of a plurality of tomographic images in which an internal part of a human body is photographed, then cuts a prescribed size of a three-dimensional partial region 3 from the volume data 2, then performs minimum value projection or maximum value projection of each voxel value of the partial region 3 to a plurality of orthogonal directions, for generating a plurality of projection images 4a-4c, then, on the basis of the plurality of projection images 4a-4c, identifies a state of the partial region 3 out of the plurality of states.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image recognition program, an image recognition method, an image recognition device, and an information processing system. [Background technology]

[0002] Medical images such as those obtained by CT (Computed Tomography) and MRI (Magnetic Resonance Imaging) are widely used to diagnose various diseases. Diagnostic imaging using medical images requires doctors to interpret a large number of images, placing a heavy burden on them. Therefore, there is a demand for technology that uses computers to assist doctors in some way in their diagnostic work.

[0003] The following diagnostic support technologies using medical images have been proposed: For example, a computing system has been proposed that uses multiple individual neural networks to process multiple two-dimensional images of biological tissue structures, and a segmentation method has been proposed that converts a three-dimensional volume containing the pericardium into three sets of two-dimensional images and then uses multiple independently trained neural networks, each processing one of the three sets. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Patent No. 9,968,257 [Patent Document 2] US Patent Application Publication No. 2020 / 0320751 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the identification process for identifying which of multiple lesion-related states an imaged region is in from a medical image, it can be difficult to distinguish between two specific states depending on the disease. For example, there are cases where a specific lesion region and a specific normal internal body part appear with similar brightness on the image, but the three-dimensional shapes of the former lesion region and the latter specific part are different. In this case, if the lesion region and the specific part appear with different shapes on the image, there is a high possibility that they can be identified. However, in some cases, the two may appear with similar shapes on the image, in which case it may be impossible to accurately identify them.

[0006] For example, in MR images of the liver taken in the hepatic phase using a linear MRI liver contrast agent (gadoxetate sodium), tumors and blood vessels appear with similar brightness. The three-dimensional shape of a tumor is spherical, while the three-dimensional shape of blood vessels is long and thin tubular. In this case, the tumor appears circular or elliptical on the MR image, but the blood vessels may also appear circular or elliptical on the MR image, which may lead to the blood vessel region being mistakenly identified as the tumor region.

[0007] In one aspect, the present invention aims to provide an image identification program, an image identification method, an image identification device, and an information processing system that are capable of identifying a state related to a lesion from a medical image with high accuracy. [Means for solving the problem]

[0008] In one proposal, an image identification program is provided that causes a computer to execute a process of extracting a three-dimensional subregion of a predetermined size from three-dimensional volume data generated based on a plurality of tomographic images of the inside of a human body, generating a plurality of projection images by minimum value projection or maximum value projection of the values ​​of each voxel of the subregion in a plurality of mutually orthogonal directions, and identifying which of a plurality of states the subregion is in based on the plurality of projection images.

[0009] Also, one proposal provides an image classification method in which a computer executes processing similar to the processing based on the image classification program described above. Furthermore, one proposal provides an image recognition device that executes processing similar to the processing based on the image recognition program described above.

[0010] Also, one proposal provides an information processing system having a learning processing unit and an image classification unit. In this information processing system, the learning processing unit cuts out a plurality of learning partial regions, each of which is a three-dimensional region of a predetermined size, from three-dimensional learning volume data generated based on a plurality of learning tomographic images of the inside of the human body, extracts a plurality of first partial regions corresponding to a first state and a plurality of second partial regions corresponding to a second state from the plurality of learning partial regions, generates a first projection image group for each of the plurality of first partial regions, including a plurality of first projection images generated by minimum value projection or maximum value projection of the value of each voxel of the corresponding first partial region in a plurality of mutually perpendicular directions, and generates a second projection image group for each of the plurality of second partial regions, including a plurality of second projection images generated by minimum value projection or maximum value projection of the value of each voxel of the corresponding second partial region in a plurality of directions, and performs machine learning using the first projection image group corresponding to each of the plurality of first partial regions as learning data corresponding to the first state and the second projection image group corresponding to each of the plurality of second partial regions as learning data corresponding to the second state, thereby generating a learning model for identifying whether the state is the first state or the second state. The image identification unit extracts an identification partial area, which is a three-dimensional area of ​​a predetermined size, from three-dimensional identification volume data generated based on multiple identification tomographic images taken of the inside of the human body, generates multiple third projection images by minimum value projection or maximum value projection of the values ​​of each voxel in the identification partial area in multiple directions, and uses a learning model to identify whether the identification partial area is in a first state or a second state based on the multiple third projection images. [Effects of the Invention]

[0011] In one aspect, it is possible to identify lesion conditions from medical images with high accuracy. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a diagram illustrating an example of a configuration and a processing example of an image identification device according to a first embodiment; [Figure 2] FIG. 10 is a diagram illustrating an example of the configuration of a diagnosis support processing system according to a second embodiment. [Figure 3] FIG. 2 is a diagram illustrating an example of a hardware configuration of an image identification device. [Figure 4] FIG. 10 is a first diagram showing a comparative example of lesion identification processing. [Figure 5] FIG. 2 is a second diagram showing a comparative example of lesion identification processing. [Figure 6] FIG. 10 is a third diagram showing a comparative example of lesion identification processing. [Figure 7] FIG. 10 is a diagram for explaining a process for generating input data for identification in the second embodiment. [Figure 8] FIG. 2 is a diagram illustrating an example of the configuration of processing functions included in a learning processing device and an image recognition device. [Figure 9] FIG. 2 is a diagram illustrating an example of the internal configuration of a learning data generation unit. [Figure 10] 10A and 10B are diagrams for explaining the process of generating annotations and distance images; [Figure 11] FIG. 10 is a diagram illustrating a first example of a distance image generating method. [Figure 12] FIG. 10 is a diagram illustrating a second example of a distance image generating method. [Figure 13] FIG. 10 is a diagram illustrating a procedure for determining a normal 3D patch. [Figure 14] FIG. 10 is a diagram showing a procedure for determining a tumor 3D patch. [Figure 15] FIG. 10 is a diagram for explaining input of learning data to a classifier learning unit. [Figure 16] 10 is a flowchart illustrating an example of the overall processing procedure of a learning process. [Figure 17]10 is a first example of a flowchart illustrating a procedure for generating a combined 2D patch. [Figure 18] 10 is a second example of a flowchart illustrating the procedure of a combined 2D patch generation process. [Figure 19] 10 is a third example of a flowchart illustrating the procedure of a combined 2D patch generation process. [Figure 20] FIG. 2 illustrates an example of the internal configuration of an input data generating unit. [Figure 21] 10 is an example of a flowchart illustrating a procedure of a classification process performed by an image classification device. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. [First embodiment] FIG. 1 is a diagram illustrating an example of the configuration and processing of an image identification device according to a first embodiment. The image identification device 1 illustrated in FIG. 1 is an information processing device that acquires a plurality of tomographic images of the inside of a human body and, based on these tomographic images, identifies which of a plurality of states related to a lesion the imaged region is in. For example, when a tomographic image of a region including the liver is captured, the state is identified as either a tumor region or a normal region. The tomographic image is, for example, a medical image such as a CT image or an MR image.

[0014] The image recognition device 1 includes a processing unit 1a. The processing unit 1a is, for example, a processor. The processing unit 1a executes the following processes. Based on the above-mentioned multiple tomographic images, the processing unit 1a generates three-dimensional volume data 2. The processing unit 1a cuts out a three-dimensional partial region 3 of a predetermined size from the generated volume data 2. This partial region 3 becomes a unit for lesion identification.

[0015] The processing unit 1a generates multiple projection images by performing minimum value projection or maximum value projection on the values ​​(e.g., brightness values) of each voxel in the partial region 3 in multiple directions that are orthogonal to each other. In the example of FIG. 1, projection images 4a to 4c are generated by performing minimum value projection in three directions. The processing unit 1a identifies which of the multiple states the partial region 3 is in based on the generated projection images 4a to 4c. This allows the state of the lesion to be identified with high accuracy.

[0016] As an example, assume that a first state and a second state are identified as the state of a lesion. In this case, whether to use minimum intensity projection or maximum intensity projection can be determined based on, for example, a condition as to which of the first state and the second state appears relatively brighter on the volume data 2, and a condition as to what shapes each of the regions of the first state and the second state can exist on the volume data 2.

[0017] In the following description, it is assumed that the first state appears darker, and that the projection images 4a to 4c are generated by minimum intensity projection. It is also assumed that the region in the first state is normally spherical on the volume data 2.

[0018] On the other hand, even within the region of the second state, there may be a region that appears with the same brightness as the first state. For example, a specific region 5 identified as the second state may appear on the volume data 2 with the same brightness as the first state. In the example of FIG. 1, such a specific region 5 appears as a long, thin tube on the volume data 2. It should be noted that such a specific region 5 may be, for example, a blood vessel.

[0019] For example, consider a case where the first state and the second state are identified based on the tomographic image from which the volume data 2 was generated or a tomographic image of an arbitrary direction generated from the volume data 2. Since the region in the first state is spherical, this region appears circular or elliptical in the tomographic image. On the other hand, a specific region 5 having a long, thin tube shape may appear long and thin in the tomographic image, or circular or elliptical. In other words, the specific region 5 may appear in the tomographic image with a similar brightness and shape to the region in the first state, and therefore the above identification method may erroneously identify the region of the specific region 5 as the first region.

[0020] To address this problem, in this embodiment, the occurrence of misidentification is suppressed by using projection images 4a to 4c that are minimum-value projected in multiple directions that are orthogonal to each other. That is, the region in the first state appears as a circle or an ellipse in all of projection images 4a to 4c. On the other hand, specific portion 5 appears as an elongated shape in at least one of projection images 4a to 4c. In the example of FIG. 1, specific portion 5 appears as an ellipse in projection image 4a, but as an elongated shape in projection images 4b and 4c. Therefore, by performing the identification process based on projection images 4a to 4c, the possibility of misidentifying the region of specific portion 5 as the first region can be suppressed, and identification accuracy can be improved.

[0021] In the example of FIG. 1, projection images 4a to 4c are generated in three directions based on the partial region 3 and are used in the identification process. However, for example, projection images in two mutually orthogonal directions may be generated, and each projection image may be used in the identification process. Even in this case, for example, the specific portion 5 is likely to appear as an elongated shape in at least one of the projection images. Therefore, the identification accuracy can be improved in this case as well.

[0022] Second Embodiment Next, a system that can distinguish between tumor regions and normal regions in the liver as a condition related to a lesion will be described.

[0023] Fig. 2 is a diagram showing an example of the configuration of a diagnosis support processing system according to the second embodiment. The diagnosis support system shown in Fig. 2 is a system that supports image diagnosis using MRI imaging, and includes MRI devices 11 and 21, a learning processing device 12, and an image recognition device 22. The image recognition device 22 is an example of the image recognition device 1 shown in Fig. 1.

[0024] The MRI devices 11 and 21 capture MR images of the human body. In this embodiment, the MRI devices 11 and 21 capture a predetermined number of axial tomographic images of the abdominal region including the liver while changing the position in the height direction (direction perpendicular to the axial plane) of the human body at predetermined intervals. In this embodiment, the MRI devices 11 and 21 perform imaging using a linear MRI liver contrast agent (gadoxetate sodium).

[0025] The image identification device 22 generates three-dimensional volume data from each tomographic image captured by the MRI device 21 and extracts 3D (dimensional) patches, which are three-dimensional regions of a certain size, from the volume data. The image identification device 22 performs lesion identification processing for each 3D patch using a classifier based on the learning model 100 generated by the learning processing device 12. In this lesion identification processing, the 3D patch is identified as a tumor region or a normal region. In this lesion identification processing, the image identification device 22 also generates a 2D image (a "combined 2D patch" described below) based on the 3D patch as input data for identification, and executes the lesion identification processing by inputting the 2D image to the classifier.

[0026] The learning processing device 12 generates, by machine learning, weights of the learning model 100 used in the lesion classification process by the image classification device 22. For this model generation process, the learning processing device 12 generates learning data from each tomographic image captured by the MRI device 11 and performs machine learning using the generated learning data. The weights of the learning model 100 generated by the learning processing device 12 are read into the image classification device 22, for example, via a network or via a portable recording medium.

[0027] Note that captured images may be input to the learning processing device 12 and the image recognition device 22 from the same MRI device 11. Furthermore, the learning processing device 12 may acquire captured images from the MRI device via a recording medium, rather than directly. Furthermore, the learning processing device 12 and the image recognition device 22 may be the same information processing device.

[0028] Fig. 3 is a diagram showing an example of the hardware configuration of an image recognition device. The image recognition device 22 is realized, for example, as a computer having the hardware configuration shown in Fig. 3. As shown in Fig. 3, the image recognition device 22 includes a processor 201, a RAM (Random Access Memory) 202, an HDD (Hard Disk Drive) 203, a GPU (Graphics Processing Unit) 204, an input interface (I / F) 205, a reader 206, and a communication interface (I / F) 207.

[0029] The processor 201 comprehensively controls the entire image recognition device 22. The processor 201 is, for example, a central processing unit (CPU), a micro processing unit (MPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or a programmable logic device (PLD). The processor 201 may also be a combination of two or more elements of a CPU, an MPU, a DSP, an ASIC, or a PLD. The processor 201 is an example of the processing unit 1a shown in FIG. 1.

[0030] The RAM 202 is used as a main storage device of the image recognition device 22. The RAM 202 temporarily stores at least a part of an OS (Operating System) program and application programs to be executed by the processor 201. The RAM 202 also stores various data necessary for processing by the processor 201.

[0031] The HDD 203 is used as an auxiliary storage device for the image identification device 22. The OS program, application programs, and various data are stored in the HDD 203. Note that other types of non-volatile storage devices such as SSDs (Solid State Drives) can also be used as the auxiliary storage device.

[0032] A display device 204a is connected to the GPU 204. The GPU 204 displays an image on the display device 204a in accordance with an instruction from the processor 201. The display device 204a may be a liquid crystal display or an organic EL (Electroluminescence) display.

[0033] An input device 205a is connected to the input interface 205. The input interface 205 transmits a signal output from the input device 205a to the processor 201. The input device 205a includes a keyboard and a pointing device. The pointing device includes a mouse, a touch panel, a tablet, a touch pad, a trackball, and the like.

[0034] A portable recording medium 206a is detachably attached to the reading device 206. The reading device 206 reads data recorded on the portable recording medium 206a and transmits the data to the processor 201. The portable recording medium 206a may be an optical disk, a semiconductor memory, or the like.

[0035] The communication interface 207 transmits and receives data to and from other devices such as the MRI device 21 via a network. The above hardware configuration can realize the processing functions of the image recognition device 22. The learning processing device 12 can also be realized as a computer with a hardware configuration as shown in FIG.

[0036] Next, a comparative example of lesion identification processing will be described with reference to FIGS. FIG. 4 is a first diagram showing a comparative example of lesion identification processing. One method of lesion identification uses a classifier generated by machine learning. For example, as shown in FIG. 4, one method involves extracting "patches," which are image regions of a certain size, from each tomographic image and performing lesion identification processing on a patch-by-patch basis. In the following description, patches extracted from tomographic images will be referred to as "2D patches" to distinguish them from three-dimensional patches (3D patches) described below.

[0037] In the example of FIG. 4, a set of tomographic images is acquired from an MRI apparatus (step S11), and a 2D patch is generated from each tomographic image included in the set of tomographic images (step S12). The 2D patch is generated by dividing the tomographic image into portions of a fixed size, such as 16 pixels by 16 pixels. Alternatively, the 2D patch may be generated by cutting out the tomographic image while shifting a window of the same size as the 2D patch on the tomographic image. For example, a 16 pixel by 16 pixel window may be shifted horizontally by 8 pixels to cut out a 16 pixel by 16 pixel 2D patch.

[0038] The 2D patches generated in this way are input to a classifier based on a learning model generated by prior machine learning. This identifies lesions in the 2D patches (step S13). In the example of Figure 4, the classifier identifies whether the patch is Lesion A, Lesion B, or normal (non-lesion).

[0039] The training data for generating the training model is also generated as 2D patches extracted from tomographic images using the same procedure as above. That is, a label indicating whether it is Lesion A, Lesion B, or normal is added to the training 2D patches, and machine learning of the classifier is performed using the labeled training 2D patches.

[0040] In MR images using a linear MRI liver contrast agent (gadoxetate sodium), tumors within an organ (here, the liver in the hepatocellular phase) are visualized darkly against the organ, but blood vessels within the organ are also visualized darkly against the organ. Therefore, when identifying tumors using the method in Figure 4, there is a possibility that blood vessel regions (i.e., normal regions) may be mistakenly identified as tumor regions.

[0041] FIG. 5 is a second diagram showing a comparative example of lesion identification processing. Three-dimensional volume data can be generated by interpolation processing from a set of tomographic images including multiple tomographic images. By using such volume data, it is possible to generate tomographic images in a direction different from that of the original tomographic images. For example, it is possible to generate sagittal plane tomographic images and coronal plane tomographic images based on a set of axial plane tomographic images.

[0042] Therefore, a method of classifying lesions at a certain classification target position can be considered, using tomographic images in multiple directions, rather than a single-direction tomographic image as in the example of Figure 4. In the example of Figure 5, an axial plane tomographic image 31, a sagittal plane tomographic image 32, and a coronal plane tomographic image 33, each of which passes through a central position 30 of the classification target, are generated, and these three tomographic images (orthogonal triplanar images) are input to a classifier.

[0043] Because blood vessels may be depicted with different shapes in cross-sectional images taken in multiple directions passing through the same central position 30, this method is believed to improve tumor identification accuracy compared to the method shown in FIG. 4. For example, in Case 1 shown in FIG. 5, a tumor 34a is located at the central position 30. Because tumors are often spherical, the tumor 34a in FIG. 5 is depicted as a circle or ellipse in the axial plane cross-sectional image 31, the sagittal plane cross-sectional image 32, and the coronal plane cross-sectional image 33. However, because blood vessels are tubular, the blood vessel may be depicted as an elongated shape in at least one cross-sectional image taken in multiple directions. For example, in Case 2 shown in FIG. 5, a blood vessel 34b extends along the sagittal plane. In this case, the blood vessel 34b is depicted as a circle or ellipse in the axial plane cross-sectional image 31, but as an elongated shape in both the sagittal plane cross-sectional image 32 and the coronal plane cross-sectional image 33. Therefore, between Case 1 and Case 2, there is a high possibility that the tumor and blood vessels can be correctly distinguished.

[0044] However, cases where blood vessels extend in a direction parallel to the tomographic image, as in Case 2, are rare. In reality, blood vessels often extend in a curved line, as in Case 34c in Figure 5. In Case 3, blood vessel 34c appears circular or elliptical in all three of axial plane tomographic image 31, sagittal plane tomographic image 32, and coronal plane tomographic image 33. Therefore, in such cases, the blood vessel region (i.e., normal region) may be mistakenly identified as a tumor region.

[0045] Figure 6 is a third diagram showing a comparative example of lesion identification processing. To address the above-mentioned problem of misidentification, one possible method is to use three-dimensional data as input data to the classifier. In Figure 6, lesion identification processing is performed using a 3D patch of a fixed size centered on the center position of the object to be identified as a unit. The 3D patch is partial volume data obtained by cutting out a cubic area of ​​a fixed size from three-dimensional volume data.

[0046] As an example, Figure 6 shows a 3D patch 35 of a region including blood vessel 34c shown in Case 3 of Figure 5. This 3D patch 35 is generated by cutting out a cubic region with one side equal to that of the 2D patch, centered on the central position 30 of the object to be identified, from volume data generated based on a set of tomographic images. By inputting this type of 3D patch 35 to a classifier 36, it is possible to reduce the possibility that a blood vessel region will be mistakenly identified as a tumor region, even if the blood vessel is not aligned with any of the axial plane tomographic image 31, the sagittal plane tomographic image 32, and the coronal plane tomographic image 33.

[0047] However, the training data for training the classifier 36 is also three-dimensional volume data, i.e., 3D patches 37a, 37b, 37c, etc. Therefore, compared to when 2D patches are used, the number of parameters for training increases, resulting in a problem of increased training processing time. Another problem is that the memory capacity required for the training process increases. Furthermore, when a classifier is used for classification processing, the classification processing time and memory capacity required for classification processing increase, compared to when 2D patches are used.

[0048] Therefore, in the second embodiment, two-dimensional image data is used as learning data and input data to the classifier, thereby reducing the time required for learning and classification processes while improving the accuracy of tumor classification.

[0049] 7 is a diagram for explaining the generation process of input data for identification in the second embodiment. In the following explanation, the direction from right to left of a human body in a standing position is defined as the X axis, the direction from back to front is defined as the Y axis, and the direction from top to bottom is defined as the Z axis. In this case, the axial plane tomographic image is an image along the XY plane, the sagittal plane tomographic image is an image along the YZ plane, and the coronal plane tomographic image is an image along the XZ plane.

[0050] In this embodiment, as input data for the discriminator, a minimum projection image for three axial directions based on a 3D patch is used. As shown in FIG. 7, for each pixel on the X-Y plane (axial plane) in the 3D patch 35, a minimum projection is performed in the Z-axis direction, thereby generating a minimum projection image 41 for the Z-axis direction. Further, for each pixel on the Y-Z plane (sagittal plane) in the 3D patch 35, a minimum projection is performed in the X-axis direction, thereby generating a minimum projection image 42 for the X-axis direction. Furthermore, for each pixel on the X-Z plane (coronal plane) in the 3D patch 35, a minimum projection is performed in the Y-axis direction, thereby generating a minimum projection image 43 for the Y-axis direction.

[0051] Note that, for example, the pixel value of the minimum projection image 41 for the Z-axis direction is obtained by the following calculation. Here, it is assumed that n tomographic images (axial plane tomographic images here) are generated from the 3D patch, and the pixel value at the coordinates (x, y) in the i-th tomographic image is g i (x, y). At this time, the pixel value h(x, y) at the coordinates (x, y) in the minimum projection image h is calculated using the following formula (1). "min{···}" indicates the minimum value among the values included in {···}. h(x, y)=min{g i (x, y)}, i = 1, 2, ···, n ···(1) Note that it is desirable to determine the value of n so that a tumor of the size to be detected is included in the 3D patch cut out from the volume data. For example, when the minimum size of the tumor to be detected is r and the distance between tomographic images is d, and it is desired to detect a tumor larger than half of the patch size (the size of one side of the 3D patch), n is determined so as to satisfy n * d / 2 < r.

[0052] For example, blood vessel 34c shown in Fig. 7 extends in a curved line, but whatever the shape of the blood vessel, the blood vessel is depicted as a long, thin shape in at least one of minimum intensity projection images 41 to 43. In the example of Fig. 7, blood vessel 34c is depicted as an oval in minimum intensity projection image 41, but is depicted as a long, thin shape in minimum intensity projection images 42 and 43. Therefore, by using such minimum intensity projection images in three directions as input data, it is possible to reduce the possibility of misidentifying a blood vessel as a tumor, and improve the classification accuracy of the classifier.

[0053] In this embodiment, the minimum value projection images in three directions generated based on the 3D patches are combined into a single image, the combined 2D patch 44. To avoid confusion, the X-axis and Y-axis of the combined 2D patch 44 are shown as the X'-axis and Y'-axis in Fig. 7.

[0054] 7, in the combined 2D patch 44, a minimum intensity projection image 41 in the Z-axis direction and a minimum intensity projection image 42 in the X-axis direction are adjacently combined with their Y coordinates matching, and a minimum intensity projection image 41 in the Z-axis direction and a minimum intensity projection image 43 in the Y-axis direction are adjacently combined with their X coordinates matching.

[0055] In this case, the X coordinate and Y coordinate in the minimum value projection image 41 are used as they are as the X' coordinate and Y' coordinate in the combined 2D patch 44. The Y coordinate in the minimum value projection image 42 is used as they are as the Y' coordinate in the combined 2D patch 44, and the value obtained by adding the patch size to the Z coordinate in the minimum value projection image 42 is used as the X' coordinate in the combined 2D patch 44. The X coordinate in the minimum value projection image 43 is used as they are as the X' coordinate in the combined 2D patch 44, and the value obtained by adding the patch size to the Z coordinate in the minimum value projection image 43 is used as the Y' coordinate in the combined 2D patch 44.

[0056] In this way, a combined 2D patch, which is two-dimensional image data, is input to the classifier. Furthermore, the training data used to train the classifier is also a combined 2D patch, which is created by combining three-directional minimum value projection images generated from the original 3D patch. That is, a label indicating whether the combined 2D patch is a tumor or normal is added to the training combined 2D patch, and the classifier is trained using the combined 2D patch. This reduces the time required for the training process and the memory capacity required for the training process compared to when 3D patches are used as training data. Furthermore, the processing time and memory capacity required for the classification process using the classifier obtained by training can also be reduced.

[0057] Note that the minimum intensity projection images generated based on the 3D patch do not necessarily have to be images projected in a direction along the coordinate axis. However, it is preferable that each minimum intensity projection image be images projected in directions perpendicular to each other. Alternatively, two minimum intensity projection images projected in two directions perpendicular to each other may be generated based on the 3D patch. Even in this case, blood vessels in the 3D patch are depicted as elongated shapes extending from one side of the image to the same side or another side in at least one of the minimum intensity projection images.

[0058] Furthermore, within the liver, tumors and blood vessels are both depicted darker than their surroundings. Therefore, the combined 2D patch is generated by combining minimum intensity projection images. However, depending on the lesion to be identified, it may be depicted brighter than the surrounding area. In this case, the combined patch can be generated by combining maximum intensity projection images.

[0059] Next, FIG. 8 is a diagram showing an example of the configuration of processing functions provided in the learning processing device and the image recognition device. The learning processing device 12 includes a storage unit 110, a learning data generation unit 120, and a classifier learning unit .

[0060] The storage unit 110 is a storage area secured in a storage device provided in the learning processing device 12. The storage unit 110 stores data indicating the learning model 100 for identifying tumors (weights of the learning model 100).

[0061] The processing of the training data generation unit 120 and the classifier training unit 130 is realized, for example, by a processor included in the learning processing device 12 executing a predetermined program. The training data generation unit 120 generates training data for training a classifier for tumor classification based on one or more sets of tomographic images obtained by imaging using the MRI device 11. The above-mentioned combined 2D patches are generated as this training data. The classifier training unit 130 performs machine learning using the generated combined 2D patches for training, thereby generating a training model 100 that represents a classifier for tumor classification. The weights of the generated training model 100 are stored in the storage unit 110.

[0062] The image recognition device 22 includes a storage unit 210 , an input data generation unit 220 , and a mass recognition unit 230 . The storage unit 210 is a storage area secured in a storage device provided in the image recognition device 22, such as the RAM 202 or the HDD 203. The storage unit 210 stores weights of a learning model 211 for organ region recognition and weights of a learning model 100 for tumor recognition.

[0063] The learning model 211 is a model representing a classifier for identifying an organ region (liver in this embodiment) in a set of tomographic images, and is generated in advance by machine learning. The weights of the learning model 100 are generated by the learning processing device 12, and are read into the image recognition device 22 via a network or a portable recording medium, for example, and stored in the storage unit 210.

[0064] The processing of input data generation unit 220 and tumor identification unit 230 is realized, for example, by processor 201 executing a predetermined program. Input data generation unit 220 generates a combined 2D patch for each position of a 3D patch based on a set of tomographic images obtained by imaging using MRI device 21. In this generation process, a classifier for organ region identification based on learning model 211 is used. Tumor identification unit 230 inputs the combined 2D patch generated by input data generation unit 220 into a classifier based on learning model 100, and identifies whether the 3D patch corresponding to the combined 2D patch is a tumor or normal.

[0065] Next, the process of generating learning data by the learning processing device 12 will be described. In MR images of an organ (liver), the image may become darker as it approaches the boundary between the center and the outside of the organ, and a gradation may occur even in normal, non-tumorous areas near the boundary. As a result, it may not be possible to correctly distinguish between normal and tumor areas near the boundary. Therefore, the training data generation unit 120 ensures that the training data for each of the tumor label and normal label includes not only data from areas toward the center, some distance from the boundary, but also a sufficient amount of data from areas near the boundary. This makes it possible to generate a classifier that can correctly distinguish between tumors and normal areas even in areas near the boundary.

[0066] Furthermore, the training data generation unit 120 ensures that the normal-labeled training data includes a sufficient amount of data from regions containing blood vessels. This enables the generation of a classifier that can more accurately distinguish between blood vessels and tumors. Furthermore, for the tumor-labeled training data, the training data generation unit 120 generates training data from 3D patches selected so that the proportion of tumor regions varies. This enables the generation of a classifier that can correctly distinguish between tumors of various sizes.

[0067] 9 is a diagram showing an example of the internal configuration of the training data generation unit 120. As shown in Fig. 9, the training data generation unit 120 includes a volume data generation unit 121, an annotation unit 122, a distance image generation unit 123, a patch center point candidate extraction unit 124, a patch center point determination unit 125, a minimum value projection image generation unit 126, and a combined patch generation unit 127.

[0068] The volume data generating unit 121 generates three-dimensional volume data based on a set of tomographic images obtained by imaging using the MRI apparatus 11. The annotation unit 122 annotates each voxel of the volume data into an intra-organ region and an extra-organ region, as well as into a tumor region and a normal region.

[0069] The distance image generating unit 123 generates a three-dimensional distance image based on the annotation result. The distance images generated include a "first distance image" showing the distance from the intra-organ region to the organ boundary (the boundary with the extra-organ region), and a "second distance image" showing the distance from the normal region to the boundary with the tumor.

[0070] The patch center point candidate extraction unit 124 extracts candidates for the center points of normal 3D patches and mass 3D patches from the volume data based on the first distance image and the second distance image. A normal 3D patch is a 3D patch corresponding to a combined 2D patch labeled as "normal." A mass 3D patch is a 3D patch corresponding to a combined 2D patch labeled as "mass." Through the processing of the patch center point candidate extraction unit 124, candidates for normal 3D patches and mass 3D patches are extracted evenly from within the organ, both from the region near the organ boundary and from the region toward the center that is some distance from the boundary.

[0071] The patch center point determiner 125 determines the final center points of the normal 3D patches and the tumor 3D patches from the extracted center point candidates based on the statistical information of the brightness within the corresponding 3D patches. Through the processing of the patch center point determiner 125, the normal 3D patches are selected so that a sufficient number of 3D patches containing blood vessels are included, and the tumor 3D patches are selected so that the proportion of the tumor region is varied.

[0072] The minimum intensity projection image generating unit 126 generates minimum intensity projection images in the three axial directions for each of the normal 3D patch and the tumor 3D patch. The combined patch generator 127 combines the minimum value projections in the three axial directions for each of the normal 3D patch and the tumor 3D patch to generate a combined 2D patch.

[0073] FIG. 10 is a diagram for explaining the annotation and distance image generation process. The annotation unit 122 acquires three-dimensional volume data 50 generated based on a set of tomographic images from the volume data generation unit 121. The annotation unit 122 annotates each voxel of the generated volume data 50 into an intra-organ region and an extra-organ region. This annotation may be performed, for example, using a classifier that identifies organ regions in the set of tomographic images. This classifier is based on a learning model (a model equivalent to the learning model 100 in FIG. 8) generated in advance by machine learning.

[0074] The annotation unit 122 also annotates each voxel of the generated volume data 50 into a tumor region and a normal region. This annotation is performed, for example, manually by the user.

[0075] Based on the annotation results, distance image generator 123 generates binary image 51a that distinguishes between intra-organ regions and extra-organ regions, and binary image 51b that distinguishes between tumor regions and normal regions. Binary image 51a is a three-dimensional binary image in which, among the voxels of volume data 50, voxels in the intra-organ region are set to "1" and voxels in the extra-organ region are set to "0." Binary image 51b is a three-dimensional binary image in which, among the voxels of volume data 50, voxels in the normal region are set to "1" and voxels in the tumor region are set to "0."

[0076] Next, distance image generator 123 generates first distance image 52a based on binary image 51a and volume data 50. This first distance image 52a is a three-dimensional distance image in which the distance from the intra-organ region to the organ boundary is set for each voxel of volume data 50. That is, in first distance image 52a, a distance of "0" is set for each voxel in the extra-organ region, and a value according to the distance to the organ boundary is set for each voxel in the intra-organ region. In the intra-organ region, the distance value increases toward the center.

[0077] Distance image generator 123 also generates second distance image 52b based on binary image 51b and volume data 50. Second distance image 52b is a three-dimensional distance image in which the distance from the normal region to the boundary with the tumor is set for each voxel in volume data 50. That is, in second distance image 52b, a distance of "0" is set for each voxel in the tumor region, and a value corresponding to the distance to the boundary with the tumor is set for each voxel in the normal region. In the normal region, the distance value increases with increasing distance from the tumor region.

[0078] Fig. 11 is a diagram showing a first example of a distance image generation method. In the first example shown in Fig. 11, the distance to each voxel in the volume data 50 is calculated by moving a three-dimensional mask from one end of the volume data 50 to the opposite end. In Fig. 11, as an example, the mask is raster-scanned on a tomographic image in the XY plane (axial plane tomographic image), and then sequentially moved in the Z-axis direction.

[0079] First, a mask 61a for forward scanning is used. Mask 61a includes voxel B0, the target of distance calculation, and 13 voxels B1 to B13 surrounding it. Furthermore, as an initial setting for the distance image, distance image generator 123 sets "0" to voxels in volume data 50 whose corresponding binary image value is "0," and sets M to voxels whose corresponding binary image value is "1." M is set to a value greater than the maximum predicted distance.

[0080] The distance image generation unit 123 performs the following distance calculation while moving the mask 61a so that the position of voxel B0 is raster scanned in the X direction starting from the origin of the tomographic image on the XY plane, and this raster scanning is then moved in the Z direction. If the distance value of voxel B0 is not "0," the distance image generation unit 123 extracts the minimum value from the distance values ​​of the surrounding voxels B1 to B13 and updates the distance value of voxel B0 with the minimum value plus "1." On the other hand, if the distance value of voxel B0 is "0," the distance image generation unit 123 does not update the distance value but leaves it as it is.

[0081] Once the above process has been performed up to the final position, the reverse operation mask 61b is used. Mask 61b includes voxel B0, the target of distance calculation, and 13 surrounding voxels B21 to B33. Distance image generation unit 123 performs the following distance calculation while moving mask 61b so that voxel B0 is moved from the final position in the opposite direction to the above process. If the distance value of voxel B0 is not "0," distance image generation unit 123 extracts the minimum value from the distance values ​​of the surrounding voxels B21 to B33 and updates the distance value of voxel B0 by adding "1" to this minimum value. On the other hand, if the distance value of voxel B0 is "0," distance image generation unit 123 does not update the distance value but maintains it as is.

[0082] In the process of generating the first distance image using the above method, the distance values ​​of the voxels B1 to B13 and B21 to B33 that are not included in the volume data 50 are set to “0.” On the other hand, in the process of generating the second distance image using the above method, the distance values ​​of the voxels B1 to B13 and B21 to B33 that are not included in the volume data 50 are set to M.

[0083] Fig. 12 is a diagram showing a second example of a distance image generation method. In the second example shown in Fig. 12, the distance calculation process is simplified by calculating the distance using a two-dimensional mask. According to the method shown in Fig. 12, the accuracy of distance calculation is lower than that of the method shown in Fig. 11, but the time required for the distance calculation process can be shortened.

[0084] Here, as in FIG. 11, it is assumed that the mask is raster scanned on a tomographic image in the XY plane (axial plane tomographic image) and then sequentially moved in the Z-axis direction. First, a mask 62a for forward scanning is used. Mask 62a includes pixel P0, the pixel for which distance is to be calculated, and its four adjacent pixels P1 to P4. Furthermore, as an initial setting for the distance image, distance image generator 123 sets "0" to pixels (i.e., voxels) in each tomographic image of volume data 50 whose corresponding binary image value is "0," and sets M to pixels whose corresponding binary image value is "1." M is set to a value greater than the maximum predicted distance.

[0085] The distance image generation unit 123 performs the following distance calculation while moving the mask 62a so that the position of pixel P0 is raster scanned in the X direction starting from the origin of the tomographic image on the XY plane, and this raster scanning is then moved in the Z direction. If the distance value of pixel P0 is not "0," the distance image generation unit 123 extracts the minimum value from the distance values ​​of surrounding pixels P1 to P4 and updates the distance value of pixel P0 to the minimum value plus "1." On the other hand, if the distance value of pixel P0 is "0," the distance image generation unit 123 does not update the distance value but maintains it as is.

[0086] Once the above process has been performed up to the final position, mask 62b for reverse operation is then used. Mask 62b includes pixel P0, the pixel for which distance is to be calculated, and its four adjacent pixels P11 to P14. Distance image generation unit 123 performs the following distance calculation while moving mask 62b so that pixel P0 is moved from the final position in the direction opposite to the above process. If the distance value of pixel P0 is not "0," distance image generation unit 123 extracts the minimum value from the distance values ​​of the surrounding pixels P11 to P14 and updates the distance value of pixel P0 to the minimum value plus "1." On the other hand, if the distance value of pixel P0 is "0," distance image generation unit 123 does not update the distance value but maintains it as is.

[0087] In the process of generating the first distance image using the above method, the distance value of pixels P1 to P4, P11 to P14 that are not included in the volume data 50 is set to "0." On the other hand, in the process of generating the second distance image using the above method, the distance value of pixels P1 to P4, P11 to P14 that are not included in the volume data 50 is set to M.

[0088] 13 is a diagram showing the procedure for determining a normal 3D patch. Normal 3D patches corresponding to the learning data (combined 2D patches) labeled as "normal" are determined by the processing of the patch center point candidate extraction unit 124 and the patch center point determination unit 125. In this determination processing, an "interior normal 3D patch" existing in a region toward the center of the organ that is some distance from the boundary with the outside of the organ, and a "boundary normal 3D patch" existing in a region near the boundary are individually determined.

[0089] In the following description, the distance to the voxel at coordinates (x, y, z) on the first distance image is defined as D1(x, y, z), and the distance to the voxel at coordinates (x, y, z) on the second distance image is defined as D2(x, y, z).

[0090] First, the patch center point candidate extraction unit 124 extracts N1 voxels that satisfy both of the following conditions C1a and C2a from among the voxels in the organ region of the volume data, as candidates for the center point of the internal normal 3D patch (step S21a). Note that S indicates the patch size (the length of one side of the 3D patch). Specifically, the patch center point candidate extraction unit 124 extracts all voxels that satisfy both conditions C1a and C2a from the organ region, and then randomly extracts N1 voxels from among them.

[0091] (Condition C1a)D1(x,y,z)>(S / 2)*3 1 / 2 (Condition C2a)D2(x,y,z)>(S / 2)*3 1 / 2 Condition C1a indicates that the distance value of the first distance image is greater than half the length of the diagonal of the 3D patch. In other words, a center point (voxel) that satisfies condition C1a exists in a region toward the center, some distance from the boundary with the outside of the organ. Furthermore, a center point that satisfies condition C2a indicates that the distance value of the second distance image is greater than half the length of the diagonal of the 3D patch. In other words, a center point (voxel) that satisfies condition C2a exists in a region outside the tumor region (normal region). Therefore, the above process extracts N1 candidates for the center point of the internal tumor 3D patch.

[0092] Furthermore, the patch center point candidate extraction unit 124 extracts N1 voxels that satisfy both of the following conditions C1b and C2b from among the voxels in the organ region of the volume data, as candidates for the center point of the boundary-normal 3D patch (step S21b). Specifically, the patch center point candidate extraction unit 124 extracts all voxels that satisfy both conditions C1b and C2b from the organ region, and then randomly extracts N1 voxels from among them.

[0093] (Condition C1b)D1(x,y,z)≦(S / 2)*3 1 / 2 (Condition C2b)D2(x,y,z)>(S / 2)*3 1 / 2 Condition C1b indicates that the distance value of the first distance image is less than or equal to the length of the diagonal of the 3D patch. That is, the central points (voxels) that satisfy condition C1b exist in the region near the boundary with the outside of the organ. Also, condition C2b is the same as condition C2a, and the central points (voxels) that satisfy condition C2b exist in the region outside the tumor region (normal region). Therefore, N1 candidate central points of the boundary tumor 3D patch are extracted by the above process.

[0094] By the processing of steps S21a and S21b above, candidate central points of the normal 3D patch are evenly extracted from both the region at a certain distance from the boundary and the region near the boundary. Next, the patch center point determination unit 125 narrows down the central points of the internal normal 3D patch to N2 (<N1) and also narrows down the central points of the boundary normal 3D patch to N2. In this narrowing process, based on the statistical information of the luminance in the 3D patch, the final normal 3D patch is determined so that the distribution of the luminance varies among the 3D patches. Thereby, a sufficient number of normal 3D patches in the determined normal 3D patches that include regions with blood vessels are included.

[0095] Here, the number of the finally determined internal normal 3D patches and boundary normal 3D patches is made the same, but it is not necessarily the same number, and it may be about the same number. Also, these ratios may be intentionally changed for sensitivity adjustment or the like.

[0096] Specifically, the patch center point determination unit 125 executes the narrowing process according to the following procedure. For each of the candidate central points extracted in step S21a, the patch center point determination unit 125 calculates the average value (step S22a) and the standard deviation of the luminance (step S22b) of each voxel in the corresponding 3D patch. Based on the calculation results in steps S22a and S22b, the patch center point determination unit 125 extracts N2 central points corresponding to the final internal normal 3D patch from the candidate central points so that the average value and the standard deviation of the luminance are dispersed (step S23a).

[0097] As mentioned above, in MR images, blood vessel regions are depicted darker than the surrounding organ regions. Therefore, by extracting internal normal 3D patches with different brightness distribution patterns using the above procedure, regions containing blood vessels and regions not containing blood vessels are extracted in a balanced manner. As a result, it is possible to generate a classifier that can correctly distinguish blood vessel regions from normal regions.

[0098] Furthermore, for each of the center point candidates extracted in step S21b, the patch center point determination unit 125 calculates the average brightness value (step S22c) and the standard deviation of brightness (step S22d) of each voxel in the corresponding 3D patch. Based on the calculation results in steps S22c and S22d, the patch center point determination unit 125 extracts N center points corresponding to the final boundary-normal 3D patch from the center point candidates so that the average brightness values ​​and the standard deviations of brightness are dispersed (step S23b).

[0099] This allows for the extraction of boundary-normal 3D patches with different brightness distribution patterns, resulting in a balanced extraction of regions that contain blood vessels and regions that do not, making it possible to generate a classifier that can correctly distinguish blood vessel regions from normal regions.

[0100] In the process of narrowing down N1 candidates for center points to N2, only one of the average brightness value and the standard deviation of brightness may be used. However, since there are cases where the average brightness value differs significantly even if the standard deviation is the same, using both of these makes it possible to extract regions that include blood vessels and regions that do not include blood vessels without bias.

[0101] 14 is a diagram showing the procedure for determining a tumor 3D patch. The patch center point candidate extraction unit 124 and patch center point determination unit 125 perform processing to determine a tumor 3D patch corresponding to the training data (combined 2D patch) labeled as a "tumor." The tumor 3D patch is determined using a procedure similar to that for normal 3D patches. That is, this determination process separately determines an "internal tumor 3D patch" that exists in a region inside the organ toward the center, some distance from the boundary with the outside of the organ, and a "boundary tumor 3D patch" that exists in a region near the boundary.

[0102] First, the patch center point candidate extraction unit 124 extracts N1 voxels that satisfy both of the following conditions C1c and C2c from among the voxels in the organ region of the volume data, as candidates for the center point of the internal tumor 3D patch (step S31a). Specifically, the patch center point candidate extraction unit 124 extracts all voxels that satisfy both conditions C1c and C2c from the organ region, and then randomly extracts N1 voxels from among them.

[0103] (Condition C1c)D1(x,y,z)>(S / 2)*3 1 / 2 (Condition C2c)D2(x,y,z)≦(S / 2)*3 1 / 2 Condition C1c indicates that the distance value of the first distance image is greater than half the length of the diagonal of the 3D patch. In other words, a center point (voxel) that satisfies condition C1c exists in a region toward the center, some distance from the boundary with the outside of the organ. Furthermore, a center point that satisfies condition C2c indicates that the distance value of the second distance image is less than half the length of the diagonal of the 3D patch. In other words, a center point (voxel) that satisfies condition C2c exists in the tumor region.

[0104] Also, the patch center point candidate extraction unit 124 extracts N1 voxels that satisfy both of the following conditions C1d and C2d from the voxels in the organ region in the volume data as candidates for the center point of the boundary tumor 3D patch (step S31b). As a specific process, the patch center point candidate extraction unit 124 extracts all voxels that satisfy both conditions C1d and C2d from the organ region, and then randomly extracts N1 of them.

[0105] (Condition C1d) D1(x,y,z)≦(S / 2)*3 1 / 2 (Condition C2d) D2(x,y,z)≦(S / 2)*3 1 / 2 Condition C1d indicates that the distance value of the first distance image is less than or equal to the length of the diagonal of the 3D patch. That is, the center point (voxel) that satisfies condition CId exists in the region near the boundary with the outside of the organ. Also, condition C2d is the same as condition C2c, and the center point (voxel) that satisfies condition C2d exists in the tumor region.

[0106] Through the processes of steps S31a and S31b above, candidates for the center point of the tumor 3D patch are evenly extracted from both the region at a certain distance from the boundary and the region near the boundary. Next, the patch center point determination unit 125 narrows down the center points of the internal tumor 3D patches to N2 (<N1) and also narrows down the center points of the boundary tumor 3D patches to N2. In this narrowing process, based on the statistical information of the luminance in the 3D patch, the final tumor 3D patch is determined so that the distribution of the luminance varies between the 3D patches. As a result, the final tumor 3D patch is determined so that the size of the tumor region varies between the determined 3D patches.

[0107] Here, the number of the finally determined internal tumor 3D patches and boundary tumor 3D patches is made the same, but it is not necessarily required to be the same number, and it is sufficient if the numbers are about the same. Also, these ratios may be intentionally changed for sensitivity adjustment or the like.

[0108] Specifically, the patch center point determination unit 125 executes the narrowing process according to the following procedure. For each of the center point candidates extracted in step S31a, the patch center point determination unit 125 calculates the average brightness value (step S32a) and the standard deviation of brightness (step S32b) of each voxel in the corresponding 3D patch. Based on the calculation results in steps S32a and S32b, the patch center point determination unit 125 extracts N center points corresponding to the final internal tumor 3D patch from the center point candidates so that the average brightness value and the standard deviation of brightness are dispersed (step S33a).

[0109] As mentioned above, tumor regions appear darker in MR images than the regions within the surrounding organs. Therefore, by extracting internal tumor 3D patches with different brightness distributions using the above procedure, it becomes possible to extract internal tumor 3D patches that include tumors of various shapes, such as tumors of different sizes and tumors with different internal gradations. As a result, it becomes possible to generate a classifier that can correctly identify tumor regions as tumors.

[0110] Furthermore, for each of the center point candidates extracted in step S31b, the patch center point determination unit 125 calculates the average brightness value (step S32c) and the standard deviation of brightness (step S32d) of each voxel in the corresponding 3D patch. Based on the calculation results in steps S32c and S32d, the patch center point determination unit 125 extracts N center points corresponding to the final border tumor 3D patch from the center point candidates so that the average brightness value and the standard deviation of brightness are dispersed (step S33b).

[0111] This allows for the extraction of 3D patches of boundary tumors with different brightness distributions, resulting in the extraction of 3D patches of internal tumors containing tumors of various shapes, making it possible to generate a classifier that can correctly identify tumor regions as tumors.

[0112] As with the normal 3D patch determination process, the tumor 3D patch determination process may use only one of the mean brightness value or the standard deviation of brightness to narrow down the N1 center point candidates to N2. However, since there are cases where the mean values ​​differ significantly even if the standard deviations are the same, using both of these makes it possible to extract regions containing tumors of various shapes.

[0113] FIG. 15 is a diagram for explaining input of learning data to the classifier learning unit. For each of the interior normal 3D patches extracted by the patch center point determination unit 125, the minimum value projection image generation unit 126 generates minimum value projection images in the three axial directions, and the combined patch generation unit 127 combines the generated minimum value projection images to generate a combined 2D patch. Similarly, for each of the boundary normal 3D patches extracted by the patch center point determination unit 125, the minimum value projection image generation unit 126 generates minimum value projection images in the three axial directions, and the combined patch generation unit 127 combines the generated minimum value projection images to generate a combined 2D patch. Normal labels are attached to these combined 2D patches.

[0114] Furthermore, for each internal tumor 3D patch extracted by the patch center point determiner 125, the minimum intensity projection image generator 126 generates minimum intensity projection images in the three axial directions, and the combined patch generator 127 combines the generated minimum intensity projection images to generate a combined 2D patch. Also, for each border tumor 3D patch extracted by the patch center point determiner 125, the minimum intensity projection image generator 126 generates minimum intensity projection images in the three axial directions, and the combined patch generator 127 combines the generated minimum intensity projection images to generate a combined 2D patch. A tumor label is attached to these combined 2D patches.

[0115] In this way, a combined 2D patch labeled as normal and a combined 2D patch labeled as tumor are generated and input to the classifier training unit 130. The classifier training unit 130 uses these combined 2D patches as training data to generate, by machine learning, a learning model for a classifier that distinguishes between normal and tumor for each combined 2D patch (i.e., each 3D patch).

[0116] In practice, a combined 2D patch labeled as normal and a combined 2D patch labeled as tumor are generated by the above process using a plurality of sets of tomographic images, and are input to the classifier training unit 130.

[0117] Since minimum intensity projection images in three axial directions are used as training data, it is possible to generate a highly accurate classifier that is less likely to mistakenly classify blood vessels for tumors. Furthermore, since the training data input to the classifier training unit 130 is 2D data rather than 3D data, the time required for training can be shortened compared to when 3D data is used, and the memory capacity required for training can be reduced.

[0118] The same or approximately the same number of combined 2D patches labeled with normality and the same number of combined 2D patches labeled with masses are generated and input to the classifier training unit 130. However, for purposes such as sensitivity adjustment, the ratio between the number of input combined 2D patches labeled with normality and the number of input combined 2D patches labeled with masses may be intentionally changed.

[0119] Alternatively, machine learning using a combined 2D patch based on the internal normal 3D patch and the internal mass 3D patch and machine learning using a combined 2D patch based on the boundary normal 3D patch and the boundary mass 3D patch may be performed separately. In this case, a separate classifier is generated by each machine learning. During the classification process, it is determined whether the 3D patch that formed the combined 2D patch is located in a region toward the center of the organ, some distance from the organ boundary, or near the organ boundary. The classification process is performed using a classifier that corresponds to the result of the determination.

[0120] Next, the processing of the learning processing device 12 will be described using a flowchart. FIG. 16 is an example of a flowchart showing the overall processing procedure of the learning process. [Step S41] The volume data generation unit 121 acquires a set of tomographic images from the MRI apparatus 11. The volume data generation unit 121 generates three-dimensional volume data based on the acquired set of tomographic images.

[0121] [Step S42] The annotation unit 122 annotates each voxel of the generated volume data into an intra-organ region and an extra-organ region, as well as into a tumor region and a normal region.

[0122] [Step S43] Based on the annotation results of the intra-organ region and the extra-organ region, the distance image generation unit 123 generates a first binary image representing the intra-organ region. This first binary image is a three-dimensional binary image in which voxels in the intra-organ region are set to "1" and voxels in the extra-organ region are set to "0."

[0123] Furthermore, distance image generator 123 generates a second binary image showing the normal region (non-tumor region) based on the annotation results of the tumor region and the normal region. This second binary image is a three-dimensional binary image in which "1" is set to the voxels of the normal region and "0" is set to the voxels of the tumor region.

[0124] [Step S44] The distance image generator 123 generates a first distance image based on the first binary image and the volume data. This first distance image is a three-dimensional distance image in which a distance of "0" is set for each voxel in the extra-organ region and a value corresponding to the distance to the organ boundary is set for each voxel in the intra-organ region.

[0125] Furthermore, distance image generator 123 generates a second distance image based on the second binary image and the volume data. This second distance image is a three-dimensional distance image in which a distance of "0" is set for each voxel in the tumor region and a value corresponding to the distance to the boundary with the tumor is set for each voxel in the normal region.

[0126] [Step S45] Using the first distance image and the second distance image, a process of generating a combined 2D patch based on the internal normal 3D patch is executed. [Step S46] Using the first distance image and the second distance image, a process of generating a combined 2D patch based on the boundary-normal 3D patch is executed.

[0127] [Step S47] Using the first distance image and the second distance image, a process of generating a combined 2D patch based on the internal mass 3D patch is executed. [Step S48] Using the first distance image and the second distance image, a process for generating a combined 2D patch based on the border mass 3D patch is executed.

[0128] The order of steps S45 to S48 is not limited to the above order, and steps S45 to S48 may be executed in parallel. [Step S49] The classifier training unit 130 performs machine learning using the combined 2D patches as training data to generate a learning model 100 of a classifier that distinguishes between normal and tumor tissue. For example, the classifier training unit 130 generates the learning model 100 by machine learning using a neural network. The classifier training unit 130 stores the weights of the generated learning model 100 in the storage unit 110.

[0129] In practice, the processes of steps S41 to S48 are performed using a plurality of sets of tomographic images, and in step S49, machine learning is performed using the obtained combined 2D patches. 17 to 19 are example flowcharts showing the procedure of the combined 2D patch generation process. The processes in Figures 17 to 19 correspond to the processes in steps S45 to S48 in Figure 16. That is, the processes in Figures 17 to 19 are executed for each of the interior normal 3D patch, boundary normal 3D patch, interior mass 3D patch, and boundary mass 3D patch.

[0130] [Step S51] The patch center point candidate extraction unit 124 selects one voxel in the organ region in the volume data. [Step S52] The patch center point candidate extraction unit 124 determines whether the distance value of the selected voxel in the first distance image satisfies condition C1. In step S45, the above-mentioned condition C1a is used as condition C1. In step S46, the above-mentioned condition C1b is used as condition C1. In step S47, the above-mentioned condition C1c is used as condition C1. In step S48, the above-mentioned condition C1d is used as condition C1. If condition C1 is satisfied, the process proceeds to step S53; if condition C1 is not satisfied, the process proceeds to step S55.

[0131] [Step S53] The patch center point candidate extraction unit 124 determines whether the distance value of the selected voxel in the second distance image satisfies condition C2. In step S45, the above-mentioned condition C2a is used as condition C2. In step S46, the above-mentioned condition C2b is used as condition C2. In step S47, the above-mentioned condition C2c is used as condition C2. In step S48, the above-mentioned condition C2d is used as condition C2. If condition C2 is satisfied, the process proceeds to step S54; if condition C2 is not satisfied, the process proceeds to step S55.

[0132] [Step S54] The patch center point candidate extraction unit 124 selects the voxel selected in step S51 as a candidate for the center point of the corresponding 3D patch. The "corresponding 3D patch" is an internal normal 3D patch in step S45, a boundary normal 3D patch in step S46, an internal mass 3D patch in step S47, or a boundary mass 3D patch in step S48.

[0133] [Step S55] The patch center point candidate extraction unit 124 determines whether all voxels in the organ region in the volume data have been selected in step S51. If there are unselected voxels, the process proceeds to step S51, where one unselected voxel is selected. On the other hand, if all voxels have been selected, the process proceeds to step S56.

[0134] [Step S56] The patch center point candidate extraction unit 124 randomly selects N1 center point candidates from the selected center point candidates. This narrows down the center point candidates. In the following processing, the narrowed down N1 center points are processed.

[0135] [Step S57] The patch center point determination unit 125 selects one of the center point candidates. [Step S58] The patch center point determination unit 125 extracts a 3D patch centered on the selected center point from the volume data. Note that voxels in the 3D patch that are not included in the volume data may be interpolated based on multiple nearby voxels, or may be treated as having no data.

[0136] [Step S59] The patch center point determination unit 125 calculates the average brightness value of each voxel in the extracted 3D patch and the standard deviation of the brightness of each voxel. [Step S60] The patch center point determination unit 125 determines whether all center point candidates have been selected in step S57. If there are unselected center point candidates, the process proceeds to step S57, where the unselected center point candidates are selected. On the other hand, if all center point candidates have been selected, the process proceeds to step S61.

[0137] Next, a process is executed to narrow down the N1 center point candidates so that the luminance distribution varies among the 3D patches corresponding to the center points. The processes in steps S61 to S67 below are an example of such a narrowing down process procedure.

[0138] [Step S61] The patch center point determination unit 125 generates a two-dimensional histogram based on the average luminance value calculated in step S58 and the standard deviation of luminance value calculated in step S59. In this two-dimensional histogram, the frequency of each of the average and standard deviation is counted in units of a predetermined width. For example, the width of the average value is set to "20" and the width of the standard deviation is set to "5."

[0139] [Step S62] The patch center point determination unit 125 sets the number of blocks N3. For example, if the brightness varies from "100" to "220," the brightness is divided into 6 parts, using (220-100) / 20=6. If the standard deviation varies from "5" to "50," the standard deviation is divided into 9 parts, using (50-5) / 5=9. In this case, the number of blocks N3 to be divided is set to 6 x 9 = 45.

[0140] The patch center point determination unit 125 sets the initial number of patches N4 in each block by calculating N2 / N3. [Step S63] The patch center point determination unit 125 sorts the combinations of the average luminance value and the standard deviation of luminance value in the two-dimensional histogram in descending order of frequency.

[0141] [Step S64] The patch center point determination unit 125 divides the sorted combinations into N3 blocks, and selects one block in descending order of frequency. [Step S65] The patch center point determination unit 125 randomly extracts up to N4 3D patches from among the 3D patches corresponding to center point candidates included in the block.

[0142] [Step S66] The number of 3D patches extracted in step S65 is set to N5. The patch center point determination unit 125 updates the current N3 to (N3-1), and updates the initial number of patches N4 by calculating (N1-N5) / N3 using the updated N3, N1, and N5.

[0143] [Step S67] The patch center point determination unit 125 determines whether all divided blocks have been selected in step S64. If there are any unselected blocks, the process proceeds to step S64, where the unselected blocks are selected.

[0144] On the other hand, if all blocks have been selected, the process proceeds to step S68. In this case, the 3D patch extracted in step S65 is the 3D patch based on the center point narrowed down from the center point candidates. In step S45, an internal normal 3D patch is determined. In step S46, a boundary normal 3D patch is determined. In step S47, an internal mass 3D patch is determined. In step S48, a boundary mass 3D patch is determined. In the following processes, these 3D patches are processed.

[0145] [Step S68] The minimum intensity projection image generation unit 126 selects one 3D patch. [Step S69] The minimum intensity projection image generating unit 126 generates minimum intensity projection images for each of the three axial directions based on the selected 3D patch.

[0146] [Step S70] The combined patch generation unit 127 combines the generated minimum intensity projection images using the method shown in Fig. 7 to generate a combined 2D patch. The combined patch generation unit 127 then adds a label to the generated combined 2D patch. In steps S45 and S46, a label indicating normal is added, and in steps S47 and S48, a label indicating tumor is added.

[0147] [Step S71] The combined patch generation unit 127 determines whether all of the corresponding 3D patches have been selected in step S68. If there are any unselected 3D patches, the process proceeds to step S68, where the unselected 3D patches are selected. On the other hand, if all 3D patches have been selected, the combined 2D patch generation process ends. In this case, the process proceeds to step S49 in FIG. 16, where classifier training is performed using the generated labeled combined 2D patches.

[0148] Next, the processing of the image identification device 22 will be described. 20 is a diagram showing an example of the internal configuration of the input data generation unit 220. As shown in Fig. 20, the input data generation unit 220 includes a volume data generation unit 221, an organ region identification unit 222, a 3D patch generation unit 223, a minimum intensity projection image generation unit 224, and a combined patch generation unit 225.

[0149] The volume data generating unit 221 generates three-dimensional volume data based on a set of tomographic images obtained by imaging using the MRI device 21. The organ region identifying unit 222 identifies an organ region (here, a liver region) from the volume data using a classifier for organ region identification based on the learning model 211.

[0150] The 3D patch generation unit 223 selects a center point of a 3D patch from an organ region in the volume data, and generates a 3D patch centered on the selected center point. The minimum intensity projection image generating unit 224 generates minimum intensity projection images in the three axial directions for each 3D patch corresponding to each of the determined center points.

[0151] The combined patch generator 225 combines the generated minimum intensity projection images for each 3D patch to generate a combined 3D patch, and inputs the combined 3D patch to the mass identifier 230 . FIG. 21 is an example of a flowchart showing a classification processing procedure performed by the image classification device.

[0152] [Step S81] The volume data generation unit 221 acquires a set of tomographic images from the MRI apparatus 21. The volume data generation unit 221 generates three-dimensional volume data based on the acquired set of tomographic images.

[0153] [Step S82] The organ region identification unit 222 uses a classifier for organ region identification based on the learning model 211 to identify an organ region from the volume data. This learning model 211 is generated by deep learning using, for example, a large number of tomographic images as teacher images and teacher labels indicating whether each pixel in the tomographic image is in an intra-organ region or an extra-organ region. In this case, in step S82, each tomographic image included in the tomographic image set is input to a classifier based on such learning model 211, and it is determined whether each pixel in each tomographic image is in an intra-organ region or an extra-organ region. Then, based on the determination result for each tomographic image, voxels in the organ region are identified from among the voxels in the volume data.

[0154] [Step S83] The 3D patch generation unit 223 selects the center point of a 3D patch from among the voxels in the organ region. Here, the 3D patch is generated, for example, by dividing the volume data into equal-sized voxels. Alternatively, the 3D patch may be generated by moving a three-dimensional window of the same size as the 3D patch by a predetermined distance (for example, half the patch size). In step S83, one of the center points of the 3D patches generated in this manner is selected. Note that the size of the 3D patch is the same as that of the 3D patch generated during the learning process.

[0155] [Step S84] The 3D patch generation unit 223 extracts a 3D patch centered on the selected center point from the volume data. For voxels of the 3D patch that are not included in the volume data, the values ​​of the nearest voxels are used, or values ​​of multiple nearby voxels are interpolated.

[0156] [Step S85] The minimum intensity projection image generating unit 224 generates minimum intensity projection images for each of the three axial directions based on the extracted 3D patch. [Step S86] The combined patch generating unit 225 combines the generated minimum intensity projection images using the method shown in FIG. 7 to generate a combined 2D patch.

[0157] [Step S87] The mass identifier 230 inputs the generated combined 2D patch to a classifier based on the learning model 100 to identify whether the 3D patch corresponding to the combined 2D patch is a mass or a normal patch.

[0158] [Step S88] The tumor identifier 230 determines whether all of the center points of the 3D patches to be generated have been selected in step S83. If there are any unselected center points, the process proceeds to step S83, where the unselected center points are selected as the center points of the 3D patches. On the other hand, if all applicable center points have been selected, the identification process ends. In this case, as the identification result, for example, identification result information is output in which information indicating whether each 3D patch is a tumor or normal is added.

[0159] The above process performs lesion classification using minimum intensity projection images from multiple directions based on 3D patches, enabling highly accurate classification of tumors or normal areas for each 3D patch. This reduces the possibility of misidentifying blood vessel regions as tumors. Furthermore, because 2D data, rather than 3D data, is input to the classifier, the time required for classification processing can be shortened compared to when 3D data is used, and the memory capacity required for classification processing can be reduced.

[0160] The processing functions of the devices (e.g., image recognition device 1, learning processing device 12, image recognition device 22) shown in each of the above embodiments can be realized by a computer. In this case, a program describing the processing content of the functions that each device should have is provided, and the processing functions are realized on the computer by executing the program on the computer. The program describing the processing content can be recorded on a computer-readable recording medium. Examples of computer-readable recording media include magnetic storage devices, optical discs, and semiconductor memories. Examples of magnetic storage devices include hard disk drives (HDDs) and magnetic tapes. Examples of optical discs include CDs (Compact Discs), DVDs (Digital Versatile Discs), and Blu-ray Discs (BD, registered trademark).

[0161] When distributing a program, for example, the program is recorded on a portable recording medium such as a DVD or CD and sold. Alternatively, the program can be stored in a storage device of a server computer and transferred from the server computer to other computers via a network.

[0162] A computer that executes a program stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. The computer then reads the program from its own storage device and executes processing in accordance with the program. Note that the computer can also read the program directly from a portable recording medium and execute processing in accordance with that program. The computer can also execute processing in accordance with the program received each time a program is transferred from a server computer connected via a network. [Explanation of symbols]

[0163] 1. Image recognition device 1a Processing section 2 Volume data 3 partial area 4a~4c Projected images 5 Specific parts

Claims

1. On the computer, A three-dimensional partial region of a predetermined size is extracted from three-dimensional volume data generated based on a plurality of tomographic images of the inside of the human body, generating a plurality of projection images by performing minimum value projection or maximum value projection on the values ​​of each voxel of the partial region in a plurality of directions orthogonal to each other; Identifying which of a plurality of states the partial region is in based on the plurality of projection images. Execute the process, The identification is performed using a learning model generated by machine learning; The learning model is extracting a plurality of learning partial regions, each of which is a three-dimensional region of the predetermined size, from an organ region including a specific organ in three-dimensional learning volume data based on a plurality of learning tomographic images obtained by photographing the inside of a human body; extracting, from the plurality of learning partial regions, a plurality of first partial region candidates corresponding to a first state among the plurality of states and a plurality of second partial region candidates corresponding to a second state among the plurality of states; extracting a plurality of first partial regions corresponding to the first state from the plurality of first partial region candidates such that distances from a boundary with an outside of the organ region are distributed among the first partial region candidates; extracting a plurality of second partial regions corresponding to the second state from the plurality of second partial region candidates such that the distances from the boundary are distributed among the second partial region candidates; generating a first projection image group including a plurality of first projection images generated by minimum value projection or maximum value projection of the value of each voxel of the corresponding first partial region in the plurality of directions for each of the plurality of first partial regions; generating a second projection image group including a plurality of second projection images generated by minimum value projection or maximum value projection of the value of each voxel of the corresponding second partial region in the plurality of directions for each of the plurality of second partial regions; performing machine learning using the first projection image group corresponding to each of the plurality of first partial regions as learning data corresponding to the first state, and using the second projection image group corresponding to each of the plurality of second partial regions as learning data corresponding to the second state; generated by a model generation process including Image recognition program.

2. causing the computer to further perform processing to combine the plurality of projection images to generate a two-dimensional combined image; the identification includes identifying which of the plurality of states the partial region is in based on the combined image. The image recognition program according to claim 1.

3. The specific organ is a liver, and each of the plurality of first partial region candidates corresponding to the first state includes a tumor region, and each of the plurality of second partial region candidates corresponding to the second state does not include the tumor region. The image recognition program according to claim 1.

4. The computer A three-dimensional partial region of a predetermined size is extracted from three-dimensional volume data generated based on a plurality of tomographic images of the inside of the human body, generating a plurality of projection images by performing minimum value projection or maximum value projection on the values ​​of each voxel of the partial region in a plurality of directions orthogonal to each other; Identifying which of a plurality of states the partial region is in based on the plurality of projection images; The identification is performed using a learning model generated by machine learning; The learning model is extracting a plurality of learning partial regions, each of which is a three-dimensional region of the predetermined size, from an organ region including a specific organ in three-dimensional learning volume data based on a plurality of learning tomographic images obtained by photographing the inside of a human body; extracting, from the plurality of learning partial regions, a plurality of first partial region candidates corresponding to a first state among the plurality of states and a plurality of second partial region candidates corresponding to a second state among the plurality of states; extracting a plurality of first partial regions corresponding to the first state from the plurality of first partial region candidates such that distances from a boundary with an outside of the organ region are distributed among the first partial region candidates; extracting a plurality of second partial regions corresponding to the second state from the plurality of second partial region candidates such that the distances from the boundary are distributed among the second partial region candidates; generating a first projection image group including a plurality of first projection images generated by minimum value projection or maximum value projection of the value of each voxel of the corresponding first partial region in the plurality of directions for each of the plurality of first partial regions; generating a second projection image group including a plurality of second projection images generated by minimum value projection or maximum value projection of the value of each voxel of the corresponding second partial region in the plurality of directions for each of the plurality of second partial regions; performing machine learning using the first projection image group corresponding to each of the plurality of first partial regions as learning data corresponding to the first state, and using the second projection image group corresponding to each of the plurality of second partial regions as learning data corresponding to the second state; generated by a model generation process including Image identification methods.

5. A three-dimensional partial region of a predetermined size is extracted from three-dimensional volume data generated based on a plurality of tomographic images of the inside of the human body, generating a plurality of projection images by performing minimum value projection or maximum value projection on the values ​​of each voxel of the partial region in a plurality of directions orthogonal to each other; a processing unit that identifies which of a plurality of states the partial region is in based on the plurality of projection images, The identification is performed using a learning model generated by machine learning; The learning model is extracting a plurality of learning partial regions, each of which is a three-dimensional region of the predetermined size, from an organ region including a specific organ in three-dimensional learning volume data based on a plurality of learning tomographic images obtained by photographing the inside of a human body; extracting, from the plurality of learning partial regions, a plurality of first partial region candidates corresponding to a first state among the plurality of states and a plurality of second partial region candidates corresponding to a second state among the plurality of states; extracting a plurality of first partial regions corresponding to the first state from the plurality of first partial region candidates such that distances from a boundary with an outside of the organ region are distributed among the first partial region candidates; extracting a plurality of second partial regions corresponding to the second state from the plurality of second partial region candidates such that the distances from the boundary are distributed among the second partial region candidates; generating a first projection image group including a plurality of first projection images generated by minimum value projection or maximum value projection of the value of each voxel of the corresponding first partial region in the plurality of directions for each of the plurality of first partial regions; generating a second projection image group including a plurality of second projection images generated by minimum value projection or maximum value projection of the value of each voxel of the corresponding second partial region in the plurality of directions for each of the plurality of second partial regions; performing machine learning using the first projection image group corresponding to each of the plurality of first partial regions as learning data corresponding to the first state, and using the second projection image group corresponding to each of the plurality of second partial regions as learning data corresponding to the second state; generated by a model generation process including Image recognition device.

6. An information processing system having a learning processing unit and an image classification unit, The learning processing unit a plurality of learning partial regions, each of which is a three-dimensional region of a predetermined size, are extracted from an organ region including a specific organ in three-dimensional learning volume data generated based on a plurality of learning tomographic images obtained by photographing the inside of a human body; extracting a plurality of first partial region candidates corresponding to a first state and a plurality of second partial region candidates corresponding to a second state from the plurality of learning partial regions; extracting a plurality of first partial regions corresponding to the first state from the plurality of first partial region candidates such that distances from a boundary with an outside of the organ region are distributed among the first partial region candidates; extracting a plurality of second partial regions corresponding to the second state from the plurality of second partial region candidates such that the distances from the boundary are distributed among the second partial region candidates; generating a first projection image group including a plurality of first projection images generated by minimum value projection or maximum value projection of the value of each voxel of the corresponding first partial region in a plurality of directions orthogonal to each other, for each of the plurality of first partial regions; generating a second projection image group including a plurality of second projection images generated by minimum value projection or maximum value projection of the value of each voxel of the corresponding second partial region in the plurality of directions for each of the plurality of second partial regions; performing machine learning using the first projection image group corresponding to each of the plurality of first partial regions as learning data corresponding to the first state and the second projection image group corresponding to each of the plurality of second partial regions as learning data corresponding to the second state, thereby generating a learning model for identifying whether the state is the first state or the second state; The image identification unit extracting a classification partial region, which is a three-dimensional region of the predetermined size, from three-dimensional classification volume data generated based on a plurality of classification tomographic images obtained by photographing the inside of a human body; generating a plurality of third projection images by minimum value projection or maximum value projection of the value of each voxel of the identification partial region in the plurality of directions; using the learning model to identify whether the classification partial region is in the first state or the second state based on the plurality of third projection images; Information processing system.

Images