Training data generation method and abnormality determination method

Abstract

To facilitate determination of abnormalities occurring in various modes in parts of an object from a three-dimensional image of the object.SOLUTION: A training data generation method for use in machine learning for determining abnormalities in an object from a three-dimensional image of the object, includes: generating a plurality of two-dimensional images (41-43) of a specific image size, the partial images of which are projected on a two-dimensional plane from a plurality of directions on the basis of the partial images (33) cropped from a three-dimensional image (32) to include a portion of the object and a vicinity thereof; generating a superimposed image (34) by superimposing images (51 and 52) representing a position of each pixel in the images of the specific image size on a plurality of two-dimensional images; and associating the superimposed image with abnormality information obtained by determining abnormalities in parts of the object, to generate training data to be used for generating a trained model by machine learning. The trained model outputs abnormality information of parts of the input image from an input image being input as the superimposed image of the object.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to a method for generating training data used in machine learning for determining an abnormality in an object from a three-dimensional image of the object, and a method for determining an abnormality. [Background technology]

[0002] Patent Document 1 discloses an image classification device that classifies each pixel of a 3D image containing an object such as the blood vessels of the heart, captured by a CT (Computed Tomography) device or the like, into multiple classes (e.g., coronary arteries and coronary veins). The image classification device of Patent Document 1 detects feature points (e.g., the aortic valve, mitral valve, and apex of the heart) contained in the 3D image and sets a reference axis in the 3D image based on the feature points. The image classification device of Patent Document 1 generates a 2D image by projecting the object contained in the 3D image in a specific projection direction using the reference axis as a reference, and classifies each pixel of the object into multiple classes based on the 2D image, thereby enabling accurate classification with a small amount of calculation. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2018-139693 [Non-patent literature]

[0004] [Non-Patent Document 1] Murase, R., Suganuma, M., & Okatani, T. (2020). How can CNNs use image position for segmentation?. arXiv preprint arXiv:2005.03463. Summary of the Invention [Problem to be solved by the invention]

[0005] The present invention aims to provide a method for generating training data to be used in machine learning, which can make it easier to determine abnormalities that occur in various ways in parts of an object from a three-dimensional image of the object, and a method for determining abnormalities. [Means for solving the problem]

[0006] One aspect of the present invention provides a method for generating training data used in machine learning to determine anomalies in an object from a 3D image of the object. The training data generation method includes generating a plurality of 2D images of a specific image size in which partial images of the object are projected from multiple directions onto a 2D plane based on partial images cut out from the 3D image so as to include the portion of the object and its vicinity; generating superimposed images by superimposing images showing the positions of each pixel in the image having the specific image size on the plurality of 2D images; and associating the superimposed images with anomaly information determining anomalies in the portion of the object to generate training data used to generate a trained model by machine learning. The trained model outputs anomaly information for the portion of the input image from the input image input as the superimposed image of the object.

[0007] Another aspect of the present invention provides an anomaly detection method for detecting an anomaly in an object from a 3D image of the object. The anomaly detection method includes generating a plurality of 2D images of a specific image size by projecting the partial images from a plurality of directions onto a 2D plane based on partial images cut out from the 3D image so as to include a portion of the object and its vicinity; generating superimposed images by superimposing an image showing a position within the image having the specific image size on the plurality of 2D images; and inputting the superimposed images into a trained model based on machine learning to output anomaly information indicating an anomaly in a portion of the object. The trained model is generated by machine learning based on training data that associates input images input as superimposed images of the object with anomaly information indicating an anomaly in a portion of the input image. [Effects of the Invention]

[0008] According to the present invention, based on an image cut out from a 3D image of an object, including a portion of the object and its vicinity, a superimposed image is generated by superimposing an image showing the position of each pixel on a plurality of 2D images obtained by projecting the image onto a 2D plane from a plurality of directions. This makes it possible to easily determine abnormalities occurring in various ways in portions of the object using the superimposed image generated from the 3D image of the object. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing the configuration of an imaging diagnostic system according to embodiment 1. [Figure 2] A block diagram illustrating the configuration of an image processing device in an image diagnostic system. [Figure 3] A diagram for explaining a CT image of a diagnostic target in an image diagnostic system. [Figure 4] 10 is a flowchart illustrating an example of an abnormality determination operation in an image processing device; [Figure 5] FIG. 10 is a diagram for explaining an abnormality determination operation in the image processing device. [Figure 6] 1 is a flowchart illustrating a process for generating a superimposed image in an image processing device; [Figure 7] FIG. 10 is a diagram for explaining a process for generating a superimposed image. [Figure 8] 1 is a flowchart illustrating a process for generating training data in an image processing device; [Figure 9] 1 is a flowchart illustrating a training process in an image processing device. [Figure 10] FIG. 10 shows the results of performance evaluation of a part classification model and an abnormality determination model in an image processing device. [Figure 11] FIG. 10 is a diagram for explaining a two-dimensional image according to a first modification of the first embodiment. [Figure 12] FIG. 10 is a diagram for explaining a two-dimensional image according to a second modification of the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, a method for generating training data and a method for determining an anomaly according to an embodiment of the present invention will be described with reference to the accompanying drawings. In the following description, like components are designated by like reference numerals, and redundant descriptions thereof will be omitted as appropriate.

[0011] (Embodiment 1) In this embodiment, an image processing device that executes a training data generation method and an abnormality determination method according to one embodiment of the present invention, and an image diagnostic system that includes the image processing device will be described.

[0012] 1. Configuration The configuration of the diagnostic imaging system according to the first embodiment will be described with reference to FIGS.

[0013] 1-1. System Overview FIG. 1 is a diagram showing the configuration of an imaging diagnostic system 100 according to this embodiment. The imaging diagnostic system 100 in FIG. 1 includes a CT device 10 that captures CT (Computed Tomography) images using, for example, a microfocus X-ray tube, and an image processing device 20 that functions as an example of an information processing device according to the present invention, such as by performing image processing on the CT images. The CT image is an example of a three-dimensional image in this embodiment. The CT device 10 generates image data representing the captured CT image. The image data of the CT image is acquired by the image processing device 20 from, for example, the CT device 10 via a communication network or the like.

[0014] This system 100 is applied to image diagnosis, for example, to determine abnormalities within a living organism, such as a rat, from CT images taken of the organism using the CT device 10. Applications of such image diagnosis include developmental toxicity testing of compounds such as pesticides on the skeleton of a living organism. In developmental toxicity testing, dissection of fetuses born from rat mothers administered the compound to prepare skeletal specimens to evaluate abnormalities such as bone teratogenicity requires several weeks of experimentation and stable procedures. For example, using CT images obtained using microfocus X-rays or the like to determine abnormalities within the rat's body can shorten experiment time, reduce procedures, and enable observation of the rat in a live state.

[0015] However, there is a concern that visual evaluation of CT images may be difficult to achieve consistently and efficiently. For example, the abnormalities evaluated in skeletal developmental toxicity tests include a variety of types, such as bone loss, splitting, deformation, emergence, and fusion, and the locations of abnormalities within the entire skeleton also vary. For this reason, evaluators must observe individual bones by, for example, rotating, zooming in, and zooming out on CT images obtained as 3D images. The present system 100 enables consistent and efficient evaluation of CT images through image analysis of the CT images.

[0016] 1-2. Image processing device configuration An example of the configuration of the image processing device 20 in this system 100 will be described with reference to FIG.

[0017] 2 is a block diagram illustrating the configuration of the image processing device 20. The image processing device 20 is configured by, for example, a computer.

[0018] 2 includes a CPU (Central Processing Unit) 21 that performs arithmetic processing, a storage device 22 that stores various data and computer programs, and an input / output interface 26 for data communication with other devices. Hereinafter, interface will be abbreviated as "I / F."

[0019] The CPU 21, which is an example of a processor, executes a control program 23 stored in the storage device 22 to realize predetermined functions including the execution of a training data generation method and an anomaly determination method according to the present invention. The control program 23 may be provided via a communication network. The image processing device 20 is not limited to the CPU 21 and may include various processors such as an MPU or a GPU, or may include one or more processors.

[0020] The storage device 22 is configured with a semiconductor storage device such as a hard disk drive (HDD) or a solid state drive (SSD), and stores the control program 23, etc. The storage device 22 may store models trained by machine learning, such as a body part classification model 24 and an abnormality determination model described below, along with parameters of the models and related programs, etc. The storage device 22 may include a temporary storage element configured with a RAM such as a DRAM or an SRAM, and may function as a work area for the CPU 21.

[0021] The input / output I / F 26 is a communication circuit that performs communication in accordance with various standards, such as IEEE802.11, 4G, or 5G. The input / output I / F 26 may perform wired communication in accordance with standards such as Ethernet (registered trademark). The input / output I / F 26 is connectable to a communication network, such as an intranet or the Internet. The image processing device 20 may communicate directly with other devices via the input / output I / F 26, or may communicate via an access point. The input / output I / F 26 may include a communication circuit that allows communication with other devices without going through a communication network, and may be configured as a communication interface including a connection terminal such as USB or HDMI (registered trademark).

[0022] 1-3.About CT images A CT image for which image diagnosis is performed in the image diagnosis system 100 of this embodiment will be described with reference to FIG.

[0023] 3 is a diagram illustrating a CT image 30 of a diagnostic object in the diagnostic imaging system 100. The CT image 30 is obtained by imaging, for example, a rat using the CT device 10. In FIG. 3, the rat's skeleton appears bright in the CT image 30.

[0024] The CT image 30 is obtained as a three-dimensional image in which positions within the image are represented in a Cartesian coordinate system consisting of three mutually orthogonal axes, X, Y, and Z. In the CT image 30, for example, the CT value at each position is reflected in the shade on the image.

[0025] 2.Operation The operation of the diagnostic imaging system 100 configured as above will be described below.

[0026] 2-1. Abnormality determination In the image diagnostic system 100 of this embodiment, the image processing device 20 judges abnormalities in a living organism such as a rat from a CT image 30 of the living organism. Such an abnormality judgment operation will be described with reference to FIGS.

[0027] Fig. 4 is a flowchart illustrating an abnormality determination operation in the image processing device 20. Each process in this flowchart is executed by, for example, the CPU 21 of the image processing device 20. Fig. 5 is a diagram for explaining the abnormality determination operation in the image processing device 20.

[0028] First, the CPU 21 acquires image data of a CT image 30 from, for example, the CT device 10 via the input / output I / F 26 (S1). The CT image 30 is acquired as image data of a three-dimensional image and is stored, for example, in the storage device 22 of the image processing device 20. It is not essential that the CPU 21 acquires the image data directly from the CT device 10. For example, the image data may be stored in a storage device on a communication network (not shown) from the CT device 10, and the CPU 21 may acquire the image data from the storage device via the input / output I / F 26.

[0029] Next, the CPU 21 performs a process of separating individual bones in the CT image 30 (S2). An example of a CT image 32 after separation is shown in Fig. 5. This process is performed by clustering using the k-nearest neighbor method or the like based on the CT values in the CT image 30. For example, the processes from step S3 onwards are performed for each separated bone site.

[0030] Based on the CT image 32 after the bones have been separated, the CPU 21 projects images cut out from the CT image 32 onto a two-dimensional plane from multiple directions for the target 11 to be diagnosed as having an abnormality as a bone of the diagnosis target, and generates a superimposed image 34 by superimposing the obtained two-dimensional images (S3). In this superimposed image generation process (S3), the CPU 21 cuts out an area including the target 11 and its vicinity from the CT image 32, and adds information indicating the position of each pixel in the image to the two-dimensional image obtained by projecting the cut-out images onto the two-dimensional plane, thereby generating the superimposed image 34. The superimposed image generation process (S3) will be described in detail later.

[0031] CPU 21 classifies the bone parts of object 11 using the image classification of superimposed image 34 (S4). CPU 21 inputs superimposed image 34 to part classification model 24 stored in storage device 22, for example, and outputs the part classification result. Part classification model 24 is generated by a training process, which will be described later, as a trained model using machine learning for image classification. In the example of FIG. 5, the part in superimposed image 34 is classified as ribs.

[0032] The CPU 21 determines whether there is an abnormality in the bone of the object 11 based on the superimposed image 34, for example, according to the body part classified in step S4 (S5). In this embodiment, the CPU 21 determines whether the bone is abnormal, i.e., whether the bone is normal or abnormal (S5). The CPU 21 inputs the superimposed image 34 into an abnormality determination model 25-1 for ribs, which is stored in the storage device 22, for example, and outputs a determination result of normal or abnormal. In this embodiment, the abnormality determination models 25-1, 25-2, etc. for each body part are generated as trained models by machine learning through a training process described below. Hereinafter, the abnormality determination models 25-1, 25-2, etc. will be collectively referred to as abnormality determination models 25.

[0033] The CPU 21 outputs the determination result of step S5, for example, via the input / output I / F 26 (S6). The CPU 21 may output the determination result to, for example, a terminal device external to the image processing device 20, or may output the determination result to a display device such as a display. Furthermore, the image processing device 20 may be equipped with a display, and the determination result may be output to the display. Thereafter, the CPU 21 ends the processing of this flowchart.

[0034] According to the above process, a superimposed image 34 including the bones of the object of determination 11 and their vicinity is generated from the CT image 30 as a three-dimensional image (S3), and an abnormality is determined (S5) for the parts of the object of determination 11 classified in the superimposed image 34 (S4). In this way, an abnormality can be determined for the bones of the object of determination 11 using the superimposed image 34 generated from the three-dimensional image.

[0035] The above describes an example in which bones shown in the CT image 30 are separated in the image processing device 20 (S2). This processing may be performed by an information processing device or the like external to the image processing device 20. In this case, image data of the separated CT image 32 may be acquired in step S1, and the processing of step S2 may not be executed.

[0036] 2-1-1. Generation of superimposed images The details of the superimposed image generation process in step S3 of FIG. 4 will be described with reference to FIGS.

[0037] Fig. 6 is a flowchart illustrating the process (S3) of generating a superimposed image in the image processing device 20. Fig. 7 is a diagram for explaining the process (S3) of generating a superimposed image. Each process shown in the flowchart in Fig. 6 starts in a state where the CT image 32 from which the bones have been separated in step S2 of Fig. 4 is held in the storage device 22.

[0038] First, CPU 21 cuts out a region including, for example, the bone of object 11 and the vicinity of the bone from CT image 32 after bone separation (S31). The region including the vicinity of the bone is set to a predetermined range, for example, a cube with a side length of 160 pixels centered on object 11, from the viewpoint of including bones adjacent to the cut-out image, and multiple types of ranges may be set depending on the location of the bone. Fig. 7(A) shows an example of partial images 33 cut out from CT image 32 in step S31 for each of two bones of object 11.

[0039] When determining bone abnormalities using a CT image 32, if an image obtained by cutting out only the bones of the object of determination 11 is used, for example, the image may lack features for distinguishing the parts of the object of determination 11, making it difficult to perform accurate determination. In the image processing device 20 of this embodiment, a partial image 33 is cut out so as to include the bones surrounding the object of determination 11, so that information that makes it easier to distinguish the parts, such as the arrangement of the bones, can be reflected in the image used for determination. Furthermore, with such a partial image 33, abnormalities that are difficult to determine based on bone morphology alone, such as bone loss (or disappearance) and appearance, can be expressed as features on the image based on the relationship with the arrangement of surrounding bones, etc.

[0040] Next, based on image data of partial image 33 cut out from CT image 32 as a three-dimensional image, CPU 21 generates two-dimensional images by projecting partial image 33 onto a two-dimensional plane from three directions (S32). Fig. 7(B) shows an example in which partial image 33 is projected onto a two-dimensional plane from three mutually orthogonal directions (XY directions, YZ directions, and ZX directions in the figure) to generate two-dimensional images 41, 42, and 43. The XY, YZ, and ZX directions shown in Fig. 7(B) are directions in which the XY plane, YZ plane, and ZX plane in the coordinate axes similar to those of CT image 30 in Fig. 3 are viewed from above, respectively. CPU 21 generates each of two-dimensional images 41 to 43 at a common image size, for example, depending on the range from which partial image 33 was cut out in step S31 (S32).

[0041] The projection of the partial image 33 onto a two-dimensional plane in step S32 can be performed using a projection method such as maximum intensity projection (MIP), digitally reconstructed radiographs (DRR), average projection, or volume rendering. Alternatively, the two-dimensional images 41-43 may be generated at any cross section using multi-planar reconstruction (MPR). The two-dimensional images 41-43 obtained by projecting the three-dimensional image 33 from multiple directions can be treated as two-dimensional images in image processing for abnormality detection while retaining three-dimensional information, thereby reducing the processing load and data volume of the CPU 21, etc.

[0042] The CPU 21 generates a superimposed image 34 by superimposing each of the two-dimensional images 41 to 43 from three directions and a position image having the same image size as each of the two-dimensional images 41 to 43 (S33). The position image is information indicating the position of each pixel in a two-dimensional image of that image size. Fig. 7(C) shows an example in which the superimposed image 34 is generated by superimposing each of the two-dimensional images 41 to 43 with position images 51 and 52 indicating the position of the image in the width direction and height direction, respectively. For example, the superimposed image 34 is generated as a five-channel image including the three two-dimensional images 41 to 43 and the two position images 51 and 52.

[0043] The pixel values I1 and I2 of the position images 51 and 52 are expressed by the following equations. In the following equations, i and j represent the coordinates in the width and height directions, respectively, of images of the same image size as the two-dimensional images 41 to 43. W and H represent the width and height of the image, respectively. λ is an adjustment value that adjusts the weight of each pixel value of the position images 51 and 52 in the superimposed image 34.

number

[0044] In the superimposed image 34 of this embodiment, by superimposing the position images 51 and 52, it is possible to explicitly include information indicating the position of each pixel in the image. For example, in medical images such as CT images 30, subjects are often photographed at similar positions and sizes, and such information indicating positions within the image is expected to improve the accuracy and robustness of image recognition processing (see, for example, Non-Patent Document 1). The position images 51 and 52 allow the superimposed image 34 to reflect information such as the positional relationship between the bone of the object of determination 11 and surrounding bones, making it possible to perform abnormality determination using the superimposed image 34.

[0045] In FIG. 6, the CPU 21 generates the superimposed image 34 (S33), and then ends the processing of this flowchart.

[0046] According to the above process, a region including the vicinity of the bone is extracted from the CT image 32 (S31), and the extracted partial image 33 is projected onto a two-dimensional plane from three directions to generate two-dimensional images 41-43 (S32). Then, each of the two-dimensional images 41-43 is superimposed on the position images 51 and 52 to generate a superimposed image 34 (S33). As a result, for example, a range including nearby bones for the bones of the object of determination 11 can be included in the superimposed image 34, and further, by adding the position images 51 and 52 indicating the position of each image, the positional relationship between the bones, etc., can be reflected in the superimposed image 34.

[0047] Furthermore, by superimposing two-dimensional images 41 to 43 obtained by projecting a three-dimensional CT image 32 from multiple directions, it is possible to utilize the information of the three-dimensional image while treating the superimposed image 34 as a two-dimensional image, for example, when determining abnormalities. In this way, it is possible to easily determine abnormalities occurring in various ways in the bones of various parts of a living organism such as a rat from a three-dimensional image of the living organism.

[0048] 2-2. Training data generation process The image processing device 20 of this embodiment generates training data for machine learning to construct the region classification model 24 and the abnormality determination model 25 used in the processes of steps S4 and S5 in Fig. 4. The training data generation process will be described with reference to Fig. 8.

[0049] 8 is a flowchart illustrating a process for generating training data in the image processing device 20. The process of this flowchart is executed, for example, before a training process (FIG. 9) described later, and each step is executed by the CPU 21.

[0050] First, the CPU 21 acquires a data set including a plurality of CT images 30 that have been captured by, for example, the CT device 10 and stored in the storage device 22 (S11). For example, the data set includes each CT image 30 associated with information such as the location of each bone shown in each CT image 30 and a label indicating whether each bone is normal or abnormal.

[0051] Next, the CPU 21 performs a process of separating bones in each CT image 30 of the acquired data set, similar to, for example, step S2 in FIG. 4 (S12).

[0052] The CPU 21 determines whether or not there is a CT image 30 in the data set for which the superimposed image 34 has not yet been generated (i.e., the next step S3 has not been executed) (S13). For example, a flag or the like indicating whether or not the superimposed image 34 has been generated may be associated with and managed for each CT image 30 in the data set.

[0053] If there is a CT image 30 for which a superimposed image 34 has not yet been generated (YES in S13), the CPU 21 selects, for example, one CT image 30, and executes processing (S3) to generate a superimposed image for each bone that may be the separated determination target 11 in the CT image 30. Thereafter, the CPU 21 performs the determination of step S13 again, and repeats the processing of steps S13 and S3 while there is a CT image 30 for which a superimposed image 34 has not yet been generated (YES in S13).

[0054] On the other hand, if the data set does not contain a CT image 30 for which a superimposed image 34 has not yet been generated (NO in S13), the CPU 21 refers to the information included in the data set and generates training data for part classification that associates each superimposed image 34 with the bone part in that superimposed image 34 (S14). In this embodiment, the training data for part classification is generated by associating, for example, each superimposed image 34 with a label indicating whether or not it corresponds to that part, for each bone part.

[0055] Furthermore, the CPU 21 generates training data for abnormality determination in which each superimposed image 34 is associated with a label indicating whether the bone in the superimposed image 34 is normal or abnormal (S15). In this embodiment, the training data for abnormality determination is generated, for example, for each part of the bone.

[0056] Thereafter, the CPU 21 ends the processing of this flowchart. The processing of steps S14 and S15 may be executed in the reverse order to that of the example of FIG.

[0057] According to the above processing, from the data set of CT images 30, for example, superimposed images 34 are generated (S3) for bones that can be determination targets 11 in each CT image 30, and training data for site classification and training data for abnormality determination are generated (S14, 15). Using this training data, machine learning can be performed for site classification model 24 that classifies bone sites from superimposed images 34, and abnormality determination model 25 that determines bone abnormalities from superimposed images 34.

[0058] In the above processing, it is not essential that the processing of step S14 be performed before step S15. For example, step S14 may be performed after step S15, or steps S14 and S15 may be performed in parallel. Furthermore, step S14 may be performed, for example, in an information processing device external to image processing device 20. Furthermore, for example, if abnormality determination model 25 is generated as a single model that can be commonly applied to superimposed images 34 of all parts, rather than for each part, step S14 may not be performed, and training data for part classification may not be generated.

[0059] 2-3. Training process The process of generating the region classification model 24 and the abnormality determination model 25 by machine learning using the training data generated as described above in the image processing device 20 of this embodiment will be described with reference to FIG.

[0060] 9 is a flowchart illustrating training processing in the image processing device 20. The processing of this flowchart is started, for example, in a state where the training data generated by the processing of FIG. 8 is stored in the storage device 22. Each processing of this flowchart is executed by the CPU 21, for example, before the operation of abnormality determination shown in FIG.

[0061] CPU 21 acquires training data for body part classification from storage device 22 (S21). CPU 21 generates body part classification model 24 by machine learning based on the training data (S22). Body part classification model 24 is generated as an image classification model by deep learning using, for example, a Convolutional Neural Network (CNN). Body part classification model 24 of this embodiment is configured, for example, for each body part, by combining multiple binary classification models that classify an input image as either that body part or another body part. The classification results of body part classification model 24 may be determined, for example, by using the output of the final layer of the CNN in each binary classification model as the confidence level of the classification result, and the classification result of the model with the highest confidence level may be used.

[0062] Furthermore, the CPU 21 acquires training data for abnormality determination from the storage device 22 (S23). The CPU 21 generates an abnormality determination model 25 for each part by machine learning based on the training data (S24). Each abnormality determination model 25 is generated as an image classification model by deep learning using CNN, for example, so as to classify an input image as normal or abnormal.

[0063] After generating the abnormality determination model 25 (S24), the CPU 21 ends the processing of this flowchart.

[0064] According to the above processing, part classification model 24 and abnormality determination model 25 are generated as trained models by machine learning using the respective training data for part classification and abnormality determination (S22, S24). With these trained models 24, 25, in image processing device 20, for example, in accordance with the result of classifying a bone part by part classification model 24, abnormality determination can be performed by abnormality determination model 25 for that part (S4, S5 in FIG. 4).

[0065] In the above processing, it is not essential to execute the processing of steps S21 and S23 before steps S23 and S24. The execution order of steps S21 to S24 is not limited to the above example, and for example, steps S21 and S22 may be executed after steps S23 and S24 are executed, or steps S21 and S22 and S23 and S24 may be executed in parallel.

[0066] 3. Effects etc. As described above, in this embodiment, the training data generation process (S11 to S15 in FIG. 8 ) provides a method for generating training data used in machine learning to identify abnormalities in a living organism (an example of an object), such as a rat, from a CT image 30 (an example of a three-dimensional image) of the living organism. This method includes generating a plurality of two-dimensional images 41 to 43 of a specific image size by projecting the partial images 33 onto a two-dimensional plane from multiple directions, based on the partial images 33 cut out from the CT image 30 (including a CT image 32 obtained by separating the bones from the CT image 30) so as to include bones, which are an example of parts of the living organism, and areas near the bones (S3, S31 to S32). This method also includes generating a superimposed image 34 by superimposing position images 51 and 52, which are an example of images indicating the position of each pixel in the image having the specific image size, onto the plurality of two-dimensional images 41 to 43 (S3, S33). This method also includes generating (S15) training data used to generate an abnormality determination model 25 as an example of a trained model by machine learning by associating the superimposed image 34 with whether the bones in the living body are normal or abnormal (an example of abnormality information that determines whether an abnormality exists). The abnormality determination model 25 outputs a determination result of whether the bones in the input image are normal or abnormal, based on the input image input as the superimposed image 34 of the living body.

[0067] According to the above-described method for generating training data, a superimposed image 34 is generated by superimposing position images 51 and 52 on two-dimensional images 41 to 43 obtained by projecting from multiple directions partial image 33, including bones and their vicinity, extracted from a three-dimensional CT image 30. Such superimposed image 34 can reflect information such as the positional relationship between nearby bones in the image, and therefore an abnormality determination model 25 can be generated from training data using superimposed image 34, making it easier to determine abnormalities occurring in various ways in bones from CT image 30.

[0068] In this embodiment, the multiple directions when projecting partial image 33 onto a two-dimensional plane include, for example, three mutually orthogonal directions, the XY direction, the YZ direction, and the ZX direction, as shown in Fig. 7(B). In this way, by projecting from multiple directions, three-dimensional information in partial image 33 can be included in each of two-dimensional images 41 to 43. Furthermore, for example, when partial image 33 is cut out within a cubic area, it is easy to generate each of two-dimensional images 41 to 43 so that they have a common image size.

[0069] In this embodiment, the abnormality information indicates whether or not a bone in a living body is abnormal, and the abnormality determination model 25 classifies an input image according to whether or not the bone in the input image is abnormal (S15, S5). As a result, it is possible to determine whether the bone is normal or abnormal from the input image by image analysis using the abnormality determination model 25.

[0070] In this embodiment, training data for abnormality determination is generated for multiple bone sites (an example of one or more types of parts) in a living body (S15), and abnormality determination models 25 are generated for each site based on the training data for each site (S23, S24). As a result, for example, each abnormality determination model 25 only needs to determine abnormalities for one site, and can be generated so as to be able to perform determination with high accuracy even with a small amount of training data.

[0071] In this embodiment, generating a plurality of two-dimensional images 41-43 from a partial image 33 of a three-dimensional image includes generating each two-dimensional image by maximum intensity projection, average projection, volume rendering, or multiplanar reconstruction (S3, S32). Such generation of two-dimensional images from three-dimensional images is not limited to the above examples, and various image reconstruction techniques can be used.

[0072] The training process (S21 to S25 in FIG. 9) in this embodiment provides a method for generating a trained model. The training process includes generating an abnormality determination model 25 (an example of a trained model) by performing machine learning based on training data for abnormality determination generated by the training data generation method described above (S23, S24). This makes it possible to generate an abnormality determination model 25 that can easily determine abnormalities that occur in various ways in the bones of a living body from three-dimensional images such as CT images 30, based on the training data for abnormality determination that associates superimposed images 34 with abnormality information such as whether the bone is abnormal or normal.

[0073] As described above, in this embodiment, the abnormality determination operations (S1 to S6 in FIG. 4) in the image processing device 20 provide an abnormality determination method for determining an abnormality in a living organism (an example of an object), such as a rat, from a CT image 30 (an example of a three-dimensional image) of the living organism. This method includes generating, in a specific image size, a plurality of two-dimensional images 41 to 43 in which the partial images 33 are projected onto a two-dimensional plane from a plurality of directions, based on the partial images 33 cut out from the CT image 30 (including a CT image 32 in which the bones are separated from the CT image 30) so as to include bones, which are an example of parts of the living organism, and the vicinity of the bones (S3, S31 to S32). This method also includes generating a superimposed image 34 in which position images 51 and 52, which are an example of images indicating the position of each pixel in the image having the specific image size, are superimposed on the plurality of two-dimensional images 41 to 43 (S3, S33). This method includes inputting the superimposed image 34 as an example of a trained model by machine learning into an abnormality determination model 25, and outputting a determination result of abnormality or normality (an example of abnormality information) that determines an abnormality in the bones of the living body (S5, S6). The abnormality determination model 25 is generated by machine learning based on abnormality determination training data that associates an input image input as the superimposed image 34 of the living body with abnormality information on the bones in the input image (S24, S23, S15). This abnormality determination method using the superimposed image 34 makes it easier to determine abnormalities that occur in various ways in the bones of the living body from the CT image 30.

[0074] The control program 23 in this embodiment is an example of a computer program for causing the CPU 21 (an example of a processor) of the image processing device 20 (an example of a computer) to execute at least one of the above-mentioned training data generation method, learned model generation method, and anomaly determination method.

[0075] The image processing device 20 in this embodiment is an example of an information processing device that includes a CPU 21 (an example of a processor) and a storage device 22 that stores a control program 23 executed by the CPU 21.

[0076] According to the control program 23 and image processing device 20 described above, the method for generating training data, the method for generating a learned model, and / or the method for determining abnormalities in this embodiment can be executed, making it easier to determine abnormalities that occur in various ways in the bones of a living body from the CT image 30.

[0077] (Example) An example of the first embodiment will be described with reference to FIG.

[0078] In this example, a site classification model 24 and an abnormality determination model 25 were constructed by machine learning for each of the rat's skull, nasal bone, cervical vertebrae, and ribs as diagnostic targets, and the performance of each was evaluated.

[0079] In this performance evaluation, the accuracy and recall of each model 24, 25 were evaluated using test data in which superimposed images 34 were associated with the correct answers for parts or normal and abnormal, separate from the training data used to construct the models. For example, the recall of anomaly detection indicates the degree to which the determination results by the anomaly detection model 25 reproduce the correct answer among the data in the test data for which the correct answer was anomaly. In particular, in anomaly detection, it is practically important to reduce the number of false negatives, where anomalies are mistaken for normal, and an increase in the recall is required.

[0080] The results of this performance evaluation are shown in Figure 10. In Figure 10, the abnormality determination results for the skull, for which no abnormality data was available, are omitted. As shown in Figure 10, in both the body part classification model 24 and the abnormality determination model 25, the accuracy rate for each body part was over 95%, confirming that accurate classification or determination is possible. Furthermore, the recall rate for abnormality determination was relatively high, at 100% for the nasal bone and ribs, and approximately 85% for the tibia, suggesting that the number of false negatives has been reduced.

[0081] (Other embodiments) As described above, the first embodiment has been described as an example of the present invention. However, the present invention is not limited to these embodiments, and other embodiments may exist. Other embodiments will be described below.

[0082] In the first embodiment described above, as shown in FIG. 7B, for example, two-dimensional images 41, 42, and 43 are generated from a partial image 33 of a CT image 32 as a three-dimensional image by projecting them onto three cross sections along the mutually orthogonal X, Y, and Z axes. Projections onto two-dimensional planes such as the cross sections are not limited to the three directions shown in FIG. 7B. Such modifications of the first embodiment will be described with reference to FIGS. 11 and 12. FIGS. 11 and 12 are diagrams for explaining two-dimensional images in first and second modifications of the first embodiment, respectively. Below, an example will be described in which a partial image 33 similar to that shown in FIG. 7B can be regarded as a cube having three axes X, Y, and Z of a Cartesian coordinate system.

[0083] Fig. 11 shows an example of projecting partial image 33 in six directions, i.e., two diagonal directions in each of the XY plane, YZ plane, and ZX plane. In the example of Fig. 11, partial image 33 is projected from each direction onto a two-dimensional plane that maximizes the area of a cube representing partial image 33, with each diagonal line serving as a perpendicular line.

[0084] Fig. 12 shows an example of projecting partial image 33 in four directions different from the three directions shown in Fig. 7(B) and the six directions shown in Fig. 11. In the example of Fig. 12, the +Y direction is upward and the -Y direction is downward, and the four directions are directions from each vertex on the top surface of a cube showing partial image 33 to the farthest vertex among the vertices on the bottom surface. In the example of Fig. 12, partial image 33 is projected from each direction onto a two-dimensional plane that is orthogonal to each direction and shares two vertices with each vertex on the top or bottom surface of the cube.

[0085] For example, in the superimposed image generation process (S3) similar to that of the first embodiment, the CPU 21 generates two-dimensional images by projecting the partial image 33 onto a two-dimensional plane from the six directions in FIG. 11 or the four directions in FIG. 12, instead of the three directions in step S32 in FIG. 6. Alternatively, the CPU 21 may generate two-dimensional images from nine directions by adding the six directions in FIG. 11 to the three directions in step S32, seven directions by adding the four directions in FIG. 12 to the three directions, or thirteen directions by adding the four directions in FIG. 12 to the nine directions. Furthermore, from these multiple directions, two or more directions (e.g., three directions) that experimentally improve the accuracy of anomaly detection may be selected, and two-dimensional images from those directions may be generated. A mechanism for automatically setting the number of directions that maximizes classification accuracy may be provided.

[0086] As described above, in each of the modified examples, the multiple directions in which the partial image 33 is projected onto the two-dimensional plane include two or more directions selected from the three mutually orthogonal directions shown in Fig. 7(B) and a direction different from the three directions. For example, multiple directions can be selected and used to make it easier to determine abnormalities from the two-dimensional image obtained by projection.

[0087] In the above embodiments, an example has been described in which the region classification model 24 of the image processing device 20 is configured as a binary classification model for each region. In this embodiment, the region classification model 24 may be generated as a multi-value classification model that classifies three or more regions. In this case, for example, in the training data generation process (FIG. 8), the superimposed image 34 is associated with a label indicating each region to generate training data for region classification (S14).

[0088] In each of the above embodiments, an example has been described in which the abnormality determination model 25 of the image processing device 20 is generated as abnormality determination models 25-1, 25-2, etc. for each body part. In the present embodiment, the abnormality determination model 25 may be generated as a binary classification model that determines whether a plurality of body parts are abnormal or normal. In this case, the abnormality determination process (S5 in FIG. 4), the training data generation process (FIG. 8), and the training process (FIG. 9) are executed so that the superimposed image 34 of the plurality of body parts is used as the input image for the abnormality determination model 25.

[0089] In the above-described embodiments, examples have been described in which the abnormality determination model 25 determines whether or not the bone of the object 11 is abnormal, i.e., whether the bone is normal or abnormal. In the present embodiment, the abnormality determination model 25 may determine the type of abnormality in addition to whether or not the bone is abnormal. In the example of embodiment 1, in the training data for abnormality determination, each superimposed image 34 may be associated with a type of abnormality, such as bone loss, splitting, deformation, appearance, or fusion.

[0090] In the above embodiments, examples have been described in which the region classification model 24 and the anomaly determination model 25 are generated as image classification models using CNN. The models 24 and 25 are not limited to CNN and may be realized using various image recognition technologies. For example, machine learning such as SVM may be performed using features extracted from the superimposed image 34 by feature extraction. The anomaly determination model 25 is not limited to image classification and may be a trained model generated by machine learning to detect areas in an image that may be determined to be abnormal, or classify each pixel into such areas.

[0091] In each of the above embodiments, an example has been described in which a predetermined range is cut out (S31) from the CT image 32 in the process of generating a superimposed image (S3). In this embodiment, for example, an area including the bones of the object 11 and its vicinity may be cut out by detecting an object in the image.

[0092] In the above-described embodiments, examples have been described in which the training process ( FIG. 9 ) and the abnormality determination operation ( FIG. 4 ) are executed in the image processing device 20. For example, the training process and / or the abnormality determination operation may be executed by an information processing device or the like external to the image processing device 20. In this case, the training data generated in the training data generation process ( FIG. 8 ) and / or the part classification model 24 and the abnormality determination model 25 generated by executing the training process may be output to the outside of the image processing device 20 via the input / output I / F 26.

[0093] In the above embodiments, an image diagnostic system 100 has been described that determines abnormalities in a living organism, such as a rat, based on a CT image 30 of the living organism captured by the CT device 10. The image diagnostic system of the present embodiment is not limited to CT images 30 captured by the CT device 10, and may be applied to image diagnosis using, for example, MRI images or three-dimensional microscope images. The image diagnostic system may also be applied to inspecting abnormalities in various non-living objects, such as product production lines or buildings, using three-dimensional (3D) scanner images of the objects. Furthermore, the image diagnostic system may be applied to various applications, such as face recognition using 3D scanner images of faces, trait evaluation using 3D scanner images of plants, and estimation of the binding state or physiological activity of molecules or proteins based on structural images of biomolecules.

[0094] For example, by extracting a partial image corresponding to a molecular substructure from a three-dimensional image showing the molecular structure as a three-dimensional structure, the partial image can be projected from multiple directions (e.g., three directions) to create a two-dimensional image that expresses the structural characteristics of the molecule as a three-dimensional structure. Furthermore, by adding positional information indicating the position within the image, such as the positional image in embodiment 1, to the resulting image, the positional relationship between atoms constituting the molecule can be emphasized. For example, the superimposed image with the added positional information can be used as a molecular descriptor of a molecule having physiological activity, and machine learning of a classification model that determines whether or not the molecule is physiologically active based on the molecular descriptor can be performed. A classification model trained in this way can be used to determine whether or not an unknown molecule is physiologically active based on the superimposed image of the molecule.

[0095] Furthermore, by extracting a portion of the binding site from a 3D image showing the state of a protein bound to other molecules as a complex 3D structure and creating a 3D partial image, the arrangement of the binding site can be expressed. Furthermore, for example, in a superimposed image generated by adding positional information similar to that described above, the positional relationships between atoms or molecules contained in the 3D structure can be emphasized. An image diagnostic system using such superimposed images that express structural features can be used, for example, to determine the quality of the binding state using various machine learning techniques.

[0096] Furthermore, by generating partial images by cutting out parts of each plant individual from an image of the whole or part of the individual plant taken with a 3D scanner or the like, the morphological characteristics of the plant can be expressed for each cut-out part. Furthermore, by adding position information to the partial images projected onto a two-dimensional plane, the positional relationship of each part, for example, according to the plant's stem, branch, or leaf, can be emphasized and expressed. Such images can be used, for example, by an image diagnostic system using machine learning or the like to determine whether the photographed plant is in an abnormal state, including diseased parts, or in a normal state without diseased parts.

[0097] Furthermore, an image of a person's face captured by a 3D scanner can be cut out from the entire face or portions corresponding to certain parts (e.g., eyes, nose, and mouth) and projected onto a two-dimensional plane from multiple directions (e.g., three directions), thereby expressing features such as facial shape. A superimposed image in which positional information is added to the image can emphasize the positional relationship according to the distance between parts, making it easier to accurately express the facial features of each person. An image diagnostic system using such superimposed images can be used, for example, as an authentication system that determines an abnormality when a face other than that of a specific individual is detected by facial recognition. The facial image is not limited to a 3D scanned image of the face, but can also be an image captured from three directions, for example. Instead of cutting out a partial image, the imaging range and imaging direction can be set.

[0098] In addition, partial images obtained by cutting out the entire structure or a portion of the structure from a 3D scanned image of various structures and projecting it onto a two-dimensional plane from multiple directions, such as three directions, can represent the structure or shape of the structure. Furthermore, partial images with added positional information can emphasize the positional relationship between the constituent parts of the structure. An image diagnostic system using such partial images may diagnose the presence or absence of abnormalities in the entire structure by, for example, detecting the presence or absence of abnormalities in the shape, etc., of each part of the structure.

[0099] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. In other words, embodiments obtained by combining technical means modified appropriately within the scope of the claims are also included in the technical scope of the present invention.

[0100] Aspects of the present disclosure The following describes exemplary aspects of the present disclosure.

[0101] A first aspect of the present disclosure is a method for generating training data used in machine learning to determine an abnormality in an object from a 3D image of the object. The training data generation method includes generating a plurality of 2D images of a specific image size in which partial images of the object are projected from multiple directions onto a 2D plane based on partial images cut out from the 3D image so as to include a portion of the object and its vicinity; generating superimposed images in which images indicating the positions of each pixel in the image having the specific image size are superimposed on the plurality of 2D images; and associating the superimposed images with anomaly information determining an abnormality in a portion of the object to generate training data used to generate a trained model by machine learning. The trained model outputs anomaly information for the portion in the input image from an input image input as a superimposed image of the object.

[0102] In a second aspect, in the method for generating training data according to the first aspect, the plurality of directions includes three directions that are orthogonal to each other.

[0103] In a third aspect, in the training data generation method of the first or second aspect, the anomaly information indicates whether a part of an object is abnormal or not, and the trained model classifies an input image based on whether a part of the input image is abnormal or not.

[0104] In a fourth aspect, in the training data generation method of any of the first to third aspects, training data is generated for each type for one or more types of parts of an object, and a trained model is generated for each type based on the training data for each type.

[0105] In a fifth aspect, in the training data generation method of the first, third or fourth aspect, the multiple directions include two or more directions selected from directions including three directions that are perpendicular to each other and a direction different from the three directions.

[0106] In a sixth aspect, in the training data generation method of any of the first to fifth aspects, generating multiple two-dimensional images from partial images of a three-dimensional image includes generating each two-dimensional image using maximum intensity projection, DRR, mean value projection, volume rendering, or multiplanar reconstruction.

[0107] The seventh aspect includes generating a trained model by performing machine learning based on training data generated by the training data generation method of any one of the first to sixth aspects.

[0108] An eighth aspect is an anomaly detection method for detecting an anomaly in an object from a three-dimensional image of the object. The anomaly detection method includes generating a plurality of two-dimensional images of a specific image size in which partial images are projected onto a two-dimensional plane from a plurality of directions based on partial images cut out from the three-dimensional image so as to include a portion of the object and the vicinity of the portion, generating superimposed images in which images showing positions within the images having the specific image size are superimposed on the plurality of two-dimensional images, and inputting the superimposed images into a trained model using machine learning to output anomaly information that determines an anomaly in a portion of the object. The trained model is generated by machine learning based on training data that associates input images input as superimposed images of the object with anomaly information that determines an anomaly in a portion of the input image.

[0109] In a ninth aspect, in the abnormality determination method according to the eighth aspect, the plurality of directions include three directions that are orthogonal to one another.

[0110] In a tenth aspect, in the anomaly determination method described in the eighth or ninth aspect, the anomaly information indicates whether or not a part of the object is abnormal, and the trained model classifies the input image based on whether or not a part of the input image is abnormal.

[0111] In an eleventh aspect, in the anomaly determination method described in any of the eighth to tenth aspects, training data is generated for each type of part of the object for one or more types of parts, and a trained model is generated for each type based on the training data for each type.

[0112] In a twelfth aspect, in the abnormality determination method described in the eighth, tenth or eleventh aspect, the multiple directions include two or more directions selected from directions including three directions that are perpendicular to each other and a direction different from the three directions.

[0113] In a thirteenth aspect, in the abnormality determination method described in any of the eighth to twelfth aspects, generating multiple two-dimensional images from partial images of a three-dimensional image includes generating each two-dimensional image using maximum value projection, DRR, mean value projection, volume rendering, or multiplanar reconstruction.

[0114] A fourteenth aspect is a computer program for causing a processor of a computer to execute the method according to any one of the first to thirteenth aspects.

[0115] A fifteenth aspect is an information processing device comprising a processor and a storage device that stores the computer program according to the fourteenth aspect, which is executed by the processor. [Industrial Applicability]

[0116] The present invention is applicable to a method for generating training data used in machine learning for determining an abnormality in an object from a three-dimensional image of the object, and to an abnormality determination method. [Explanation of symbols]

[0117] 100 Diagnostic Imaging System 10 CT device 20 Image processing device 21 CPU 22 Storage device 23 Control Program 24 Body Part Classification Model 25 Anomaly detection model 26 Input / Output Interface 30,32 CT images 33 Partial Image 34 Superimposed Images

Claims

1. 1. A method for generating training data used in machine learning to determine an abnormality in an object from a three-dimensional image of the object, comprising: generating a plurality of two-dimensional images of a specific image size by projecting the partial images onto a two-dimensional plane from a plurality of directions based on partial images cut out from the three-dimensional image so as to include a portion of the object and the vicinity of the portion; generating a superimposed image by superimposing an image indicating the position of each pixel in the image having the specific image size on the plurality of two-dimensional images; and generating training data to be used for generating a trained model by machine learning by associating the superimposed image with anomaly information that determines an anomaly in the portion of the object; The trained model outputs the anomaly information for the part in an input image input as a superimposed image of the object. How training data is generated.

2. The plurality of directions includes three directions that are orthogonal to each other. The method for generating training data according to claim 1 .

3. the anomaly information indicates whether the part of the object is abnormal; The trained model classifies the input image based on whether the portion of the input image is abnormal. The method for generating training data according to claim 1 .

4. the training data is generated for each type of part of one or more types of parts of the object; The trained model is generated for each type based on the training data for each type. The method for generating training data according to claim 1 .

5. The plurality of directions include two or more directions selected from three directions orthogonal to each other and a direction different from the three directions. The method for generating training data according to claim 1 .

6. Generating the plurality of two-dimensional images from the partial images of the three-dimensional image includes generating each of the two-dimensional images by maximum intensity projection, DRR, mean value projection, volume rendering, or multiplanar reconstruction. The method for generating training data according to claim 1 .

7. and generating the trained model by performing machine learning based on the training data generated by the training data generation method according to claim 1. How to generate a trained model.

8. 1. An abnormality determination method for determining an abnormality in an object from a three-dimensional image of the object, comprising: generating a plurality of two-dimensional images of a specific image size by projecting the partial images onto a two-dimensional plane from a plurality of directions based on partial images cut out from the three-dimensional image so as to include a portion of the object and the vicinity of the portion; generating a superimposed image by superimposing an image indicating a position within the image having the specific image size on the plurality of two-dimensional images; and inputting the superimposed image into a trained model based on machine learning, and outputting anomaly information that determines an anomaly in the part of the object; The trained model is generated by machine learning based on training data that associates an input image input as a superimposed image of the object with anomaly information that determines an anomaly in the part of the input image. Abnormality determination method.

9. The plurality of directions includes three directions that are orthogonal to each other. The abnormality determination method according to claim 8.

10. the anomaly information indicates whether the part of the object is abnormal; The trained model classifies the input image based on whether the portion of the input image is abnormal. The abnormality determination method according to claim 8.

11. the training data is generated for each type of part of one or more types of parts of the object; The trained model is generated for each type based on the training data for each type. The abnormality determination method according to claim 8.

12. The plurality of directions include two or more directions selected from three directions orthogonal to each other and a direction different from the three directions. The abnormality determination method according to claim 8.

13. Generating the plurality of two-dimensional images from the partial images of the three-dimensional image includes generating each of the two-dimensional images by maximum intensity projection, DRR, mean value projection, volume rendering, or multiplanar reconstruction. The abnormality determination method according to claim 8.

14. A computer program product for causing a computer processor to carry out the method of any one of claims 1 to 13.

15. a processor; and a storage device storing the computer program of claim 14 for execution by the processor. Information processing device.

Images