Image generation device, trained model for image generation, data structure for image generation, program for image generation, and diagnostic support method, for generating images in which objects a1 and an are respectively identified from biological image including two or more overlapping projected objects a1-n

Abstract

To provide an image generation device, a trained model for image generation, a data structure for image generation, a program for image generation, and a diagnostic support method, for generating images in which objects A1 and AN are respectively identified from a biological image including two or more overlapping projected objects A1-N.SOLUTION: In the present invention, a convolutional neural network having excellent image recognition is trained using original images created by pseudo-projecting digital phantoms including a plurality of overlapping projected objects and correct label images obtained by selectively projecting only the plurality of objects. In this way, a trained model for image generation is constructed that generates images in which objects A1 and AN are respectively identified from a biological image including two or more overlapping projected objects A1-N.SELECTED DRAWING: None

Description

[Technical Field]

[0001] The present invention is directed to a method for projecting two or more objects A 1-N From a biometric image including the object A1 and the object A N The present invention relates to an image generation device that generates images that identify each of the above, a trained model for image generation, a data structure for image generation, a program for image generation, and a diagnostic support method. [Background technology]

[0002] In chest X-ray examinations, lateral images are taken in addition to frontal images in order to evaluate the shape of the thorax, lungs, and diaphragm when viewed from the side. Recently developed dynamic chest X-ray examinations also evaluate dynamics and function based on changes in shape when viewed from the side. However, because the left and right lungs and diaphragm are projected overlapping to the side, it is difficult to evaluate them separately. Although technology has been developed to automatically recognize overlapping left and right lungs as a single region, separating the left and right lungs has been technically difficult (Non-Patent Documents 1 and 2).

[0003] A special MRI test (pseudo-dynamic MRI) has been proposed to assess dynamic and functional functions using lateral tomographic images (see Non-Patent Document 3). However, due to the cost and complexity, its introduction into routine clinical practice is not practical. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] https: / / aapm.onlinelibrary.wiley.com / doi / 10.1118 / 1.598331 [Non-patent document 2] https: / / ietresearch.onlinelibrary.wiley.com / doi / 10.1049 / iet-ipr.2016.0526 [Non-patent document 3] https: / / ietresearch.onlinelibrary.wiley.com / doi / 10.1049 / iet-ipr.2016.0526 Summary of the Invention [Problem to be solved by the invention]

[0005] In recent years, dynamic chest radiography (DCR) has been developed, enabling pulmonary function imaging diagnosis in a general radiography room. Diaphragmatic movement and changes in density within the lung field are quantified using DCR frontal images, which capture respiratory status, and are used to evaluate pulmonary function on a lung-by-lung basis. However, because the left and right lungs are projected overlapping on DCR lateral images, their use is limited to visual evaluation of thoracic movement. Therefore, if it were possible to distinguish between the left and right lungs on DCR lateral images, this would provide a useful index for quantitative evaluation of pulmonary function, expanding the scope of diagnosis. Therefore, the present invention provides a method for projecting two or more objects A that are overlapped. 1-N From a biometric image including the object A1 and the object A N The present invention aims to provide an image generation device that generates images that identify each of the above, a trained model for image generation, a data structure for image generation, a program for image generation, and a diagnostic support method. [Means for solving the problem]

[0006] It is extremely difficult to recognize and separate biological images containing multiple overlapping objects (especially images of the left and right lungs overlapping on a lateral chest X-ray image) with the naked eye. Therefore, in the present invention, a convolutional neural network (CNN) with excellent image recognition capabilities is trained with an original image created by pseudo-projecting a digital phantom containing multiple overlapping projected objects, and a ground truth label image in which only the multiple objects are selectively projected. 1-N From a biometric image including the object A1 and the object A N We constructed a trained model for image generation that generates images that identify each of the following: Furthermore, by using a lateral DCR taken of a biological image of the respiratory state as a trained model for image generation, it was confirmed that a DCR side image in which the left and right lungs are distinguished, and even a DCR side image in which the left and right lungs are separated, could be obtained, thereby completing the present invention.

[0007] 1. Two or more overlapping objects A 1-N From a biometric image including the object A1 and the object A N An image generating device that generates an image in which each of the following is identified: Object A 1-N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M and the original image X 1-M Object A 1-N The correct labeled image X is 1-M a learning data storage unit that stores the above as learning data; Original image X 1-M and the correct labeled image X 1-M Using the overlapping projections, the object A 1-N From the image containing the object A1 and the object A N a model construction unit that constructs a learning model for generating images that identify each of the above; Two or more overlapping projected objects A 1-N a biometric image receiving unit for receiving a biometric image including: The biometric image received by the biometric image receiving unit is applied to the learning model constructed by the model construction unit, whereby two or more overlapping projected objects A are projected. 1-N From a biometric image including the object A1 and the object A N an image generating unit that generates images that identify each of the An image generating device comprising: 2. Furthermore, the object A1 and the object A N 2. The image generating device according to claim 1, further comprising an image synthesis unit that synthesizes the identified images with a biometric image. 3. Furthermore, the object A1 and the object A NThe images that have been identified are combined with the biometric image to form the target A1 and target A2. N 2. The image generating device according to claim 1, further comprising a separated image generating unit that generates separated images of the above. 4. The model construction unit further constructs a projection image of two or more overlapping objects A. 1-N Two or more objects A are projected overlapping with the biological image of a normal case including 1-N 3. The image generating device according to claim 1 or 2, wherein the disease case Y can be identified using a biological image of the disease case Y including: 5. Furthermore, the object A1 and the object A N 3. The image generating device according to claim 1 or 2, further comprising a diagnostic support unit that matches the identified and / or separated images with a known disease medical database to present predicted diseases. 6. Original image X 1-M 3. The image generating device according to claim 1 or 2, wherein the image is generated by pseudo-projecting a virtual human phantom. 7. Correctly labeled image X 1-M is object A 1-N 3. The image generating device according to claim 1 or 2, which is configured only from the image. 8. A diagnostic support device according to paragraph 1 or 2 above, wherein the learning model is constructed from one or more of the following: 1)CNN 2) U-Net 3) Mask R-CNN 4) TransUnet 5) Swin U-net 6) Generative Adversarial Networks 9. Object A 1-N 3. The image generating device according to claim 1 or 2, wherein the combination is any one of the following: 1) Left lung (A1) and right lung (A2) 2) Left lung (A1), right lung (A2) and diaphragm (A3) 3) Left lung (A1), right lung (A2), and heart and great vessels (A3) 4) Left lung (A1), right lung (A2) and bronchus (A3) 10. Object A 1-Nare the left and right lungs, and the learning model is U-Net. 11. Two or more overlapping objects A 1-N From a biometric image including the object A1 and the object A N A trained model for image generation that generates images that identify each of the following: The trained model is 1) Object A as training data 1-N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M and the original image X 1-M Object A 1-N The correct labeled image X is 1-M , and 2) Original image X 1-M and the correct labeled image X 1-M By machine learning using 1-N From a biometric image including the object A1 and the object A N A learning model for generating images that identify each of the following: A pre-trained model for image generation. 12. Object A 1-N are the left and right lungs, and the training model is U-Net. 13. Two or more overlapping objects A 1-N From a biometric image including the object A1 and the object A N A data structure for image generation that generates an image in which each of the following is identified: The data structure is: 1) Object A as training data 1-N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M and the original image X 1-M Object A 1-N The correct labeled image X is 1-M , and 2) Original image X 1-M and the correct labeled image X 1-MBy machine learning using 1-N From a biometric image including the object A1 and the object A N The trained model data for generating images that identify each of the following is included. Data structures for image generation. 14. Object A 1-N are the left and right lungs, and the learning model is U-Net. 15. Two or more overlapping objects A 1-N From a biometric image including the object A1 and the object A N An image generating program for generating an image in which each of the following is identified: The program 1) Object A 1-N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M and the original image X 1-M Object A 1-N The correct labeled image X is 1-M By performing machine learning using 1-N From a biometric image including the object A1 and the object A N A trained model generation process for generating images that respectively identify the 2) Two or more overlapping projected objects A 1-N By applying a biological image including the target A1 and the target A to the trained model, N creating an image in which each of the A program for generating images. 16. Object A 1-N are the left and right lungs, and the learning model is U-Net. 17. Two or more overlapping objects A 1-N From a biometric image including the object A1 and the object A N A diagnostic support method for providing an image in which each of the following is identified: The method comprises: 1) Two or more overlapping projected objects A 1-NBy applying a biological image including 1-N From a biometric image including the object A1 and the object A N creating an image in which each of the images is identified; Here, the trained model is a) As training data, object A 1-N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M and the original image X 1-M Object A 1-N The correct labeled image X is 1-M , and b) The original image X 1-M and the correct labeled image X 1-M By machine learning using 1-N From a biometric image including the object A1 and the object A N A learning model for generating images that identify each of the following: Diagnostic support methods. 18. Object A 1-N are the left and right lungs, and the learning model is U-Net. 19. The trained model further includes: two or more overlapping projected objects A 1-N Two or more objects A are projected overlapping with the biological image of a normal case including 1-N 19. The diagnostic support method according to claim 17 or 18, wherein the method is capable of learning a biological image of disease case Y including the above and discriminating disease case Y. [Effects of the Invention]

[0008] Two or more overlapping projected objects A 1-N From a biometric image including the object A1 and the object A N The present invention provides an image generation device that generates images that identify each of the above, a trained model for image generation, a data structure for image generation, a program for image generation, and a diagnostic support method. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing an example of the functional arrangement of an image generating apparatus according to a first embodiment. [Figure 2] FIG. 10 is a block diagram showing an example of the functional arrangement of an image generating apparatus according to a second embodiment. [Figure 3] FIG. 10 is a block diagram showing an example of the functional arrangement of an image generating apparatus according to a third embodiment. [Figure 4] FIG. 10 is a block diagram showing an example of the functional arrangement of an image generating apparatus according to a fourth embodiment. [Figure 5] Original image and correct labeled image created by pseudoprojection onto a virtual human phantom. [Figure 6] Geometric arrangement in 2D projection [Figure 7] The flow of machine learning using U-Net. [Figure 8] Images obtained from the training model. DETAILED DESCRIPTION OF THE INVENTION

[0010] (Subject of the present invention) The present invention is directed to a method for projecting two or more objects A 1-N From a biometric image including the object A1 and the object A N The present invention relates to an image generating device that generates images in which each of the above is identified, and further to a separated image, a trained model for image generation, a data structure for image generation, an image generation program, and a diagnostic support method.

[0011] (First embodiment) A first embodiment of the present invention will be described below with reference to the accompanying drawings. 1-N From a biometric image including the object A1 and the object A N 1 is a block diagram showing an example of the functional configuration of an image generating device 1 that generates images in which the above-mentioned items are identified. As shown in Fig. 1, the image generating device 1A according to the first embodiment includes components (functional blocks) of a learning data storage unit 2, a model construction unit 3, a trained model 4, a biometric image acceptance unit 5, and an image generating unit 6. For convenience of explanation, the components are separated, but the components may be combined or may include other components. The same applies to the other embodiments described below. The above configuration can be configured by either hardware or software. For example, when configured by software, each of the above functional blocks is actually configured with a computer's CPU, RAM, ROM, etc., and is realized by the operation of a program stored in a recording medium such as RAM, ROM, a hard disk, or a semiconductor memory. Furthermore, each configuration does not need to be mounted on the same hardware, and part of the configuration may be mounted on another medium (e.g., cloud). The same applies to the other embodiments described below.

[0012] (biometric images) The term "biological image" as used herein is not particularly limited, but includes, for example, DCR images, plain X-ray images, and X-ray fluoroscopic images. Furthermore, in this specification, "two or more objects A that are projected to overlap" 1-N "Biological image including" is not limited to, but means an image in which objects such as organs, tissues (blood vessels, bronchi, skeleton) in the image are projected and superimposed, and examples include the following: 1) Left lung (A1) and right lung (A2) 2) Left lung (A1), right lung (A2) and diaphragm (A3) 3) Left lung (A1), right lung (A2), and heart and great vessels (A3) 4) Left lung (A1), right lung (A2) and bronchus (A3) In addition, A 1-N "N" in the above expression means an integer equal to or greater than 2. For example, if N is 2, there are two overlapping projected objects (A1, A2), and if N is 3, there are three overlapping projected objects (A1, A2, A3). N is selected from, for example, 2, 3, 4, 5, 6, 7, 8, 9, 10, 20, 50, 100, etc., but is not particularly limited.

[0013] Learning data storage unit 2 The learning data storage unit 2 stores the object A 1-N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M (Original image X 1-M Data group) and the original image X 1-M Object A 1-N The correct labeled image X is 1-M (Correct label image X 1-M Each of the image data is accepted as training data. Note that a portion of the training data (for example, about 20%) may be divided into verification data. Hereinafter, the training data will also include the verification data. The division method is not particularly limited, but the holdout method, cross-validation method, etc. can be used. In addition, X 1-M "M" in the above expression means an integer of 2 or more, for example, 10, 50, 100, 500, 1000, 2000, 3000, 4000, 4200, 5000, 10000, etc. In detail, when M is 5, X 1、 X 2、 X 3、 X 4、 It becomes X5.

[0014] (Virtual human phantom, 3D data of real human body) In this specification, the term "virtual human phantom" refers to a digital phantom that mathematically models a 4D CT scan of a real human body, with thousands of defined structures and parameterized models of cardiac and respiratory motion. By pseudoprojecting these structures, it is possible to generate a large amount of training data without irradiating a patient. The model may be a voxelized model, a boundary representation model, or the like. Preferred digital phantoms include the XCAT phantom (see Segars et al. Med. Phys. 37 (9), 2010), the MIRD phantom, and the VIP-Man, with the XCAT phantom being particularly preferred. Three-dimensional data of the actual human body can be obtained from CT and MRI.

[0015] (pseudo photography) The term "pseudo-radiography" as used herein refers to, but is not limited to, an X-ray simulator developed at Duke University, which can be used to project a created NURBS file two-dimensionally from a virtual radiation source (see: Realistic CT simulation using the 4D XCAT phantom. Med. Phys. 2008; 35(8): 3800-3808).

[0016] (Correct label image) In this specification, the "correct labeled image" refers to the object A 1-N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M Mark each object with a marker (e.g., color L ) means an image that has been given a color. Lの "L" means an integer of 2 or more, for example, selected from 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 20, 50, 100, etc. For example, object A 1-N If there are three objects, color 1 is assigned to object A1, color 2 to object A2, color 3 to object A3, color 4 to the overlapping area of ​​object A1 and object A2, color 5 to the overlapping area of ​​object A1 and object A3, color 6 to the overlapping area of ​​object A1, object A2 and object A3, and color 7 to the overlapping area of ​​object A1, object A2 and object A3. Furthermore, it is preferable that the color be applied to the entire surface (total surface area) of the object, but it can also be applied to the periphery or center of the object. Furthermore, the correct labeled image is preferably a projection of two or more objects A that are projected in a substantially overlapping manner. 1-N It is preferable that the image is composed of only two or more objects A that are projected in an overlapping manner. 1-N This means that the image may contain images that are unavoidably present in the image.

[0017] Model Construction Section 3 The model construction unit 3 may, as necessary, store the learning data A received by the learning data storage unit 2. 1-N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M Data group and original image X 1-M Object A 1-N The correct labeled image X is 1-M Generate multiple explanatory variables for learning from a set of image data. The model construction unit 3 uses the data group received by the learning data storage unit 2 and, if necessary, the generated explanatory variables (hereinafter, the learning explanatory variables may also be referred to as learning data) to generate a model of the overlapping projected object A. 1-N From a biometric image including the object A1 and the object A N A learning model is constructed to generate images (output images) that identify each of the above. Furthermore, the constructed learning model is stored in the learned model 4. Note that the "identified image" refers to the image in which object A1 is identified as object A. N For example, the image of object A1 and object A2 N are labeled with different colors.

[0018] The model construction unit 3 constructs a learning model by applying known machine learning using the above-mentioned learning data. For example, CNN, U-Net, Mask R-CNN, TransUnet, Swin U-net, generative adversarial networks (GANs, e.g., Cycle GAN, pix2pix, etc.) can be used.

[0019] 〇Trained model 4 The trained model 4 stores and updates the trained model constructed by the model construction unit 3. The trained model 4 also has a function of updating the trained model by repeating training by the model construction unit 3. Additional learning model 4-1 Furthermore, the trained model 4 is able to detect overlapping projected objects A 1-N From the image containing the object A1 and the object A N In addition to training to generate images that identify each of the objects, 1-N Two or more objects A are projected overlapping with the biological image of a normal case including 1-N By learning a biological image of disease case Y including the above, disease case Y can be made distinguishable. For example, examples of disease case Y include respiratory disease, interstitial pneumonia, chronic obstructive pulmonary disease, lung cancer, and the like.

[0020] Biometric Image Reception Unit 5 The biometric image receiving unit 5 receives two or more overlapping projected objects A. 1-N The subjects from which the biometric images are taken include healthy individuals, patients suspected of having disease case Y, and patients who have been definitively diagnosed with disease case Y.

[0021] Image generation unit 6 The image generation unit 6 applies the biometric image received by the biometric image receiving unit 5 to the trained model 4 constructed by the model construction unit 3, thereby generating two or more overlapping projected objects A. 1-N From a biometric image including the object A1 and the object A N An image (output image) is generated that identifies each of the above. Furthermore, if necessary, the Dice coefficient and the Jaccard coefficient can be calculated as indicators of image accuracy. The Dice coefficient and the Jaccard coefficient are indices that measure the similarity of sets. The preferred range of the Dice coefficient is 0.8 or more, preferably 0.9 or more. The preferred range of the Jaccard coefficient is 0.8 or more, preferably 0.9 or more. In the following example, it has been confirmed that the Dice coefficient is 0.948 and the Jaccard coefficient is 0.903.

[0022] According to the image generating device of the first embodiment, two or more overlapping projected objects A are generated using a learning model. 1-N From a biometric image including the object A1 and the object A N It is possible to create an image in which each of these is identified (see output image in Figure 7).

[0023] (Second embodiment) An image generating device according to the second embodiment will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the functional configuration of an image generating device 1B according to the second embodiment. In Fig. 2, components having the same reference numerals as those shown in Fig. 1 have the same functions, and therefore redundant explanations will be omitted here.

[0024] The image generating device of the second embodiment includes an image synthesis unit 7 in addition to the components of the first embodiment. The image synthesis unit 7 synthesizes the object A1 and the object A N The image synthesis unit 7 synthesizes the identified images with the biological image. Specifically, the image synthesis unit 7 sets the transparency (0.1 to 0.9) of the labeled images output from the learning model (makes them semi-transparent) and synthesizes them with the original clinical image (see Figure 8: Label-Transparent Image). The above synthesis method is not particularly limited, but an example of an A1A2 separated (left and right lung separated) image is shown below. Pixels that exist only in A1 (right lung) are replaced with a one-hot vector representation in which only the first channel is 1 (= [1,0,0,0]), pixels that exist only in A2 (left lung) are replaced with a one-hot vector representation in which only the second channel is 1 (= [0,1,0,0]), pixels that exist in both lungs are replaced with a one-hot vector representation in which only the third channel is 1 (= [0,0,1,0]), and pixels that do not exist in the lungs are replaced with a one-hot vector representation in which only the fourth channel is 1 (= [0,0,0,1]). Next, an image is generated by overlaying a specific color on the original image with a set transparency (e.g., 0.5, 0.1 to 0.9), such as color 1 (green) if the first channel is 1, color 2 (blue) if the second channel is 1, and color 3 (light blue) if the third channel is 1. Furthermore, if necessary, the above number of channels (4 channels) is reduced to 3 channels and converted to PNG.

[0025] According to the image generating device of the second embodiment, the object A1 and the object A2 are generated by using a learning model. N A composite clinical image can be created in which each of the following is identified.

[0026] (Third embodiment) An image generating device according to the third embodiment will be described with reference to Fig. 3. Fig. 3 is a block diagram showing an example of the functional configuration of an image generating device 1C according to the third embodiment. In Fig. 3, components having the same reference numerals as those shown in Figs. 1 and 2 have the same functions, and therefore redundant explanations will be omitted here.

[0027] The image generating device of the third embodiment includes a separated image generating unit 8 in addition to the first and second embodiments. The separated image creating unit 8 separates the object A1 and the object A N are synthesized with the original clinical images to obtain separated images A1 and A2. N (See Figure 8: Separated Image 1, Separated Image 2). The separated images can be created by a method similar to the image synthesis method described in the image generating device.

[0028] According to the image generating device of the third embodiment, the object A1 and the object A2 are generated by using a learning model. N A composite clinical image can be created by separating each of the above.

[0029] (Fourth embodiment) An image generating device according to a fourth embodiment will be described with reference to Fig. 4. Fig. 4 is a block diagram showing an example of the functional configuration of an image generating device 1D according to the fourth embodiment. In Fig. 4, components having the same reference numerals as those shown in Figs. 1 to 3 have the same functions, and therefore redundant explanations will be omitted here.

[0030] The image generating device of the fourth embodiment includes a diagnosis support unit 9 in addition to the first to third embodiments. The diagnostic support unit 9 matches the images generated by the image generation unit 6, the images generated by the image synthesis unit 7, and / or the images generated by the separated image creation unit 8 with a known disease medical database to present a predicted disease. The diagnostic support unit 9 may also include a device for accessing the known disease medical database online.

[0031] The image generating device according to the fourth embodiment can present predicted diseases using a learning model.

[0032] (Two or more objects A that are projected overlapping 1-N From a biometric image including the object A1 and the object A N A trained image generation model that generates images that identify each of the following: The learning model includes: 1) Object A as training data 1-N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M and the original image X 1-M Object A 1-N The correct labeled image X is 1-M 2) Original image X 1-M and the correct labeled image X 1-M By machine learning using 1-N From a biometric image including the object A1 and the object A N A learning model for generating images that distinguish between Additionally, the learning model may further include: 3) Two or more overlapping projected objects A 1-N Two or more objects A are projected overlapping with the biological image of a normal case including 1-N A biological image of disease case Y containing 4) The learning model in 2) above is machine-learned using the biological images in 3) above, and is a learning model that can distinguish disease case Y.

[0033] (Two or more overlapping objects A 1-N From a biometric image including the object A1 and the object A N (Image generation data structure that generates images that identify each of the The image generation data structure includes the following: 1) Object A as training data 1-N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M and the original image X 1-M Object A 1-N The correct labeled image X is 1-M 2) Original image X 1-M and the correct labeled image X 1-M By machine learning using 1-N From a biometric image including the object A1 and the object A N Pre-trained model data for generating images that identify each of the Additionally, the image generation data structure may further include: 3) Two or more overlapping projected objects A 1-N Two or more objects A are projected overlapping with the biological image of a normal case including 1-N A biological image of disease case Y containing 4) The learning model in 2) above is machine-learned using the biological images in 3) above, and the trained model data can distinguish disease case Y.

[0034] (Two or more overlapping objects A 1-N From a biometric image including the object A1 and the object A N (Image generation program that generates images that identify each of the following) This image generation program includes the following: 1) Object A 1-N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M and the original image X 1-M Object A 1-N The correct labeled image X is 1-M By performing machine learning using 1-N From a biometric image including the object A1 and the object A N A trained model generation process for generating images that identify each of the 2) Two or more overlapping projected objects A 1-N By applying a biological image including the target A1 and the target A to the trained model, N A process of creating an image in which each of the In addition, the image generating program may further include the following. 3) The object A1 and the object A N An image synthesis step of synthesizing the identified images with a biological image. 4) The object A1 and the object A N The images that have been identified are synthesized with the biometric image, and then the target A1 and the target A2 are synthesized. N A step of creating separated images by creating separate images of the

[0035] (Two or more overlapping objects A 1-N From a biometric image including the object A1 and the object A N (a diagnostic support method that provides images that identify each of the following) The diagnostic support method includes the following. 1) Two or more overlapping projected objects A 1-N By applying a biological image including 1-N From a biometric image including the object A1 and the object A N A process of creating an image in which each of the The trained model also includes: a) As training data, object A 1-N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M and the original image X 1-M Object A 1-N The correct labeled image X is 1-M b) The original image X 1-M and the correct labeled image X 1-M By machine learning using 1-N From a biometric image including the object A1 and the object A N A learning model for generating images that distinguish between

[0036] The above diagnostic support method further includes: 1-N Two or more objects A are projected overlapping with the biological image of a normal case including 1-N It may also include a trained model that can learn biological images of disease case Y including the disease case Y and can distinguish disease case Y.

[0037] The present invention will be described below with reference to examples, but the present invention is not limited to these examples in any way. [Example]

[0038] In this embodiment, two or more objects A are projected to overlap each other. 1-N From a biometric image including the object A1 and the object A N We constructed a trained model for image generation that generates images that identify each of the following:

[0039] (1) Data used and creation of dataset To create the training dataset, we used the XCAT phantom, a virtual human phantom that mathematically models a 4D CT image of a real human body (see Segars et al. Med. Phys. 37 (9), 2010). In this example, DCR lateral images of 150 phases per virtual human phantom in a breathing state (n=28) were pseudo-projected, and a total of 4200 sets of original images (original images X created by pseudo-projecting the virtual human phantom including the left and right lungs) were obtained. 4200 ) and the corresponding correct labeled image X 4200 was obtained as a data set (Figure 5). In the correct labeled image, the area of ​​the right lung, the area of ​​the left lung, and the areas of the left and right lungs are labeled with different colors.

[0040] Using the X-ray simulator "CT projector" developed at Duke University, a collaborative research partner, the created NURBS file was projected in two dimensions from a virtual radiation source to create a pseudo chest X-ray image. The imaging conditions were the same as those for the DCR actually used: tube voltage 100 kV, tube current time product 0.2 mAs / pulse, Cu additional filter: 0.2 mm, source to image-receptor distance (SID) 2.0 m, and the images were set to be projected in the dorsoventral direction. The geometrical arrangement during 2D projection is shown in Figure 6. The virtual human phantom represents continuous respiratory states with a set number of phases. To reproduce the pixel size used in clinical practice, the matrix size was set to 1024 × 1024 pixels and the pixel size was set to 417.0 μm. The output image was adjusted to approximate the actual DCR histogram and converted to an 8-bit grayscale image in PNG format.

[0041] (2) Building a trained model using a convolutional neural network In this example, a left and right lung separation model for DCR lateral images was constructed using the network structure of U-Net, a type of CNN. U-Net is a CNN structure proposed for medical image segmentation, and is a symmetric network with an Encoder-Decoder structure. In detail, the network model was trained using a dataset of 4,200 pairs of original images and their corresponding correctly labeled images created in (1) above, and tested by inputting biological images from actual clinical cases. An output image was obtained in which the left and right lungs were identified using markers (Figure 7: Output image).

[0042] (3) Results A labeled image (output image) was obtained from the learning model (2) above, which inputted biological images of actual clinical cases. The labeled image was then made semi-transparent and combined with the original clinical image to create a labeled transparent image. Furthermore, left and right lung images (separated images 1 and 2) were created from the labeled transmission images. An example of the resulting image is shown in Figure 8. We confirmed that the left and right lungs could be separated in a live image of an actual clinical case (a 79-year-old male with suspected right upper lobe lung cancer). Furthermore, the accuracy of the trained CNN model was verified (tested) using images of normal subjects taken with DCR lateral views and images of clinical cases (n=122) including respiratory disease cases. In this verification, it was possible to distinguish between clinical cases including respiratory disease cases and normal subjects.

[0043] (General remarks) The trained model for image generation of the present invention can output a DCR lateral image with the left and right lungs separated by inputting lateral chest X-ray video images that capture the respiratory state. Furthermore, it has become possible to evaluate pulmonary function, such as changes in lung area and diaphragm movement, using these images. Currently, pulmonary function imaging diagnosis of lateral views can be performed using specialized imaging techniques with expensive medical equipment such as CT and MRI. However, due to the cost, examination time, and patient exposure to CT, it is not performed as a routine examination. By combining the trained model for image generation of this invention with X-ray examination, simple and rapid pulmonary function imaging diagnosis support for lateral views can be realized for the purposes of initial examination and follow-up observation (especially the dynamic and functional evaluation of the thorax, lungs, and diaphragm from the side).

Claims

1. Two or more overlapping objects A 1-N From a biometric image containing 1 and Object A N An image generating device that generates an image in which each of the following is identified: Object A 1-N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M and the original image X 1-M Object A 1-N The correct labeled image X is 1-M a learning data storage unit that stores the above as learning data; Original image X 1-M and the correct labeled image X 1-M Using the overlapping projections, the object A 1-N Object A from an image containing 1 and Object A N a model construction unit that constructs a learning model for generating images that identify each of the above; Two or more overlapping projected objects A 1-N a biometric image receiving unit for receiving a biometric image including: The biometric image received by the biometric image receiving unit is applied to the learning model constructed by the model construction unit, whereby two or more overlapping projected objects A are projected. 1-N From a biometric image containing 1 and Object A N an image generating unit that generates images that identify each of the An image generating device comprising:

2. Furthermore, the object A 1 and Object A N The image generating device according to claim 1 , further comprising an image combining unit that combines the images obtained by identifying each of the above with a biometric image.

3. Furthermore, the object A 1 and Object A N The images that have been identified are combined with the biometric image to create the target A. 1 and object A N 2. The image generating device according to claim 1, further comprising a separated image generating unit that generates separated images of the respective components.

4. The model construction unit further constructs a projection image of two or more objects A that are overlapped and projected. 1-N Two or more objects A are projected overlapping with the biological image of a normal case including 1-N 3. The image generating device according to claim 1, wherein the disease case Y is distinguishable using a biological image of the disease case Y including:

5. Furthermore, the object A 1 and object A N 3. The image generating device according to claim 1, further comprising a diagnostic support unit that matches the identified and / or separated images with a known disease medical database to present predicted diseases.

6. Original image X 1-M 3. The image generating apparatus according to claim 1, wherein the image is generated by pseudo-projecting a virtual human phantom.

7. The correct labeled image X 1-M is object A 1-N 3. The image generating device according to claim 1, wherein the image generating device is configured from only the image.

8. The diagnosis support device according to claim 1 , wherein the learning model is constructed from one or more of the following: 1) CNN 2) U-Net 3) Mask R-CNN 4) TransUnet 5) Swin U-net 6) Generative Adversarial Networks

9. The object A 1-N 3. The image generating device according to claim 1, wherein: 1) Left lung (A 1 ) and right lung (A 2 ) 2) Left lung (A 1 ), right lung (A 2 ) and diaphragm (A 3 ) 3) Left lung (A 1 ), right lung (A 2 ) and heart and great vessels (A 3 ) 4) Left lung (A 1 ), right lung (A 2 ) and bronchial (A 3 )

10. The object A 1-N are the left and right lungs, and the learning model is a U-Net.

11. Two or more overlapping objects A 1-N From a biometric image containing 1 and Object A N A trained model for image generation that generates images that identify each of the following: The trained model is 1) Object A as training data 1-N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M and the original image X 1-M Object A 1-N The correct labeled image X is 1-M , and 2) The original image X 1-M and the correct labeled image X 1-M By machine learning using 1-N From a biometric image containing 1 and Object A N A learning model for generating images that identify each of the following: A pre-trained model for image generation.

12. The object A 1-N are the left and right lungs, and the training model is a U-Net.

13. Two or more overlapping objects A 1-N From a biometric image containing 1 and Object A N A data structure for image generation that generates an image in which each of the following is identified: The data structure is: 1) Object A as training data 1-N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M and the original image X 1-M Object A 1-N The correct labeled image X is 1-M , and 2) The original image X 1-M and the correct labeled image X 1-M By machine learning using 1-N From a biometric image containing 1 and Object A N The trained model data for generating images that identify each of the following is included. Data structures for image generation.

14. The object A 1-N are the left and right lungs, and the learning model is a U-Net.

15. Two or more overlapping objects A 1-N From a biometric image containing 1 and Object A N An image generating program for generating an image in which each of the following is identified: The program 1) Object A 1-N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M and the original image X 1-M Object A 1-N The correct labeled image X is 1-M By performing machine learning using 1-N From a biometric image containing 1 and Object A N A trained model generation process for generating images that respectively identify the 2) Two or more overlapping projected objects A 1-N By applying the trained model to a biological image including 1 and Object A N creating an image in which each of the A program for generating images.

16. The object A 1-N are the left and right lungs, and the learning model is U-Net.

17. Two or more overlapping objects A 1-N From a biometric image containing 1 and Object A N A diagnostic support method for providing an image in which each of the following is identified: The method comprises: 1) Two or more overlapping projected objects A 1-N By applying a biological image including 1-N From a biometric image containing 1 and Object A N creating an image in which each of the images is identified; Here, the trained model is a) Object A as training data 1-N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M and the original image X 1-M Object A 1-N The correct labeled image X is 1-M , and b) The original image X 1-M and the correct labeled image X 1-M By machine learning using 1-N From a biometric image containing 1 and Object A N A learning model for generating images that identify each of the following: Diagnostic support methods.

18. The object A 1-N are the left and right lungs, and the learning model is U-Net.

19. The trained model further includes two or more overlapping projected objects A 1-N Two or more objects A are projected overlapping with the biological image of a normal case including 1-N The diagnostic support method according to claim 17 or 18, wherein the method is capable of learning a biological image of a disease case Y including the following and discriminating the disease case Y.

Images