Image generation device, trained model for image generation, data structure for image generation, program for image generation, and diagnostic support method, for generating image containing only an shadows or image with reduced an shadows from biological image including an shadows

Abstract

To provide an image generation device, a trained model for image generation, a data structure for image generation, a program for image generation, and a diagnostic support method, for generating an image with reduced AN shadows from a biological image including the AN shadows.SOLUTION: A trained model for image generation is constructed by training a generative adversarial network, which is a type of a generative model, using original images X1-M created by pseudo-projecting virtual human body phantoms including a target tissue or organ O that contains AN and a target region T, and correct images X1-M. The trained model thus generates, from a biological image including AN shadows, an image containing only the AN shadows.SELECTED DRAWING: None

Description

[Technical Field]

[0001] The present invention is N A from biological images including shadows N Image with only shadows or A N The present invention relates to an image generation device that generates an image with reduced shadows, a trained model for image generation, a data structure for image generation, an image generation program, and a diagnostic support method. [Background technology]

[0002] In chest X-ray examinations, lateral images are taken in addition to frontal images in order to evaluate the shape of the thorax, lungs, and diaphragm when viewed from the side. Recently developed dynamic chest X-ray examinations also evaluate dynamics and function based on changes in shape when viewed from the side. However, because the left and right lungs and diaphragm are projected overlapping to the side, it is difficult to evaluate them separately. Although technology has been developed to automatically recognize overlapping left and right lungs as a single region, separating the left and right lungs has been technically difficult (Non-Patent Documents 1 and 2).

[0003] A special MRI test (pseudo-dynamic MRI) has been proposed to assess dynamic and functional functions using lateral tomographic images (see Non-Patent Document 3). However, due to the cost and complexity, its introduction into routine clinical practice is not practical. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] https: / / aapm.onlinelibrary.wiley.com / doi / 10.1118 / 1.598331 [Non-patent document 2] https: / / ietresearch.onlinelibrary.wiley.com / doi / 10.1049 / iet-ipr.2016.0526 [Non-patent document 3] https: / / ietresearch.onlinelibrary.wiley.com / doi / 10.1049 / iet-ipr.2016.0526 Summary of the Invention [Problem to be solved by the invention]

[0005] In recent years, dynamic chest radiography (DCR) has been developed, which enables pulmonary function imaging diagnosis in a general radiography room. Diaphragmatic movement and changes in density within the lung field are quantified using DCR frontal images that capture respiratory status, and this is used to evaluate pulmonary function on a lung-by-lung basis. On moving chest X-ray images of respiratory status, areas of impaired pulmonary ventilation are detected as decreased changes in lung density, but it has been suggested that breast shadows can cause false positives. Furthermore, the diaphragm and subdiaphragmatic organs, which are projected onto the lungs in a frontal image, hinder the analysis of pulmonary function and the detection of lesions in the relevant region. Furthermore, accurate preoperative assessment of tumor invasion and adhesion is important for appropriate surgical planning of thoracic diseases. Furthermore, because the mediastinum contains many vital organs, such as the heart, great vessels, trachea, and esophagus, early detection of tumors is desirable. However, mediastinal shadows make it difficult to detect tumors near the mediastinum and assess their invasion. Therefore, in the present invention, A N A from biological images including shadows N The objective of the present invention is to provide an image generation device that generates images with reduced shadows, a trained model for image generation, a data structure for image generation, a program for image generation, and a diagnostic support method. [Means for solving the problem]

[0006] It is extremely difficult to recognize and separate tissues hidden in the shadows of other tissues and organs, or tumors in those tissues, in biological images using the naked eye. Therefore, in this invention, we use a generative adversarial network, which is a type of generative model, as Nand an original image X created by pseudo-projecting a virtual human phantom containing target tissues and organs O containing target region T. 1-M and correct image X 1-M By learning A N A from biological images including shadows N We have constructed a trained model for image generation that generates images with only shadows. N Generated from a biological image including shadows A N By subtracting the image with only shadows, A N It was confirmed that an image with reduced shadows was generated. Furthermore, A N By using biological images including shadows as a pre-trained model for image generation, N Generation of shadow-only images,A. N Generate images with reduced shadows, and N The tumor that was hidden and overlapping with the shadow and could not be distinguished was identified as A. N The present invention was completed after confirming that this is possible on images with reduced shadows.

[0007] 1.A N A from biological images including shadows N Image with only shadows or A N An image generating device for generating an image with reduced shadows, comprising: 1) A N and an original image X created by pseudo-projecting three-dimensional data of a virtual human phantom or real human body containing target tissue / organ O containing target region T. 1-M and correct image X 1-M a learning data storage unit that stores the above as learning data; 2) Original image X 1-M and the correct image X 1-M Using A N A from biological images including shadows N Image with only shadows or A N a model construction unit that constructs a learning model for generating an image with reduced shadows; 3) A N a biometric image receiving unit that receives a biometric image including a shadow; and 4) Applying the biometric image accepted by the biometric image accepting unit to the learning model constructed by the model constructing unit, A N A from biological images including shadows N Image with only shadows or A N an image generation unit that generates an image with reduced shadows; An image generating device comprising: Correct image X 1-M teeth, a)A N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M From A N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M Image X obtained by subtracting 1-M (Difference image X 1-M ) b) A N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M In A N The masked image part where the target area T part that does not overlap with A is masked. N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M Image X created by applying 1-M (Masking image X 1-M ) c) A N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M (A N Image X with 1-M ) d) A with bone structure removed N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M From bone structure and A N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M Image X obtained by subtracting 1-M (Bone structure removed difference image X 1-M ) Selected from one of the following: Image generating device. 2. Correct image X 1-M The imaging device according to the preceding paragraph 1, which includes a pseudo-tumor in or near a target tissue / organ O. 3. Correct image X 1-M is the difference image X 1-M or the bone structure removed difference image X 1-M 3. The imaging device according to claim 1 or 2, 4. Correct image X 1-M is the masked image X 1-M 3. The imaging device according to claim 1 or 2, 5. The above A N The image with reduced shadows is the original image X 1-M A multiplied by a weighting factor from N 3. The imaging device according to claim 1 or 2, wherein the image is obtained by subtracting an image of only shadows. 6. Original image X 1-M 3. The image generating device according to claim 1 or 2, wherein the image is generated by pseudo-projecting a virtual human phantom. 7. The above A N 4. The imaging apparatus according to the preceding paragraph 3, wherein the target tissue / organ O is the aorta, the heart, the superior vena cava, and / or the inferior vena cava, and the target tissue / organ O is the lung. 8. The above A N 5. The imaging device according to claim 4, wherein the target region T is a lung field. 9. The above A N 4. The imaging apparatus according to the preceding paragraph 3, wherein the target tissue / organ O is a breast and the target tissue / organ O is a lung. 10. The image generating device according to the preceding paragraph 2, wherein the model constructing section is further capable of detecting a tumor present in or near the target tissue / organ O. 11.Furthermore, the above-mentioned A N 6. The image generating device according to item 1 or 5, further comprising a diagnostic support unit that matches the image with reduced shadows with a known disease medical database to present predicted diseases. 12. A diagnostic support device as described in paragraph 1 or 2 above, wherein the learning model is constructed from one or more of the following: 1) Generative Adversarial Networks 2) pix2pix 3) CycleGAN 13.A N A from biological images including shadows N Image with only shadows or A N A trained model for image generation that generates an image with reduced shadows, The trained model is 1) As training data, A N and an original image X created by pseudo-projecting three-dimensional data of a virtual human phantom or real human body containing target tissue / organ O containing target region T. 1-M and correct image X 1-M , and 2) Original image X 1-M and the correct image X 1-M By using machine learning, A N A from biological images including shadows N Image with only shadows or A N a learning model for generating an image with reduced shading; Here, the correct image X 1-M teeth, a)A N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M From A N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M Image X obtained by subtracting 1-M (Difference image X 1-M ) b) A N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M In A N The masked image part where the target area T part that does not overlap with A is masked. N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M Image X created by applying 1-M (Masking image X 1-M ) c) A NImage X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M (A N Image X with 1-M ) d) A with bone structure removed N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M From bone structure and A N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M Image X obtained by subtracting 1-M (Bone structure removed difference image X 1-M ) Selected from one of the following: A pre-trained model for image generation. 14.A N A from biological images including shadows N Image with only shadows or A N 1. A data structure for generating an image with reduced shadows, comprising: The data structure is: 1) As training data, A N and an original image X created by pseudo-projecting three-dimensional data of a virtual human phantom or real human body containing target tissue / organ O containing target region T. 1-M and correct image X 1-M , and 2) Original image X 1-M and the correct image X 1-M By using machine learning, A N A from biological images including shadows N Image with only shadows or A N Including data of a trained model for generating an image with reduced shading, Here, the correct image X 1-M teeth, a)A N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M From A N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M Image X obtained by subtracting 1-M(Difference image X 1-M ) b) A N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M In A N The masked image part where the target area T part that does not overlap with A is masked. N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M Image X created by applying 1-M (Masking image X 1-M ) c) A N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M (A N Image X with 1-M ) d) A with bone structure removed N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M From bone structure and A N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M Image X obtained by subtracting 1-M (Bone structure removed difference image X 1-M ) Selected from one of the following: Data structures for image generation. 15.A N A from biological images including shadows N Image with only shadows or A N An image generation program for generating an image with reduced shadows, The program 1) A N and an original image X created by pseudo-projecting three-dimensional data of a virtual human phantom or real human body containing target tissue / organ O containing target region T. 1-M and correct image X 1-M By using machine learning, A N A from biological images including shadows N Image with only shadows or A Na trained model generation step for generating an image with reduced shadows; and 2) A N By applying the trained model to biological images containing shading, A N Image with only shadows or A N creating an image with reduced shadows; Here, the correct image X 1-M teeth, a)A N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M From A N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M Image X obtained by subtracting 1-M (Difference image X 1-M ) b) A N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M In A N The masked image part where the target area T part that does not overlap with A is masked. N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M Image X created by applying 1-M (Masking image X 1-M ) c) A N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M (A N Image X with 1-M ) d) A with bone structure removed N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M From bone structure and A N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M Image X obtained by subtracting 1-M (Bone structure removed difference image X 1-M ) Selected from one of the following: A program for generating images. 16.A N A from biological images including shadows N A diagnostic support method for providing an image with reduced shadows, comprising: The method comprises: 1) A N By applying the trained model to biological images containing shading, A N A from biological images including shadows N Image with only shadows or A N creating an image with reduced shadows; and 2) Matching the image created in 1) with a known disease medical database to present predicted diseases; Here, the trained model is 1) As training data, A N and an original image X created by pseudo-projecting three-dimensional data of a virtual human phantom or real human body containing target tissue / organ O containing target region T. 1-M and correct image X 1-M , and 2) Original image X 1-M and the correct image X 1-M By using machine learning, A N Image with only shadows or A N a learning model for generating an image with reduced shading; Here, the correct image X 1-M teeth, a)A N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M From A N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M Image X obtained by subtracting 1-M (Difference image X 1-M ) b) A N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M In A NThe masked image part where the target area T part that does not overlap with A is masked. N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M Image X created by applying 1-M (Masking image X 1-M ) c) A N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M (A N Image X with 1-M ) d) A with bone structure removed N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M From bone structure and A N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M Image X obtained by subtracting 1-M (Bone structure removed difference image X 1-M ) is selected from one of and The A N The image with only shadows is the original image X 1-M Multiply by the weighting factor to get A N Converted into an image with reduced shadows, Diagnostic support methods. [Effects of the Invention]

[0008] A N A from biological images including shadows N Provided are a device for generating images with only shadows, a trained model for image generation, a data structure for image generation, a program for image generation, and a diagnostic support method. [Brief explanation of the drawings]

[0009] [Figure 1] Creation of pseudo X-ray moving images (Example 1). [Figure 2] Pix2pix network structure (Example 1). [Figure 3]Configuration of generator and discriminator (Example 1). [Figure 4] Breast shadow reduction processed image 1 of a virtual human phantom (Example 1). [Figure 5] Breast shadow reduction processed image 2 of virtual human phantom (Example 1). [Figure 6] Breast shadow reduction processed image of a clinical case (Example 1). [Figure 7] Procedure for preparing a virtual human phantom with a simulated tumor inserted near the mediastinum and a projection image of the virtual human phantom with the tumor inserted (16 virtual human phantoms in total, one in each location) (Example 2). [Figure 8] Procedure for creating difference images used in the training dataset: (a) original image, (b) image without mediastinal shadow, (c) difference image (Example 2). [Figure 9] Process for generating mediastinal shadow-reduced images (Example 2). [Figure 10] Image after mediastinal shadow reduction processing of a virtual human phantom with a solid tumor embedded (Example 2). [Figure 11] The first frame of a chest X-ray sequence from subject #3 (79 years old, male) with squamous cell carcinoma in the right lower lobe of the lung. The trajectory of the tumor shadow tracked by template matching is colored. (a) The white line represents the trajectory of the correct data, and the blue line represents the trajectory of the tracking results from the original sequence. (b) The white line represents the trajectory of the correct data, and the red line represents the trajectory of the tracking results from the sequence after mediastinal shadow reduction processing (Example 2). [Figure 12] Chest X-ray images of subject #3 (79 years old, male) with squamous cell carcinoma in the right lower lobe of the lung. The trajectory of the tumor shadow tracked using optical flow is colored. (a) The white line represents the trajectory of the correct data, and the blue line represents the trajectory of the tracking results of the original video sequence. (b) The white line represents the trajectory of the correct data, and the red line represents the trajectory of the tracking results of the video sequence after mediastinal shadow reduction processing (Example 2). [Figure 13] Tumor implantation method (Example 3). [Figure 14] Method for removing organ shadows overlapping with lung fields (Example 3). [Figure 15]Another method for removing organ shadows overlapping with lung fields (Example 4). [Figure 16] Method for generating organ shadow reduced image (Example 4). [Figure 17] Pix2pix network structure (Example 3). [Figure 18] One frame of the output video image of the diaphragm and subdiaphragmatic organ shadow reduction process obtained by deep learning using pix2pix (Example 3). [Figure 19] Each output image using a clinical case image (Example 3). [Figure 20] 10 is a frame of a moving frontal chest X-ray image before processing to reduce the shadows of the diaphragm and subdiaphragmatic organs (Example 3). [Figure 21] 10 is a frame of a moving frontal chest X-ray image after processing to reduce shadows of organs below the diaphragm (Example 3). [Figure 22] Original image (left) and output image (right, weighting factor: 0.6) (Example 4). [Figure 23] FIG. 1 is a block diagram showing an example of the functional arrangement of an image generating apparatus according to a first embodiment. [Figure 24] FIG. 10 is a block diagram showing an example of the functional arrangement of an image generating apparatus according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] (Subject of the present invention) The present invention is N A from biological images including shadows N Image with only shadows or A N The present invention relates to an image generation device with reduced shadows, a trained model for image generation, a data structure for image generation, an image generation program, and a diagnostic support method.

[0011] (First embodiment) A first embodiment of the present invention will be described below with reference to the drawings. N Image with only shadows or A N A from biological images including shadows N 1 is a block diagram showing an example of the functional configuration of an image generating device 1 that generates an image with reduced shadows. As shown in Fig. 23, the image generating device 1A according to the first embodiment includes components (functional blocks) of a learning data storage unit 2, a model construction unit 3, a trained model 4, a biometric image acceptance unit 5, and an image generating unit 6. For convenience of explanation, the components are separated, but the components may be combined or may include other components. The same applies to the other embodiments described below. The above configuration can be configured by either hardware or software. For example, when configured by software, each of the above functional blocks is actually configured with a computer's CPU, RAM, ROM, etc., and is realized by the operation of a program stored in a recording medium such as RAM, ROM, a hard disk, or a semiconductor memory. Furthermore, each configuration does not need to be mounted on the same hardware, and part of the configuration may be mounted on another medium (e.g., cloud). The same applies to the other embodiments described below.

[0012] (biometric images) The term "biological image" as used herein is not particularly limited, but includes, for example, a DCR image, a plain X-ray image, an X-ray fluoroscopic image, and the like. Furthermore, in this specification, "A N The term "biological image including shading" is not particularly limited, but may include, but is not limited to, objects A in the image such as organs, tissues, blood vessels, bronchi, and skeletons. N This means an image in which shadows of In addition, A N "N" in the above expression means an integer equal to or greater than 1. For example, when N is 2, there are shadows of at least two objects (A1, A2), and when N is 3, there are shadows of at least three objects (A1, A2, A3). N is selected from, for example, 2, 3, 4, 5, 6, 7, 8, 9, 10, 20, 50, 100, etc., but is not particularly limited. For example, A N The following can be exemplified: 1) A1: Breasts (both breasts), aorta, heart, superior vena cava, inferior vena cava, diaphragm, trachea, or bronchial skeleton 2) A 1-4: Mediastinal organs, tissues, and blood vessels (two or more of the following: aorta, heart, superior vena cava, and inferior vena cava) 3) A 1-7 : Seven or more of the following organs below the diaphragm (liver, gallbladder, pancreas, spleen, small intestine, large intestine, kidneys, blood vessels (arteries branching from the abdominal aorta, veins joining the inferior vena cava)) Furthermore, when a pair of left and right organs are projected overlapping each other, it is possible to suppress the shadow of one of them.

[0013] In this specification, the "tissue / organ of interest" refers to N This refers to tissues and organs that are targets for various diagnoses, such as lungs, blood vessels, trachea, and bronchi, by reducing shadows. In this specification, the "target region T" refers to the projection region of the target tissue / organ O that is the particular diagnostic target on the image of the target tissue / organ O. For example, if the target tissue / organ O is the lung, the target region T is the lung field, and if the target tissue / organ O is the blood vessels, trachea, or bronchi, the target region T is the projection image of the blood vessels, trachea, or bronchi.

[0014] Learning data storage unit 2 The learning data storage unit 2 accepts any one of the following as learning data: A N and an original image X created by pseudo-projecting three-dimensional data of a virtual human phantom or real human body containing target tissue / organ O containing target region T. 1-M (Original image X 1-M Data Group) Original image X 1-M From A N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M Image X obtained by subtracting 1-M (Difference image X 1-M , A N Shading-only image X 1-M : Correct image X 1-M Data Group) A N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-MThe correct image X obtained from 1-M (A N Image X with 1-M , correct image X 1-M Data Group) A N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M In A N The masked image part where the target area T part that does not overlap with A is masked. N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M Image X created by applying 1-M (Masking image X 1-M ) Note that a portion of the training data (for example, about 20%) may be divided into verification data. Hereinafter, the training data will also include the verification data. The division method is not particularly limited, but the holdout method, cross-validation method, etc. can be used. In addition, X 1-M "M" in the above expression means an integer of 2 or more, for example, 10, 50, 100, 500, 1000, 2000, 3000, 4000, 4200, 5000, 10000, etc. In detail, when M is 5, X 1、 X 2、 X 3、 X 4、 It becomes X5.

[0015] (Virtual human phantom, 3D data of real human body) In this specification, the term "virtual human phantom" refers to a digital phantom that mathematically models a 4D CT scan of a real human body, with thousands of defined structures and parameterized models of cardiac and respiratory motion. By pseudoprojecting these structures, it is possible to generate a large amount of training data without irradiating a patient. The model may be a voxelized model, a boundary representation model, or the like. Preferred digital phantoms include the XCAT phantom (see Segars et al. Med. Phys. 37 (9), 2010), the MIRD phantom, and the VIP-Man, with the XCAT phantom being particularly preferred. 3D data of the actual human body can be obtained from CT and MRI.

[0016] (pseudo photography) The term "pseudo-radiography" as used herein refers to, but is not limited to, the use of a CT projector, an X-ray simulator developed at Duke University, to obtain images by projecting a created NURBS file two-dimensionally from a virtual radiation source (see: Realistic CT simulation using the 4D XCAT phantom. Med. Phys. 2008; 35(8): 3800-3808).

[0017] (correct image) As used herein, "A N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M The correct image X obtained from 1-M " refers to the object A of the shadow to be reduced. N The range, number, and type of the hydroxyl group can be appropriately selected. For example, the following can be given as examples, but are not particularly limited.

[0018] (A N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M ) When the object A to be shaded is one (e.g., the left or right breast), an image created by pseudo-projecting three-dimensional data of a virtual human phantom or real human body from which A1 has been removed can be used as the correct image. Additionally, three-dimensional data of a virtual human phantom or a real human body from which the bone structures of the ribs, clavicles, scapulae, sternum, and vertebrae have been removed may be used as needed.

[0019] (Difference image X 1-M) Object A to be shaded N If there are 2 or more (aorta, heart, superior vena cava, inferior vena cava), A N Original images X to A created by pseudo-projecting 3D data of a virtual human phantom or real human body including N The correct image can be obtained by subtracting image X from image X created by pseudo-projecting three-dimensional data of a virtual human phantom or real human body from which the difference has been removed. Additionally, if necessary, a virtual human phantom may be used in which the bone structures of the ribs, clavicles, shoulder blades, sternum, and vertebrae have been removed.

[0020] (Masking image X 1-M ) Object A, whose shadows are reduced by photography N If overlaps with a part of the target tissue / organ O, A N In the original image X created by pseudo-projecting the 3D data of a virtual human phantom or real human body, the masked image portion in which the portion not overlapping with the target region T is masked is designated as A. N By applying the three-dimensional data of a virtual human phantom or a real human body from which the distortion has been removed to an image X created by pseudo-projection, the created image X can be regarded as the correct image (see Figure 14). In detail, a binary image of a virtual human phantom, in which only the target region T (lungs, lung fields) is projected, is used as the target A. N A virtual human phantom with only the subdiaphragmatic organs remaining is projected onto a binary image, and then a mask (Mask A) is created by removing the mask from the binary image. Then, a mask (Mask B) is created by cutting out the original image X1 in the shape of Mask A. N In the projection image X2 of the human body phantom from which the mask B has been removed, the image of the part overlapping with Mask B is replaced with Mask B, and the "object A overlapping with the target area T" is N An image X3 with the above removed can be created and used as the correct image. Also, A N After generating the shadows (see Figure 15), A N By subtracting the shadows, A NMethods for producing images with reduced shadows can also be used (see Figure 16). Additionally, three-dimensional data of a virtual human phantom or a real human body from which the bone structures of the ribs, clavicles, scapulae, sternum, and vertebrae have been removed may be used as needed.

[0021] Model Construction Section 3 The model construction unit 3 may, as necessary, store the original learning image X received by the learning data storage unit 2. 1-M Data group and correct image X 1-M Generate multiple explanatory variables for learning from a set of image data. The model construction unit 3 uses the data group received by the learning data storage unit 2 and, if necessary, the generated explanatory variables (hereinafter, the learning explanatory variables may also be referred to as learning data), and N A from biological images including shadows N Image with only shadows or A N A learning model is constructed to generate images (output images) with reduced shadows. Furthermore, the constructed learning model is stored in the learned model 4.

[0022] The model construction unit 3 constructs a learning model by applying known machine learning using the above-mentioned learning data. For example, a generative adversarial network (conditional GANs, pix2pix, CycleGAN) or the like can be used. By adjusting the batch size, the number of epochs representing the number of times of learning, and the learning rate, it is possible to build a high-performance trained model.

[0023] 〇Trained model 4 The trained model 4 stores and updates the trained model constructed by the model construction unit 3. The trained model 4 also has a function of updating the trained model by repeating training by the model construction unit 3. Additional learning model 4-1 The trained model 4 further N A from biological images including shadows N Image with only shadows or A NIn addition to training to generate images with reduced shadows, it is possible to train images that do not contain pseudo-tumors in or near the target tissue / organ O, and images that contain pseudo-tumors in or near the target tissue / organ O, making it possible to detect tumors.

[0024] Biometric Image Reception Unit 5 The biometric image receiving unit 5 is N We accept biological images (actual clinical images) containing shading. The subjects from which the biological images are taken include healthy individuals, cancer patients, patients suspected of having cancer, patients suspected of having disease case Y, and patients who have been confirmed to have disease case Y. For example, examples of disease case Y include respiratory disease, interstitial pneumonia, chronic obstructive pulmonary disease, lung cancer, and the like.

[0025] Image generation unit 6 The image generation unit 6 applies the biometric image received by the biometric image receiving unit 5 to the trained model 4 constructed by the model construction unit 3, thereby generating A N A from biological images including shadows N Image with only shadows or A N Generates an image (output image) with reduced shadows. Furthermore, if necessary, PSNR and SSIM (e.g., breast: 32.7 / 0.989, diaphragm: 30.0 / 0.946, mediastinum: 36.1 / 0.906) can also be output as indicators of image accuracy. A N Image X with reduced shadows 1-M is the original image X 1-M A multiplied by the weighting factor N Shading-only image X 1-M It can also be obtained by subtracting The weighting coefficients are exemplified as follows, but are not particularly limited to these. Bone (A N Shade):0.4~0.6 Breast:0.3~0.5 Diaphragm: 0.4~0.6 Mediastinum: 0.6~0.9

[0026] According to the image generating device of the first embodiment, a learning model is used to generate a N A from biological images including shadows N Image with only shadows or A N Images with reduced shadows can be created.

[0027] (Second embodiment) The image generating device of the second embodiment will be described with reference to Fig. 24. Fig. 24 is a block diagram showing an example of the functional configuration of an image generating device 1B according to the second embodiment. In Fig. 24, components with the same reference numerals as those shown in Fig. 23 have the same functions, and therefore redundant explanations will be omitted here.

[0028] The image generating device of the second embodiment includes a diagnosis support unit 9 in addition to the components of the first embodiment. The diagnostic support unit 9 matches the image generated by the image generation unit 6 with a known disease medical database and presents predicted diseases. The diagnostic support unit 9 may include a device for accessing the known disease medical database online.

[0029] According to the image generating device of the second embodiment, it is possible to present predicted diseases using a learning model.

[0030] (A N A from biological images including shadows N Image with only shadows or A N A pre-trained model for image generation that generates images with reduced shadows This trained model includes the following: 1) As training data, A N and an original image X created by pseudo-projecting three-dimensional data of a virtual human phantom or real human body containing target tissue / organ O containing target region T. 1-M and correct image X 1-M 2) Original image X 1-M and the correct image X 1-M By using machine learning, A N A from biological images including shadows NImage with only shadows or A N Data for a learning model to generate images with reduced shadows In addition, the original image X 1-M A multiplied by the weighting factor N By subtracting the image with only shadows, A N It is also possible to generate an image with reduced Additionally, the trained model may further include: 3) Images that do not contain pseudo-tumors in or near the target tissue / organ O, images that contain pseudo-tumors in or near the target tissue / organ O, live images of disease cases that contain tumors, and / or live images of normal cases that do not contain tumors. 4) The learning model in 2) above is a learning model that can detect tumors through machine learning using the images in 3) above.

[0031] (A N A from biological images including shadows N Image with only shadows or A N A data structure for generating images with reduced shadows The image generation data structure includes the following: 1) As training data, A N and an original image X created by pseudo-projecting three-dimensional data of a virtual human phantom or real human body containing target tissue / organ O containing target region T. 1-M and correct image X 1-M 2) Original image X 1-M and the correct image X 1-M By using machine learning, A N A from biological images including shadows N Image with only shadows or A N Data from a trained model for generating images with reduced shadows In addition, the original image X 1-M A multiplied by the weighting factor N By subtracting the image with only shadows, A N It is also possible to generate an image with reduced Additionally, the image generation data structure may further include: 3) Images that do not contain pseudo-tumors in or near the target tissue / organ O, images that contain pseudo-tumors in or near the target tissue / organ O, live images of disease cases that contain tumors, and / or live images of normal cases that do not contain tumors. 4) The learning model in 2) above is machine-learned using the biological images in 3) above, and the data of the trained model is capable of detecting tumors.

[0032] (A N A from biological images including shadows N Image with only shadows or A N An image generation program that generates images with reduced shadows This image generation program includes the following: 1) A N and an original image X created by pseudo-projecting three-dimensional data of a virtual human phantom or real human body containing target tissue / organ O containing target region T. 1-M and correct image X 1-M By using machine learning, A N A from biological images including shadows N Image with only shadows or A N A trained model generation process for generating images with reduced shadows 2) A N By applying the trained model to biological images containing shading, A N Image with only shadows or A N A process for creating images with reduced shadows In addition, the original image X 1-M A multiplied by the weighting factor N By subtracting the image with only shadows, A N It is also possible to generate an image with reduced

[0033] (A N A from biological images including shadows N A diagnostic support method for providing images with reduced shadows The diagnostic support method includes the following. 1) A N By applying the trained model to biological images containing shading, A N A from biological images including shadows N Image with only shadows or AN A process for creating images with reduced shadows 2) A process of matching the image created in 1) with a known disease medical database to present predicted diseases. Furthermore, if necessary, the A N The image with only shadows is the original image X 1-M Multiply by the weighting factor to get A N It may also include a step of converting the image into an image with reduced shadows.

[0034] The present invention will be described below with reference to examples, but the present invention is not limited to these examples in any way. [Example]

[0035] In this example, a trained model for image generation was constructed that generates images in which breast shadows are suppressed as A1 shadows.

[0036] (1) Data used and creation of dataset Using the XCAT phantom program, a virtual human phantom developed at Duke University, a joint research partner, we performed a series of measurements on 14 women (Body Mass Index (BMI) = 18.2-28.6 kg / m) breathing forcefully at a normal heart rate (60 beats / min). 2 ) was created for 10 phases of one respiratory cycle. Furthermore, we also created an XCAT phantom by removing breast shadows from the XCAT phantom. Bone shadow suppression is preferable for analyzing slight changes in pixel values ​​in the lung field. Therefore, to achieve a similar effect, we created an XCAT phantom by removing bone structures such as ribs, clavicles, and scapulas (see Segars et al. Med.Phys. 37 (9), 2010, 4D XCAT phantom for multimodality imaging research). Using an X-ray projection program (X-ray simulator), an XCAT phantom was simulated with settings of 100 kV tube voltage, 0.2 mAs, 0.4 mm focal spot size, and 200 cm focus-detector distance. The projection angle was also changed 1-2 degrees left and right from the front, creating a 0.01 mAs fluctuation during projection to pad the dataset. Finally, 14 XCAT phantoms were projected with 10 phases each in two patterns, with and without breast shadows, to create a total training dataset of 4,200 images (2,100 each with and without breast shadows) (Figure 1). The matrix size was 512 x 512, 16-bit, PNG format. The training data used is as follows: Original image X created by pseudo-projecting a virtual human phantom including A1 (breast) shadows 1-M :Image with breast shadow Image X created by pseudo-projection of a virtual human phantom with A1 (breast) shadow removed 1-M :Image without breast shadow A1 (Breast) Image obtained by subtracting an image without breast shadow from an image with breast shadow (image of breast only, subtraction image X 1-M , correct image X 1-M ) The above was performed using images from which bone structures had been removed.

[0037] (2) Building a trained model using pix2pix In this example, we used pix2pix, a type of conditional generative adversarial network. pix2pix is ​​a deep learning model that performs image conversion. It learns the correspondence between an output image (breast only) generated from an original image (with breasts) and the original image (breast only), and repeats this process until the original image and the output image become indistinguishable (Phillip Isola et al. CVPR 2017, pp. 1125-1134, Figures 2-3). Training was performed using an Intel(R) Core(TM) i7-8700 CPU, 32.0 GB of memory, and an NVIDIA GeForce GTX 1080 Ti (11 GB) GPU. The training conditions were 500 epochs and a batch size of 30.

[0038] (3) Evaluation method A seven-fold cross-validation method was used to train the system to estimate breast shadow-suppressed images from the original images (Journal of the American Statistical Association 78.382 (1983): 316-331.). Specifically, 3,600 of the 4,200 images (1,800 with breast and 1,800 with breast only) were used for training, and 600 with breast images were used for testing. Images without breast were generated by subtracting the output breast-only images from the with-breast images. Images were evaluated objectively based on the peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM). While both PSNR and SSIM are objective evaluation indices, SSIM is a more subjective index than PSNR. We also visually evaluated the degree of breast shadow reduction in the generated images compared with the original images.

[0039] PSNR is expressed by the following equation (1):

number

number

[0040] SSIM is expressed as the following number (3).

number

[0041] The original image and the breast shadow reduction image output from pix2pix are shown. The output image of the virtual human phantom showed high values ​​of PSNR 32.7dB and SSIM 0.989, and the breast shadow reduction effect was visually confirmed in both the output images of the virtual human phantom and the clinical case (Figures 4-6).

[0042] Because high PSNR and SSIM values ​​were obtained and the reduction of breast shadows was visually confirmed in clinical images, we were able to construct a trained image generation model that generates images with suppressed breast shadows. In addition, in order to build a trained model for image generation with higher performance, it is effective to increase the number of image datasets, optimize parameters such as the number of epochs, reconsider the network structure of the generator (U-Net) and classifier, and increase the matrix size. [Example]

[0043] In this example, A 1-4 We constructed a trained model for image generation that generates images in which mediastinal shadows are suppressed as shadows (aorta, heart, superior vena cava, inferior vena cava).

[0044] (1) Creation of a virtual human phantom Using the "XCAT program" developed at Duke University, a joint research partner, 26 virtual human phantoms (female:male ratio 7:19) were created. The body mass index (BMI) of the virtual human phantoms (kg / m 2 ) for men is 20.3-33.3 kg / m 2 (median 26.0 kg / m 2 , standard deviation 3.6 kg / m 2 ), female 18.2~28.6 kg / m 2 (median 22.3 kg / m 2 , standard deviation 3.5 kg / m 2 ) was. In Example 2, all virtual human phantoms were created with 150 phases per 10 seconds of one respiratory cycle to match the actual DCR. Furthermore, to reproduce lung cancer, simulated tumors were created using the "XCAT program." The tumor diameter was set to 30 mm, which indicates suspected pleural invasion, based on the TNM classification of the Lung Cancer Treatment Guidelines 2023 (Reference: Japan Lung Cancer Society. Lung Cancer Treatment Guidelines, 8th Edition. Kanehara Publishing. 2017;4-6). When creating the virtual human phantom and simulated tumor, an attenuation file containing source attenuation coefficients and a non-uniform rational B-splines (NURBS) file used to project the virtual human phantom and simulated tumor were output. An example of a 3D image of the generated virtual human phantom and simulated tumor is shown in Figure 7. A virtual human phantom with a simulated tumor inserted near the mediastinum was created by adding the NURBS file of the simulated tumor, which was created to be located around the mediastinum, to the NURBS file of the virtual human phantom, thereby recreating a lung cancer patient. Figure 7 also shows the procedure for recreating a lung cancer patient. A simulated tumor was inserted in one location in each of the 16 bodies. Next, to prepare pseudo-chest X-ray images without mediastinal organs, which are necessary for the training dataset, we rewrote the NURBS files so that the mediastinal organs (aorta, heart, superior vena cava, and inferior vena cava) of the 26 virtual human phantoms we created were replaced with lung fields.

[0045] (2) 2D projection of virtual human phantom Using the program "CT projector" developed at Duke University, a collaborative research partner, the created NURBS file was projected in two dimensions from a virtual radiation source to create simulated chest X-ray images (reference: Med. Phys. 2008; 35(8):3800-3808). The imaging conditions were the same as those for an actual DCR: tube voltage 100 kV, tube current time product 0.2 mAs / pulse, Cu additional filter: 0.2 mm, source to image-receptor distance (SID) 2.0 m, and projection in the dorsoventral direction. The geometric arrangement of the two-dimensional projection is shown in Figure 1. The virtual human phantom represents successive respiratory states using the set number of phases. To reproduce the pixel size used in clinical practice, the matrix size was set to 1024 × 1024 pixels and the pixel size to 417.0 μM (Reference: Japan Lung Cancer Society. Guidelines for the Treatment of Lung Cancer, 8th Edition. Kanehara Publishing. 2017;4-6). The output image was adjusted to approximate the actual DCR histogram and converted to an 8-bit grayscale image in PNG format.

[0046] (3) Creating a training dataset In this Example 2, a total of 6,000 pseudo chest X-ray images were prepared from 20 virtual human phantoms, including 3,000 images with mediastinal shadows (hereinafter referred to as original images) and 3,000 images without mediastinal shadows. Next, images without mediastinal shadows were subtracted from the original images (hereinafter referred to as subtraction images) to create images. The flow of creating subtraction images is shown in Figure 8. A training dataset of 6,000 images (3,000 original images and 3,000 subtraction images) was created. In addition, a total of 900 pseudo chest X-ray images with mediastinal shadows were created from six virtual human phantoms to create an evaluation dataset. The training data used is as follows: A 1-4 Original image X created by pseudo-projection of a virtual human phantom including shadows (aorta, heart, superior vena cava, inferior vena cava) 1-M :Image with mediastinal shadow A 1-4(Aorta, heart, superior vena cava, inferior vena cava) Image X created by pseudo-projection of a virtual human phantom with shadows removed 1-M :Image without mediastinal shadow A 1-4 (Aorta, Heart, Superior Vena Cava, Inferior Vena Cava) From the image with shadows, A 1-4 Image obtained by subtracting the image without 1-4 Only image, difference image X 1-M , correct image X 1-M ) The above can also be done on images from which bone structures have been removed.

[0047] (4) Development of mediastinal shadow reduction technology (4-1) Conditional Generative Adversarial Network Model In this Example 2, we adopted pix2pix, a conditional generative adversarial network (cGAN) that generates images from images (Reference: CVPR. 2017; 1125-1134). The objective function of pix2pix used in this Example 2 is shown in the following equation (4). Pix2pix requires a corresponding pair of images, but since a pseudo chest X-ray image without a mediastinal shadow could be created using a virtual human phantom that could remove any organ, a pair of images consisting of an original image (Fig. 8(a)) and a subtraction image (Fig. 8(c)) could be prepared. Pix2pix consists of a generator and a classifier, and is trained until the difference image generated from the input original image becomes indistinguishable from the original difference image. A "U-net" was used for the generator, and a skip connection was added between the encoder and decoder to prevent gradient vanishing. The encoder used a convolution layer, batch normalization, and LeakyReLU (Rectified Linear Unit: ReLU) as the activation function. The decoder used a deconvolution layer, batch normalization, and ReLU as the activation function. The convolution layer performs convolution operations on the input values ​​to reduce the dimensionality, while the deconvolution layer enlarges the image to the same size as the input image. Batch normalization was used to normalize between the batch data used for training. ReLU is a layer that outputs the result applied to the input, always outputting 0 for inputs less than 0, while LeakyReLU is a layer that multiplies inputs less than 0 by a constant value and outputs the result. In this study, the constant value was set to 0. The classifier used was PatchGAN. PatchGAN divides the image input to the classifier into patches of an arbitrary matrix size (70 x 70), outputs each as a value between 0.0 and 1.0, and takes the average as the output of the classifier (Reference: CVPR. 2017; 1125-1134). The output of the discriminator was the output from the final layer, which had a sigmoid activation function after a convolutional layer similar to the encoder. The batch size, which is the number of data used during training, was 20, and the number of epochs, which represents the number of training times, was 950.

[0048]

number

[0049] (4-2) Creation of mediastinal shadow reduction images To quantitatively evaluate the mediastinal shadow reduction technology and track the dynamics of the simulated tumor, a total of 1,800 simulated chest X-ray images were prepared from six virtual human phantoms: 900 original images and 900 images without mediastinal shadows. The 900 original images were input into a generator optimized by learning to obtain estimated difference images. Mediastinal shadow-reduced images were obtained by subtracting the estimated difference images from the original images. An overview of the mediastinal shadow reduction technology is shown in Figure 9.

[0050] (5) Evaluation method We evaluated mediastinal shadow-reduced images and mediastinal shadow-free images obtained by subtracting estimated subtraction images from the original images of six virtual human phantoms using two quantitative indices. The peak signal-to-noise ratio (PSNR) is calculated as the ratio of the square of the maximum signal value to the mean squared error (MSE). PSNR is expressed in dB, and a higher value indicates a smaller difference between the images. Structural similarity (SSIM) is an index that evaluates the similarity between images based on brightness, contrast, and structure. SSIM is expressed between 0 and 1.0, with values ​​closer to 1 indicating a higher degree of similarity between the images (IEEE Transactions on Image Processing. 2004; 13(4):600-612). PSNR was calculated using equation (1) and SSIM using equation (2).

[0051] (6) Application of mediastinal shadow reduction technology to clinical cases (6-1) Target The clinical evaluation of mediastinal shadow reduction technology involved 405 DCR cases performed for clinical purposes at the Department of Thoracic Surgery, Kanazawa University Hospital, between December 2015 and June 2023. Of the 205 cases in which preoperative imaging was performed (F:M = 140:65, age 40-88, median age 70), 33 cases (F:M = 7:27, age 42-85, median age 70) had tumors near the mediastinum, and 10 cases (F:M = 3:7, age 42-79, median age 70) in which the tumor's long diameter was 30 mm or greater and overlapped the mediastinum were included. The characteristics of the patients studied are shown in Table 1.

[0052] [Table 1]

[0053] (6-2) Acquisition of chest X-ray motion images Using a digital radiography system (prototype, Konica Minolta, Tokyo) consisting of an X-ray generator / X-ray tube (DHF-155HII / UH-6QC-07E, Hitachi, Ltd., Chiyoda, Japan) capable of emitting pulsed X-rays and an indirect video-compatible FPD (PaxScan 4343CB, Varian Medical Systems, Salt Lake City, UT, USA), images were taken in a frontal, dorsoventral upright position, with one forced breathing cycle lasting approximately 14–15 seconds (5 seconds of expiration, 2 seconds of breath-holding, 5 seconds of inspiration, 2 seconds of breath-holding) (tube voltage: 100 kV, tube current-time product: 0.2 mA / pulse, imaging rate: 15 frames / second, SID = 2.0 m, Cu-added filter: 0.2 mm) (Reference material: Journal of the Japanese Society of Radiological Technology. 2021; 77(11):1279–1287). The detector surface entrance dose in this case was less than the 1.9 mGy dose limit for two plain chest X-rays (front + side) recommended by the International Atomic Energy Agency (Reference: Radiol Phys Technol. 2016; 9(2):139-153). The chest X-ray video image matrix size was 1024 x 1024, the file format was DICOM, and the grayscale was 16-bit. However, due to limitations in the learning environment, the matrix size was rescaled to 1024 x 1024, the file format was PNG, and the grayscale was 8-bit.

[0054] (7) Tracking of target structures To evaluate the mediastinal shadow reduction technology, we performed tumor tracking on simulated tumors inserted into DCRs reproduced from XCAT and on DCRs from clinical cases in which a mass was present near the mediastinum.

[0055] (7-1) Tracking method In this Example 2, the following two object tracking technologies were compared. Template matching is a tracking method that uses a cross-correlation algorithm and is one of the OpenCV library methods (Reference: Bradski, G. (2000) The OpenCV Library. Dr. Dobb's Journal of Software Tools, 120; 122-125.). Optical flow is a method for extracting vectors formed by pixels after a specific pixel moves between two temporally consecutive images in a video sequence. The amount of tumor displacement per frame was calculated based on the amount of tumor displacement at maximum inspiration and maximum expiration of the target patient, and a search window size of 15 × 15 pixels was used.

[0056] (7-2) Tracking of virtual human phantom Tumors inserted into six virtual human phantoms used to evaluate this model were tracked. Correct tracking results were measured from a video sequence in which only the simulated tumor was projected. Template matching was performed by placing a 90x90 pixel ROI on the simulated tumor in the first frame of the video sequence. In addition, the size of the search window was set to focus tracking only on the tumor periphery. Next, in Optical Flow, feature points were placed on the simulated tumor in the first frame of the video sequence and tracked. The feature points were positioned at the top to avoid the influence of the vertebral body and diaphragm shadows. Each tracking result was used as the ground truth data, and the tracking results of the original video sequence with the simulated tumor inserted and the video sequence with reduced mediastinal shadows were compared using the maximum tracking error and Euclidean distance. The maximum tracking error represents the distance from the ground truth data to the furthest tracking position, and the Euclidean distance is an index that indicates how similar the trajectories of the simulated tumor in the original video sequence and the video sequence with reduced mediastinal shadows are to the ground truth data trajectory. The smaller this value, the higher the similarity between the trajectories. The Euclidean distance is calculated using the following equation (5):

[0057]

number

[0058] (7-3) Clinical Image Tracking This model was used to reduce mediastinal shadows in 10 cases in which the longest diameter of tumors near the mediastinum was 30 mm or greater and the mass overlapped the mediastinum. The trajectory of the lung cancer was measured by a single radiologist and used as the reference data. Each object tracking technique was used to track lung cancer in clinical images with mediastinal shadows and clinical images with reduced mediastinal shadows, and the results were compared with the reference data. The ROI used in template matching was set to add 10 pixels (4.17 mm) to the long and short diameters of the tumor for each patient.

[0059] (8) Results The PSNR and SSIM for each of the six virtual human phantoms are shown in Table 2. The average PSNR was 36.1 dB, and the average SSIM was 0.906. Five of the six virtual human phantoms had high PSNR and SSIM values, and the mediastinal shadow reduction effect after mediastinal shadow reduction processing was visually confirmed (Figure 10).

[0060] [Table 2]

[0061] Figure 11 shows the trajectory of dynamic tracking performed by template matching on the first frame of a video sequence projected in the dorsoventral direction for subject #3, who had squamous cell carcinoma in the right lower lobe of the lung. While a tracking error occurred in the exhalation frame of the original image, which identified an area different from the correct tumor location, the tumor shadow was correctly tracked in the video sequence after mediastinal shadow reduction processing.

[0062] Figure 12 shows the trajectory of dynamic tracking using optical flow in the first frame of a video sequence projected in the dorsoventral direction for subject #4, who had squamous cell carcinoma in the right lower lobe of the lung. In the original video sequence, feature points erroneously tracked the mediastinal shadow from the expiration frame (frame 100) onwards. On the other hand, in the video sequence that had undergone mediastinal shadow reduction processing, the tumor was tracked until the final frame.

[0063] Because high PSNR and SSIM values ​​were obtained and the reduction of mediastinal shadows was visually confirmed in clinical images, we were able to construct a trained image generation model that generates images with suppressed mediastinal shadows. Furthermore, even when the correct tumor position could not be tracked in the original image, the tumor shadow could be tracked correctly in the video image that had undergone mediastinal shadow reduction processing. As a result, the visibility of tumor shadows has been greatly improved, making it possible to use the system for accurate preoperative evaluation and early detection of lung cancer. [Example]

[0064] In this example, we constructed a trained model for image generation that generates images in which the shadows of subdiaphragmatic organs (liver, gallbladder, pancreas, spleen, small intestine, large intestine, kidneys, and vascular shadows (hepatic artery to renal artery)) that overlap with the lungs are suppressed.

[0065] (Creating a dataset) Using the XCAT phantom developed at Duke University, a joint research partner, we conducted a study in which participants were forced to breathe at a normal heart rate (60 breaths / min) and had a Body Mass Index (BMI) of 23-33 (kg / m). 2 Nine phases of one respiratory cycle were created for 19 subjects (14 males, 5 females). Under the same conditions, XCAT phantoms were also created with the subdiaphragmatic organs removed, with only the lungs, and with only the subdiaphragmatic organs. These were named phantom DS, phantom L, and phantom D, respectively. We also prepared 15 unprocessed XCAT phantoms and 15 phantoms with subdiaphragmatic organs removed (11 males and 4 females). A pseudo-tumor was placed in a position that was completely hidden by the subdiaphragmatic organs when projected from the front (phantom AL). A pseudo-tumor was embedded in the phantom with the subdiaphragmatic organs removed under the same conditions (phantom DSL). The pseudo-tumor was assumed to be a solid tumor measuring 1.5 to 2.0 cm (Figure 13). Using a CT projector developed at Duke University, we generated DCR images with a matrix size of 256 × 256 pixels and 512 × 512 pixels for the XCAT phantoms under the following conditions: tube voltage 100 kV, 0.2 mAs, focal spot size 0.4 mm, and focus-detector distance 200 cm. As a result, 171 pseudo chest X-ray images with diaphragmatic shadows and 135 pseudo chest X-ray images with subdiaphragmatic and tumor shadows were created (Figure 1). Additionally, 171 pseudo chest X-ray images without diaphragmatic shadows and 135 pseudo chest X-ray images without subdiaphragmatic shadows but with tumor shadows were created. Using ImageJ (ver. 1.53), we removed only the organ areas overlapping with the lung fields and left the non-overlapping areas (Figure 14). Specifically, we created a standard phantom and a virtual human phantom with the organs below the diaphragm removed. Next, we captured chest X-ray sequences of the created phantoms. The original images were then masked to remove the areas not overlapping with the lung fields, and the mask was applied to the phantom projection image with the organs removed to create the correct image. A total of 1,206 images (603 sets) were prepared for training, including images with different matrix sizes. 78% of the training dataset was used for training, and 22% for validation (Table 3). Images were in PNG format and 8-bit.

[0066] [Table 3]

[0067] (Building a learning model) In this Example 3, we used pix2pix, a conditional generative adversarial network. pix2pix is ​​a deep learning model that mainly performs image conversion. It is designed to learn the correspondence between pseudo chest X-ray videos without diaphragm shadows (output data) generated from pseudo chest X-ray videos with diaphragm shadows (source data) and pseudo chest X-ray videos with the diaphragm removed (ground truth data), and repeats learning until the ground truth data and output data become indistinguishable (Figure 17). The learning environment was an Intel(R) Core(TM) i9-9900K CPU, 32.0 GB of memory, and an NVIDIA GeForce RTX 2080 SUPER (8 GB). The conditions were 700 epochs and 20 batch sizes.

[0068] The images were evaluated by visually comparing the reduction of the diaphragm shadow with the original image, and quantitatively using the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) to evaluate the degree of difference and similarity between the output data and the correct data (References: IEEE 2010 International Conference on Pattern Recognition, pp. 2366-2369, IEEE Transactions on Medical Imaging 2004; 13(4)600-612). PSNR was calculated using number (1), and SSIM was calculated using number (3).

[0069] (result) Figure 18 shows images at maximum inspiration and maximum expiration from the output video of the process to reduce the shadows of organs below the diaphragm, obtained through deep learning using pix2pix. Figure 19 also shows images at maximum inspiration and maximum expiration when reduction processing was performed on a clinical case (matrix size: 512 x 512 pixels). Figure 20 shows the image before processing, including the tumor shadow, and Figure 21 shows the output video of a tumor overlapping the organ shadow. Reducing the diaphragm shadow improved the visibility of the lung field that was hidden in the relevant area, and the tumor was clearly depicted. Furthermore, high PSNR and SSIM values ​​were obtained under all conditions (Table 4).

[0070] [Table 4]

[0071] In both respiratory phases, we were able to reduce the shadows of organs below the diaphragm that overlap with the lung field. Furthermore, we confirmed the superiority of this model by being able to reduce only the organ shadows while leaving the tumor shadows intact. This makes it possible to use this model for pulmonary function analysis of lung fields that overlap with organ shadows and for tumor detection. Specifically, it is possible to perform pulmonary function analysis of lungs that overlap with organs below the diaphragm, and to detect the presence of tumors behind the diaphragm (determine tumor infiltration) from the frontal image. Furthermore, in order to create a better model, it would be effective to add more types of body shapes for the XCAT phantoms used in the training data, training data with tumors, optimize parameters such as the number of epochs, and reconsider the network structure of the generator (U-Net) and classifier. [Example]

[0072] In this example, unlike Example 3, a trained model for image generation was constructed using the method described in Figures 15 and 16 to generate images in which subdiaphragmatic organ shadows are suppressed. A combination of a clinical case image (original image) and the generated image is shown in Figure 22. It was confirmed that the generated image successfully reduced the shadows of the subdiaphragmatic organs that overlap with the lung field.

[0073] (General remarks) The trained model for image generation of the present invention is input with a moving chest X-ray image capturing the respiratory state, and is able to generate shadow A. N Furthermore, it is now possible to use these images to evaluate lung function, such as changes in lung area and diaphragm movement, and to detect tumors. Currently, pulmonary function imaging diagnosis can be performed using specialized imaging techniques with expensive medical equipment such as CT and MRI. However, due to the cost, examination time, and patient exposure to CT, it is not performed as a routine examination. By combining the trained image generation model of this invention with X-ray examination, simple and rapid pulmonary function imaging diagnosis support (particularly dynamic and functional evaluation of the thorax, lungs, and diaphragm from the front, and tumor detection) can be realized for the purposes of initial examination and follow-up observation.

Claims

1. A N A from biological images including shadows N Image with only shadows or A N An image generating device for generating an image with reduced shadows, comprising: 1) A N and an original image X created by pseudo-projecting three-dimensional data of a virtual human phantom or real human body including the target tissue / organ O containing the target region T. 1-M and correct image X 1-M a learning data storage unit that stores the above as learning data; 2) Original image X 1-M and the correct image X 1-M Using A N A from biological images including shadows N Image with only shadows or A N a model construction unit that constructs a learning model for generating an image with reduced shadows; 3) A N a biometric image receiving unit that receives a biometric image including a shadow; and 4) Applying the biometric image accepted by the biometric image accepting unit to the learning model constructed by the model constructing unit, A N A from biological images including shadows N Image with only shadows or A N an image generation unit that generates an image with reduced shadows; An image generating device comprising: Correct image X 1-M teeth, a) A N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M From A N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M Image X obtained by subtracting 1-M (Difference image X 1-M ) b) A N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M In A N The masked image part where the target area T part that does not overlap with A is masked. N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M Image X created by applying 1-M (Masking image X 1-M ) c) A N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M (A N Image X with 1-M ) d) A with bone structure removed N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M From bone structure and A N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M Image X obtained by subtracting 1-M (Bone structure removed difference image X 1-M ) selected from any one of Image generating device.

2. The correct image X 1-M The imaging device of claim 1 , wherein the target tissue / organ O includes a pseudo-tumor in or near the target tissue / organ O.

3. The correct image X 1-M is the difference image X 1-M or the bone structure removed difference image X 1-M 3. The imaging device according to claim 1, wherein:

4. The correct image X 1-M is the masked image X 1-M 3. The imaging device according to claim 1, wherein:

5. The above A N The image with reduced shadows is the original image X 1-M A multiplied by a weighting factor from N 3. The imaging device according to claim 1, wherein the image is obtained by subtracting an image of only shadows.

6. Original image X 1-M 3. The image generating apparatus according to claim 1, wherein the image is generated by pseudo-projecting a virtual human phantom.

7. The above A N The imaging device according to claim 3 , wherein the target tissue / organ O is the aorta, the heart, the superior vena cava, and / or the inferior vena cava, and the target tissue / organ O is the lung.

8. The above A N 5. The imaging device of claim 4, wherein the target region T is a lung field.

9. The above A N The imaging device according to claim 3 , wherein the target tissue / organ O is a breast and the target tissue / organ O is a lung.

10. The image generating device according to claim 2 , wherein the model constructing unit is further capable of detecting a tumor present in or near the target tissue / organ O.

11. Furthermore, the above-mentioned A N 6. The image generating device according to claim 1, further comprising a diagnostic support unit that matches the image with reduced shadows with a known disease medical database to present predicted diseases.

12. The diagnosis support device according to claim 1 , wherein the learning model is constructed from one or more of the following: 1) Generative Adversarial Networks 2) pix2pix 3) CycleGAN

13. A N A from biological images including shadows N Image with only shadows or A N A trained model for image generation that generates an image with reduced shadows, The trained model is 1) As training data, A N and an original image X created by pseudo-projecting three-dimensional data of a virtual human phantom or real human body including the target tissue / organ O containing the target region T. 1-M and correct image X 1-M , and 2) Original image X 1-M and the correct image X 1-M By using machine learning, A N A from biological images including shadows N Image with only shadows or A N a learning model for generating an image with reduced shading; Here, the correct image X 1-M teeth, a) A N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M From A N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M Image X obtained by subtracting 1-M (Difference image X 1-M ) b) A N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M In A N The masked image part where the target area T part that does not overlap with A is masked. N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M Image X created by applying 1-M (Masking image X 1-M ) c) A N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M (A N Image X with 1-M ) d) A with bone structure removed N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M From bone structure and A N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M Image X obtained by subtracting 1-M (Bone structure removed difference image X 1-M ) selected from any one of A pre-trained model for image generation.

14. A N A from biological images including shadows N Image with only shadows or A N 1. A data structure for generating an image with reduced shadows, comprising: The data structure is: 1) As training data, A N and an original image X created by pseudo-projecting three-dimensional data of a virtual human phantom or real human body including the target tissue / organ O containing the target region T. 1-M and correct image X 1-M , and 2) Original image X 1-M and the correct image X 1-M By using machine learning, A N A from biological images including shadows N Image with only shadows or A N Including data of a trained model for generating an image with reduced shading, Here, the correct image X 1-M teeth, a) A N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M From A N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M Image X obtained by subtracting 1-M (Difference image X 1-M ) b) A N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M In A N The masked image part where the target area T part that does not overlap with A is masked. N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M Image X created by applying 1-M (Masking image X 1-M ) c) A N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M (A N Image X with 1-M ) d) A with bone structure removed N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M From bone structure and A N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M Image X obtained by subtracting 1-M (Bone structure removed difference image X 1-M ) selected from any one of Data structures for image generation.

15. A N A from biological images including shadows N Image with only shadows or A N An image generation program for generating an image with reduced shadows, The program 1) A N and an original image X created by pseudo-projecting three-dimensional data of a virtual human phantom or real human body including the target tissue / organ O containing the target region T. 1-M and correct image X 1-M By using machine learning, A N A from biological images including shadows N Image with only shadows or A N a trained model generation step for generating an image with reduced shadows; and 2) A N By applying the trained model to biological images containing shading, A N Image with only shadows or A N creating an image with reduced shadows; Here, the correct image X 1-M teeth, a) A N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M From A N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M Image X obtained by subtracting 1-M (Difference image X 1-M ) b) A N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M In A N The masked image part where the target area T part that does not overlap with A is masked. N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M Image X created by applying 1-M (Masking image X 1-M ) c) A N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M (A N Image X with 1-M ) d) A with bone structure removed N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M From bone structure and A N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M Image X obtained by subtracting 1-M (Bone structure removed difference image X 1-M ) selected from any one of A program for generating images.

16. A N A from biological images including shadows N A diagnostic support method for providing an image with reduced shadows, comprising: The method comprises: 1) A N By applying the trained model to biological images containing shading, A N A from biological images including shadows N Image with only shadows or A N creating an image with reduced shadows; and 2) matching the image created in 1) with a known disease medical database to present predicted diseases; Here, the trained model is 1) As training data, A N and an original image X created by pseudo-projecting three-dimensional data of a virtual human phantom or real human body including the target tissue / organ O containing the target region T. 1-M and correct image X 1-M , and 2) Original image X 1-M and the correct image X 1-M By using machine learning, A N Image with only shadows or A N a learning model for generating an image with reduced shading; Here, the correct image X 1-M teeth, a) A N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M From A N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M Image X obtained by subtracting 1-M (Difference image X 1-M ) b) A N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M In A N The masked image part where the target area T part that does not overlap with A is masked. N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M Image X created by applying 1-M (Masking image X 1-M ) c) A N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M (A N Image X with 1-M ) d) A with bone structure removed N Original image X created by pseudo-projecting 3D data of a virtual human phantom or real human body, including 1-M From bone structure and A N Image X created by pseudo-projecting 3D data of a virtual human phantom or real human body from which 1-M Image X obtained by subtracting 1-M (Bone structure removed difference image X 1-M ) is selected from one of and The A N The image with only shadows is the original image X 1-M Multiply by the weighting factor to get A N Converted into an image with reduced shadows, Diagnostic support methods.

Images