Method for creating a pre-trained model and generating images

Abstract

To provide an image generation method and an image processing device capable of performing image processing on various image elements and on multiple image elements, and provide a method for creating a learned model capable of efficiently creating a learned model used for the image processing.SOLUTION: A method for creating a learned model generates a reconstructed image (60) obtained by reconstructing three-dimensional X-ray image data (80). A projection image (61) is generated from a three-dimensional model of an image element (50) through simulation. The projection image is superimposed on the reconstructed image to create a superimposed image (67). A learned model (40) is created by performing machine learning using the superimposed image, the reconstructed image or the projection image.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a method for creating a learned model and Image generation method In the law and is related thereto.

Background Art

[0002] Conventionally, a method for performing image processing using a learned model has been known. Such a method is disclosed, for example, in International Publication No. 2019 / 138438.

[0003] International Publication No. 2019 / 138438 discloses creating an image representing a specific part by performing conversion on an X-ray image of a region including a specific part of a subject using a learned model. In International Publication No. 2019 / 138438, examples of the specific part include a bone part of a subject, a blood vessel into which a contrast agent has been injected, and a stent implanted in the body. The learned model is created by performing machine learning on a first DRR (Digitally Reconstructed Radiography) image reconstructed from CT image data and a second DRR image as a teacher input image and a teacher output image, respectively. By differentiating the image obtained by the conversion using the learned model from the original image, an image with the specific part removed can be generated.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] International Publication No. 2019 / 138438 discloses image processing for a specific type of image element (specific site), such as creating an image representing a bone region, an image representing a blood vessel into which contrast agent has been injected, or an image representing a stent placed in the body. However, in order to improve the visibility of medical images in diverse usage scenarios, it is desirable to be able to perform image processing not only on a specific image element (specific site), but also on various image elements, and even on multiple image elements.

[0006] Furthermore, in the aforementioned International Publication No. 2019 / 138438, machine learning is performed using the first and second DRR images reconstructed from CT image data, which requires the preparation of CT image data that actually contains the image elements (specific regions) to be extracted. In order to efficiently perform machine learning on various image elements, it is desirable to enable machine learning not only using CT image data that actually contains the image elements to be extracted, but also to enable machine learning for image elements that are difficult to isolate and extract even if they are included in the CT image data.

[0007] This invention was made to solve the above-mentioned problems, and one of its objectives is to provide an image generation method and an image processing apparatus that can perform image processing on various image elements and on multiple image elements, and to provide a method for creating a trained model that can efficiently create a trained model used in such image processing. [Means for solving the problem]

[0008] To achieve the above objective, the method for creating a trained model in the first aspect of this invention reconstructs 3D X-ray image data into a 2D projection image. Ta Reconstructed images Multiple images for each of the multiple image elements to be extracted generate, If the 3D X-ray image data contains image elements, those elements have been removed from the reconstructed image. Through simulation, a 2D projection image is obtained from a 3D model of image elements. multiple for each image element Generate and project the image of the image elements handle The reconstructed image is superimposed to generate a superimposed image, and this superimposed image is used as the training input data. 、 By performing machine learning using projected images as training output data, Includes X-ray images of the subject. This process extracts image elements from the input image. Alternatively, the superimposed image can be used as training input data, and the reconstructed image as training output data to perform machine learning, thereby generating an image from which image elements have been removed from the input image. Create a trained model and superimpose the image 、 For each image element, By superimposing the projection image of the corresponding image element onto the corresponding reconstructed image, Multiple images are created, and each image element includes a first element which is biological tissue and a second element which is non-biological tissue. 。

[0009] This invention 2 The image generation method in this phase is: Created by the method for creating a trained model in the first phase described above. Using a pre-trained model, multiple image elements are separated and extracted from an X-ray image. By performing image-to-image operations using the extracted images, each showing a different image element, and the original X-ray image, a processed image is generated in which each image element in the X-ray image has undergone image processing.

[0011] In this specification, the term "extraction of image elements" is a broad concept that includes both generating an image representing the extracted image elements and generating an X-ray image from which the extracted image elements have been removed. More specifically, the "extraction of image elements" includes generating an image containing only the image elements and generating an image from which the image elements have been removed from the original X-ray image. Furthermore, "inter-image operation" means generating a single image by performing operations such as addition, subtraction, multiplication, and division between one image and another. More specifically, "inter-image operation" means performing pixel value calculations for each corresponding pixel between multiple images to determine the pixel value of that pixel in the resulting image. [Effects of the Invention]

[0012] According to the method for creating a trained model in the first phase described above, a superimposed image is used as training input data. This superimposed image is obtained by reconstructing 3D X-ray image data into a 2D projection image, and superimposing it with a 2D projection image generated from a 3D model of the image element to be extracted through simulation. The reconstructed image or projection image is used as training output data. Therefore, even if the 3D X-ray image data does not contain the image element to be extracted, machine learning can be performed using the image element to be extracted generated by simulation. In other words, training data can be prepared without preparing CT image data that actually contains the image element to be extracted. Furthermore, since projection images of the image element to be extracted are generated by simulation, training data can also be prepared for image elements that are included in CT image data but are difficult to separate and extract. As a result, it is possible to efficiently create trained models for image processing on various image elements, and on multiple image elements.

[0013] Furthermore, according to the image generation method in the second phase and the image processing device in the third phase, a trained model that has learned to extract specific image elements from an input image is used to separately extract multiple image elements from an X-ray image. A processed image is generated by performing inter-image operations using the multiple extracted images extracted for each image element and the X-ray image. As a result, various image elements are separately extracted as extracted images from the input X-ray image, and each extracted image can be freely added to or subtracted from the X-ray image depending on the type of extracted image element. Consequently, image processing can be performed on various image elements and on multiple image elements. [Brief explanation of the drawing]

[0014] [Figure 1] This is a block diagram showing an image processing device according to one embodiment. [Figure 2] This figure shows an example of an X-ray imaging device. [Figure 3] This diagram illustrates machine learning and trained models in relation to learning models. [Figure 4] Figure showing an example of an image element [Figure 5] Figure showing a first example of extraction of image elements by a learned model [Figure 6] Figure showing a second example of extraction of image elements by a learned model and generation of a processed image [Figure 7] Figure showing an example of performing image processing on an extracted image, different from FIG. 6 [Figure 8] Flowchart for explaining an image generation method according to an embodiment [Figure 9] Flowchart for explaining a method for creating a learned model according to an embodiment [Figure 10] Figure for explaining generation of a superimposed image used in machine learning [Figure 11] Figure for explaining a method for generating a reconstructed image<000009~ [Figure 12] Figure showing examples of teacher input data and teacher output data when the image element is bone [Figure 13] Figure showing an example of a processed image when the image element is bone [Figure 14] Figure showing examples of teacher input data and teacher output data when the image element is a device s [Figure 15] Figure showing an example of a processed image when the image element is a device [Figure 16] Figure showing variations (A) to (I) of a projection image of a device generated from a 3D model [Figure 17] Figure showing examples of teacher input data and teacher output data when the image element is noise [Figure 18] Figure showing an example of a processed image when the image element is noise [Figure 19] Figure showing examples of teacher input data and teacher output data when the image element is a blood vessel [Figure 20] Figure showing an example of a processed image when the image element is a blood vessel [Figure 21] This figure shows examples of training input data and training output data when the image element is clothing. [Figure 22] This figure shows an example of a processed image when the image element is clothing. [Figure 23] This figure shows examples of training input data and training output data when the image elements are X-ray scattered radiation components. [Figure 24] This figure shows an example of a processed image when the image element is an X-ray scattering component. [Figure 25] This figure shows the projection angle ranges for the first direction (A) and second direction (B) of X-rays in a Monte Carlo simulation. [Figure 26] This figure shows the first example of the X-ray energy spectrum in a Monte Carlo simulation. [Figure 27] This figure shows a second example of the X-ray energy spectrum in a Monte Carlo simulation. [Figure 28] This figure shows an example of a collimator image used for training input data and training output data. [Figure 29] This figure illustrates an example of extracting image elements using multiple pre-trained models. [Modes for carrying out the invention]

[0015] The following describes embodiments of the present invention based on the drawings.

[0016] Referring to Figures 1 to 28, the configuration of the image processing device 100 according to one embodiment, the image generation method according to one embodiment, and the method for creating a trained model according to one embodiment will be described.

[0017] (Configuration of the image processing device) First, the configuration of the image processing device 100 will be described with reference to Figure 1.

[0018] The image processing device 100 is configured to extract image elements 50 contained in the X-ray image 201 using a trained model 40 created by machine learning, and to perform image processing on the X-ray image 201 using the extracted image elements 50. The image processing device 100 takes the X-ray image 201 captured by the X-ray imaging device 200 as input and generates a processed image 22 as output in which image processing has been performed on each image element 50 contained in the X-ray image 201.

[0019] As shown in Figure 1, the image processing device 100 comprises an image acquisition unit 10, an extraction processing unit 20, and an image generation unit 30. The image processing device 100 is composed of a computer equipped with one or more processors 101 such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit), and one or more storage units 102 such as an HDD (Hard Disk Drive) and an SSD (Solid State Drive). The image processing device 100 is connected to a display device 103.

[0020] The image acquisition unit 10 is configured to acquire X-ray images 201. The image acquisition unit 10 is configured, for example, by an interface for communicating with an external device and the image processing device 100. The interface of the image acquisition unit 10 may include a communication interface such as a LAN (Local Area Network). The image acquisition unit 10 may include input / output interfaces such as HDMI®, DisplayPort, and USB ports. The image acquisition unit 10 can acquire X-ray images 201 from the X-ray imaging device 200 or from a server device connected via a network by communication.

[0021] X-ray image 201 is a medical image taken of a patient or subject by an X-ray imaging device 200. X-ray image 201 may be either a still image or a moving image. A moving image is a collection of still images taken at a predetermined frame rate. X-ray image 201 is a two-dimensional image. X-ray image 201 can be various images taken by methods such as plain radiography, fluoroscopy, or angiography.

[0022] The extraction processing unit 20 is configured to separately extract multiple image elements 50 from the X-ray image 201 using a trained model 40 stored in the memory unit 102. The trained model 40 is a trained model that has been trained to extract specific image elements 50 from the input image.

[0023] Image elements 50 are image portions or image information that constitute the X-ray image 201, and are defined as identical or similar groups. Image elements 50 can be anatomically classified parts of the human body. Such image elements 50 are, for example, living tissues such as bones and blood vessels. Image elements 50 can be objects introduced or implanted in the body of a subject during surgery or the like. Such image elements 50 can be, for example, devices such as catheters, guidewires, stents, surgical instruments, and fixation devices introduced into the body. Image elements 50 can be noise, artifacts, scattered X-ray components, etc. that occur during the imaging process in X-ray photography. Image elements 50 can be clothing worn by the subject that is captured in the image during photography. Clothing is a concept that includes garments, ornaments, and other attached items. For example, buttons, zippers, accessories, and metal parts of clothing may be captured in the X-ray image 201.

[0024] The pre-trained model 40 is created in advance by machine learning, which trains it to extract specific image elements 50 from an input image. The extraction processing unit 20 uses one or more pre-trained models 40 to extract the image elements 50. As a result, the extraction processing unit 20 generates multiple extracted images 21, each containing different image elements 50 extracted from the X-ray image 201. For example, the first extracted image 21 contains the first image element 50, and the second extracted image 21 contains a second image element 50 that is different from the first image element 50. The method for creating the pre-trained model 40 will be described later.

[0025] The image generation unit 30 is configured to generate a processed image 22 in which image processing has been performed on each image element 50 included in the X-ray image 201 by performing inter-image operations using a plurality of extracted images 21 extracted for each image element 50 and the X-ray image 201. The image processing includes, for example, enhancement processing or removal processing of the image element 50. Enhancement processing is a process that relatively increases the pixel value of the pixels belonging to the image element 50. Removal processing is a process that relatively decreases the pixel value of the pixels belonging to the image element 50. Removal processing includes not only complete removal from the image but also partial removal to reduce visibility. Enhancement processing may be, for example, edge enhancement processing. Removal processing may be noise reduction processing.

[0026] Inter-image operation is the process of determining the pixel value of a corresponding pixel in a processed image 22 by performing a pixel value calculation for each corresponding pixel between multiple extracted images 21 and the X-ray image 201. The content of the calculation is not particularly limited, but for example, it may be the four basic arithmetic operations of addition, subtraction, multiplication, and division. In this embodiment, inter-image operation includes performing weighted addition or weighted subtraction of individual extracted images 21 to the X-ray image 201. By performing weighted addition of extracted images 21 to the X-ray image 201, image elements 50 contained in the X-ray image 201 can be enhanced. By performing weighted subtraction of extracted images 21 to the X-ray image 201, image elements 50 contained in the X-ray image 201 can be removed. By adjusting the weight values, the degree of enhancement or removal of image elements 50 can be optimized.

[0027] In the example shown in Figure 1, the processor 101 functions as the extraction processing unit 20 and the image generation unit 30 by executing a program (not shown) stored in the memory unit 102. In other words, in the example shown in Figure 1, the extraction processing unit 20 and the image generation unit 30 are implemented as functional blocks of the processor 101. The extraction processing unit 20 and the image generation unit 30 may also be configured as separate hardware components.

[0028] Furthermore, "individual hardware" includes the fact that the extraction processing unit 20 and the image generation unit 30 are configured by separate processors. "Individual hardware" also includes the fact that the image processing device 100 includes multiple computers (PCs), and that a computer (PC) functioning as an extraction processing unit and a computer (PC) functioning as an image generation unit are provided separately.

[0029] The image processing device 100 displays the processed image 22 generated by the image generation unit 30 on the display device 103. The image processing device 100 transmits the generated processed image 22 to a server device via a network. The image processing device 100 records the generated processed image 22 in the storage unit 102.

[0030] Figure 2 shows an example configuration of the X-ray imaging apparatus 200. Figure 2 shows an example of an X-ray imaging apparatus capable of performing vascular fluoroscopy. The X-ray imaging apparatus 200 includes a top plate 210, an X-ray irradiation unit 220, and an X-ray detector 230. The top plate 210 is configured to support the subject 1 (person). The X-ray irradiation unit 220 includes an X-ray source such as an X-ray tube and is configured to irradiate X-rays toward the X-ray detector 230. The X-ray detector 230 is configured, for example, as an FPD (Flat Panel Detector) and is configured to detect X-rays irradiated from the X-ray irradiation unit 220 and transmitted through the subject 1.

[0031] In the example shown in Figure 2, the X-ray irradiation unit 220 and the X-ray detector 230 are held by a C-arm 240. The C-arm 240 is movable in a first direction 250 along its arc-shaped arm and rotatable in a second direction 252 around a rotation axis 251. This allows the X-ray imaging apparatus 200 to change the projection direction of X-rays from the X-ray irradiation unit 220 toward the X-ray detector 230 by predetermined angular ranges in the first direction 250 and the second direction 252, respectively.

[0032] (Pre-trained model) As shown in Figure 1, the extraction of image elements 50 contained in the X-ray image 201 is performed by a trained model 40 created by machine learning. As shown in Figure 3, the trained model 40 extracts the pre-trained image elements 50 from the input image and outputs an extracted image 21 that displays only the extracted image elements 50.

[0033] In this embodiment, the trained model 40 is pre-created by machine learning using a reconstructed image 60 obtained by reconstructing a 2D projected image from 3D image data, and a projected image 61 created from a 3D model of the image elements 50 by simulation.

[0034] Any machine learning method can be used, such as a fully convolutional neural network (FCN), a neural network, a support vector machine (SVM), or boosting. In this embodiment, a convolutional neural network, more preferably a fully convolutional neural network, is used for the learning model LM (trained model 40). Such a learning model LM (trained model 40) consists of an input layer 41 into which an image is input, a convolutional layer 42, and an output layer 43.

[0035] To create a trained model 40, machine learning is performed using training data 66, which includes training input data 64 and training output data 65. The training input data 64 and training output data 65 included in a single training data set 66 represent the relationship between the pre-extraction data and the post-extraction data for the same image element 50.

[0036] Machine learning is performed for each of the 50 image elements to be extracted. In other words, training data 66 is prepared for each of the 50 image elements to be extracted.

[0037] As shown in Figure 4, the multiple image elements 50 include a first element 51 which is biological tissue and a second element 52 which is non-biological tissue. The multiple image elements 50 also include at least several of the following: bone 53, blood vessels 54, a device introduced into the body 55, clothing 56, noise 57, and scattered X-ray components 58. Of these, bone 53 and blood vessels 54 correspond to the first element 51. The first element 51 may also include biological tissue other than bone 53 and blood vessels 54. Of these, the device introduced into the body 55, clothing 56, noise 57, and scattered X-ray components 58 correspond to the second element 52. The second element 52 may also include image elements other than the device 55, clothing 56, noise 57, and scattered X-ray components 58.

[0038] In the example shown in Figure 5, a single trained model 40 is configured to extract multiple image elements 50 separately. The trained model 40 has one input channel and multiple (N) output channels, where N is an integer greater than or equal to 2. When an X-ray image 201 is input to the input channel, the trained model 40 extracts N image elements 50 separately. The trained model 40 outputs the extracted 1st to Nth image elements 50 from the N output channels as the first extracted image 21-1 to the Nth extracted image 21-N, respectively.

[0039] As an example different from Figure 5, in the example in Figure 6, the trained model 40 has one input channel and N+1 output channels. When an X-ray image 201 is input to the input channel, the trained model 40 extracts multiple (N) image elements 50 from the input image without duplication. "Without duplication" means that the image information extracted in any of the extracted images (e.g., the first extracted image 21-1) is not included in any of the other extracted images (e.g., the second extracted image 21-2 to the Nth extracted image 21-N). The trained model 40 outputs the extracted 1st to Nth extracted image elements as the first extracted image 21-1 to the Nth extracted image 21-N, respectively. Then, the trained model 40 outputs the remaining image elements 59 after extraction as the N+1th extracted image 21x from the N+1th output channel.

[0040] In this case, the first extracted images 21-1 to the Nth extracted images 21-N do not contain any identical image information from each other. Furthermore, the image information that remains unextracted from the input X-ray image 201 is included in the N+1th extracted image 21x. Therefore, adding the first extracted images 21-1 to the Nth extracted images 21-N and the N+1th extracted image 21x will return to the original X-ray image 201.

[0041] As shown in the example in Figure 6, the trained model 40 is configured to extract multiple image elements 50 from the input image without duplication, and to output the extracted multiple image elements 50 and the remaining image elements 59 after extraction. As a result, the total amount of image information extracted by the extraction process does not increase or decrease compared to the input image.

[0042] As shown in Figure 6, the image generation unit 30 (see Figure 1) performs inter-image operations on the input X-ray image 201, adding or subtracting the first extracted image 21-1 to the Nth extracted image 21-N by applying a weight coefficient 23, thereby generating a processed image 22. The N+1th extracted image 21x, representing the remaining image elements 59, does not need to be used in image processing. The weight coefficient 23 is set individually, with coefficients w1 to wn corresponding to each of the first extracted image 21-1 to the Nth extracted image 21-N. The weight coefficient 23 can be, for example, a fixed value set in the storage unit 102 beforehand. However, multiple types of weight coefficients may be set depending on the intended use of the processed image 22. For example, in processing mode A, the first set value of the weight coefficient 23 is used for the first extracted image 21-1, and in processing mode B, the second set value of the weight coefficient 23 is used for the first extracted image 21-1. Furthermore, the weight coefficient 23 may be set to any value according to user input.

[0043] As shown in Figure 7, image processing may be performed on each extracted image 21 before performing inter-image operations between each extracted image 21 and the X-ray image 201. In the example in Figure 7, the image generation unit 30 is configured to perform image processing separately on some or all of the multiple extracted images 21. The processed image 22 is generated by inter-image operations between the multiple extracted images 21 after image processing and the X-ray image 201.

[0044] For example, in Figure 7, the image generation unit 30 performs the first image processing 25-1 on the first extracted image 21-1, the second image processing 25-2 on the second extracted image 21-2, ..., and the Nth image processing 25-N on the Nth extracted image 21-N. Since it may not be necessary to perform image processing on some of the image elements 50, image processing may be performed only on some of the extracted images 21.

[0045] The image processing performed on each extracted image is not particularly limited, but could be, for example, image correction processing or image interpolation processing. Image correction processing may include edge enhancement processing and noise reduction processing. Image correction processing could include, for example, contrast adjustment, line enhancement processing, and smoothing processing. Image interpolation processing is the process of interpolating the broken parts of image elements 50 that appear to be interrupted in the X-ray image 201 because they are difficult to capture, such as guide wires and catheters. For contrast adjustment and line enhancement processing, the appropriate parameters differ for each image element 50, and it is difficult to process all image elements 50 at once, but by performing image processing on each extracted image 21, optimal image processing can be performed on each image element 50.

[0046] (Image generation method) Next, the image generation method of this embodiment will be described with reference to Figure 8. The image generation method can be carried out by the image processing device 100. The image generation method of this embodiment comprises at least the following steps S2 and S5 shown in Figure 8. (S2) Using a trained model 40 that has been trained to extract specific image elements 50 from an input image, multiple image elements 50 are extracted separately from the X-ray image 201. (S5) By performing inter-image operations using the multiple extracted images 21 extracted for each image element 50 and the X-ray image 201, a processed image 22 is generated in which each image element 50 included in the X-ray image 201 has undergone image processing. Furthermore, the image generation method of this embodiment may further include steps S1, S3, S4, and S6 shown in Figure 8.

[0047] In step S1, an X-ray image 201 is acquired. Specifically, the image acquisition unit 10 (see Figure 1) acquires the X-ray image 201 taken by, for example, the X-ray imaging device 200 shown in Figure 2, through communication with the X-ray imaging device 200 or the server device.

[0048] In step S2, the extraction processing unit 20 uses the trained model 40 to separately extract multiple image elements 50 from the X-ray image 201. The extraction processing unit 20 inputs the X-ray image 201 acquired in step S1 to the trained model 40. As a result, the trained model 40 outputs the first extracted image 21-1 to the Nth extracted image 21-N, as shown in Figure 5 or Figure 6.

[0049] In step S3, as shown in Figure 7, image processing may be performed on some or all of the extracted images 21. In this case, the image generation unit 30 performs pre-set image processing on the extracted images 21 that are the target of image processing with predetermined parameters. Whether or not to perform image processing may be determined based on input from the user. Whether or not to perform image processing may be determined based on the image quality of the extracted images 21. Step S3 is optional.

[0050] In step S4, the image generation unit 30 obtains calculation parameters for each extracted image 21. The calculation parameters include, for example, the setting value of the weight coefficient 23 and the setting value of the calculation method. The setting value of the calculation method indicates whether to perform weighted addition (i.e., enhancement processing of image elements 50) or weighted subtraction (i.e., removal processing of image elements 50) for the target extracted image 21. The setting value of the calculation method and the setting value of the weight coefficient 23 are pre-set in the storage unit 102 for each type of image element 50 to be extracted.

[0051] In step S5, the image generation unit 30 performs inter-image operations using the multiple extracted images 21 extracted for each image element 50 and the X-ray image 201. The image generation unit 30 performs inter-image operations according to the parameters obtained in step S4. The image generation unit 30 multiplies each of the first extracted images 21-1 to the Nth extracted images 21-N by the corresponding weight coefficient 23 and performs inter-image operations on the X-ray image 201 according to the corresponding calculation method. As a result, the X-ray image 201, which has been weighted and added or weighted and subtracted by each extracted image 21, is generated as a processed image 22. Thus, the image generation unit 30 generates a processed image 22 (see Figure 6 or Figure 7) in which each image element 50 included in the X-ray image 201 has undergone image processing, such as enhancement or removal.

[0052] In step S6, the image processing device 100 outputs the processed image 22. The image processing device 100 outputs the processed image 22 to the display device 103 or the server device. The image processing device 100 also stores the processed image 22 in the storage unit 102.

[0053] After displaying the processed image 22 on the display device 103, the image generation unit 30 may accept input to change the calculation parameters. For example, the image generation unit 30 may accept input for the value of the weight coefficient 23, or it may accept the selection of other preset parameters. For example, the image generation unit 30 may accept a change in the image processing parameters in step S3. Then, the image generation unit 30 may regenerate the processed image 22 using the changed parameters in response to the user's input.

[0054] (How to create a pre-trained model) Next, we will explain how to create a pre-trained model. The creation of the pre-trained model 40 may be performed by the processor 101 of the image processing device 100, but it can also be performed using a machine learning computer (learning device 300, see Figure 10).

[0055] As shown in Figures 9 and 10, the method for creating a trained model in this embodiment includes the following steps S11 to S14. (S11) A reconstructed image 60 is generated by reconstructing the CT (computed tomography) image data 80 into a two-dimensional projection image. The CT image data 80 is an example of "three-dimensional X-ray image data". (S12) The simulation generates a two-dimensional projection image 61 from the three-dimensional model of the image element 50 to be extracted. (S13) The projected image 61 of the image element 50 is superimposed on the reconstructed image 60 to generate a superimposed image 67. (S14) Using the superimposed image 67 as training input data 64 (see Figure 3) and the reconstructed image 60 or projection image 61 as training output data 65 (see Figure 3), machine learning is performed to create a trained model 40 (see Figure 3) that extracts image elements 50 contained in the input image.

[0056] In this embodiment, machine learning involves inputting the teacher input data 64 (see Figure 3) and teacher output data 65 (see Figure 3), created for each image element 50, into a single learning model LM. Machine learning may be performed on a separate learning model LM for each image element 50 to be extracted.

[0057] First, in step S10 of Figure 10, CT image data 80 is acquired. CT image data 80 is 3D image data that reflects the 3D structure of subject 1, obtained by CT scanning subject 1. CT image data 80 is a 3D collection of voxel data that includes 3D position coordinates and CT values ​​at those position coordinates. For image elements of moving (pulsating) parts such as blood vessels, 4D-CT data, which includes time changes in the 3D information, may be used. This makes it possible to learn accurately even for objects that move over time.

[0058] In step S11, a reconstructed image 60 is generated by reconstructing the CT image data 80 into a two-dimensional projection image.

[0059] The reconstructed image 60 is a DRR image generated from the CT image data 80. The DRR image is a simulated X-ray image created as a two-dimensional projection image by virtual fluoroscopic projection that simulates the geometric projection conditions of the X-ray irradiation unit 220 and the X-ray detector 230 of the X-ray imaging apparatus 200 as shown in Figure 2.

[0060] Specifically, as shown in Figure 11, a virtual X-ray tube 91 and a virtual X-ray detector 92 are virtually positioned in a three-dimensional virtual space so as to be projected in a predetermined direction relative to the CT image data 80, thereby generating a three-dimensional spatial arrangement (imaging geometry) of a virtual X-ray imaging system. The arrangement of the CT image data 80, the virtual X-ray tube 91, and the virtual X-ray detector 92 is the same imaging geometry as the arrangement of the actual subject 1, the X-ray irradiation unit 220, and the X-ray detector 230 shown in Figure 2. The imaging geometry refers to the geometrical spatial arrangement relationship between the subject 1, the X-ray irradiation unit 220, and the X-ray detector 230.

[0061] Then, the pixel value of each pixel in the reconstructed image 60 is calculated by adding up the sum of the CT values ​​in each voxel that the X-rays irradiated from the virtual X-ray tube 91 passed through before reaching the virtual X-ray detector 92. By changing the imaging geometry, a simulated X-ray image can be generated at any projection angle.

[0062] In the method for creating a trained model, the training device 300 generates multiple reconstructed images 60 from a single 3D data (CT image data 80). The number of reconstructed images 60 generated can be, for example, around 100,000. The training device 300 generates multiple reconstructed images 60 by varying each parameter, such as projection angle, projection coordinates, parameters for DRR image generation, contrast, and edge enhancement. These varying reconstructed images 60 can be generated, for example, by an algorithm that generates reconstructed images 60 by randomly changing the above parameters.

[0063] In this embodiment, a superimposed image 67 is created by superimposing a projected image 61 containing the image element 50. Therefore, the reconstructed image 60 may not contain the image element 50 to be extracted, or even if it contains the image element 50, it may not have sufficient contrast to be extractable.

[0064] Furthermore, in this specification, the terms "random" and "random number" mean the absence of regularity or a sequence (set of numbers) without regularity, but do not need to be completely random, and include pseudo-random and pseudo-random numbers.

[0065] In step S12 of Figure 9, a two-dimensional projection image 61 is generated from the three-dimensional model of the image element 50 to be extracted through simulation.

[0066] The projected image 61 is a two-dimensional image representing the image elements 50 that are to be extracted by the trained model 40. The projected image 61 contains, for example, only the image elements 50. The learning device 300 acquires a three-dimensional model of the image elements 50 to be extracted and generates the projected image 61 from the three-dimensional model through simulation.

[0067] A 3D model is created, for example, by performing a CT scan of an object containing the image elements 50 to be extracted, and then extracting the image elements 50 from the obtained CT data. The 3D model can be created, for example, by using a CT image database published by a research institution. An example of a CT image database is the LIDC / IDRI (The Lung Image Database Consortium and Image Database Resource Initiative) from the National Cancer Institute in the United States, which is a lung CT image dataset. Other options include brain CT image datasets and standardized 3D models of skeletons. Furthermore, devices 55 and clothing 56 can be created using 3D CAD data. The 3D model is, for example, a 3D image data file containing only the image elements 50 to be extracted.

[0068] Furthermore, it is not necessary to generate all projection images 61 used in machine learning from a 3D model through simulation. Projection images 61 can be created by acquiring 2D data (X-ray images) that actually contain the image elements 50 to be extracted, and then separating and extracting the image elements 50 contained in the acquired images.

[0069] The learning device 300 generates multiple 2D projected images 61 of a 3D model or 2D data by varying parameters such as projection direction, translation amount, rotation amount, deformation amount, and contrast. These multiple projected images 61 can be generated by an algorithm that randomly changes variable parameters such as translation amount, rotation amount, deformation amount, and contrast of the original data (3D model, 2D data, or other projected images 61).

[0070] The learning device 300 generates multiple 2D projection images 61 for each type of image element 50. The learning device 300 generates multiple projection images 61 from one source data (3D data, 2D data, or projection image 61). The number of projection images 61 generated from one source data can be, for example, about 100,000.

[0071] In step S13, the projection image 61 generated in step S12 is superimposed on the reconstructed image 60 generated in step S11 to generate a superimposed image 67 (see Figure 10). Superimposition generates a superimposed image 67 in which the projection image 61 of the image element 50 to be extracted is included in the two-dimensional reconstructed image 60. Multiple superimposed images 67 are generated by combining multiple reconstructed images 60 and multiple projection images 61. One training data 66 (see Figure 3) includes a superimposed image 67 and one of the reconstructed images 60 and projection images 61 used to generate that superimposed image 67.

[0072] In step S14 of Figure 9, machine learning is performed. The superimposed image 67 is used as the training input data 64 input to the input layer 41 of the learning model LM in Figure 3. Then, the reconstructed image 60 or projection image 61 is used as the training output data 65 input to the output layer 43 of the learning model.

[0073] If the training output data 65 is a reconstructed image 60, the learning model LM learns to extract image elements 50 from the input image and generate an extracted image 21 that does not contain those image elements 50. If the training output data 65 is a projection image 61, the learning model LM learns to extract image elements 50 from the input image and generate an extracted image 21 that represents the extracted image elements 50. The extracted image 21 may be an image that contains only the image elements 50. Furthermore, since an image containing only the extracted image elements 50 can be generated by generating a processed image 22 that does not contain the image elements 50 and subtracting it from the input X-ray image 201, adopting a reconstructed image 60 as the training output data 65 and adopting a projection image 61 can be considered equivalent from the standpoint of image processing.

[0074] It is not necessary to create all training data 66 used for machine learning using superimposed images 67. Alternatively, a reconstructed image 60 containing the image elements 50 to be extracted may be used as the training input data 64, and the projected image 61 of the image elements 50 extracted from that reconstructed image 60 may be used as the training output data 65.

[0075] In step S15 of Figure 9, the learning device 300 determines whether or not machine learning is complete. The learning device 300 determines that machine learning is complete, for example, when machine learning has been performed for a predetermined number of iterations on all the training data 66. The learning device 300 also determines that machine learning is complete, for example, when the value of the evaluation function that evaluates the performance of the learning model LM becomes equal to or greater than a predetermined value. If the learning device 300 determines that machine learning is not complete, in step S16, it changes the training data 66 and performs the machine learning in step S14 using the next training data 66.

[0076] If machine learning is completed, in step S17, the trained model LM is stored as the trained model 40. This completes the creation of the trained model 40. The created trained model 40 is provided to the image processing device 100 via the network or by being recorded on a non-transient recording medium.

[0077] (Specific examples for each image element) Next, we will explain specific examples of superimposed images 67 (training input data 64) and reconstructed images 60 or projected images 61 (training output data 65) for each of the 50 image elements. We will also explain examples of processed images 22 using extracted images 21 for each of the 50 image elements.

[0078] <Bone part> Figure 12 shows an example where the image element 50 is a bone 53. The training input data 64 is, for example, a superimposed image 67 of a reconstructed image 60 that does not include the bone 53 and a projection image 61 that includes the bone 53.

[0079] For example, a projection image 61 is created from CT image data 80 or a 3D model of the skeleton, and a superimposed image 67 is generated by superimposing this projection image 61 onto a reconstructed image 60 from which the bones 53 have been removed. The reconstructed image 60 from which the bones 53 have been removed is generated by clipping (fixing) the CT values ​​to zero for pixels within the range of CT values ​​that the bones 53 possess. The CT values ​​of bone are generally considered to be approximately 200 HU to 1000 HU, so the threshold can be set to a predetermined value of approximately 0 HU to 200 HU.

[0080] The training output data 65 is a projection image 61 that includes only bones 53. The training output data 65 uses the projection image 61 used to generate the superimposed image 67.

[0081] Since CT image data 80 taken from a subject usually includes bone, the reconstructed image 60 generated from the CT image data 80 may be used as training input data 64. Therefore, it is not always necessary to create superimposed images 67.

[0082] In this case, the training output data 65 is a reconstructed image 60 containing only bone 53. The reconstructed image 60 containing only bone 53 is generated, for example, by clipping the CT values ​​to zero for pixels in the reconstructed image 60 containing bone 53 that have a CT value less than the CT value of bone 53.

[0083] Through machine learning, the learning model LM learns to generate extracted images 21 in which bones 53 are extracted from input images such as the training input data 64, as shown in the training output data 65.

[0084] The image processing device 100 weights and subtracts the extracted image 21 generated by the trained model 40 from the X-ray image 201 acquired by the X-ray imaging device 200. As a result, as shown in Figure 13, a processed image 22 is generated from the X-ray image 201 with the image elements 50 of the bone 53 removed. In Figure 13, for illustrative purposes, the removed image elements 50 of the bone 53 are shown with dashed lines.

[0085] <device> Figure 14 shows an example where the image element 50 is the device 55. In Figure 14, the device 55 is the guidewire.

[0086] The training input data 64 is a superimposed image 67 that includes the device 55. The reconstructed image 60 generated from the CT image data 80 does not include the image element 50. A projection image 61 of only the device 55, generated from the 3D model of the device 55, is superimposed on the reconstructed image 60. As a result, a superimposed image 67 including the device 55 is generated as shown in Figure 14.

[0087] As described above, a two-dimensional X-ray image of the device 55 may be taken, and multiple variations of the projection image 61 may be created by changing the shape, etc., through simulation. Multiple superimposed images 67 are generated using the multiple projection images 61.

[0088] In particular, as shown in Figure 16, when the image element 50 is a linear or tubular device 55, the projected image 61 of the image element 50 is generated by simulating the shape of the 3D model of the device 55 with a curve generated based on random coordinate values. In Figure 16, as an example of a device 55, examples (A) to (I) are shown of projected images 61 of a guide wire 55b holding a stent 55a, generated with a random shape by curve simulation.

[0089] The guide wires 55b in each projection image 61 are generated in different shapes by Bézier curves based on random coordinate values. A Bézier curve is a K-1 type curve obtained from K control points (where K is an integer greater than or equal to 3). By using an algorithm that randomly specifies the coordinate values ​​of the K control points, a large number of projection images 61 of devices 55 with diverse shapes can be generated.

[0090] Multiple projection images 61 of the stent 55a implanted in the body are generated by applying random translation, rotation, deformation, and contrast changes to a 3D model of a pseudo-stent.

[0091] Returning to Figure 14, the training output data 65 is a projection image 61 that includes only device 55. The training output data 65 uses the projection image 61 that was used to generate the superimposed image 67.

[0092] Through machine learning, the learning model LM learns to generate extracted images 21, in which the device 55 is extracted from input images such as the training input data 64, as shown in the training output data 65.

[0093] The image processing device 100 weights and adds the extracted image 21 generated by the trained model 40 to the X-ray image 201 acquired by the X-ray imaging device 200. As a result, as shown in Figure 15, a processed image 22 is generated from the X-ray image 201 with the image elements 50 of the device 55 emphasized. In Figure 15, the device 55 is shown in thicker lines compared to Figure 14 to indicate that it has been emphasized. The emphasis process may include not only processes that increase the pixel values, but also processes such as coloring and displaying the image elements 50 of the device 55 through image processing before inter-image operations, as shown in Figure 7.

[0094] <noise> Figure 17 shows an example where image element 50 is noise 57. Noise 57 is, for example, random noise, but in Figure 17, for illustrative purposes, it is shown as a collection of horizontal dotted lines.

[0095] The training input data 64 is a superimposed image 67 containing noise 57. The reconstructed image 60 generated from the CT image data 80 does not contain noise 57. A projection image 61 containing only randomly generated noise 57 is superimposed on the reconstructed image 60. As a result, a superimposed image 67 containing noise 57 is generated as shown in Figure 17. The noise 57 is created by randomly generating Gaussian noise following a Gaussian distribution or Poisson noise following a Poisson distribution for each projection image 61.

[0096] The training output data 65 is a projection image 61 containing only noise 57. The training output data 65 uses the projection image 61 used to generate the superimposed image 67.

[0097] Through machine learning, the learning model LM learns to generate extracted images 21 from input images like the training input data 64, with noise 57 extracted, as in the training output data 65.

[0098] The image processing device 100 weights and subtracts the extracted image 21 generated by the trained model 40 from the X-ray image 201 acquired by the X-ray imaging device 200. As a result, as shown in Figure 18, a processed image 22 is generated from the X-ray image 201 with the noise 57 image element 50 removed. The processed image 22 in Figure 18 shows that the noise 57 has been removed from the X-ray image 201, which contains noise 57 as shown in the training input data 64 in Figure 17.

[0099] <Vessel> Figure 19 shows an example where the image element 50 is a blood vessel 54. The blood vessel 54 is a contrast-enhanced vessel that has been imaged after the introduction of a contrast agent. Figure 19 shows an example of a cerebral blood vessel in the head, but other blood vessels may also be used. The blood vessel could be, for example, the coronary arteries of the heart.

[0100] The training input data 64 is a superimposed image 67 that includes blood vessels 54. The reconstructed image 60 generated from CT image data 80 acquired without contrast does not contain much of the blood vessels 54 (it does not have sufficient contrast). A projection image 61 of only the blood vessels 54, generated from a 3D model of the blood vessels 54, is superimposed on the reconstructed image 60. As a result, a superimposed image 67 that includes the blood vessels 54 is generated, as shown in Figure 19.

[0101] When the image element 50 is a blood vessel 54, the projection image 61 of the image element 50 is generated by a simulation that randomly changes the shape of the 3D model of the blood vessel 54. The blood vessel in the projection image 61 is subjected to random translation, rotation, deformation, contrast changes, etc., by the simulation. That is, similar to the device 55 shown in Figure 16, variations of the projection image 61 of the blood vessel 54 with random changes are generated. As described above, the projection image 61 of the blood vessel 54 may also be created from CT image data of the contrast-enhanced blood vessel acquired by contrast imaging.

[0102] The training output data 65 is a projection image 61 that includes only blood vessels 54. The training output data 65 uses the projection image 61 used to generate the superimposed image 67.

[0103] Through machine learning, the learning model LM learns to generate extracted images 21 in which blood vessels 54 are extracted from input images such as the training input data 64, as shown in the training output data 65.

[0104] The image processing device 100 weights and adds the extracted image 21 generated by the trained model 40 to the X-ray image 201 acquired by the X-ray imaging device 200. As a result, as shown in Figure 20, a processed image 22 is generated from the X-ray image 201 in which the image element 50 of the blood vessel 54 is emphasized. The processed image 22 in Figure 20 shows that the blood vessel 54 is emphasized in the X-ray image 201 which includes the image element 50 of the blood vessel 54, as in the training input data 64 in Figure 19.

[0105] <clothing> Figure 21 shows an example where image element 50 is clothing 56. In Figure 21, examples of clothing 56 include buttons on clothing and a necklace worn by the subject.

[0106] The training input data 64 is a superimposed image 67 that includes clothing 56. The reconstructed image 60 generated from the CT image data 80 does not include clothing 56. A projection image 61 of only clothing 56, generated from a 3D model of clothing 56, is superimposed on the reconstructed image 60. As a result, a superimposed image 67 including clothing 56 is generated as shown in Figure 21. The 3D model of clothing 56 may be created from a CT image of clothing 56 only, or it may be created from CAD data, for example. As described above, a 2D X-ray image of clothing 56 may be taken and used as the projection image 61. Random translation, rotation, deformation, contrast changes, etc., are applied to the projection image 61 by simulation.

[0107] The training output data 65 is a projection image 61 that includes only the clothing 56. The training output data 65 uses the same data as the projection image 61 used to generate the superimposed image 67.

[0108] Through machine learning, the learning model LM learns to generate extracted images 21, in which clothing 56 is extracted from input images such as the training input data 64, as shown in the training output data 65.

[0109] The image processing device 100 weights and subtracts the extracted image 21 generated by the trained model 40 from the X-ray image 201 acquired by the X-ray imaging device 200. As a result, as shown in Figure 22, a processed image 22 is generated in which the image elements 50 of the clothing 56 have been removed from the X-ray image 201. In Figure 22, for convenience, the portion of the removed image element 50 of the clothing 56 is shown with a dashed line to illustrate the removal.

[0110] <Scattered radiation component> Figure 23 shows an example where the image element 50 is the scattered X-ray component 58.

[0111] The training input data 64 is a superimposed image 67 that includes the scattered radiation component 58. In the reconstructed image 60 generated from the CT image data 80, the scattered radiation component 58 is not included in the reconstruction calculation. A projection image 61 containing only the scattered radiation component 58, generated by a Monte Carlo simulation that models the acquisition environment of the input image, is superimposed on the reconstructed image 60. As a result, a superimposed image 67 that includes the scattered radiation component 58 is generated.

[0112] In Monte Carlo simulation, for example, as shown in Figure 2 (or Figure 11), a three-dimensional imaging environment model 85 is created, which models the imaging environment of the input image (X-ray image 201) from the X-ray irradiation unit 220 to the X-ray detector 230. Then, each X-ray photon emitted from the X-ray irradiation unit 220 and detected by the X-ray detector 230 is calculated (simulated) as a probability phenomenon using random numbers. That is, in the simulation, the projection direction of the X-rays, the shape (body shape) of the subject, and physical properties related to the interaction with the photons are assumed. Then, interactions such as absorption and scattering phenomena that occur when the X-ray photons pass through the subject are calculated as probability phenomena using random numbers. In Monte Carlo simulation, a predetermined number of photons is calculated, and a projection image 61 formed by the X-ray photons detected by the virtual X-ray detector 230 is generated. The predetermined number of photons can be any number sufficient to create an image, for example, about 10 billion.

[0113] In this embodiment, multiple projection images 61 of the image element 50, which is the scattered radiation component 58, are generated by changing the projection angle over the projection angle range that the X-ray imaging apparatus 200 can capture in the imaging environment model 85. For example, in the example shown in Figure 2, the projection direction can be changed to a first direction 250 and a second direction 252 by moving the C-arm 240. Therefore, as shown in Figure 25, multiple projection images 61 are generated by Monte Carlo simulation at multiple projection angles that are changed to different angle values ​​over the entire projection angle range of ±α degrees in the first direction 250 and ±β degrees in the second direction 252. The projection angle may be changed at equal angular intervals over the entire projection angle range, or it may be a predetermined number of randomly changed values ​​within the projection angle range. Furthermore, projection images 61 may be generated at angle values ​​outside the projection angle range but near the limits of the projection angle (±α degrees, ±β degrees).

[0114] Furthermore, in this embodiment, multiple projection images 61 of the image element 50, which is the scattered radiation component 58, are generated by changing the energy spectrum of the virtual radiation source in the imaging environment model 85. That is, the projection images 61 are created by Monte Carlo simulation under multiple conditions in which the energy spectrum of the X-rays irradiated by the X-ray irradiation unit 220, which is assumed to be the imaging environment model 85, is changed to different spectra.

[0115] Generally, the lower the photon energy of the X-rays, the more easily they are absorbed in the subject's body, and the less scattered radiation component 58 is generated. The higher the photon energy of the X-rays, the less easily they are absorbed in the subject's body, and the more scattered radiation component 58 is generated. Therefore, for example, a projection image 61 based on the first energy spectrum 111 shown in Figure 26 and a projection image 61 based on the second energy spectrum 112 shown in Figure 27 are created. The second energy spectrum 112 is a spectrum with relatively higher energy than the first energy spectrum 111. In Figures 26 and 27, the horizontal axis of the graph shows the energy of the X-ray photons [keV], and the vertical axis of the graph shows the relative intensity of the X-rays (i.e., the number of X-ray photons detected).

[0116] Furthermore, due to differences in absorption spectra, a beam hardening phenomenon occurs during the detection of the energy spectrum of the X-rays irradiated onto the subject, resulting in a relative bias towards higher energies. While the reconstructed image 60 generated from CT image data 80 cannot simulate the image quality changes caused by the beam hardening phenomenon, the projection image 61 obtained by Monte Carlo simulation can simulate the effects of the beam hardening phenomenon.

[0117] Figure 23 shows, as an example, a projection image 61 of the scattered ray component 58 due to Compton scattering obtained by Monte Carlo simulation. In this embodiment, other scattered ray components 58 such as Rayleigh scattering may also be obtained. Furthermore, scattered ray components 58 due to multiple scattering as well as single scattering may also be obtained. These various scattered ray components 58 may be created as separate projection images 61 and machine learning may be performed to extract them separately, or a projection image 61 that displays all the scattered ray components 58 together may be created and machine learning may be performed to extract all the scattered ray components 58 at once.

[0118] The training output data 65 is a projection image 61 containing only the scattered radiation component 58. The training output data 65 uses the same data as the projection image 61 used to generate the superimposed image 67.

[0119] Through machine learning, the learning model LM learns to generate extracted images 21 from input images such as the training input data 64, by extracting the scattered radiation component 58, as shown in the training output data 65.

[0120] The image processing device 100 weights and subtracts the extracted image 21 generated by the trained model 40 from the X-ray image 201 acquired by the X-ray imaging device 200. As a result, as shown in Figure 24, a processed image 22 is generated in which the image element 50 of the scattered radiation component 58 is removed from the X-ray image 201. Since the scattered radiation component 58 is a factor that reduces the contrast of the X-ray image, the processed image 22 can improve the contrast by removing the scattered radiation component 58. The processed image 22 in Figure 24 shows that the contrast has been improved from the X-ray image 201, which had reduced contrast due to the scattered radiation component 58 as shown in the training input data 64 in Figure 23, by removing the scattered radiation component 58.

[0121] <Collimator image> In this embodiment, a portion of each training input data 64 and each training output data 65 created includes a collimator image 68 whose imaging range is limited by a collimator (not shown) provided by the X-ray imaging apparatus 200 (X-ray irradiation unit 220). In the collimator image 68, an image is formed only in a portion of the image, and no image information is included in the area shielded by the collimator. A portion of each training input data 64 and each training output data 65 includes multiple collimator images 68 in which the shape of the X-ray irradiation range (i.e., image area) and the image quality parameters affected by the collimator are randomly varied by simulation based on images actually captured using the collimator. The image quality parameters affected by the collimator include the degree of transmission (contrast), edge blurring, and noise content.

[0122] The collimator image 68 is generated, for example, by removing the image portion outside the simulated irradiation range from the superimposed image 67, the reconstructed image 60, and the projected image 61, and by applying image processing that simulates the effect of the collimator. The collimator image 68 may also be generated from an image actually taken using the collimator. This improves the robustness of the image element extraction process against changes in the X-ray irradiation range and image quality caused by the use of the collimator.

[0123] As described above, machine learning is performed for each type of image element 50, and the processed image 22 is generated by the image processing device 100.

[0124] In each of the above specific examples, for the sake of explanation, the individual image elements 50 and the processed image 22 were described separately. However, when the actual image processing device 100 generates the processed image 22, the X-ray image 201 input to the image processing device 100 contains multiple of the above-mentioned bones 53, blood vessels 54, devices 55, clothing 56, noise 57, and scattered radiation components 58. The image processing device 100 generates an extracted image 21 by extracting individual image elements 50 from the input X-ray image 201 using the trained model 40, and performs inter-image operations. As a result, a processed image 22 is generated in which each of the multiple image elements 50 has undergone enhancement or removal processing.

[0125] For example, in the example shown in Figure 7, N=6, the first extracted image 21-1 represents bone 53, the second extracted image 21-2 represents the device 55, the third extracted image (let's call it 21-3) represents noise 57, the fourth extracted image (let's call it 21-4) represents blood vessels 54, the fifth extracted image (let's call it 21-5) represents clothing 56, and the sixth extracted image (let's call it 21-6) represents the scattered radiation component 58.

[0126] Here are some examples of how the processed image 22 can be used.

[0127] For example, the processed image 22 is applied to an X-ray image of the front of a subject's chest taken by simple radiography. In this case, bone 53, noise 57, clothing 56, and scattered radiation components 58 are removed from the processed image 22. Removal of bone 53 improves the visibility of areas of interest such as the heart and lungs. In addition, removal of noise 57 and scattered radiation components 58 improves the overall visibility of the image. Since the image elements 50 of clothing 56 can be removed, X-ray imaging can be performed without the subject removing clothing, accessories, etc. that contain metal, etc. This brings useful effects such as improved work efficiency and reduced waiting time for subjects when X-ray imaging of a large number of subjects is performed continuously, such as in mass screenings.

[0128] Furthermore, the processed image 22 can be applied to X-ray fluoroscopic images in interventional radiology (IVR), such as catheter treatment using an X-ray angiography system. In this case, bone 53, noise 57, and scattered radiation components 58 are removed from the processed image 22. Devices 55 such as catheters, guidewires, and stents, and blood vessels 54 are highlighted in the processed image 22. The removal of bone 53, noise 57, and scattered radiation components 58 improves the visibility of the fluoroscopic image. The highlighting of devices 55 and blood vessels 54 improves the visibility of the region of interest and the device being manipulated in catheter treatment.

[0129] (Effects of this embodiment) In this embodiment, the following effects can be obtained.

[0130] According to the method for creating the trained model 40 of this embodiment, a superimposed image 67 is used as the training input data 64. This superimposed image 67 is obtained by superimposing a reconstructed image 60, which is obtained by reconstructing CT image data 80 into a two-dimensional projection image, with a two-dimensional projection image 61 generated from a three-dimensional model of the image element 50 to be extracted by simulation. The reconstructed image 60 or projection image 61 is used as the training output data 65. This allows machine learning to be performed using the image element 50 to be extracted generated by simulation, even if the CT image data 80 does not contain the image element 50 to be extracted. In other words, training data can be prepared without having to prepare CT image data 80 that actually contains the image element 50 to be extracted. Furthermore, since the projection image 61 of the image element 50 to be extracted is generated by simulation, training data can be prepared even for image elements 50 that are difficult to separate and extract even if they are included in the CT image data 80. As a result, it is possible to efficiently create a trained model 40 for image processing on various image elements 50, and on multiple image elements 50.

[0131] According to the image generation method and image processing apparatus 100 of this embodiment, a trained model 40 that has been trained to extract specific image elements 50 from an input image is used to separately extract multiple image elements 50 from an X-ray image 201. An image processing image 22 is generated by performing inter-image operations using the multiple extracted images 21 extracted for each image element 50 and the X-ray image 201. As a result, various image elements 50 are separately extracted as extracted images 21 from the input X-ray image 201, and each extracted image 21 can be freely added to or subtracted from the X-ray image 201 according to the type of extracted image element 50. Consequently, image processing can be performed on various image elements 50, and on multiple image elements 50.

[0132] Furthermore, in the above embodiment, additional effects can be obtained by configuring it as follows.

[0133] In other words, in this embodiment, multiple superimposed images 67 are created for each of several different image elements 50, and each of the several image elements 50 includes a first element 51 which is biological tissue and a second element 52 which is non-biological tissue. With this configuration, it is possible to create a trained model 40 that can perform image processing on image elements 50 of biological tissue such as bone 53 and blood vessels 54, and image processing on image elements 50 of non-biological tissue such as a device 55 introduced into the body and clothing 56 worn by the subject, in a combined manner. By using such a trained model 40 in the image processing device 100, it is possible to perform image processing on image elements 50 of biological tissue and image elements 50 of non-biological tissue in a combined manner.

[0134] Furthermore, in this embodiment, multiple superimposed images 67 are created for each of several different image elements 50, and each of the several image elements 50 includes at least several of the following: bone 53, blood vessels 54, a device introduced into the body 55, clothing 56, noise 57, and scattered X-ray components 58. With this configuration, a trained model 40 capable of complexly performing image processing on various image elements 50 according to various usage scenarios of the X-ray image 201 can be created. By using such a trained model 40 in the image processing device 100, complex image processing on various image elements 50 according to various usage scenarios of the X-ray image 201 can be performed.

[0135] Furthermore, in this embodiment, the image element 50 includes a linear or tubular device 55, and the projected image 61 of the image element 50 is generated by simulating the shape of the 3D model of the device 55 with a curve generated based on random coordinate values. With this configuration, training data for learning the image element 50 of long, flexible devices 55 such as guidewires and catheters can be generated in a large quantity of diverse shapes through simulation. As a result, efficient machine learning can be performed without having to prepare a large amount of 3D CT data of the device 55 actually placed inside the subject's body.

[0136] Furthermore, in this embodiment, the image element 50 includes a blood vessel 54, and the projected image 61 of the image element 50 is generated by a simulation that randomly changes the shape of the 3D model of the blood vessel 54. With this configuration, a large amount of training data for learning the image element 50 of the blood vessel 54, which bends into long and complex shapes, can be generated by simulation, and this data can include diverse individual differences. As a result, efficient machine learning can be performed without having to prepare a large amount of 3D CT data from various subjects.

[0137] Furthermore, in this embodiment, the image element 50 includes the scattered X-ray component 58, and the projected image 61 of the image element 50 is generated by a Monte Carlo simulation that models the imaging environment of the input image. With this configuration, the projected image 61 of the scattered X-ray component 58, which is difficult to separate and extract from actual 3D CT data or 2D X-ray images 201, can be generated by Monte Carlo simulation. By creating a trained model 40 that extracts the scattered X-ray component 58 using the projected image 61 generated in this way, image processing to remove the scattered X-ray component 58 from an X-ray image 201 taken of an actual subject can be realized without complex and high-load computational processing such as Monte Carlo simulation. Therefore, for example, the contrast in areas such as the abdomen and bones, where the scattered X-ray component 58 has a large impact, can be effectively improved. In addition, it is also possible to perform imaging without using an X-ray absorption grid, which is used to reduce the impact of the scattered X-ray component 58 in X-ray imaging, and remove the scattered X-ray component 58 by image processing. In that case, the amount of X-rays can be reduced by the amount of X-ray absorption grid not used, so the radiation dose to the subject can be effectively reduced.

[0138] Furthermore, in this embodiment, multiple projection images 61 of the image element 50 are generated by changing the projection angle over the projection angle range (±α, ±β) that the X-ray imaging device 200 can capture in the imaging environment model 85. With this configuration, a highly versatile trained model 40 can be created that effectively extracts scattered radiation components 58 not only when imaging is performed only from a specific projection direction (such as the front or side of the chest) as in simple X-ray imaging, but also when X-ray imaging is performed at various projection angles or while changing the projection angle in X-ray image-guided therapy.

[0139] Furthermore, in this embodiment, multiple projection images 61 of the image element 50 are generated by changing the energy spectrum of the virtual radiation source in the imaging environment model 85. In actual medical settings, X-ray imaging is performed under various imaging conditions with different energy spectra depending on the area being imaged, etc. However, with the above configuration, even when the X-ray image 201 is captured under various energy spectra, a highly versatile trained model 40 capable of effectively extracting the scattered radiation component 58 can be created.

[0140] Furthermore, in this embodiment, machine learning includes inputting training input data 64 and training output data 65 created for each image element 50 to a single training model LM. The trained model 40 is configured to extract multiple image elements 50 from the input image without duplication and to output the extracted multiple image elements 50 and the remaining image elements 59 after extraction. With this configuration, a trained model 40 can be provided that extracts image elements 50 in such a way that adding all the extracted multiple image elements 50 and the remaining image elements 59 returns to the original input image. In other words, even when multiple image elements 50 are extracted from a single X-ray image 201, there is no loss of image information contained in the X-ray image 201, nor is there an unintended increase in image information. Doctors and others who perform diagnoses using X-ray images 201 find the basis for their diagnosis in the image information contained in the original image, even if they perform various image processing to improve visibility. Therefore, a trained model 40 that can extract image elements 50 without losing image information can provide doctors and others with reliable images even when performing complex image processing.

[0141] Furthermore, in this embodiment, image processing is performed separately on some or all of the multiple extracted images 21, and the processed image 22 is generated by inter-image calculations between the multiple extracted images 21 after image processing and the X-ray image 201. With this configuration, by utilizing the fact that multiple image elements 50 can be extracted separately using the trained model 40, image processing such as correction processing and interpolation processing can be performed independently on the extracted image 21 of each extracted image element 50. Here, for example, if we were to try to perform image processing that acts only on a specific image element 50 on the X-ray image 201 before extraction, the image processing algorithm would become complex and computationally intensive, and other image elements 50 besides the specific image element 50 might be negatively affected. In contrast, when image processing is performed on the extracted image 21, since there are no other image elements 50, high image processing accuracy can be obtained even by simply performing filtering processing on the entire extracted image 21. Furthermore, by performing inter-image operations using the extracted image 21 after image processing, it is possible to easily generate a processed image 22 in which not only are multiple image elements 50 simply emphasized or removed, but high-precision image processing is performed specifically on each individual image element 50.

[0142] Furthermore, in this embodiment, the inter-image operation includes weighted addition or weighted subtraction of individual extracted images 21 to the X-ray image 201. With this configuration, a processed image 22 can be obtained in which multiple extracted image elements 50 are individually emphasized or removed. In addition, since the degree of emphasis or removal of each image element 50 can be appropriately set by adjusting the weight coefficient 23, a processed image 22 can be generated that provides high visibility for the required image elements 50 depending on the various usage scenarios of the X-ray image 201.

[0143] (modified version) It should be noted that the embodiments disclosed herein are illustrative and not restrictive in all respects. The scope of the present invention is indicated by the claims rather than by the description of the embodiments above, and further includes all modifications (exceptions) within the meaning and scope equivalent to the claims.

[0144] For example, in the above embodiment, an example was shown where the machine learning device (learning device 300) and the image processing device 100 are separate devices, but the present invention is not limited to this. In the present invention, machine learning may be performed in the image processing device. Also, the learning device 300 may be configured as a server computer located on the cloud.

[0145] Furthermore, while the above embodiments (see Figures 5 to 7) show an example in which multiple image elements 50 are extracted using one trained model 40, the present invention is not limited to this. In the present invention, multiple image elements 50 may be extracted using multiple trained models. For example, as shown in Figure 29, one trained model 40 may be provided for each image element 50 to be extracted. In Figure 29, trained model 40-1, trained model 40-2, ..., trained model 40-N are provided. One trained model 40 extracts one (one type) of image element 50. In addition, multiple trained models 40 created to extract multiple image elements 50 may be provided.

[0146] Furthermore, although the above embodiment shows an example in which a plurality of image elements 50 include at least a plurality of bones 53, blood vessels 54, devices 55, clothing 56, noise 57, and scattered radiation components 58, the present invention is not limited thereto. The image elements 50 may include image elements other than bones 53, blood vessels 54, devices 55, clothing 56, noise 57, and scattered radiation components 58. The image elements 50 may be specific structural parts, such as specific organs in the body. Also, for example, among the bones 53, image elements 50 of bones in a specific area, or among the blood vessels 54, image elements 50 of blood vessels in a specific area, may be extracted separately from other bones and other blood vessels. The image elements 50 do not have to include bones 53, blood vessels 54, devices 55, clothing 56, noise 57, and scattered radiation components 58.

[0147] Furthermore, although the above embodiment shows an example in which bone 53 is removed and blood vessels 54 and devices 55 are enhanced by inter-image calculations, the present invention is not limited thereto. Bone 53 may be enhanced, or one or both of blood vessels 54 and devices 55 may be removed.

[0148] Furthermore, although the above embodiment shows two examples of inter-image operations: weighted addition and weighted subtraction, the present invention is not limited thereto. Inter-image operations may also be addition or subtraction without weight coefficients. The enhancement process of image elements 50 may be performed by multiplication with or without weight coefficients. The removal process of image elements 50 may be performed by division with or without weight coefficients.

[0149] [Aspect] Those skilled in the art will understand that the exemplary embodiments described above are specific examples of the following embodiments.

[0150] (Item 1) A reconstructed image is generated by reconstructing 3D X-ray image data into a 2D projection image. Through simulation, a 2D projection image is generated from a 3D model of the image elements to be extracted. The projected image of the aforementioned image element is superimposed on the reconstructed image to generate a superimposed image. A method for creating a trained model, which performs machine learning using the superimposed image as training input data and the reconstructed image or the projected image as training output data to create a trained model that extracts the image elements contained in the input image.

[0151] (Item 2) Multiple superimposed images are created for each of several different image elements. The method for creating a trained model as described in item 1, wherein the plurality of image elements include a first element which is biological tissue and a second element which is non-biological tissue.

[0152] (Item 3) Multiple superimposed images are created for each of several different image elements. The method for creating a trained model as described in item 1, wherein the plurality of image elements include at least a few of bone, blood vessels, devices introduced into the body, clothing, noise, and scattered X-ray components.

[0153] (Item 4) The aforementioned image element includes a linear or tubular device, The method for creating a trained model as described in item 1, wherein the projected image of the image element is generated by simulating the shape of the 3D model of the device with a curve generated based on random coordinate values.

[0154] (Item 5) The aforementioned image element includes blood vessels, The method for creating a trained model as described in item 1, wherein the projected image of the image element is generated by a simulation that randomly changes the shape of the three-dimensional model of blood vessels.

[0155] (Item 6) The aforementioned image element includes the scattered X-ray component, The method for creating a trained model as described in item 1, wherein the projected image of the image element is generated by a Monte Carlo simulation that models the shooting environment of the input image.

[0156] (Item 7) The method for creating a trained model as described in item 6, wherein the projected images of the aforementioned image elements are generated in multiple quantities by changing the projection angle over a projection angle range that can be captured by the X-ray imaging device in the imaging environment model.

[0157] (Item 8) The method for creating a trained model as described in item 6, wherein the projected images of the image elements are generated in multiple quantities by changing the energy spectrum of a virtual source in the imaging environment model.

[0158] (Item 9) The machine learning includes inputting the training input data and training output data created for each image element into a single learning model. The method for creating the trained model described in item 1, wherein the trained model is configured to extract the multiple image elements from the input image without overlap, and to output the extracted multiple image elements and the residual image elements remaining after extraction, respectively.

[0159] (Item 10) Using a pre-trained model that has been trained to extract specific image elements from an input image, multiple image elements are extracted separately from an X-ray image. An image generation method that generates a processed image in which each image element included in the X-ray image has undergone image processing by performing inter-image operations using multiple extracted images extracted for each image element and the X-ray image.

[0160] (Item 11) The image generation method according to item 10, wherein the image processing includes enhancement or removal processing.

[0161] (Item 12) The image generation method described in item 10, wherein the multiple image elements include a first element which is biological tissue and a second element which is non-biological tissue.

[0162] (Item 13) The image generation method according to item 10, wherein the multiple image elements include at least several of the following: bone, blood vessels, devices introduced into the body, clothing, noise, and scattered X-ray components.

[0163] (Item 14) Image processing is performed separately on some or all of the above-mentioned extracted images. The image generation method described in item 10, wherein the processed image is generated by an inter-image operation between the plurality of extracted images after image processing and the X-ray image.

[0164] (Item 15) The image generation method according to item 10, wherein the inter-image operation includes weighted addition or weighted subtraction of individual extracted images to the X-ray image.

[0165] (Item 16) The image generation method according to item 10, wherein the trained model is configured to extract the plurality of image elements from the input image without overlap, and to output the extracted plurality of image elements and the residual image elements remaining after extraction, respectively.

[0166] (Item 17) The image generation method described in item 10, wherein the pre-trained model is created in advance by machine learning using a reconstructed image obtained by reconstructing a two-dimensional projection image from three-dimensional image data and a projection image created from a three-dimensional model of the image elements by simulation.

[0167] (Item 18) An image acquisition unit that acquires X-ray images, An extraction processing unit that separately extracts multiple image elements from the X-ray image using a trained model that has been trained to extract specific image elements from an input image, An image processing apparatus comprising: an image generation unit that generates a processed image in which each image element included in the X-ray image has undergone image processing by performing inter-image operations using a plurality of extracted images extracted for each image element and the X-ray image. [Explanation of Symbols]

[0168] 10 Image acquisition unit 20 Extraction Processing Unit 21(21-1, 21-2, 21-N) Extracted Images 22 Processed Images 30 Image generation unit 40 (40-1, 40-2, 40-N) pre-trained models 50 image elements 51. First Element 52 Second Element 53 bones 54 Blood vessels 55 devices 56 Clothing 57 Noise 58 Scattered radiation component 59 Residual Image Elements 60 reconstructed images 61 Projection image 64 Teacher Input Data 65. Teacher output data 67 Superimposed images 80 CT image data (3D X-ray image data) 85 Shooting Environment Models 100 Image Processing Devices 111 First energy spectrum (energy spectrum) 112 Second energy spectrum (energy spectrum) 200 X-ray imaging equipment 201 X-ray image LM Learning Model

Claims

1. Reconstructed images are generated by reconstructing 3D X-ray image data into 2D projection images, and multiple reconstructed images are created for each of the multiple image elements to be extracted. If the three-dimensional X-ray image data contains the image elements, the reconstructed image has the image elements removed. Through simulation, multiple two-dimensional projection images are generated from the three-dimensional model of the image element, for each image element. The projected image of the aforementioned image element is superimposed on the corresponding reconstructed image to generate a superimposed image. By performing machine learning using the superimposed image as training input data and the projected image as training output data, a trained model is created that performs a process to extract the image elements contained in the input image, which includes an X-ray image of the subject, or by performing machine learning using the superimposed image as training input data and the reconstructed image as training output data, a trained model is created that performs a process to generate an image from which the image elements have been removed from the input image. Multiple superimposed images are created for each image element by superimposing the projection image of the corresponding image element onto the corresponding reconstructed image. A method for creating a trained model, wherein the plurality of image elements include a first element which is biological tissue and a second element which is non-biological tissue.

2. The method for creating a trained model according to claim 1, wherein the first element includes either bone or blood vessels, and the second element includes at least one of a device introduced into the body, clothing, noise, and scattered X-ray components.

3. The aforementioned image element includes a linear or tubular device as the second element, which is the non-biological tissue. The method for creating a trained model according to claim 1, wherein the projected image of the image element is generated based on the shape of a three-dimensional model of the device, which is generated by performing a simulation using a curve generated based on the irregular and random three-dimensional coordinate values ​​of the device implanted in the body.

4. 、 The first element includes blood vessels, The method for creating a trained model according to claim 1, wherein the projected image of the image element is generated based on the shape of the three-dimensional model of the blood vessel, which is generated by performing a simulation that randomly and without regularity changes the shape of the three-dimensional model of the blood vessel.

5. The second element includes the scattered radiation component of X-rays, The method for creating a trained model according to claim 1, wherein the projected image of the image element is an image showing the X-ray scattering component generated by a Monte Carlo simulation that models the arrangement of the equipment when the input image is taken and the shooting environment of the input image based on the X-ray energy spectrum.

6. The machine learning includes inputting the training input data and training output data created for each image element to a single learning model, The method for creating a trained model according to claim 1, wherein the trained model is configured to extract the plurality of image elements from the input image without overlap, and to output the extracted plurality of image elements and the residual image elements remaining after extraction, respectively.

7. Using the trained model described in Claim 1, multiple image elements are extracted separately from the X-ray image, By performing image operations using the multiple extracted images extracted for each image element and the X-ray image, a processed image is generated in which each image element included in the X-ray image has undergone image processing. An image generation method wherein the plurality of image elements include a first element which is biological tissue and a second element which is non-biological tissue.

8. The image generation method according to claim 7, wherein the first element includes either bone or blood vessels, and the second element includes at least one of a device introduced into the body, clothing, noise, and scattered X-ray components.

9. The image generation method according to claim 7, wherein the trained model is configured to extract the plurality of image elements from the input image without overlap, and to output the extracted plurality of image elements and the residual image elements remaining after extraction, respectively.

Images