Method for manufacturing a learned model, image generation method, and image processing apparatus

Abstract

The method for producing the learned model generates a reconstructed image (60) obtained by reconstructing three-dimensional X-ray image data (80). A projection image (61) is generated based on a three-dimensional model of image elements (50) through simulation. The projection image and the reconstructed image are superimposed to generate a superimposed image (67). The learned model (40) is produced by performing machine learning using the superimposed image and the reconstructed image or using the superimposed image and the projection image.

Description

Technical Field

[0001] The present invention relates to a method for producing a learned model, an image generation method, and an image processing apparatus. Background Art

[0002] Conventionally, a method for performing image processing using a learned model has been known. Such a method is disclosed, for example, in International Publication No. 2019 / 138438.

[0003] In the above International Publication No. 2019 / 138438, the following is disclosed: An X-ray image of a region including a specific part of a subject is transformed using a learned model, thereby producing an image representing the specific part. In the above International Publication No. 2019 / 138438, as the specific part, the bone part of the subject, a blood vessel injected with a contrast agent, and a stent placed in the body are exemplified. A learned model is produced by performing machine learning using, as a training input image and a training output image, a first DRR (Digitally Reconstructed Radiography) image and a second DRR image respectively reconstructed from CT image data. An image from which a specific part has been removed can be generated by subtracting an image obtained by performing a transformation using the learned model from the original image.

[0004] Prior Art Documents

[0005] Patent Documents

[0006] Patent Document 1: International Publication No. 2019 / 138438 Summary of the Invention

[0007] Problems to be Solved by the Invention

[0008] In the above International Publication No. 2019 / 138438, the following is disclosed: Image processing for a specific one type of image element (specific part) such as producing an image representing the bone part as the specific part, producing an image representing a blood vessel injected with a contrast agent as the specific part, and producing an image representing a stent placed in the body as the specific part. However, it is desired to perform image processing not only for a limited specific one image element (specific part), but also for various image elements and for a plurality of image elements, in order to improve the visual recognition of medical images in diverse usage scenarios.

[0009] In addition, in the above-mentioned International Publication No. 2019 / 138438, the first DRR image and the second DRR image reconstructed from CT image data are used for machine learning. Therefore, it is necessary to prepare CT image data that actually contains the image elements (specific parts) to be extracted. In order to efficiently perform machine learning for various image elements, it is desirable to be able to perform machine learning without being limited to using CT image data that actually contains the image elements to be extracted, or to be able to perform machine learning for image elements that, although contained in the CT image data, are difficult to extract separately.

[0010] The present invention has been completed to solve the above problems. One object of the present invention is to provide an image generation method and an image processing apparatus that can perform image processing for various image elements and for a plurality of image elements, and to provide a method for manufacturing a learned model that can efficiently manufacture a learned model used in such image processing.

[0011] Means for Solving the Problem

[0012] To achieve the above object, in the method for manufacturing a learned model according to the first aspect of the present invention, a reconstructed image obtained by reconstructing three-dimensional X-ray image data into a two-dimensional projection image is generated. A two-dimensional projection image is generated based on a three-dimensional model of an image element to be extracted by simulation. The projection image of the image element is superimposed on the reconstructed image to generate a superimposed image. Machine learning is performed by using the superimposed image as training input data and the reconstructed image or the projection image as training output data to manufacture a learned model for performing processing for extracting the image elements contained in the input image.

[0013] In the image generation method according to the second aspect of the present invention, a learned model that has learned the processing of extracting specific image elements from an input image is used to extract a plurality of image elements from an X-ray image, and an inter-image operation using the X-ray image and the plurality of extracted images extracted for each image element is performed to generate a processed image obtained by performing image processing on each image element contained in the X-ray image.

[0014] The image processing apparatus according to the third aspect of the present invention includes: an image acquisition unit that acquires an X-ray image; an extraction processing unit that uses a learned model that has learned the processing of extracting specific image elements from an input image to extract a plurality of image elements from the X-ray image; and an image generation unit that generates a processed image obtained by performing image processing on each image element contained in the X-ray image by performing an inter-image operation using the X-ray image and the plurality of extracted images extracted for each image element.

[0015] In addition, in this specification, the process of "extracting" an image element is a broad concept that includes both generating an image representing the extracted image element through the extraction of the image element and generating an X-ray image from which the image element has been removed by the extraction. More specifically, the process of "extracting" an image element includes generating an image of only the image element and generating an image obtained by removing the image element from the original X-ray image. Additionally, "image arithmetic" refers to generating one image by performing arithmetic operations such as addition, subtraction, multiplication, and division between a certain image and other images. More specifically, "image arithmetic" refers to performing arithmetic processing on pixel values for each corresponding pixel among multiple images to determine the pixel value of that pixel in the arithmetic result image.

[0016] Effects of the Invention

[0017] According to the method for manufacturing a learned model of the first aspect described above, the reconstructed image obtained by reconstructing three-dimensional X-ray image data into a two-dimensional projection image and the superimposed image obtained by superimposing the two-dimensional projection image generated based on the three-dimensional model of the image element to be extracted through simulation are used as training input data, and the reconstructed image or the projection image is used as training output data. Therefore, even if the three-dimensional X-ray image data does not contain the image element to be extracted, machine learning can be performed using the image element to be extracted generated through simulation. That is, even if CT image data actually containing the image element to be extracted is not prepared, training data can be prepared in advance. Additionally, the projection image of the image element to be extracted is generated through simulation. Therefore, for image elements that are included in CT image data but are difficult to separate and extract, training data can also be prepared in advance. As a result, the manufacturing of a learned model for performing image processing on various image elements and multiple image elements can be efficiently carried out.

[0018] Furthermore, according to the image generation method of the second aspect and the image processing apparatus of the third aspect described above, a learned model that has learned the process of extracting specific image elements from an input image is used to extract multiple image elements from an X-ray image respectively. By performing image arithmetic using the X-ray image and the multiple extracted images extracted for each image element, a processed image is generated. Therefore, based on extracting various image elements from the input X-ray image as extracted images, according to the types of the extracted image elements, the respective extracted images can be freely added or subtracted from the X-ray image. As a result, image processing can also be performed on various image elements and multiple image elements. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a block diagram showing an image processing apparatus based on one embodiment.

[0020] Figure 2 This is a diagram showing an example of an X-ray imaging apparatus.

[0021] Figure 3 This is a diagram for explaining machine learning for a learning model and a learned model.

[0022] Figure 4 This is a diagram showing an example of an image element.

[0023] Figure 5 This is a diagram showing a first example of extracting an image element by a learned model.

[0024] Figure 6 This is a diagram showing a second example of extracting an image element by a learned model and the generation of a processed image.

[0025] Figure 7 This is a diagram showing Figure 6 an example of performing image processing on an extracted image differently.

[0026] Figure 8 This is a flowchart for explaining an image generation method based on one embodiment.

[0027] Figure 9 This is a flowchart for explaining a method for creating a learned model based on one embodiment.

[0028] Figure 10 This is a diagram for explaining the generation of an overlaid image used in machine learning.

[0029] Figure 11 This is a diagram for explaining a method for generating a reconstructed image.

[0030] Figure 12 This is a diagram showing an example of training input data and training output data when the image element is a bone.

[0031] Figure 13 This is a diagram showing an example of a processed image when the image element is a bone.

[0032] Figure 14 This is a diagram showing an example of training input data and training output data when the image element is a device.

[0033] Figure 15 This is a diagram showing an example of a processed image when the image element is a device.

[0034] Figure 16 This is a diagram showing deformations (A) to (I) of a projection image of a device generated based on a three-dimensional model.

[0035] Figure 17 It is a diagram showing examples of training input data and training output data in the case where the image element is noise.

[0036] Figure 18 It is a diagram showing an example of a processed image in the case where the image element is noise.

[0037] Figure 19 It is a diagram showing examples of training input data and training output data in the case where the image element is a blood vessel.

[0038] Figure 20 It is a diagram showing an example of a processed image in the case where the image element is a blood vessel.

[0039] Figure 21 It is a diagram showing examples of training input data and training output data in the case where the image element is clothing.

[0040] Figure 22 It is a diagram showing an example of a processed image in the case where the image element is clothing.

[0041] Figure 23 It is a diagram showing examples of training input data and training output data in the case where the image element is a scattered ray component of X-rays.

[0042] Figure 24 It is a diagram showing an example of a processed image in the case where the image element is a scattered ray component of X-rays.

[0043] Figure 25 It is a diagram showing the projection angle ranges of X-rays in the first direction (A) and the second direction (B) in a Monte Carlo simulation.

[0044] Figure 26 It is a diagram showing a first example of the energy spectrum of X-rays in a Monte Carlo simulation.

[0045] Figure 27 It is a diagram showing a second example of the energy spectrum of X-rays in a Monte Carlo simulation.

[0046] Figure 28 It is a diagram showing an example of a collimator image used in training input data and training output data.

[0047] Figure 29 It is a diagram showing an example of extracting image elements by multiple learned models. Detailed implementation mode

[0048] Hereinafter, embodiments for embodying the present invention will be described based on the drawings.

[0049] Refer to Figures 1 to 28To describe the structure of the image processing apparatus 100 according to an embodiment, the image generation method according to an embodiment, and the method for producing a learned model according to an embodiment.

[0050] (Structure of the image processing apparatus)

[0051] First, with reference to Figure 1 the structure of the image processing apparatus 100 will be described.

[0052] The image processing apparatus 100 is configured to extract the image elements 50 included in the X-ray image 201 using the learned model 40 produced by machine learning, and perform image processing of the X-ray image 201 using the extracted image elements 50. The image processing apparatus 100 takes the X-ray image 201 captured by the X-ray imaging apparatus 200 as an input, and generates a processed image 22 obtained by performing image processing on each of the image elements 50 included in the X-ray image 201 as an output.

[0053] As Figure 1 shown, the image processing apparatus 10 has an image acquisition unit 10, an extraction processing unit 20, and an image generation unit

[0054] 30. The image processing apparatus 100 is constituted by a computer including one or more processors 101 such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit), and one or more storage units 102 such as an HDD (Hard Disk Drive) and an SSD (Solid State Drive). The image processing apparatus 100 is connected to a display device 103.

[0054] The image acquisition unit 10 is configured to acquire the X-ray image 201. The image acquisition unit 10 is constituted by, for example, an interface for connecting an external device to the image processing apparatus 100 in a communicable manner. Regarding the image acquisition unit 10, the interface can include a communication interface such as a LAN (Local Area Network). The image acquisition unit 10 can include input / output interfaces such as HDMI (registered trademark), Display Port (display interface), and USB ports. The image acquisition unit 10 can acquire the X-ray image 201 from the X-ray imaging apparatus 200 or from a server device connected via a network through communication.

[0055] The X-ray image 201 is a medical image obtained by photographing a patient or a subject with the X-ray imaging device 200. The X-ray image 201 can be either a still image or a moving image. The moving image is a collection of still images captured at a prescribed frame rate. The X-ray image 201 is a two-dimensional image. The X-ray image 201 can be various images captured by general X-ray photography, fluoroscopic photography, angiographic photography, etc.

[0056] The extraction processing unit 20 is configured to extract a plurality of image elements 50 from the X-ray image 201 respectively by using the learned model 40 stored in the storage unit 102. The learned model 40 is a learned model that has learned the process of extracting specific image elements 50 from the input image.

[0057] The image element 50 is an image part or image information constituting the X-ray image 201 and is defined in the form of the same or the same kind of set. The image element 50 can be a part of the human body classified anatomically. Such an image element 50 is, for example, a biological tissue such as bone or blood vessel. The image element 50 can be an object introduced or left in the body of the subject during surgery or the like. Such an image element 50 can be, for example, devices such as a catheter, a guide wire, a stent, a surgical instrument, and a fixing device introduced into the body. The image element 50 can be noise, artifacts, scattered X-ray components, etc. generated during the imaging process in X-ray photography. The image element 50 can be clothing worn by the subject and photographed during the photography. In addition, clothing refers to the concept including clothes, ornaments, or other wearable items. For example, buttons, zippers, accessories, metal parts, etc. of the clothing are photographed in the X-ray image 201.

[0058] The learned model 40 is pre-produced by machine learning that learns the process of extracting specific image elements 50 from the input image. The extraction processing unit 20 uses one or more learned models 40 to perform the extraction of the image elements 50. Thereby, the extraction processing unit 20 generates a plurality of extraction images 21 obtained by extracting different image elements 50 from the X-ray image 201. For example, the first image element 50 is included in the first extraction image 21, and the second image element 50 different from the first image element 50 is included in the second extraction image 21. The method for producing the learned model 40 will be described later.

[0059] The image generation unit 30 is configured to generate a processed image 22 obtained by performing image processing on each image element 50 included in the X-ray image 201 by performing inter-image calculation using the X-ray image 201 and a plurality of extracted images 21 extracted for each image element 50. The image processing includes, for example, enhancement processing of the image element 50 or removal processing of the image element 50. In addition, the enhancement processing is a process of relatively increasing the pixel value of the pixels belonging to the image element 50. The removal processing is a process of relatively decreasing the pixel value of the pixels belonging to the image element 50. The removal processing includes not only completely removing from the image but also decreasing the visual recognition by partial removal. In addition, the enhancement processing can be, for example, edge enhancement processing. The removal processing can be noise removal processing.

[0060] The inter-image calculation refers to calculating the pixel value of corresponding pixels in the processed image 22 by performing pixel value calculation for each corresponding pixel between the plurality of extracted images 21 and the X-ray image 201. The content of the calculation is not particularly limited, but can be, for example, the four arithmetic operations of addition, subtraction, multiplication, and division. In the present embodiment, the inter-image calculation includes weighted addition or weighted subtraction of each extracted image 21 with respect to the X-ray image 201. By performing weighted addition of the extracted image 21 with respect to the X-ray image 201, enhancement processing of the image element 50 included in the X-ray image 201 can be performed. By performing weighted subtraction of the extracted image 21 with respect to the X-ray image 201, removal processing of the image element 50 included in the X-ray image 201 can be performed. By adjusting the value of the weight, the enhancement degree or removal degree of the image element 50 can be optimized.

[0061] In Figure 1 the example, the processor 101 executes a program (not shown) stored in the storage unit 102, thereby functioning as the extraction processing unit 20 and the image generation unit 30. That is, in Figure 1 the example, the extraction processing unit 20 and the image generation unit 30 are implemented as functional blocks of the processor 101. The extraction processing unit 20 and the image generation unit 30 may also be configured as independent hardware.

[0062] In addition, the independent hardware includes a case where the extraction processing unit 20 and the image generation unit 30 are constituted by different processors. The independent hardware includes the following cases: The image processing apparatus 100 includes a plurality of computers (PCs), and the computer (PC) functioning as the extraction processing unit and the computer (PC) functioning as the image generation unit are respectively provided.

[0063] The image processing apparatus 100 causes the display apparatus 103 to display the processed image 22 generated by the image generation unit 30. The image processing apparatus 100 transmits, for example, the generated processed image 22 to a server apparatus via a network. The image processing apparatus 100 records, for example, the generated processed image 22 in the storage unit 102.

[0064] Figure 2 shows a structural example of the X-ray imaging apparatus 200. Figure 2 Shows an example of an X-ray angiography apparatus capable of performing fluoroscopic imaging of blood vessels. The X-ray imaging apparatus 200 includes a top plate 210, an X-ray irradiation unit 220, and an X-ray detector 230. The top plate 210 is configured to support the subject 1 (person). The X-ray irradiation unit 220 includes an X-ray source such as an X-ray tube and is configured to irradiate the X-ray detector 230 with X-rays. The X-ray detector 230 is constituted by, for example, an FPD (Flat Panel Detector) and is configured to detect the X-rays that are irradiated from the X-ray irradiation unit 220 and pass through the subject 1.

[0065] In Figure 2 's example, the X-ray irradiation unit 220 and the X-ray detector 230 are held by the C-arm 240. The C-arm 240 can move along the arc-shaped arm portion in the first direction 250 and can rotate about the rotation axis 251 in the second direction 252. Thereby, the X-ray imaging apparatus 200 can change the projection direction of the X-rays from the X-ray irradiation unit 220 toward the X-ray detector 230 by a predetermined angular range in each of the first direction 250 and the second direction 252.

[0066] (Learned model)

[0067] As Figure 1 shown, the extraction process of the image elements 50 included in the X-ray image 201 is performed by the learned model 40 created by machine learning. As Figure 3 shown, the learned model 40 extracts the pre-learned image elements 50 from the input image and outputs an extraction image 21 that only shows the extracted image elements 50.

[0068] In the present embodiment, the learned model 40 is pre-made by performing machine learning using the reconstructed image 60 obtained by reconstructing a two-dimensional projection image from three-dimensional image data and the projection image 61 created based on the three-dimensional model of the image elements 50 by simulation.

[0069] As a machine learning method, any method such as Fully Convolutional Networks (FCN), neural networks, Support Vector Machines (SVM), Boosting methods, etc. can be used. For the learning model LM (learned model 40) of the present embodiment, a convolutional neural network can be used, and more preferably, a fully convolutional neural network is used. This learning model LM (learned model 40) is configured to include an input layer 41 for inputting an image, a convolutional layer 42, and an output layer 43.

[0070] To produce the learned model 40, machine learning using learning data 66 including training input data 64 and training output data 65 is performed. The training input data 64 and the training output data 65 included in one piece of learning data 66 have a relationship of data before extraction and data after extraction for the same image element 50.

[0071] Regarding a plurality of image elements 50 to be extracted, machine learning is performed for each image element. That is, learning data 66 is prepared for each image element 50 to be extracted.

[0072] As Figure 4 shown, the plurality of image elements 50 include a first element 51 that is a biological tissue and a second element 52 that is a non-biological tissue. In addition, the plurality of image elements 50 include at least a plurality of bone 53, blood vessel 54, a device 55 introduced into the body, clothing 56, noise 57, and a scattered ray component 58 of X-rays. The bone 53 and the blood vessel 54 among them correspond to the first element 51. The first element 51 may also include biological tissues other than the bone 53 and the blood vessel 54. The device 55 introduced into the body, the clothing 56, the noise 57, and the scattered ray component 58 of X-rays among them correspond to the second element 52. The second element 52 may also include image elements other than the device 55, the clothing 56, the noise 57, and the scattered ray component 58.

[0073] In Figure 5 the example shown, one learned model 40 is configured to extract a plurality of image elements 50 respectively. The learned model 40 has one input channel and a plurality of (N) output channels. N is an integer of 2 or more. When the X-ray image 201 is input to the input channel, the learned model 40 extracts the first image element 50 to the Nth image element 50 respectively. The learned model 40 outputs the extracted first image element 50 to the Nth image element 50 from the N output channels as the first extracted image 21-1 to the Nth extracted image 21-N.

[0074] As different from Figure 5 In Figure 6In the example, the learned model 40 has 1 input channel and N + 1 output channels. When the X-ray image 201 is input to the input channel of the learned model 40, a plurality of (N) image elements 50 are extracted from the input image without repetition. Without repetition means that the image information included in any extracted image (e.g., the first extracted image 21-1) is not included in other extracted images (e.g., the second extracted image 21-2 to the Nth extracted image 21-N). The learned model 40 outputs the first to Nth extracted image elements as the first extracted image 21-1 to the Nth extracted image 21-N, respectively. Then, the learned model 40 outputs the remaining residual image elements 59 after extraction as the (N + 1)th extracted image 21x from the (N + 1)th output channel.

[0075] In this case, the first extracted image 21-1 to the Nth extracted image 21-N do not contain the same image information as each other. Moreover, the image information that remains unextracted in the input X-ray image 201 is included in the (N + 1)th extracted image 21x. Therefore, when the first extracted image 21-1 to the Nth extracted image 21-N are added to the (N + 1)th extracted image 21x, the original X-ray image 201 is restored.

[0076] Thus, in Figure 6 the example, the learned model 40 is configured to extract a plurality of image elements 50 from the input image without repetition, and output the plurality of extracted image elements 50 and the remaining residual image elements 59 after extraction, respectively. As a result, compared with the input image, the total of the image information extracted by the extraction process does not increase or decrease.

[0077] As Figure 6 shown, the image generation unit 30 (refer to Figure 1 ) performs an inter-image operation of adding or subtracting the first extracted image 21-1 to the Nth extracted image 21-N after considering the weight coefficients 23 and then adding them to the input X-ray image 201 to generate the processed image 22. The (N + 1)th extracted image 21x representing the residual image elements 59 is not required for image processing. As the weight coefficients 23, coefficients w1 to wn are separately set corresponding to the first extracted image 21-1 to the Nth extracted image 21-N, respectively. The weight coefficients 23 can be, for example, fixed values set in advance in the storage unit 102. However, multiple types of weight coefficients can also be set according to the use of the processed image 22, etc. For example, in the case where the first set value of the weight coefficient 23 is used for the first extracted image 21-1 in the processing mode A, and the second set value of the weight coefficient 23 is used for the first extracted image 21-1 in the processing mode B. In addition, the weight coefficients 23 can also be set to arbitrary values according to the user's operation input.

[0078] AsFigure 7 As shown, before performing the image - to - image operation between each extracted image 21 and the X - ray image 201, image processing can be performed on each of the extracted images 21 separately. In Figure 7 this example, the image generation unit 30 is configured to perform image processing on some or all of the multiple extracted images 21 separately. The processed image 22 is generated by performing an image - to - image operation between the multiple processed extracted images 21 and the X - ray image 201.

[0079] For example, in Figure 7 , the image generation unit 30 performs first image processing 25 - 1 on the first extracted image 21 - 1, second image processing 25 - 2 on the second extracted image 21 - 2, ……, and Nth image processing 25 - N on the Nth extracted image 21 - N. Depending on the image element 50, there may be cases where no image processing is required, so image processing can also be performed only on some of the extracted images 21.

[0080] The image processing performed on each extracted image is not particularly limited, but for example, it can be image correction processing or image interpolation processing. Image correction processing can include edge enhancement processing and noise removal processing. Image correction processing can be, for example, contrast adjustment, line enhancement processing, smoothing processing, etc. Image interpolation processing is, for example, a process of interpolating the interrupted part for image elements 50 such as guide wires and catheters that are difficult to capture in the X - ray image 201 and thus appear to be interrupted midway. For each image element 50, the appropriate parameters for contrast adjustment, line enhancement processing, etc. are different, and it is difficult to uniformly process all image elements 50. However, by performing image processing on each extracted image 21, optimal image processing can be performed on each individual image element 50.

[0081] (Image generation method)

[0082] Next, with reference to Figure 8 , the image generation method of this embodiment will be described. The image generation method can be implemented by the image processing device 100. The image generation method of this embodiment at least includes Figure 8 the following steps S2 and S5 shown.

[0083] (S2) Using the learned model 40 that has learned the process of extracting specific image elements 50 from the input image, extract multiple image elements 50 from the X - ray image 201 respectively.

[0084] (S5) By performing an image - to - image operation using the X - ray image 201 and the multiple extracted images 21 extracted for each image element 50, generate a processed image 22 obtained by performing image processing on each image element 50 included in the X - ray image 201.

[0085] In addition, the image generation method of the present embodiment may further include Figure 8 the steps S1, S3, S4, and S6 shown in

[0086] In step S1, an X-ray image 201 is acquired. Specifically, the image acquisition unit 10 (refer to Figure 1 ) acquires, through communication with the X-ray imaging apparatus 200 or the server apparatus, the X-ray image 201 taken by the X-ray imaging apparatus 200 shown, for example, in Figure 2 .

[0087] In step S2, the extraction processing unit 20 uses the learned model 40 to extract a plurality of image elements 50 from the X-ray image 201 respectively. The extraction processing unit 20 inputs the X-ray image 201 acquired in step S1 into the learned model 40. Thus, the learned model 40 outputs the first extracted image 21-1 to the Nth extracted image 21-N as shown in Figure 5 or Figure 6 .

[0088] In step S3, as shown in Figure 7 , image processing may be performed on a part or all of the plurality of extracted images 21. In this case, the image generation unit 30 performs preset image processing on the extracted image 21 that is the object of the image processing, with prescribed parameters. Whether to perform the image processing may be determined according to an input from the user. Whether to perform the image processing may be determined according to the image quality of the extracted image 21. Step S3 may not be performed.

[0089] In step S4, the image generation unit 30 acquires operation parameters for each of the extracted images 21. The operation parameters include, for example, setting values of the weight coefficients 23 and setting values of the operation methods. The setting value of the operation method indicates whether weighted addition (i.e., enhancement processing of the image elements 50) or weighted subtraction (i.e., removal processing of the image elements 50) is to be performed on the extracted image 21 that is the object. The setting value of the operation method and the setting value of the weight coefficients 23 are preset in the storage unit 102 for each type of the image elements 50 to be extracted.

[0090] In step S5, the image generation unit 30 performs an inter-image operation using the X-ray image 201 and the plurality of extracted images 21 extracted for each image element 50. The image generation unit 30 performs the inter-image operation according to the parameters obtained in step S4. The image generation unit 30 multiplies each of the first extracted image 21-1 to the Nth extracted image 21-N by its corresponding weight coefficient 23, and performs the inter-image operation with the X-ray image 201 by its corresponding operation method. As a result, an X-ray image 201 obtained by performing weighted addition or weighted subtraction using each of the extracted images 21 is generated as the processed image 22. Thus, the image generation unit 30 generates the processed image 22 (refer to Figure 6 or Figure 7 ) obtained by performing image processing on each image element 50 included in the X-ray image 201, and this image processing is enhancement processing or removal processing.

[0091] In step S6, the image processing apparatus 100 outputs the processed image 22. The image processing apparatus 100 outputs the processed image 22 to the display apparatus 103 or the server apparatus. In addition, the image processing apparatus 100 stores the processed image 22 in the storage unit 102.

[0092] After causing the display apparatus 103 to display the processed image 22, the image generation unit 30 may accept an operation input for changing the operation parameters. For example, the image generation unit 30 may accept an input of the value of the weight coefficient 23, and may also accept the selection of other preset parameters. For example, the image generation unit 30 may accept a change in the parameters of the image processing in step S3. Then, the image generation unit 30 may re-generate the processed image 22 using the changed parameters according to the user's operation input.

[0093] (Method for manufacturing a learned model)

[0094] Next, the method for manufacturing a learned model will be described. The manufacturing of the learned model 40 may be implemented by the processor 101 of the image processing apparatus 100, but can be executed using a computer for machine learning (learning apparatus 300, refer to Figure 10 ).

[0095] As Figure 9 and Figure 10 shown, the method for manufacturing a learned model according to the present embodiment includes the following steps S11 to S14.

[0096] (S11) Generate a reconstructed image 60 obtained by reconstructing CT (Computed Tomography) image data 80 into a two-dimensional projection image. The CT image data 80 is an example of "three-dimensional X-ray image data".

[0097] (S12) Generate a two-dimensional projection image 61 based on the three-dimensional model of the image element 50 to be extracted through simulation.

[0098] (S13) Superimpose the projection image 61 of the image element 50 on the reconstructed image 60 to generate a superimposed image 67.

[0099] (S14) By using the superimposed image 67 as the training input data 64 (refer to Figure 3 ) and using the reconstructed image 60 or the projection image 61 as the training output data 65 (refer to Figure 3 ) for machine learning, a learned model 40 for processing to extract the image element 50 included in the input image is produced (refer to Figure 3 ).

[0100] In the present embodiment, the machine learning includes inputting the training input data 64 (refer to Figure 3 ) and the training output data 65 (refer to Figure 3 ) produced for each image element 50 into one learning model LM. Machine learning can also be performed for different learning models LM for each image element 50 to be extracted.

[0101] First, in Figure 9 step S10, CT image data 80 is acquired. The CT image data 80 is three-dimensional image data obtained by performing CT photography on the subject 1 and reflecting the three-dimensional structure of the subject 1. The CT image data 80 is a three-dimensional aggregate of voxel data including three-dimensional position coordinates and the CT value at that position coordinate. For image elements of parts with movement (pulsation) such as blood vessels, four-dimensional 4D-CT data that also includes time variation in the three-dimensional information can also be used. Thereby, learning can be performed with high precision for objects that move over time.

[0102] In step S11, a reconstructed image 60 obtained by reconstructing the CT image data 80 into a two-dimensional projection image is generated.

[0103] The reconstructed image 60 is a DRR image generated based on the CT image data 80. The DRR image is a simulated X-ray image as a two-dimensional projection image produced through a virtual perspective projection obtained by simulating the geometric projection conditions of the X-ray irradiation unit 220 and the X-ray detector 230 of the X-ray imaging apparatus 200 as shown in Figure 2 .

[0104] Specifically, as shown in Figure 11As shown, in a three-dimensional virtual space, a virtual X-ray tube 91 and a virtual X-ray detector 92 are virtually arranged so as to form a predetermined projection direction with respect to the CT image data 80, thereby generating a three-dimensional spatial configuration (imaging geometry) of a virtual X-ray imaging system. The configuration of the CT image data 80 and the virtual X-ray tube 91 and the virtual X-ray detector 92 is set to be consistent with Figure 2 The imaging geometry shown is the same as the arrangement of the actual subject 1, the X-ray irradiation unit 220, and the X-ray detector 230. The imaging geometry refers to the geometric arrangement relationship between the subject 1, the X-ray irradiation unit 220, and the X-ray detector 230 in three-dimensional space.

[0105] The pixel value of each pixel in the reconstructed image 60 is then calculated by summing the CT values of each voxel that the X-rays pass through from the virtual X-ray tube 91 until they reach the virtual X-ray detector 92. By changing the imaging geometry, a simulated X-ray image can be generated at any projection angle.

[0106] In the method for creating a learned model, a learning device 300 generates multiple reconstructed images 60 based on a single set of three-dimensional data (CT image data 80). The number of reconstructed images 60 generated can be, for example, approximately 100,000. The learning device 300 generates the multiple reconstructed images 60 by varying various parameters such as projection angle, projection coordinates, parameters for generating DRR images, contrast, and edge enhancement. For example, the aforementioned parameters can be randomly varied and the algorithm for generating the reconstructed images 60 can be used to generate multiple, mutually different reconstructed images 60.

[0107] In this embodiment, the superimposed image 67 is created by superimposing the projected image 61 including the picture element 50 . Therefore, the reconstructed image 60 may not include the picture element 50 to be extracted, or even if it does include the picture element 50 , it may not have a contrast that can be extracted.

[0108] In this specification, the terms "random" and "random numbers" mean irregularity or irregular number sequences (sets of numbers), but do not necessarily need to be completely random and include pseudo-random and pseudo-random numbers.

[0109] exist Figure 9 In step S12 , a two-dimensional projection image 61 is generated based on the three-dimensional model of the image element 50 to be extracted by simulation.

[0110] The projection image 61 is a two-dimensional image representing the image element 50 that is the extraction target of the learned model 40. The projection image 61 only includes, for example, the image element 50. The learning device 300 acquires a three-dimensional model of the image element 50 that is the extraction target, and generates the projection image 61 through simulation based on the three-dimensional model.

[0111] The three-dimensional model is produced, for example, by performing CT imaging on an object including the image element 50 that is the extraction target, and extracting the image element 50 from the obtained CT data. For example, a three-dimensional model can be produced using a CT image database publicly available from a research institution or the like. Regarding the CT image database, for example, as a lung CT image dataset, there is the LIDC / IDRI (The Lung Image Data base Consortium and Image Database Resource Initiative) of the National Cancer Institute of the United States. In addition to this, a brain CT image dataset, a three-dimensional model of a standardized bone, etc. can also be used. Also, regarding the device 55 and the clothing 56, a three-dimensional CAD data can be used for production. The three-dimensional model is, for example, three-dimensional image data that only includes the image element 50 that is the extraction target.

[0112] In addition, it is not necessary to generate all the projection images 61 used in machine learning through simulation based on the three-dimensional model. The projection image 61 can be produced by acquiring two-dimensional data (X-ray image) that actually includes the image element 50 to be extracted, and separating and extracting the image element 50 included in the acquired image.

[0113] The learning device 300 makes parameters such as the projection direction, translation amount, rotation amount, deformation amount, and contrast different for the three-dimensional model or the two-dimensional data, to generate a plurality of two-dimensional projection images 61. A plurality of projection images 61 can be generated by an algorithm that randomly changes variable parameters such as the translation amount, rotation amount, deformation amount, and contrast for the original data (three-dimensional model, two-dimensional data, other projection images 61).

[0114] The learning device 300 generates a plurality of two-dimensional projection images 61 for each type of the image element 50. The learning device 300 generates a plurality of projection images 61 based on one piece of original data (three-dimensional data, two-dimensional data, or projection image 61). The number of generated projection images 61 based on one piece of original data can be, for example, about 100,000.

[0115] In step S13, the projection image 61 generated in step S12 is superimposed on the reconstructed image 60 generated in step S11, to generate a superimposed image 67 (refer to Figure 10). By superimposing, a superimposed image 67 is generated, which includes the projection image 61 of the image element 50 to be extracted in the two-dimensional reconstructed image 60. By combining multiple reconstructed images 60 and multiple projection images 61, multiple superimposed images 67 are also generated. 1 learning data 66 (refer to Figure 3 ) includes any one of the reconstructed image 60 and the projected image 61 used when generating the superimposed image 67, as well as the superimposed image 67.

[0116] exist Figure 9 In step S14, machine learning is performed. Figure 3 The superimposed image 67 is used as the training input data 64 of the input layer 41 of the learning model LM. Furthermore, the reconstructed image 60 or the projected image 61 is used as the training output data 65 input to the output layer 43 of the learning model.

[0117] When the training output data 65 is a reconstructed image 60, the learning model LM learns to extract image elements 50 from the input image and generate an extracted image 21 that does not contain the image elements 50. When the training output data 65 is a projected image 61, the learning model LM learns to extract image elements 50 from the input image and generate an extracted image 21 that represents the extracted image elements 50. The extracted image 21 can be an image that contains only the image elements 50. Furthermore, an image containing only the extracted image elements 50 can be generated by generating a processed image 22 that does not contain the image elements 50 and subtracting it from the input X-ray image 201. Therefore, using the reconstructed image 60 as the training output data 65 and using the projected image 61 as the training output data 65 can be considered equivalent from the perspective of image processing.

[0118] Furthermore, it is not necessary to use the superimposed image 67 to create all the training data 66 used in machine learning. Instead, a reconstructed image 60 that actually includes the image element 50 to be extracted may be used as training input data 64, and a projection image 61 of the image element 50 extracted from the reconstructed image 60 may be used as training output data 65.

[0119] exist Figure 9 In step S15, the learning device 300 determines whether machine learning is complete. For example, if machine learning has been repeated a predetermined number of times on all learning data 66, the learning device 300 determines that machine learning is complete. For example, if the value of an evaluation function for evaluating the performance of the learning model LM exceeds a predetermined value, the learning device 300 determines that machine learning is complete. If the learning device 300 determines that machine learning is not complete, the learning device 300 changes the learning data 66 in step S16 and executes the machine learning of step S14 using the next learning data 66.

[0120] After machine learning is completed, in step S17, the learned learning model LM is stored as the learned model 40. Thus, the production of the learned model 40 is completed. The produced learned model 40 is provided to the image processing apparatus 100 via a network, or recorded on a non-transitory recording medium and provided to the image processing apparatus 100.

[0121] (Specific example of each image element)

[0122] Next, specific examples of the superimposed image 67 (training input data 64) and the reconstructed image 60 or the projection image 61 (training output data 65) for each image element 50 will be described. In addition, an example of the processed image 22 using the extracted image 21 for each image element 50 will be described.

[0123] 〈Bone part〉

[0124] Figure 12 An example in which the image element 50 is the bone 53 is shown. The training input data 64 is, for example, the superimposed image 67 of the reconstructed image 60 not including the bone 53 and the projection image 61 including the bone 53.

[0125] For example, the projection image 61 is produced based on the CT image data 80 or the three-dimensional model of the bone and superimposed on the reconstructed image 60 from which the bone 53 has been removed to generate the superimposed image 67. The reconstructed image 60 from which the bone 53 has been removed is generated by limiting (fixing) the CT value to zero for the pixels within the range of the CT value of the bone 53. Generally, the CT value of the bone part is about 200 HU to about 1000 HU, so a prescribed value of about 0 HU to 200 HU may be set as the threshold.

[0126] The training output data 65 is the projection image 61 including only the bone 53. For the training output data 65, the projection image 61 used when generating the superimposed image 67 is used.

[0127] In addition, since the CT image data 80 obtained by photographing the subject usually includes the bone part, the reconstructed image 60 generated based on the CT image data 80 may also be used as the training input data 64. Therefore, it is not necessary to produce the superimposed image 67.

[0128] In this case, the training output data 65 is the reconstructed image 60 including only the bone 53. The reconstructed image 60 including only the bone 53 is generated, for example, by limiting the CT value to zero for the pixels having a CT value less than the CT value of the bone 53 in the reconstructed image 60 including the bone 53.

[0129] Through machine learning, a learning model LM learns to generate an extraction image 21 obtained by extracting a bone 53 from an input image such as training input data 64 as training output data 65.

[0130] An image processing apparatus 100 performs weighted subtraction of the extraction image 21 generated by a learned model 40 on an X-ray image 201 acquired by an X-ray imaging apparatus 200. Thus, as Figure 13 shown, a processed image 22 is generated by removing the image elements 50 of the bone 53 from the X-ray image 201. In Figure 13 , for ease of explanation, the image elements 50 of the removed bone 53 are shown by dashed lines.

[0131] <Device>

[0132] Figure 14 An example in which the image element 50 is a device 55 is shown. In Figure 14 , the device 55 is a guide wire.

[0133] The training input data 64 is a superimposed image 67 including the device 55. The reconstructed image 60 generated based on the CT image data 80 does not include the image element 50. The projection image 61 of only the device 55 generated based on the three-dimensional model of the device 55 is superimposed on the reconstructed image 60. As a result, a superimposed image 67 including the device 55 is generated as Figure 14 shown.

[0134] As described above, a two-dimensional X-ray image of the device 55 may also be taken, and deformations of the projection image 61 after changing the shape and the like may be produced by simulation. A plurality of superimposed images 67 are generated using a plurality of projection images 61.

[0135] In particular, as Figure 16 shown, in the case where the image element 50 is a linear or tubular device 55, the projection image 61 of the image element 50 is generated by simulating the shape of the three-dimensional model of the device 55 with a curve generated based on random coordinate values. In Figure 16 , as an example of the device 55, examples (A) to (I) in which the projection image 61 of a guide wire holding a stent 55a is generated in a random shape by curve simulation are shown.

[0136] Based on a Bezier curve with random coordinate values as a base point, the guide wire 55b in each projection image 61 is generated in a different shape. The Bezier curve is a (K - 1)-th order curve obtained based on K (K is an integer of 3 or more) control points. By an algorithm for randomly specifying the coordinate values of the K control points, a large number of projection images 61 of the device 55 in various shapes can be generated.

[0137] By applying random translation, rotation, deformation, and contrast change to the pseudo-stent after three-dimensional modeling, a plurality of individual projection images 61 of the stent 55a left in the body are generated.

[0138] Return to Figure 14 , the training output data 65 only contains the projection image 61 of the device 55. For the training output data 65, the projection image 61 used when generating the superimposed image 67 is used.

[0139] Through machine learning, the learning model LM learns to generate an extraction image 21 obtained by extracting the device 55 based on an input image such as the training input data 64.

[0140] The image processing device 100 performs weighted addition of the extraction image 21 generated by the learned model 40 on the X-ray image 201 acquired by the X-ray imaging device 200. Thus, as Figure 15 shown, a processed image 22 is generated in which the image elements 50 of the device 55 are enhanced with respect to the X-ray image 201. In Figure 15 , compared with Figure 14 , the device 55 is shown in bold, thereby showing the enhanced image. The enhancement process not only includes the process of increasing the pixel value, but can also include the process of coloring and displaying the image elements 50 of the device 55 through the image processing before the image operation shown by Figure 7 .

[0141] 〈Noise〉

[0142] Figure 17 An example where the image element 50 is noise 57 is shown. The noise 57 is, for example, random noise, but in Figure 17 , for the sake of easy explanation, it is shown as a set of dotted lines in the horizontal direction.

[0143] The training input data 64 is a superimposed image 67 containing noise 57. The reconstructed image 60 generated based on the CT image data 80 does not contain noise 57. Only the projection image 61 of the randomly generated noise 57 is superimposed on the reconstructed image 60. As a result, a superimposed image 67 containing noise 57 is generated as Figure 17 . The noise 57 is made by randomly generating Gaussian noise following the Gaussian distribution and Poisson noise following the Poisson distribution in each projection image 61.

[0144] The training output data 65 only contains the projection image 61 of the noise 57. For the training output data 65, the projection image 61 used when generating the superimposed image 67 is used.

[0145] Through machine learning, the learning model LM learns to generate an extraction image 21 obtained by extracting noise 57 based on an input image such as the training input data 64, like the training output data 65.

[0146] The image processing device 100 performs weighted subtraction of the extraction image 21 generated by the learned model 40 on the X-ray image 201 acquired by the X-ray imaging device 200. As a result, as Figure 18 shown, a processed image 22 is generated by removing the image elements 50 with noise 57 from the X-ray image 201. Figure 18 The processed image 22 shows the case where noise 57 is removed from the X-ray image 201 containing noise 57, such as Figure 17 the training input data 64.

[0147] 〈Blood Vessels〉

[0148] Figure 19 An example where the image element 50 is a blood vessel 54 is shown. The blood vessel 54 is an angiographic blood vessel obtained by introducing a contrast agent and imaging. Figure 19 An example of the cerebral blood vessels in the head is shown, but it can also be other blood vessels. For example, the blood vessel can be the coronary artery of the heart.

[0149] The training input data 64 is a superimposed image 67 containing the blood vessel 54. In the reconstructed image 60 generated based on the CT image data 80 taken without contrast, the blood vessel 54 is hardly contained (there is not enough contrast). The projection image 61 of only the blood vessel 54 generated based on the three-dimensional model of the blood vessel 54 is superimposed on the reconstructed image 60. As a result, a superimposed image 67 containing the blood vessel 54 is generated, such as Figure 19 shown.

[0150] In the case where the image element 50 is the blood vessel 54, the projection image 61 of the image element 50 is generated by a simulation that randomly changes the shape of the three-dimensional model of the blood vessel 54. For the blood vessel in the projection image 61, random translation, rotation, deformation, contrast change, etc. are implemented through simulation. That is, similar to the Figure 16 device 55 shown, a deformation of the projection image 61 of the blood vessel 54 with random changes is generated. As described above, the projection image 61 of the blood vessel 54 can also be made based on the CT image data of the angiographic blood vessel taken by contrast.

[0151] The training output data 65 is only the projection image 61 containing the blood vessel 54. For the training output data 65, the projection image 61 used when generating the superimposed image 67 is used.

[0152] Through machine learning, the learning model LM learns to generate an extraction image 21 obtained by extracting blood vessels 54, such as the training output data 65, based on an input image such as the training input data 64.

[0153] The image processing apparatus 100 performs weighted addition of the extraction image 21 generated by the learned model 40 on the X-ray image 201 acquired by the X-ray imaging apparatus 200. Thus, as Figure 20 shown, a processed image 22 is generated in which the image elements 50 of the blood vessels 54 are enhanced with respect to the X-ray image 201. Figure 20 The processed image 22 of Figure 19 shows a case where the blood vessels 54 are enhanced and displayed in the X-ray image 201 including the image elements 50 of the blood vessels 54, such as the training input data 64 of

[0154] 〈Clothing〉

[0155] Figure 21 is an example showing that the image element 50 is clothing 56. In Figure 21 , as an example of the clothing 56, buttons of the clothes and a necklace worn by the subject are shown.

[0156] The training input data 64 is a superimposed image 67 including the clothing 56. The reconstructed image 60 generated based on the CT image data 80 does not include the clothing 56. The projection image 61 including only the clothing 56 generated based on the three-dimensional model of the clothing 56 is superimposed on the reconstructed image 60. As a result, a superimposed image 67 including the clothing 56, such as Figure 21 is generated. The three-dimensional model of the clothing 56 can be produced based on a CT image including only the clothing 56, and can be produced based on CAD data or the like, for example. As described above, a two-dimensional X-ray image of the clothing 56 can also be taken as the projection image 61. For the projection image 61, random translation, rotation, deformation, contrast change, etc. are performed by simulation.

[0157] The training output data 65 is a projection image 61 including only the clothing 56. For the training output data 65, the same data as the projection image 61 used when generating the superimposed image 67 is used.

[0158] Through machine learning, the learning model LM learns to generate an extraction image 21 obtained by extracting the clothing 56, such as the training output data 65, based on an input image such as the training input data 64.

[0159] The image processing apparatus 100 performs weighted subtraction of the extraction image 21 generated by the learned model 40 on the X-ray image 201 acquired by the X-ray imaging apparatus 200. Thus, as Figure 22As shown, a processed image 22 is generated by removing the image element 50 of the clothing 56 from the X-ray image 201. In Figure 22 In order to facilitate the description of the removal, the portion of the image element 50 of the removed clothing 56 is shown by a dashed line.

[0160] 〈Scattered ray component〉

[0161] Figure 23 An example in which the image element 50 is the scattered ray component 58 of the X-ray is shown.

[0162] The training input data 64 is a superimposed image 67 including the scattered ray component 58. In the reconstructed image 60 generated based on the CT image data 80, the scattered ray component 58 is not included in the reconstruction operation. The projection image 61 including only the scattered ray component 58 generated by Monte Carlo simulation that models the imaging environment of the input image is superimposed on the reconstructed image 60. As a result, a superimposed image 67 including the scattered ray component 58 is generated.

[0163] In the Monte Carlo simulation, for example, a three-dimensional model of the imaging environment of the input image (X-ray image 201) from the X-ray irradiation unit 220 to the X-ray detector 230 as shown in Figure 2 (or Figure 11 ) is created to obtain the imaging environment model 85. Then, as a random phenomenon using random numbers, each X-ray photon emitted from the X-ray irradiation unit 220 and detected by the X-ray detector 230 is calculated (simulated). That is, in the simulation, the physical properties related to the projection direction of the X-ray, the shape (body shape) of the subject, and the interaction with photons are assumed. Then, using random numbers as a random phenomenon, interactions such as absorption and scattering phenomena generated when the X-ray photon passes through the subject are calculated. In the Monte Carlo simulation, the calculation of a specified number of photons is performed, and a projection image 61 formed by the X-ray photons detected by the virtual X-ray detector 230 is generated. The specified number of photons only needs to be a sufficient number for imaging, for example, about 10 billion.

[0164] In the present embodiment, the projection angle is changed within the entire projection angle range that the X-ray imaging apparatus 200 in the imaging environment model 85 can photograph, to generate a plurality of projection images 61 of the image element 50 as the scattered ray component 58. For example, in Figure 2 In the example shown, the projection direction can be changed along the first direction 250 and the second direction 252 by moving the C-arm 240. Therefore, as Figure 25As shown, multiple projection angles that vary as different angular values throughout a projection angle range of 250 ± α degrees in a first direction and 252 ± β degrees in a second direction are used to generate a Monte Carlo simulation-based projection image 61. The projection angles can be changed at equal angular intervals throughout the projection angle range, or can be set to be changed randomly by a specified number of values within the projection angle range. Also, a projection image 61 can be generated at angular values outside the projection angle range and near the boundaries (±α degrees, ±β degrees) of the projection angles.

[0165] In addition, in the present embodiment, the energy spectrum of the virtual ray source in the imaging environment model 85 is changed to generate multiple projection images 61 of the image elements 50 that are scattered ray components 58. That is, multiple conditions in which the energy spectrum of the X-rays irradiated by the X-ray irradiation unit 220 assumed to be the imaging environment model 85 is changed to different spectra are used to produce the Monte Carlo simulation-based projection image 61.

[0166] Generally speaking, the lower the photon energy of the X-rays, the more easily it is absorbed in the subject's body, and thus the less likely it is to generate scattered ray components 58. The higher the photon energy of the X-rays, the more difficult it is to be absorbed in the subject's body, and thus the more likely it is to generate scattered ray components 58. Therefore, for example, a projection image 61 based on Figure 26 the first energy spectrum 111 shown and a projection image 61 based on Figure 27 the second energy spectrum 112 shown are produced. The second energy spectrum 112 is a spectrum with relatively higher energy compared to the first energy spectrum 111. In Figure 26 and Figure 27 , the horizontal axis of the curve graph represents the energy [keV] of the X-ray photons, and the vertical axis of the curve graph represents the relative intensity of the X-rays (i.e., the number of detected X-ray photons).

[0167] In addition, due to differences in absorption spectra, a beam hardening phenomenon occurs in which the energy spectrum of the X-rays irradiated onto the subject is relatively biased towards the high energy side during the process of detecting the energy spectrum. In the reconstructed image 60 generated based on the CT image data 80, the image quality change caused by the beam hardening phenomenon cannot be simulated, but in the Monte Carlo simulation-based projection image 61, the influence of the beam hardening phenomenon can be simulated.

[0168] As an example, Figure 23Shows the projection image 61 of the scattered ray component 58 based on Compton scattering obtained by Monte Carlo simulation. In the present embodiment, scattered ray components 58 other than Compton scattering such as Rayleigh scattering can be obtained. It is also possible to obtain not only the scattered ray component 58 based on single scattering but also the scattered ray component 58 based on multiple scattering. Machine learning can be performed in such a way that these various scattered ray components 58 are made into different projection images 61 and extracted separately, or in such a way that a projection image 61 in which various scattered ray components 58 are summarized and displayed is made to uniformly extract various scattered ray components 58.

[0169] The training output data 65 is a projection image 61 that only contains the scattered ray component 58. For the training output data 65, the same data as the projection image 61 used when generating the superimposed image 67 is used.

[0170] Through machine learning, the learning model LM learns to generate an extraction image 21 obtained by extracting the scattered ray component 58 based on an input image such as the training input data 64.

[0171] The image processing device 100 performs weighted subtraction of the extraction image 21 generated by the learned model 40 on the X-ray image 201 acquired by the X-ray imaging device 200. Thus, as Figure 24 shown, a processed image 22 is generated by removing the image element 50 of the scattered ray component 58 from the X-ray image 201. The scattered ray component 58 is the cause of the decrease in the contrast of the X-ray image. Therefore, in the processed image 22, the contrast can be improved by removing the scattered ray component 58. Figure 24 The processed image 22 of Figure 23 shows the case where the scattered ray component 58 is removed from the X-ray image 201 whose contrast has decreased due to the scattered ray component 58 such as the training input data 64 of

[0172] 〈Collimator Image〉

[0173] In the present embodiment, in a part of each of the produced training input data 64 and each of the training output data 65, there is included a collimator image 68 in which the imaging range is limited by a collimator (not shown) provided in the X-ray imaging apparatus 200 (X-ray irradiation unit 220). In the collimator image 68, an image is formed only in a partial area in the image, and no image information is included in the area shielded by the collimator. A part of each of the training input data 64 and each of the training output data 65 includes a plurality of collimator images 68, and the plurality of collimator images 68 are images obtained by randomly varying the shape of the X-ray irradiation range (i.e., the image area) and the parameters of the image quality affected by the collimator based on simulation using the actually captured images with the collimator. The parameters of the image quality affected by the collimator are the penetration degree (contrast), the blurring degree of edges, the noise content, and the like.

[0174] The collimator image 68 is generated, for example, by removing the image portions outside the simulated irradiation range from the superimposed image 67, the reconstructed image 60, and the projection image 61 and performing image processing simulating the influence of the collimator. The collimator image 68 can also be generated based on the actually captured images with the collimator. Thereby, the robustness of the extraction process of the image element 50 can be improved against the changes in the X-ray irradiation range and the image quality caused by the use of the collimator.

[0175] Machine learning for each type of the image element 50 and generation of the processed image 22 by the image processing apparatus 100 are performed as described above.

[0176] In each of the above specific examples, for the sake of convenience of explanation, each of the image elements 50 and the processed image 22 has been described separately. However, when the processed image 22 is actually generated by the image processing apparatus 100, the X-ray image 201 input to the image processing apparatus 100 includes a plurality of the above-described bone 53, blood vessel 54, device 55, clothing 56, noise 57, and scattered ray component 58. The image processing apparatus 100 generates an extraction image 21 obtained by extracting each of the image elements 50 by the learned model 40 based on the input X-ray image 201, and performs inter-image calculation. As a result, a processed image 22 obtained by performing enhancement processing or removal processing on each of the plurality of image elements 50 is generated.

[0177] For example, in Figure 7 the example shown, N = 6, the first extraction image 21-1 represents the bone 53, the second extraction image 21-2 represents the device 55, the third extraction image (assumed to be 21-3) represents the noise 57, the fourth extraction image (assumed to be 21-4) represents the blood vessel 54, the fifth extraction image (assumed to be 21-5) represents the clothing 56, and the sixth extraction image (assumed to be 21-6) represents the scattered ray component 58.

[0178] Regarding the usage scenarios of processing the image 22, several examples are shown.

[0179] For example, the processed image 22 is applied to an X-ray image of the anterior chest of a subject generated by ordinary X-ray photography. In this case, in the processed image 22, the bone 53, noise 57, clothing 56, and scattered ray component 58 are removed. By removing the bone 53, the visual recognition of regions of interest such as the heart and lungs is improved. In addition, by removing the noise 57 and scattered ray component 58, the visual recognition of the entire image is improved. Since the image elements 50 of the clothing 56 can be removed, the subject can undergo X-ray photography without taking off clothes containing metals or accessories. This has useful effects such as improving work efficiency and reducing the waiting time of the subject, for example, in the case of mass X-ray photography of a large number of subjects continuously, such as in group medical examinations.

[0180] In addition, for example, the processed image 22 is applied to an X-ray fluoroscopic image in treatments based on X-ray images, such as catheter treatment using an X-ray angiography device (IVR; Interventional Radiology). In this case, in the processed image 22, the bone 53, noise 57, and scattered ray component 58 are removed. In the processed image 22, devices 55 such as catheters, guidewires, and stents, and blood vessels 54 are enhanced. By removing the bone 53, noise 57, and scattered ray component 58, the visual recognition of the fluoroscopic image is improved. By enhancing the devices 55 and blood vessels 54, the visual recognition of the regions of interest and the devices being operated during catheter treatment is improved.

[0181] (Effects of this embodiment)

[0182] In this embodiment, the following effects can be obtained.

[0183] According to the method for manufacturing the learned model 40 of the present embodiment, the reconstructed image 60 obtained by reconstructing the CT image data 80 into a two-dimensional projection image and the superimposed image 67 obtained by superimposing the two-dimensional projection image 61 generated by simulation based on the three-dimensional model of the image element 50 to be extracted are used as the training input data 64, and the reconstructed image 60 or the projection image 61 is used as the training output data 65. Thus, even if the CT image data 80 does not contain the image element 50 to be extracted, machine learning can be performed using the image element 50 to be extracted generated by simulation. That is, even if the CT image data 80 actually containing the image element 50 to be extracted is not prepared, the training data can be prepared in advance. In addition, since the projection image 61 of the image element 50 to be extracted is generated by simulation, for the image element 50 that is difficult to separate and extract even if it is included in the CT image data 80, the training data can be prepared in advance. As a result, the manufacturing of the learned model 40 for performing image processing on various image elements 50 and multiple image elements 50 can be efficiently performed.

[0184] According to the image generation method and the image processing apparatus 100 of the present embodiment, using the learned model 40 that has learned the process of extracting specific image elements 50 from the input image, a plurality of image elements 50 are respectively extracted from the X-ray image 201, and the processed image 22 is generated by performing an inter-image operation using the X-ray image 201 and the plurality of extracted images 21 extracted for each image element 50. Thus, on the basis of respectively extracting various image elements 50 from the input X-ray image 201 as the extracted images 21, according to the types of the extracted image elements 50, the respective extracted images 21 and the X-ray image 201 can be freely added or subtracted. As a result, image processing can be performed for various image elements 50 and multiple image elements 50.

[0185] In addition, in the example of the above embodiment, by configuring as follows, further effects can be obtained.

[0186] That is, in the present embodiment, a plurality of superimposed images 67 are created for each of the plurality of different image elements 50. The plurality of image elements 50 include a first element 51 that is a biological tissue and a second element 52 that is a non-biological tissue. With such a configuration, a learned model 40 can be created that can comprehensively perform image processing on the image elements 50 for biological tissues such as bone 53 and blood vessel 54, and image processing on the image elements 50 for non-biological tissues such as the device 55 introduced into the body and the clothing 56 worn by the subject. By using such a learned model 40 in the image processing apparatus 100, it is possible to comprehensively perform image processing on the image elements 50 for biological tissues and image processing on the image elements 50 for non-biological tissues.

[0187] In addition, in the present embodiment, a plurality of superimposed images 67 are created for each of the plurality of different image elements 50. The plurality of image elements 50 include at least a plurality of bone 53, blood vessel 54, device 55 introduced into the body, clothing 56, noise 57, and scattered ray component 58 of X-rays. With such a configuration, a learned model 40 can be created that can comprehensively perform image processing on the diverse image elements 50 corresponding to various usage scenarios of the X-ray image 201. By using such a learned model 40 in the image processing apparatus 100, it is possible to comprehensively perform image processing on the diverse image elements 50 corresponding to various usage scenarios of the X-ray image 201.

[0188] In addition, in the present embodiment, the image element 50 includes a linear or tubular device 55, and the projection image 61 of the image element 50 is generated by simulating the shape of the three-dimensional model of the device 55 with a curve generated based on random coordinate values. With such a configuration, it is possible to generate, by simulation, a large amount of training data for the image elements 50 for learning devices 55 such as guide wires and catheters that are long and curved into various shapes in diverse shapes. As a result, even without preparing in advance a large amount of three-dimensional CT data in a state where a large number of devices 55 are actually arranged in the subject's body, efficient machine learning can be performed.

[0189] In addition, in the present embodiment, the image element 50 includes a blood vessel 54, and the projection image 61 of the image element 50 is generated by a simulation that randomly changes the shape of the three-dimensional model of the blood vessel 54. With such a configuration, it is possible to generate, by simulation, a large amount of training data for the image elements 50 for learning blood vessels 54 that are long and curved into complex shapes, including diverse individual differences. As a result, even without preparing in advance a large amount of three-dimensional CT data of various subjects, efficient machine learning can be performed.

[0190] In addition, in the present embodiment, the image element 50 includes a scattered ray component 58 of X-rays, and the projection image 61 of the image element 50 is generated by Monte Carlo simulation that models the imaging environment of the input image. With such a configuration, it is possible to generate, by Monte Carlo simulation, the projection image 61 of the scattered ray component 58 of X-rays that is difficult to separately extract from actual three-dimensional CT data and two-dimensional X-ray images 201. Using the projection image 61 generated in this way to produce the learned model 40 for extracting the scattered ray component 58, it is possible to implement image processing for removing the scattered ray component 58 from the X-ray image 201 obtained by actually imaging a subject without performing complex and high-load computational processing such as Monte Carlo simulation. Therefore, it is possible to effectively improve the contrast in, for example, the abdomen and near the bone where the influence of the scattered ray component 58 is large. In addition, for example, the following utilization can also be performed: In X-ray imaging, imaging is performed without using an X-ray absorption grid for reducing the influence of the scattered ray component 58, and the scattered ray component 58 is removed by image processing. In this case, the X-ray dose can be reduced by an amount equivalent to not using the X-ray absorption grid, so that the radiation dose received by the subject can be effectively reduced.

[0191] In addition, in the present embodiment, the projection angle is varied within the entire projection angle range (±α, ±β) that the X-ray imaging apparatus 200 in the imaging environment model 85 can image, to generate a plurality of projection images 61 of the image element 50. With such a configuration, it is possible to produce a highly versatile learned model 40 that can effectively extract the scattered ray component 58 not only when imaging is performed only from a specific projection direction (front of the chest, side of the chest, etc.) as in ordinary X-ray imaging, but also when X-ray imaging is performed at various projection angles or while varying the projection angle in X-ray image-based treatment.

[0192] In addition, in the present embodiment, the energy spectrum of the virtual radiation source in the imaging environment model 85 is varied to generate a plurality of projection images 61 of the image element 50. In an actual medical site, X-ray imaging is performed under various imaging conditions with different energy spectra according to the imaging region and the like. According to the above structure, it is possible to produce a highly versatile learned model 40 that can effectively extract the scattered ray component 58 even when X-ray images 201 are imaged with various energy spectra.

[0193] In addition, in the present embodiment, machine learning includes inputting training input data 64 and training output data 65 created for each image element 50 into one learning model LM. The learned model 40 is configured to extract a plurality of image elements 50 from the input image without repetition, and output the extracted plurality of image elements 50 and the remaining residual image elements 59 after extraction respectively. With such a configuration, it is possible to provide a learned model 40 that can extract the image elements 50 in such a way that when all the extracted plurality of image elements 50 and the residual image elements 59 are added together, they are restored to the original input image. That is to say, even when a plurality of image elements 50 are extracted from one X-ray image 201, there is no loss of the image information contained in the X-ray image 201 or unexpected increase in the image information. Since even after performing various image processing for improving visual recognition and doctors using the X-ray image 201 for diagnosis, etc. need to find the basis for diagnosis in the image information contained in the original image, according to the learned model 40 that can extract the image elements 50 without loss of image information, it is possible to provide an image that doctors, etc. can trust even when performing complex image processing.

[0194] In addition, in the present embodiment, image processing is performed on a part or all of the plurality of extracted images 21, and a processed image 22 is generated by inter-image operation between the processed plurality of extracted images 21 and the X-ray image 201. With such a configuration, in the case where the learned model 40 can be used to extract a plurality of image elements 50 respectively, image processing such as correction processing and interpolation processing can be independently performed on the extracted images 21 of the respective image elements 50. Here, for example, when it is desired to perform image processing that only acts on specific image elements 50 on the X-ray image 201 before extraction, the image processing algorithm is complex and high-load, and it may also have an adverse effect on image elements 50 other than the specific image elements 50. In contrast, when performing image processing on the extracted image 21, there are no other image elements 50, so even if only simple filtering processing, etc. for the whole of the extracted image 21 is performed, high image processing accuracy can be obtained. Moreover, by using the processed extracted image 21 for inter-image operation, it is possible to simply generate a processed image 22 that not only enhances or removes a plurality of image elements 50, but also performs high-precision image processing limited to each image element 50.

[0195] In addition, in the present embodiment, the inter-image operation includes weighted addition or weighted subtraction of each extracted image 21 for the X-ray image 201. According to such a configuration, a processed image 22 can be obtained in which each of the plurality of extracted image elements 50 is individually enhanced or removed. In addition, the enhancement degree or removal degree of each image element 50 can be appropriately set by adjusting the weight coefficient 23, so that a processed image 22 with high visual recognition for the necessary image elements 50 can be generated corresponding to various usage scenarios of the X-ray image 201.

[0196] (Modification example)

[0197] In addition, it should be considered that the embodiments disclosed this time are illustrative rather than restrictive in all aspects. The scope of the present invention is not shown by the description of the above embodiments, but by the claims, and also includes all changes (modification examples) within the meaning and scope equivalent to the claims.

[0198] For example, in the above embodiment, an example is shown in which the device for performing machine learning (learning device 300) and the image processing device 100 are different devices, but the present invention is not limited thereto. In the present invention, machine learning can also be performed in the image processing device. In addition, the learning device 300 can also be composed of a server computer provided on the cloud.

[0199] In addition, in the above embodiment (refer to Figures 5 to 7 ), an example is shown in which a plurality of image elements 50 are extracted by one learned model 40, but the present invention is not limited thereto. In the present invention, a plurality of learned models can also be used to extract a plurality of image elements 50. For example, as Figure 29 shown, the learned model 40 can also be set one by one for each image element 50 to be extracted. In Figure 29 , the learned models 40-1, 40-2,..., 40-N are provided. One learned model 40 extracts one (one type of) image element 50. In addition, a plurality of learned models 40 can also be provided to extract a plurality of image elements 50.

[0200] In addition, in the above-described embodiment, an example is shown in which a plurality of image elements 50 include at least a plurality of bone 53, blood vessel 54, device 55, clothing 56, noise 57, and scattered ray component 58, but the present invention is not limited thereto. The image element 50 may also include image elements other than the bone 53, blood vessel 54, device 55, clothing 56, noise 57, and scattered ray component 58. For example, the image element 50 may also be a specific structural part such as a specific organ in the body. In addition, for example, the image element 50 of a specific part of the bone in the bone 53 and the image element 50 of a specific part of the blood vessel in the blood vessel 54 may be extracted separately from other bones and other blood vessels. The image element 50 may not include the bone 53, blood vessel 54, device 55, clothing 56, noise 57, and scattered ray component 58.

[0201] In addition, in the above-described embodiment, an example is shown in which a removal process is performed on the bone 53 and an enhancement process is performed on the blood vessel 54 and the device 55 by inter-image operation, but the present invention is not limited thereto. An enhancement process may be performed on the bone 53, and a removal process may be performed on one or both of the blood vessel 54 and the device 55.

[0202] In addition, in the above-described embodiment, two examples of weighted addition and weighted subtraction are shown as examples of inter-image operation, but the present invention is not limited thereto. The inter-image operation may also be addition or subtraction without a weight coefficient. The enhancement process of the image element 50 may also be performed by multiplication with or without a weight coefficient. The removal process of the image element 50 may also be performed by division with or without a weight coefficient.

[0203] [Mode]

[0204] Those skilled in the art can understand that the above exemplary embodiments are specific examples of the following modes.

[0205] (Item 1)

[0206] A method for producing a learned model, wherein,

[0207] A reconstructed image obtained by reconstructing three-dimensional X-ray image data into a two-dimensional projection image is generated,

[0208] A two-dimensional projection image is generated by simulation based on a three-dimensional model of an image element to be extracted,

[0209] The projection image of the image element is superimposed on the reconstructed image to generate a superimposed image,

[0210] A learned model for processing to extract the image elements included in the input image is produced by performing machine learning with the superimposed image as training input data and the reconstructed image or the projection image as training output data.

[0211] (Item 2)

[0212] According to the method for producing the learned model described in Item 1, wherein

[0213] A plurality of the superimposed images are produced for each of a plurality of different image elements.

[0214] The plurality of image elements include a first element being a biological tissue and a second element being a non-biological tissue.

[0215] (Item 3)

[0216] According to the method for producing the learned model described in Item 1, wherein

[0217] A plurality of the superimposed images are produced for each of a plurality of different image elements.

[0218] The plurality of image elements include at least a plurality of bone, blood vessels, devices implanted in the body, clothing, noise, and scattered ray components of X-rays.

[0219] (Item 4)

[0220] According to the method for producing the learned model described in Item 1, wherein,

[0221] The image elements include linear or tubular devices.

[0222] The projection image of the image elements is generated by simulating the shape of the three-dimensional model of the device with a curve generated based on random coordinate values.

[0223] (Item 5)

[0224] According to the method for producing the learned model described in Item 1, wherein,

[0225] The image elements include blood vessels.

[0226] The projection image of the image elements is generated by a simulation in which the shape of the three-dimensional model of the blood vessels is randomly changed.

[0227] (Item 6)

[0228] According to the method for producing the learned model described in Item 1, wherein,

[0229] The image elements include scattered ray components of X-rays.

[0230] The projection image of the image element is generated by Monte Carlo simulation that models the imaging environment of the input image.

[0231] (Item 7)

[0232] According to the method for producing a learned model described in Item 6, wherein,

[0233] The projection angle is varied within the entire range of projection angles that can be captured by the X-ray imaging apparatus in the imaging environment model to generate a plurality of the projection images of the image element.

[0234] (Item 8)

[0235] According to the method for producing a learned model described in Item 6, wherein,

[0236] The energy spectrum of the virtual radiation source in the imaging environment model is varied to generate a plurality of the projection images of the image element.

[0237] (Item 9)

[0238] According to the method for producing a learned model described in Item 1, wherein,

[0239] The machine learning includes inputting the training input data and the training output data produced for each of the image elements into one learning model,

[0240] The learned model is configured to extract a plurality of the image elements from the input image without repetition and output the plurality of the extracted image elements and the remaining residual image elements after extraction respectively.

[0241] (Item 10)

[0242] An image generation method, wherein,

[0243] A plurality of image elements are respectively extracted from an X-ray image using a learned model that has learned the process of extracting specific image elements from an input image,

[0244] An image operation is performed using the X-ray image and a plurality of extracted images extracted for each image element to generate a processed image obtained by performing image processing on each of the image elements included in the X-ray image.

[0245] (Item 11)

[0246] According to the image generation method described in Item 10, wherein,

[0247] The image processing includes enhancement processing or removal processing.

[0248] (Item 12)

[0249] According to the image generation method described in Item 10, wherein,

[0250] The plurality of image elements include a first element that is a biological tissue and a second element that is a non-biological tissue.

[0251] (Item 13)

[0252] According to the image generation method described in Item 10, wherein,

[0253] The plurality of image elements include at least a plurality of bones, blood vessels, devices introduced into the body, clothing, noise, and scattered ray components of X-rays.

[0254] (Item 14)

[0255] According to the image generation method described in Item 10, wherein,

[0256] Image processing is performed on a part or all of the plurality of extracted images,

[0257] The processed image is generated by performing an inter-image operation between the plurality of extracted images after image processing and the X-ray image.

[0258] (Item 15)

[0259] According to the image generation method described in Item 10, wherein,

[0260] The inter-image operation includes performing weighted addition or weighted subtraction of each extracted image on the X-ray image.

[0261] (Item 16)

[0262] According to the image generation method described in Item 10, wherein,

[0263] The learned model is configured to extract the plurality of image elements from the input image without repetition and output the extracted plurality of image elements and the remaining residual image elements after extraction respectively.

[0264] (Item 17)

[0265] According to the image generation method described in Item 10, wherein,

[0266] The learned model is pre-produced by performing machine learning using a reconstructed image and a projection image. The reconstructed image is obtained by reconstructing a three-dimensional image data into a two-dimensional projection image, and the projection image is produced by simulation based on a three-dimensional model of the image element.

[0267] (Item 18)

[0268] An image processing apparatus includes:

[0269] an image acquisition unit that acquires an X-ray image;

[0270] an extraction processing unit that extracts a plurality of image elements from the X-ray image using a learned model that has learned a process of extracting specific image elements from an input image; and

[0271] an image generation unit that generates a processed image obtained by performing image processing on each image element included in the X-ray image by performing inter-image calculation using the X-ray image and a plurality of extracted images extracted for each image element.

[0272] Explanation of Reference Numerals

[0273] 10: Image acquisition unit; 20: Extraction processing unit; 21 (21-1, 21-2, 21-N): Extracted image; 22: Processed image; 30: Image generation unit; 40 (40-1, 40-2, 40-N): Learned model; 50: Image element; 51: First element; 52: Second element; 53: Bone; 54: Blood vessel; 55: Device; 56: Clothing; 57: Noise; 58: Scattered ray component; 59: Residual image element; 60: Reconstructed image; 61: Projection image; 64: Training input data; 65: Training output data; 67: Overlaid image; 80: CT image data (three-dimensional X-ray image data); 85: Imaging environment model; 100: Image processing apparatus; 111: First energy spectrum (energy spectrum); 112: Second energy spectrum (energy spectrum); 200: X-ray imaging apparatus; 201: X-ray image; LM: Learning model.

Claims

1. A method for fabricating a learned model, wherein, a reconstructed image obtained by reconstructing three-dimensional X-ray image data into a two-dimensional projection image is generated, a two-dimensional projection image is generated by simulation based on a three-dimensional model of an image element to be extracted, the projection image of the image element is superimposed on the reconstructed image to generate a superimposed image, a learned model for performing a process of extracting a plurality of the image elements included in an input image and generating a plurality of extraction images capturing the plurality of the image elements is fabricated by performing machine learning with the superimposed image as training input data and the reconstructed image or the projection image as training output data.

2. The method for fabricating a learned model according to claim 1, wherein, a plurality of the superimposed images are fabricated for each of a plurality of different image elements, the plurality of image elements include a first element that is a biological tissue and a second element that is a non-biological tissue, or, the plurality of image elements include at least a plurality of bone, blood vessels, a device introduced into the body, clothing, noise, and a scattered ray component of X-rays.

3. The method for fabricating a learned model according to claim 1, wherein, the image element includes a linear or tubular device, the projection image of the image element is generated by simulating the shape of the three-dimensional model of the device with a curve generated based on random coordinate values.

4. The method for fabricating a learned model according to claim 1, wherein, the image element includes blood vessels, the projection image of the image element is generated by a simulation that randomly varies the shape of the three-dimensional model of the blood vessels.

5. The method for fabricating a learned model according to claim 1, wherein, the image element includes a scattered ray component of X-rays, the projection image of the image element is generated by Monte Carlo simulation that models the imaging environment of the input image.

6. The method for fabricating a learned model according to claim 5, wherein, the projection angle is varied within the entire projection angle range that can be captured by an X-ray imaging device in the imaging environment model to generate a plurality of the projection images of the image element, or, the energy spectrum of a virtual radiation source in the imaging environment model is varied to generate a plurality of the projection images of the image element.

7. The method for fabricating a learned model according to claim 1, wherein, the machine learning includes inputting the training input data and the training output data fabricated for each of the image elements into one learning model, the learned model is configured to: extract a plurality of the image elements from the input image without repetition and output the extracted plurality of the image elements and the remaining residual image elements after extraction respectively.

8. An image generation method, wherein, a plurality of image elements are respectively extracted from an X-ray image using a learned model that has learned a process of extracting image elements from an input image By performing an inter-image operation using the X-ray image and a plurality of extracted images extracted for each image element, a processed image obtained by performing image processing on each image element included in the X-ray image is generated. The learned model is pre-produced by performing machine learning using a reconstructed image and a projection image. The reconstructed image is obtained by reconstructing three-dimensional image data into a two-dimensional projection image, and the projection image is produced by simulation based on a three-dimensional model of the image element.

9. The image generation method according to claim 8, wherein the image processing includes enhancement processing or removal processing.

10. The image generation method according to claim 8, wherein the plurality of image elements include a first element that is a biological tissue and a second element that is a non-biological tissue, or the plurality of image elements include at least a plurality of bone, blood vessels, devices introduced into the body, clothing, noise, and scattered ray components of X-rays.

11. The image generation method according to claim 8, wherein image processing is performed on part or all of the plurality of extracted images respectively, and the processed image is generated by an inter-image operation between the plurality of extracted images after image processing and the X-ray image.

12. The image generation method according to claim 8, wherein the inter-image operation includes weighted addition or weighted subtraction of each extracted image with respect to the X-ray image.

13. The image generation method according to claim 8, wherein the learned model is configured to extract the plurality of image elements from the input image without repetition, and output the extracted plurality of image elements and the remaining residual image elements after extraction respectively.

14. An image processing apparatus includes: an image acquisition unit that acquires an X-ray image; an extraction processing unit that uses a learned model that has learned the process of extracting image elements from an input image to extract a plurality of image elements from the X-ray image respectively; and an image generation unit that generates a processed image obtained by performing image processing on each image element included in the X-ray image by performing an inter-image operation using the X-ray image and a plurality of extracted images extracted for each image element, Among them, the learned model is pre-produced by performing machine learning using a reconstructed image and a projection image. The reconstructed image is obtained by reconstructing three-dimensional image data into a two-dimensional projection image, and the projection image is produced by simulation based on a three-dimensional model of the image element.

15. A method for producing a learned model, wherein a reconstructed image obtained by reconstructing three-dimensional X-ray image data into a two-dimensional projection image is generated, a two-dimensional projection image is generated by simulating the shape of a three-dimensional model of a linear or tubular curved device image element to be extracted with a curve, and the projection image of the image element is superimposed on the reconstructed image to generate a superimposed image. A learned model for processing to extract the image elements included in the input image is created by performing machine learning using the superimposed image as training input data and the reconstructed image or the projection image as training output data.

Images