Substance information image processing device, image processing method, program, and image processing system

Abstract

To provide an image processing device, an image processing method, and a program that allow accurate substance information to be acquired even from CT image data acquired from a conventional X-ray CT apparatus equipped with an energy integration type X-ray detector.SOLUTION: An image processing device includes: a model acquisition unit for acquiring a first learned model that has learned using a teacher data set including, in teacher data, first CT image data based on first detection data imaged by a first X-ray CT apparatus and first substance information based on the first detection data; a CT image data acquisition unit for acquiring second CT image data based on second detection data imaged by a second X-ray CT apparatus equipped with an energy integration type radiation detector using a detection system different from that of the first X-ray CT apparatus; and inference means for inferring second substance information from the second CT image data.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The disclosure of this specification relates to an image processing device, an image processing method, and a program that enable the acquisition of highly accurate material information from CT image data acquired using an energy-integrating radiation detector. [Background technology]

[0002] It is known that photon-counting X-ray CT scanners (PCCT: Photon Counting Computed Tomography) and dual-energy X-ray CT scanners (DECT: Dual Energy Computed Tomography) can detect the amount of X-ray transmission in each X-ray energy range, and can distinguish materials by using the characteristics of materials whose X-ray attenuation coefficients differ depending on the X-ray energy range (see Patent Document 1). For example, by visualizing material information in a differentiated form between iodine contrast agents for angiography and calcification, it becomes easier to determine abnormalities in blood vessels. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-13679 A Summary of the Invention [Problem to be solved by the invention]

[0004] In clinical practice, X-ray CT devices equipped with energy-integrated X-ray detectors other than PCCT or DECT are used. However, it is not possible to obtain discrimination information from detection data from conventional X-ray CT devices equipped with energy-integrated radiation detectors.

[0005] The present invention aims to provide an image processing device, an image processing method, a program, and an image processing system that enable the acquisition of highly accurate material information from CT image data acquired by a conventional X-ray CT device equipped with an energy-integrating X-ray detector. [Means for solving the problem]

[0006] An image processing device according to one embodiment of the present invention is characterized in that it comprises a model acquisition unit that acquires a first learned model trained using a training data set including first CT image data based on first detection data captured by a first X-ray CT device and first material information based on the first detection data, a CT image data acquisition unit that acquires second CT image data based on second detection data captured by a second X-ray CT device equipped with an energy-integrating radiation detector using a detection method different from that of the first X-ray CT device, and an inference means that infers second material information from the second CT image data. Effect of the Invention

[0007] The image processing device according to the present invention can obtain material information with high accuracy even from CT image data obtained by a conventional X-ray CT device equipped with an energy-integral type X-ray detector. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram showing an example of an image processing system according to first to third embodiments. [Diagram 2] 1 is a block diagram showing an example of a hardware configuration of an image processing device according to first to third embodiments. [Diagram 3] 1 is a block diagram showing an example of a functional configuration of an image processing device according to a first embodiment. [Figure 4] 1 is a block diagram showing an example of a functional configuration of an image processing device according to a first embodiment. [Diagram 5] FIG. 4 is a flow chart illustrating an example of a first learning process according to the first embodiment. [Figure 6] 1 is a block diagram showing an example of a functional configuration of an image processing device according to a first embodiment. [Figure 7] FIG. 4 is a flow chart showing an example of an image display process according to the first embodiment. [Figure 8] FIG. 2 is a diagram showing an example of a display screen of the image processing device according to the first to third embodiments. [Figure 9] FIG. 11 is a block diagram showing an example of a functional configuration of an image processing device according to a second embodiment. [Figure 10] FIG. 11 is a flow diagram illustrating an example of a second learning process according to the second embodiment. [Figure 11] FIG. 13 is a block diagram showing an example of a functional configuration of an image processing device according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the present invention will be described in detail based on the preferred embodiment with reference to the attached drawings. Note that the same numbers are used for the items described in other embodiments, and the description thereof will be omitted as appropriate. Also, the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the configurations shown in the drawings.

[0010] <First embodiment> (System Configuration) FIG. 1 is a block diagram showing an example of an image processing system 100 according to the present embodiment. The image processing system 100 according to the present embodiment shown in FIG. 1 is composed of an image processing device 101, a first X-ray CT device 102, a second X-ray CT device 103, and a LAN (Local Area Network) 104 connecting these devices. Here, the first X-ray CT device 102 is a photon counting type X-ray CT device. The second X-ray CT device 103 is an X-ray CT device having a detection method different from that of the first X-ray CT device 102, specifically, a conventional type X-ray CT device equipped with an energy integral type X-ray detector. The image processing device 101 uses a teacher data set including first CT image data and first material information in the teacher data based on the first detection data acquired from the first X-ray CT device 102. Using a trained model trained using the teacher data set, material information is generated from second CT image data reconstructed based on detection data captured by the second X-ray CT device 103. Here, the substance information is, for example, substance decomposition information, and substance decomposition information is information about an area in an image that corresponds to a specified substance, information indicating whether or not a substance is present in an image, information indicating the likelihood of a specified substance being present in a specified area or pixel in an image, etc.

[0011] Here, the image processing device 101 may have a function as an image viewer that displays the generated substance information by superimposing it on the image data. The LAN 104 is a network made up of communication devices that comply with standards such as IEEE (Institute of Electrical and Electronics Engineers) 802.3ab.

[0012] The configuration of the image processing system 100 is not limited to this, and the image processing device 101 may be connected to a storage device such as a database that stores CT image data captured by the first X-ray CT device 102, the second X-ray CT device 103, etc. Also, the image processing system 100 may be configured to include the image processing device 100 and a storage device such as a database that stores CT image data.

[0013] (Hardware configuration) Fig. 2 is a block diagram showing an example of a hardware configuration of an image processing device 101 according to this embodiment. The image processing device 101 shown in Fig. 2 is composed of a storage medium 201, a ROM (Read Only Memory) 202, a CPU (Central Processing Unit) 203, and a RAM (Random Access Memory) 204. It also includes a LAN interface 205, an input interface 208, a display interface 206, and an internal bus 211. A keyboard 209 and a mouse 210 are connected to the image processing device 101 via the input interface 208. A display 207 is connected to the image processing device 101 via the display interface 206.

[0014] The storage medium 201 is a storage medium such as an SSD (Solid State Drive) that stores an OS (Operating System), a processing program for performing various processes according to the present embodiment, and various information according to the present embodiment. The ROM 202 stores a program such as a BIOS (Basic Input Output System) for initializing hardware, reading out the OS stored in the storage medium 201, and starting it. The CPU 203 performs arithmetic processing when executing the BIOS, OS, and processing program. The RAM 204 temporarily stores the processing program and various data when the CPU 203 executes the BIOS, OS, and processing program. The LAN interface 205 is an interface that supports standards such as IEEE802.3ab and performs communication via the LAN 104. The display 207 is a display device such as an LCD (Liquid Crystal Display) that displays a user interface screen. The display interface 206 converts screen information to be displayed on the display 207 into a signal for display control, and outputs the signal to the display 207. The input interface 208 receives signals based on key presses, button clicks, coordinate movement, and the like from a keyboard 209 and a mouse 210. An internal bus 211 transmits signals when communication is performed between each block.

[0015] (Functional configuration) Fig. 3 is a block diagram showing an example of a functional configuration of an image processing device 101 according to this embodiment. The image processing device 101 shown in Fig. 3 includes a first image reconstruction unit 310, a first substance information generation unit 312, and a first machine learning unit 314. The image processing device 101 further includes a second image reconstruction unit 316, a CT image data acquisition unit 318, a first inference unit 320 that acquires second substance information, and a display control unit 301. As will be described later with reference to Figs. 4 and 6, the functional configuration may be realized as an image processing system including a plurality of image processing devices, or the learning function and the inference function may be configured separately in different image processing devices.

[0016] The first image reconstruction unit 310 acquires a plurality of first detection data 330 obtained by imaging each of a plurality of subjects by the first X-ray CT device 102 from the first X-ray CT device 102. Note that the first detection data 330 may not be acquired directly from the first X-ray CT device 102 via a LAN, but may be acquired from a storage unit or the like that stores the first detection data 330.

[0017] Next, the first image reconstruction unit 310 reconstructs the first CT image data 311 based on each of the multiple first detection data 330. Here, the first detection data 330 is a sinogram divided into X-ray energy ranges. The sinogram is data arranged with the arrangement of detectors as the X-axis and the projection position (rotation angle) as the Y-axis. Moreover, the image reconstruction in the first image reconstruction unit 310 integrates the first detection data 330 in the X-ray energy direction (energy integration), and obtains the first CT image data from the energy-integrated sinogram by a known back projection method or iterative approximation image reconstruction method. Moreover, the first image reconstruction unit 310 generates multiple first CT image data 311 for each of the multiple first detection data 330.

[0018] The first substance information generating unit 312 acquires the same multiple first detection data 330 as above from the first X-ray CT device 102, and generates first substance information 313. Here, the first substance information 313 is information indicating an area in the image corresponding to a predetermined substance, and is a mask image having voxel values ​​(pixel values) according to the type of substance. Furthermore, multiple pieces of first substance information 313 are generated for each of the multiple first detection data 330.

[0019] In the first material information generating unit 312, the first material information 313 is generated in the following procedure. First, the first material information generating unit 312 reconstructs the first detection data 330 for each X-ray energy range to obtain monochromatic X-ray image data for each X-ray energy range. Next, the first material information generating unit 312 evaluates the degree of agreement between the distribution in the X-ray energy direction of each voxel value obtained from the monochromatic X-ray image data for each X-ray energy range and the distribution in the X-ray energy direction of HU values ​​assumed from the attenuation coefficient of the material for each X-ray energy range.

[0020] For example, when there are two types of X-ray energy ranges, 80 kV and 140 kV, the ratio of HU values ​​estimated from the attenuation coefficient is approximately 1.7 for iodine and approximately 1.3 for calcium. In this case, the first material information generating unit 312 evaluates that the ratio of voxel values ​​in the monochromatic X-ray images of 80 kV and 140 kV is close to 1.7 as a high degree of agreement with iodine, and close to 1.3 as a high degree of agreement with calcium. Next, the first material information generating unit 312 identifies a material whose degree of agreement meets a predetermined standard for each voxel in the image. Finally, the first material information generating unit 312 sets a numerical value (label value) previously assigned to the identified material at a position on the mask image corresponding to each voxel in the image. Note that the number of X-ray energy ranges, the ratio of HU values ​​between X-ray energy ranges, and the material to be identified are merely examples and are not limited thereto.

[0021] The first machine learning unit 314 performs machine learning using a teacher data set including the first CT image data 311 and the first substance information 313 in the teacher data, and generates the first trained model 315. The first machine learning unit 314 may perform machine learning using a teacher data set including a plurality of first CT image data 311 and a plurality of first substance information 313 in the teacher data, and generate the first trained model 315. Specifically, the first trained model 315 is an image generation model that inputs the first CT image data 311 and outputs the first substance information 313, and is a model based on a CNN (Convolutional neural network) such as U-Net. In addition, in the machine learning, various parameters in the CNN are updated so that the error between the output value obtained by inputting the first CT image data 311 to the CNN and the first substance information 313 (generated from the same first detection data 330) that forms a pair with the input first CT image data 311 is reduced, and this is repeated for multiple images. Here, the first trained model 315 may be an image generation model based on a method other than CNN, such as a deep neural network (DNN), a vision transformer, or a support vector machine (SVM). In an image generation model based on an algorithm mainly for classification tasks, such as an SVM, a mask image is generated by estimating the materiality of each region obtained by dividing an image, such as by a sliding window method, and combining the estimation results. The generation of a mask image by an image generation model is also called segmentation.

[0022] The second image reconstruction unit 316 acquires second detection data 340 from the second X-ray CT device 103 and reconstructs second CT image data 317. The second X-ray device 103 is a CT device equipped with an energy integral type radiation detector using a detection method different from that of the first X-ray CT device 102. Here, the second detection data 340 is a sinogram, which is data arranged with the arrangement of detectors as the X axis and the projection position (rotation angle) as the Y axis. In addition, in the image reconstruction in the second image reconstruction unit 316, second CT image data 317 (multiple tomographic image data) is obtained from the second detection data 340 by a known back projection method or iterative approximation image reconstruction method. Note that the second detection data 340 may not be acquired directly from the second X-ray CT device 103 via a LAN, but may be acquired from a storage unit or the like that stores the second detection data 340.

[0023] The CT image data acquisition unit 318 acquires second CT image data 317 based on the second detection data 340 from a storage unit or the like. Here, the second CT image data 317 is image data of an inference target.

[0024] A first inference unit 320 that acquires the second substance information causes a first trained model 315 that has been subjected to a learning process by a first machine learning unit 314 to infer second substance information 321 from second CT image data 317. Specifically, the first inference unit 320 inputs the second CT image data 317 into the first trained model 315 and acquires the output obtained as the second substance information 321.

[0025] The display control unit 301 displays the second CT image data 317 and the second substance information 321 on the display 207 together with a display screen 601, which will be described later with reference to FIG.

[0026] (First learning process flow) The learning process of the first inference model according to this embodiment will be described with reference to Fig. 4 and Fig. 5. Fig. 4 shows an example of the functional configuration of an image processing device 101 related to the learning process. Fig. 5 is a flow diagram showing an example of a first learning process 400 according to this embodiment.

[0027] The first learning process 400 shown in Fig. 5 is started automatically or manually prior to the image display process 500 described with reference to Figs. 6 and 7. The first learning process 400 may be repeatedly executed multiple times.

[0028] In step S401, the first image reconstruction unit 310 and / or the first material information generating unit 312 acquires first detection data 330 obtained by imaging the subject with the first X-ray CT device 102 from the first X-ray CT device 102 via the LAN 104. A known internet protocol such as HTTP (Hypertext Transfer Protocol) is used to acquire the first detection data 330 via the LAN 104. Note that here, the first detection data 330 may be acquired from a storage device such as a database that stores data captured by the first X-ray CT device 102.

[0029] In step S402, the first image reconstruction unit 310 reconstructs the first CT image data 311 from the first detection data 330 acquired in step S401.

[0030] In step S403, the first substance information generation unit 312 generates the first substance information 313 from the first detection data 330 acquired in step S401.

[0031] In step S404, the first machine learning unit 314 determines whether acquisition of the teacher data has been completed. Specifically, it determines whether the number of the first CT image data 311 reconstructed in step S402 and the first substance information 313 generated in step S403 has reached a preset number. If acquisition of the teacher data has not been completed (No in step S404), the process from step S401 is repeated, and if acquisition of the teacher data has been completed (Yes in step S404), the process proceeds to step S405.

[0032] In step S405, the first machine learning unit 314 executes machine learning processing using the teacher dataset to generate a first trained model 315, and then ends the processing. Here, the teacher dataset includes the plurality of first CT image data 311 reconstructed in step S402 and the plurality of first substance information 313 generated in step S403.

[0033] (Image display processing flow) The image display processing step according to this embodiment will be described with reference to Fig. 6 and Fig. 7. Fig. 6 shows an example of the functional configuration of an image processing device 101 related to the image display processing step.

[0034] 7 is a flow diagram showing an example of the image display processing 500 according to this embodiment. The image display processing 500 is started based on a user operation when the user views a CT image.

[0035] In step S501, the second image reconstruction unit 316 determines whether or not a selection operation of the second detection data 340 has been performed based on an operation input via the keyboard 209 or mouse 210 on the display screen 601 described later with reference to Fig. 8 (hereinafter, the selection target is also simply referred to as "image"). If a selection operation has been performed (Yes in step S501), the process proceeds to step S511. If a selection operation has not been performed (No in step S501), the process proceeds to step S502.

[0036] In step S511, the second image reconstruction unit 316 acquires the second detection data 340 selected in step S501 from the second X-ray CT device 103 via the LAN 104. A known internet protocol such as HTTP (Hypertext Transfer Protocol) is used to acquire the second detection data 340 via the LAN 104. Note that the second detection data 340 may not be acquired directly, but may be acquired from a storage device such as a database that stores data captured by the second X-ray CT device 103.

[0037] In step S512, the second image reconstruction unit 316 reconstructs second CT image data 317 from the second detection data 340 acquired in step S511.

[0038] In step S513, the display control unit 301 displays the second CT image data 317 reconstructed in step S512 on the display 207 together with a display screen 601, which will be described later with reference to FIG.

[0039] In step S502, the first inference unit 320 determines whether the second CT image data 317 is being displayed in step S513 and whether a substance information display operation has been performed. Note that steps up to step S513 may be executed by another device or entity, and the process may start from a step in which the CT image data acquisition unit 318 acquires the second CT image data 317.

[0040] Here, the presence or absence of a substance information display operation is determined based on an operational input via the keyboard 209 or mouse 210 on a display screen 601, which will be described later with reference to Fig. 8. If an image is being displayed (i.e., the second CT image data 317 is selected) and a substance information display operation has been performed (Yes in step S502), the process proceeds to step S521. If an image is not being displayed (i.e., the second CT image data 317 is not selected) or a substance information display operation has not been performed (No in step S502), the process proceeds to step S503.

[0041] In step S521, the model acquisition unit 319 acquires the first trained model 315 generated in the first machine learning process 400, and transmits the model to the first inference unit 320 for acquiring second material information. The first inference unit 320 infers second material information 321 from the second CT image data 317 reconstructed in step S512.

[0042] In step S522, the display control unit 301 superimposes the second substance information 321 generated in step S521 on the second CT image data 317 reconstructed in step S512, and displays it on the display 207 together with a display screen 601 described later with reference to FIG. 8.

[0043] In step S503, the display control unit 301 does not perform superimposition processing of the second substance information 321. That is, when the second CT image data 317 is displayed, only the second CT image data 317 is displayed on the display 207 together with a display screen 601 described later with reference to FIG.

[0044] In step S504, a control unit (not shown) determines whether or not a termination operation has been performed based on an operation input via the keyboard 209 or the mouse 210 on a display screen 601, which will be described later with reference to Fig. 8. If a termination operation has been performed (Yes in step S504), the process ends, and if a termination operation has not been performed (No in step S504), the process is repeated from step S501.

[0045] (display screen) Fig. 8 is a diagram showing an example of the display screen 601 of the image processing device 101 according to this embodiment. In Fig. 8, Fig. 8(a) is an example of the display screen 601 on which no substance information is displayed, and Fig. 8(b) is an example of the display screen 601 on which substance information is displayed.

[0046] A display screen 601 in FIG. 6 is displayed on the display 207 by the display control unit 301, and comprises an image display area 602, an image selection button 603, substance information display check boxes 604-1 and 604-2, and an end button 605.

[0047] The image selection button 603 is a button for selecting an image, and when it is clicked with the mouse 210, a screen (not shown) for selecting an image to be displayed by the display control unit 301 is displayed on the display 207, and the user can select an image to be displayed via the screen. When the image to be displayed is selected by the user, the second image reconstruction unit 316 reconstructs second CT image data 317 from the second detection data 340 corresponding to the image to be displayed, and the display control unit 301 displays the second CT image data 317 in the image display area 602. Note that the above steps may be omitted as appropriate by the CT image data acquisition unit 318 acquiring the second CT image data reconstructed in advance from the second detection data 340 as described above.

[0048] The image display area 602 is an area for displaying an image, and the display control unit 301 displays, for example, a cross section of the second CT image data 317. The areas 606-1 to 606-6 are examples of areas having a voxel value (HU value) in a predetermined range in the cross section to be displayed.

[0049] The substance information display checkbox 604-1 is a checkbox for specifying whether or not to display substance information, and 604-1 is an example where it is not checked. If the substance information display checkbox 604-1 is not checked, the display control unit 301 does not display the second substance information 321 in the image display area 602. Therefore, areas 606-1 to 606-6 having voxel values ​​(HU values) within a predetermined range are displayed without the substances being discriminated.

[0050] The substance information display checkbox 604-2 is an example in which the substance information display is checked, and the display control unit 301 displays the second substance information 319 in the image display area 602 by superimposing it on the second CT image data 317.

[0051] Regions 607-1 to 607-4 and regions 608-1 to 608-3 are an example of a display in which substances are discriminated. Regions 607-1 to 607-4 indicate regions identified as iodine, and regions 608-1 to 608-3 indicate regions identified as calcium.

[0052] An end button 605 is a button for selecting the end of processing, and when it is clicked with the mouse 210, a control unit (not shown) ends the image display processing 500 and terminates the display on the display screen 601.

[0053] (Modification of the first embodiment) The image processing apparatus 101 of the first embodiment may be an image processing workstation, a console of an X-ray CT apparatus, or an image processing server.

[0054] Furthermore, the image processing device 101 of the first embodiment may be configured not to select the second detection data 340 based on a user operation, or to display the second substance information 319 by the display control unit 301. In this case, the first image reconstruction unit 310 and / or the first substance information generation unit 312 may acquire (receive) the second detection data 340 output by an external device (e.g., a server or viewer, not shown) in step S501, or may receive designation of the second detection data 340 from the external device in step S501, and acquire the designated second detection data 340 from the external device in step S511. When the second detection data 340 is acquired (received) in step S501, the process of step S511 is not necessary. Furthermore, following the process of step S513, the process of step S521 is performed, and in step S521, the first inference unit 320 outputs the second substance information 321 to an external device such as a server or viewer (not shown). In this case, the processes of steps S502, S503, and S522 are unnecessary.

[0055] Furthermore, the first detection data 330 in the first embodiment may be stored in a device (e.g., a separately provided server (not shown)) other than the first X-ray CT device 102 and acquired by the first image reconstruction unit 310 and / or the first substance information generation unit 312 of the image processing device 101. In this case, in step S401, the first image reconstruction unit 310 and / or the first substance information generation unit 312 acquire the first detection data 330 from the device.

[0056] Furthermore, the first image reconstruction unit 310 of the first embodiment may be provided in a device (for example, the first X-ray CT device 102) other than the image processing device 101. In this case, the first machine learning unit 314 of the image processing device 101 may be configured to acquire the first CT image data 311 from the device. Alternatively, the first machine learning unit 314 of the image processing device 101 may be configured to acquire the first CT image data 311 stored in a separately provided server (not shown). In this case, in step S402 of the first embodiment, instead of the process in which the first image reconstruction unit 310 reconstructs the first CT image data 311, the first machine learning unit 314 acquires the first CT image data 311 stored externally.

[0057] Similarly, the first substance information generation unit 312 of the first embodiment may be provided in a device (for example, the first X-ray CT device 102) other than the image processing device 101. In this case, the first machine learning unit 314 of the image processing device 101 may be configured to acquire the first substance information 313 from the device. Alternatively, the first machine learning unit 314 of the image processing device 101 may be configured to acquire the first substance information 313 stored in a separately provided server (not shown). In this case, in step S403 of the first embodiment, instead of the process in which the first substance information generation unit 312 generates the first substance information 313, the first machine learning unit 314 may acquire the first substance information 313 stored externally.

[0058] In addition, in the case where both the first CT image data 311 and the first substance information 313 are acquired from outside the image processing device 101, the first detection data 330 is unnecessary, so the process of step S401 can be omitted.

[0059] Furthermore, the first machine learning unit 314 of the first embodiment may be provided in a device (not shown) other than the image processing device 101, and the first trained model 315 may be stored in a separately provided server (not shown) and acquired by a model acquisition unit 319 of the image processing device 101. In this case, the processes of steps S401 to S404 of the first embodiment may be omitted. Furthermore, in step S405, instead of the process in which the first machine learning unit 314 generates the first trained model 315, the second substance information generation unit 318 acquires the first trained model 315 stored in the server (not shown).

[0060] Furthermore, the second detection data 340 of the first embodiment may be stored in a separately provided server (not shown) and acquired by the second image reconstruction unit 316 of the image processing device 101. In this case, in step S511, the second detection data 340 stored in the server (not shown) is acquired.

[0061] Furthermore, the second image reconstruction unit 316 of the first embodiment may be provided in a device (for example, the second X-ray CT device 103) other than the image processing device 101. In this case, the configuration may be such that the acquisition unit 318 of the CT image data of the image processing device 101 acquires the second CT image data 317 from the device. Alternatively, the acquisition unit 318 of the CT image data may acquire the second CT image data 317 stored in a separately provided server (not shown). In this case, in step S512 of the first embodiment, instead of the process in which the second image reconstruction unit 316 reconstructs the second CT image data 317, the acquisition unit 318 of the CT image data acquires the second CT image data 317 stored externally.

[0062] Moreover, the first substance information 313 and the second substance information 321 in the first embodiment are material discrimination information. The material discrimination information may be label information for the entire image indicating the presence or absence of a predetermined material. Alternatively, it may be a bounding box indicating the presence of a material and label information, or it may be information other than a mask image, such as annotation information such as an arrow and label information.

[0063] Moreover, the first X-ray CT device 102 may be a dual-energy type X-ray CT device.

[0064] Alternatively, the X-ray CT device may be equipped with both a photon-counting detector and an integral detector. In this case, the first image reconstruction unit 310 reconstructs first CT image data 311 from detection data obtained from the integral detector, and the first material information generation unit 312 generates first material information 313 from detection data obtained from the photon-counting detector. That is, the first X-ray CT device 102 is an X-ray CT device capable of acquiring first CT image data 311 based on first detection data 330 and first material information 313 based on the first detection data. On the other hand, the second X-ray CT device 103 is an X-ray CT device incapable of acquiring material information.

[0065] In addition, the first machine learning unit 314 of the first embodiment may perform machine learning using data that includes pairs of image data and substance information acquired by any means, to generate a first trained model 315.

[0066] In addition, the first machine learning unit 314 of the first embodiment may generate a first trained model 315 by performing additional learning on a model generated by performing machine learning using the above-mentioned teacher dataset.

[0067] In addition, the first machine learning unit 314 of the first embodiment may generate a first trained model 315 by performing additional learning using the above-mentioned teacher dataset on a model generated by performing machine learning using any teacher data.

[0068] In addition, the first machine learning unit 314 of the first embodiment may perform additional learning on a model acquired by any means using the first CT image data 311 and the first substance information 313 as training data to generate a first trained model 315.

[0069] Furthermore, the first image reconstruction unit 310 and the first substance information generation unit 312 of the first embodiment may set as teacher data a pair of the first CT image data 311 and the first substance information 313, which have the same first detection data 330. Furthermore, the first CT image data 311 and the first substance information 313 constituting the teacher data may be output to an external device such as a server (not shown).

[0070] In addition, the first substance information 313 and the second substance information 321 in the first embodiment may be a mask image of multiple or multiple channels provided for each substance to be discriminated. Also, the first trained model 315 may be a plurality of trained models for each substance to be discriminated.

[0071] Furthermore, each voxel value (pixel value) of the first material information 313 in the first embodiment may be a continuous value that indicates the degree of coincidence with the distribution in the X-ray energy direction for each material to be discriminated, as a likelihood of the material, instead of binary information indicating whether the degree of coincidence is high or low. The material information obtained by discriminating the material using the likelihood may be a voxel value (pixel value).

[0072] Furthermore, the first machine learning unit 314 may perform machine learning using a training data set including intermediate data such as monochromatic X-ray images used in generating the first material information 313 and the first CT image data 311 as training data, to generate the first trained model 315. In this case, the second material information generation unit 318 generates the second material information 321 using the intermediate data such as monochromatic X-ray images generated by the first trained model 315.

[0073] As described above, according to this embodiment and the modified example of this embodiment, a machine-learned model can be constructed using data generated from PCCT or DECT as training data. By applying this trained model, material information can be obtained with high accuracy even from CT image data acquired from a conventional X-ray CT device equipped with an energy-integrated X-ray detector.

[0074] <Second embodiment> In addition to the processing of the first embodiment, the image processing device of this embodiment infers the second CT image data from the first detection data using a second trained model trained using a teacher data set including the first detection data and the second CT image data in the teacher data. Note that the image processing system 100, the hardware configuration of the image processing device 101, the first machine learning processing 400, the image display processing 500, and the display screen 601 of this embodiment are the same as those of the first embodiment, and therefore the description thereof will be omitted.

[0075] (Functional configuration) 9 is a block diagram showing an example of the functional configuration of an image processing device 101 according to this embodiment. The same numbers are used for items described in other embodiments, and descriptions thereof will be omitted as appropriate. The image processing device 101 shown in FIG. 9 further includes a second machine learning unit 710 in addition to the configuration of the image processing device 101 of the first embodiment described in FIG.

[0076] The second machine learning unit 710 executes machine learning using a teacher data set to generate a second trained model 711. The teacher data set includes a plurality of first detection data 330 obtained by imaging each of a plurality of subjects, and a plurality of second CT image data 317 corresponding to each of the plurality of first detection data 330. Here, the first detection data 330 and the second CT image data 317 are data obtained by imaging the same subject by both the first X-ray CT device 102 and the second X-ray CT device 103. In addition, the second machine learning unit 710 executes a registration process so that the second CT image data 317 used as teacher data has the same subject position on the image as the first CT image data 311 reconstructed by the first image reconstruction unit 310 from the first detection data. Note that the first image reconstruction unit 310 reconstructs the first CT image data 311 used in the registration process using a method such as the energy integral and back projection method or the iterative approximation image reconstruction method described in the first embodiment. The second trained model 711 generated by the second machine learning unit 710 is an image generation model that inputs the first detection data 330 and outputs the second CT image data 317, and is a model based on a CNN (Convolutional neural network) such as U-Net.

[0077] In addition, in the learning process by the second machine learning unit 710, various parameters in the CNN are updated so as to reduce the error between the output value obtained by inputting the first detection data 330 to the CNN and the second CT image data 317 (generated from the second detection data 340 capturing the same subject as the first detection data 330) that is paired with the input first detection data 330 and has been subjected to the above-mentioned alignment, and this is repeated for multiple images. Here, the second trained model 711 may be an image generation model based on something other than a CNN, such as a DNN (Deep Neural Network) or a vision transformer.

[0078] In addition, the first image reconstruction unit 310 may use a second trained model 711 trained by the second machine learning unit 710 to reconstruct the first CT image data 311 from the first detection data 330.

[0079] (Second learning process flow) Fig. 10 is a flow diagram showing an example of a second machine learning process 800 according to the second embodiment. The second machine processing 800 is started automatically or manually prior to the first machine learning process 400 described with reference to Fig. 4. The second machine learning process 800 may be repeatedly executed multiple times.

[0080] In step S801, the control units (not shown) of the first X-ray CT device 102 and the second X-ray CT device 103 capture the same subject based on an operation from a user. The images may be captured at different time periods, and if one of the data exists, it may be substituted by acquiring the other data that does not exist. If both data exist, this step may be skipped as appropriate. Alternatively, it may be substituted by acquiring the captured data from a storage unit instead of capturing the image.

[0081] In step S802, the second machine learning unit 710 determines whether acquisition of the teacher data has been completed. Specifically, it determines whether the number of teacher data pairs of the first detection data 330 acquired in step S401 and the second CT image data 317 reconstructed in step S512 has reached a preset number. If acquisition of the teacher data has not been completed (No in step S802), the process from step S801 is repeated, and if acquisition of the learning data has been completed (Yes in step S802), the process proceeds to step S803.

[0082] In step S803, the second machine learning unit 710 executes a process of aligning the second CT image data 317 with the first CT image data 311 reconstructed from the first detection data 330. Machine learning is executed using a teacher data set including the plurality of first detection data 330 and the aligned plurality of second CT image data 317 in the teacher data, and when a second trained model 711 is generated, the process ends.

[0083] (Modification of the second embodiment) The second machine learning unit 710 of the second embodiment may be provided in a device (not shown) other than the image processing device 101, and the second trained model 711 may be stored in a separately provided server (not shown) and acquired by the first image reconstruction unit 310 of the image processing device 101. In this case, the processes of steps S801 to S802 of the second embodiment may be omitted. Also, in step S803, instead of the process in which the second machine learning unit 710 generates the second trained model 711, the first image reconstruction unit 310 acquires the second trained model 711 stored in the server (not shown).

[0084] The second trained model 711 of the second embodiment may be an image generation model that receives the first CT image data reconstructed from the first detection data 330 by energy integration and back projection or iterative image reconstruction, and outputs the second CT image data 317. In this case, the second machine learning unit 710 performs machine learning using a teacher data set including the reconstructed first CT image data as input and the second CT image data 317 as output, and generates the second trained model 711. The first image reconstruction unit 310 acquires the first CT image data reconstructed from the first detection data 330 by energy integration and back projection or iterative image reconstruction. The first image reconstruction unit 310 may input the first CT image data to the second trained model 711 and output the second CT image data. The first trained model may be generated using a teacher data set including the second CT image data and the first material information paired together.

[0085] Furthermore, the first detection data 330 and the second detection data 340 may be generated so that the subject is aligned in the image when the image is captured. Specifically, a detector for acquiring the first detection data 330 and a detector for acquiring the second detection data 340 are provided on the same movable part of the X-ray CT device, and the image is captured. In this case, in step S803, the image alignment process between the second CT image data 317 and the first CT image data 311 by the second machine learning unit 710 is not required.

[0086] In addition, the second machine learning unit 710 of the second embodiment may generate the second trained model 711 using training data that includes data that includes pairs of detection data and image data acquired by any other means.

[0087] Furthermore, the second machine learning unit 710 of the second embodiment may acquire a model generated by performing machine learning using a teacher data set including the first detection data 330 and the second CT image data 330 in the teacher data. Furthermore, the model may be additionally trained using a teacher data set including a pair of detection data and image data acquired by any means in the teacher data to generate a second trained model 711.

[0088] In addition, the second machine learning unit 710 of the second embodiment may perform additional learning using the above-mentioned teacher dataset on a model generated by machine learning using a teacher dataset acquired by any means, thereby generating a second trained model 711.

[0089] In addition, the second machine learning unit 710 of the second embodiment may perform additional learning on a model acquired by any means using the first detection data 330 and the second CT image data 330 as training data to generate a second trained model 711.

[0090] As described above, according to this embodiment and the modified example of this embodiment, a machine-learned model can be constructed using data generated from PCCT or DECT as training data. By applying this trained model, material information can be obtained with high accuracy even from CT image data acquired from a conventional X-ray CT device equipped with an energy-integrated X-ray detector.

[0091] Furthermore, by using the second trained model 711, the acquired first CT image data 311 becomes an image similar to the second CT image data 317 (i.e., an image similar to an image taken by an integral type X-ray CT device), so that the first trained model 315 is optimized for the second CT image data 317, enabling material decomposition with improved performance.

[0092] <Third embodiment> The image processing device 101 of this embodiment, unlike the first embodiment, acquires a third trained model using a teacher data set including a first sinogram based on the first detection data and first substance information in the teacher data. In addition, the third trained model is used to infer third substance information from a second sinogram based on the second detection data. Note that the image processing system 100, the hardware configuration of the image processing device 101, and the display screen 601 of this embodiment are the same as those of the first embodiment, and therefore their explanations are omitted. In addition, the first machine learning processing 400 and the image display processing 500 are similar in that image data is replaced with a sinogram, and image reconstruction is replaced with creation of a sinogram, and therefore their explanations are omitted.

[0093] (Functional configuration) Fig. 11 is a block diagram showing an example of a functional configuration of an image processing device 101 according to this embodiment. The same numbers are used for items described in other embodiments, and their description will be omitted. In the image processing device 101 of Fig. 11, the first image reconstruction unit 310 of the image processing device 101 of the first embodiment described in Fig. 3 is replaced with a first sinogram creation unit 910, the first machine learning unit 314 is replaced with a third machine learning unit 914, and the first inference unit 320 is replaced with a second inference unit 918. In addition, a second sinogram creation unit 916 is added.

[0094] The first sinogram creation unit 910 acquires the first detection data 330 from the first X-ray CT device 102 and generates a first sinogram 911. Here, the first detection data 330 is a sinogram divided into X-ray energy ranges, and the first sinogram 911 is an energy-integrated sinogram. Therefore, the first sinogram is generated by integrating the first detection data 330 in the X-ray energy direction (energy integration).

[0095] The third machine learning unit 914 performs machine learning using the multiple first sinograms 911 and the multiple first substance information 313 as teacher data to generate a third trained model 915. Specifically, the third trained model 915 is an image generation model that inputs the first sinogram 911 and outputs the first substance information 313, and is, for example, a model based on a CNN (Convolutional neural network) such as U-Net. In addition, in the machine learning, various parameters in the CNN are updated so that the error between the output value obtained by inputting the first sinogram 911 and the first substance information 313 (generated from the same first detection data 330) that forms a pair with the input first sinogram 911 is reduced, and this is repeated for multiple images. Here, the third trained model 915 may be an image generation model based on a model other than a CNN, such as a DNN (Deep Neural Network), a vision transformer, or an SVM (Support Vector Machine). In image generation models based on algorithms that are primarily used for classification tasks, such as SVM, a mask image is generated by estimating the material-likelihood of each region into which an image is divided, using methods such as the sliding window method, and combining the estimation results. The generation of a mask image using an image generation model is also called segmentation.

[0096] The second sinogram creation unit 916 creates a second sinogram 917 in which the arrangement of the detectors is the X-axis and the projection position (rotation angle) is the Y-axis from the second detection data 340. If the second detection data 340 is a sinogram acquired from an energy-integration type detector, the second sinogram 917 will be the same as the second detection data 340.

[0097] A second inference unit 918 infers third material information 919 for the second sinogram 917 using the third trained model 915. Specifically, the second sinogram 917 is input to the third trained model 915, and the output obtained is regarded as the third material information 919.

[0098] The display control unit 301 displays the second CT image data 317 and the third substance information 919 on the display 207 together with a display screen 601, which will be described later with reference to FIG.

[0099] (Modification of the third embodiment) The first sinogram creation unit 910, the first substance information generation unit 312, the third machine learning unit 914, and the second sinogram creation unit 916 of the image processing device 101 of embodiment 3 may be provided in a device other than the image processing device 101, as in the modified example of the first embodiment.

[0100] The first sinogram creation unit 910 of the image processing device 101 of the third embodiment may perform machine learning using a teacher data set including a plurality of first detection data 330 and a plurality of second sinograms 917 in the teacher data. The first sinogram 911 may be created using the generated trained model. Specifically, the trained model is an image generation model that inputs the first detection data 330 and outputs the second sinogram 917, and is, for example, a model based on a CNN (Convolutional neural network) such as U-Net. In machine learning, various parameters in the CNN are updated so that the error between the output value obtained by inputting the first detection data 330 and the second sinogram 917 (generated from the second detection data 340 obtained by capturing an image of the same subject as the first detection data 330) that forms a pair with the input first detection data 330 is reduced, and this is repeated for multiple images. Here, the trained model may be an image generation model based on a model other than a CNN, such as a DNN (Deep Neural Network) or a vision transformer.

[0101] As described above, according to this embodiment and the modified example of this embodiment, a machine-learned model can be constructed using data generated from PCCT or DECT as training data. By applying this trained model, material information can be obtained with high accuracy even from CT image data acquired from a conventional X-ray CT device equipped with an energy-integrated X-ray detector.

[0102] Furthermore, by using the sinogram before reconstruction, a trained model is created that is not affected by the reconstruction function, etc., and material information can be obtained with reduced effects due to differences in the reconstruction function.

[0103] (Other Examples) The present invention can also be realized by executing the following process: That is, software (programs) that realize the functions of the above-described embodiments are supplied to a system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device reads and executes the programs.

[0104] Disclosure according to an embodiment of the present invention includes the following configurations and methods.

[0105] (Configuration 1) a model acquisition unit that acquires a first trained model trained using a training data set including first CT image data based on first detection data captured by a first X-ray CT device and first material information based on the first detection data; a CT image data acquisition unit that acquires second CT image data based on second detection data captured by a second X-ray CT device including an energy integral type radiation detector using a detection method different from that of the first X-ray CT device; 13. An image processing apparatus comprising: an inference means for inferring second material information from the second CT image data using the first trained model.

[0106] (Configuration 2) 2. The image processing apparatus according to configuration 1, wherein the first X-ray CT apparatus is either a dual energy type CT apparatus or a photon counting type CT apparatus.

[0107] (Configuration 3) 3. The image processing device according to configuration 1 or 2, further comprising a first substance information acquisition unit that acquires the first substance information from the first detection data.

[0108] (Configuration 4) The image processing device according to any one of configurations 1 to 3, wherein the first X-ray CT device is an X-ray CT device capable of acquiring first CT image data based on first detection data and material information based on the first detection data, and the second X-ray CT device is an X-ray CT device incapable of acquiring material information based on second detection data.

[0109] (Configuration 5) 5. The image processing device according to any one of configurations 1 to 4, wherein the first substance information is acquired based on pixel values ​​constituting a plurality of image data corresponding to each of a plurality of energy ranges reconstructed from the first detection data.

[0110] (Configuration 6) 6. The image processing device according to any one of configurations 1 to 5, wherein the first substance information and the second substance information are information indicating an area in an image that corresponds to a predetermined substance.

[0111] (Configuration 7) 7. The image processing device according to any one of configurations 1 to 6, wherein the first substance information and the second substance information are information indicating whether or not a predetermined substance is present in an image.

[0112] (Configuration 8) 7. The image processing device according to any one of configurations 1 to 6, wherein the first substance information and the second substance information are information indicating the likelihood of a predetermined substance for each predetermined region or pixel in an image.

[0113] (Configuration 9) The first CT image data constituting the training data set includes image data generated from the first detection data using a second trained model trained using a training data set including the first detection data and the second CT image data as training data. 9. The image processing device according to any one of configurations 1 to 8.

[0114] (Configuration 10) The first CT image data constituting the training data set includes image data generated from the first CT image data by using a second trained model trained using a training data set including the first CT image data and the second CT image data as training data. 10. The image processing device according to any one of configurations 1 to 9.

[0115] (Configuration 11) The second CT image data constituting the teaching data is image data that has been aligned so that the position of the subject on the image is the same as that of the first CT image data generated from the first detection data. 11. The image processing device according to configuration 9 or 10.

[0116] (Configuration 12) The second CT image data constituting the teaching data is image data generated from the second detection data captured so that the position of the captured subject on the image is the same as the position of the captured subject in the first CT image data. 12. The image processing device according to claim 11,

[0117] (Configuration 13) The first trained model is a trained model that inputs CT image data and outputs material information, The inference means infers the second material information by inputting the second CT image data to the first trained model. 13. The image processing device according to any one of configurations 1 to 12.

[0118] (Configuration 14) The second trained model is a trained model that inputs the first detection data and outputs the second CT image data, The first CT image data constituting the training data includes CT image data generated by inputting the first detection data into the second trained model. 10. The image processing device according to configuration 9.

[0119] (Configuration 15) using a third trained model trained using data including a first sinogram and first material information based on the first detection data captured by the first X-ray CT device as training data; An image processing device comprising an inference means for inferring third material information from a second sinogram based on second detection data captured by a second X-ray CT device having a detection method different from that of the first X-ray CT device.

[0120] (Configuration 16) a model acquisition step of acquiring a first trained model trained using a training data set including first CT image data based on first detection data captured by a first X-ray CT device and first material information based on the first detection data; a CT image data acquisition step of acquiring second CT image data based on second detection data captured by a second X-ray CT device including an energy integral type radiation detector using a detection method different from that of the first X-ray CT device; An image processing method comprising: an inference step of inferring second material information from the second CT image data using the first trained model.

[0121] (Configuration 17) using a third trained model trained using data including a first sinogram and first material information based on the first detection data captured by the first X-ray CT device as training data; An image processing method comprising an inference step of inferring third material information from a second sinogram based on second detection data captured by a second X-ray CT device having a detection method different from that of the first X-ray CT device.

[0122] (Configuration 18) 18. A program for executing the image processing according to configuration 16 or 17 on a computer.

[0123] (Configuration 19) a model acquisition unit that acquires a first trained model trained using a training data set including first CT image data based on first detection data captured by a first X-ray CT device and first material information based on the first detection data; a CT image data acquisition unit that acquires second CT image data based on second detection data captured by a second X-ray CT device including an energy integral type radiation detector using a detection method different from that of the first X-ray CT device; An image processing system comprising an inference means for inferring second material information from the second CT image data using the first trained model.

Claims

1. a model acquisition unit that acquires a first trained model trained using a training data set including first CT image data based on first detection data captured by a first X-ray CT device and first material information based on the first detection data; a CT image data acquisition unit that acquires second CT image data based on second detection data captured by a second X-ray CT device including an energy integral type radiation detector using a detection method different from that of the first X-ray CT device; An image processing device comprising an inference means for inferring second material information from the second CT image data using the first trained model.

2. 2. The image processing apparatus according to claim 1, wherein the first X-ray CT device is either a dual energy type CT device or a photon counting type CT device.

3. 3. The image processing apparatus according to claim 1, further comprising a first substance information acquisition unit that acquires the first substance information from the first detection data.

4. 3. The image processing device according to claim 1, wherein the first X-ray CT device is an X-ray CT device capable of acquiring first CT image data based on first detection data and material information based on the first detection data, and the second X-ray CT device is an X-ray CT device incapable of acquiring material information based on second detection data.

5. 3. The image processing device according to claim 1, wherein the first substance information is acquired based on pixel values ​​constituting a plurality of image data corresponding to each of a plurality of energy ranges reconstructed from the first detection data.

6. 3. The image processing apparatus according to claim 1, wherein the first substance information and the second substance information are information indicating an area in an image that corresponds to a predetermined substance.

7. 3. The image processing apparatus according to claim 1, wherein the first substance information and the second substance information are information indicating whether or not a predetermined substance is present in the image.

8. 3. The image processing apparatus according to claim 1, wherein the first substance information and the second substance information are information indicating a likelihood of a predetermined substance for each predetermined region or pixel in the image.

9. The first CT image data constituting the teacher data set includes image data generated from the first detection data using a second trained model trained using a teacher data set including the first detection data and the second CT image data as teacher data.

2. The image processing device according to claim 1,

10. The first CT image data constituting the teacher data set includes image data generated from the first CT image data using a second trained model trained using a teacher data set including the first CT image data and the second CT image data as teacher data.

2. The image processing device according to claim 1,

11. The second CT image data constituting the teaching data is image data that has been aligned so that the position of the subject on the image is the same as that of the first CT image data generated from the first detection data.

11. The image processing device according to claim 9,

12. The second CT image data constituting the teaching data is image data generated from the second detection data captured so that the position of the captured subject on the image is the same as the position of the captured subject on the image in the first CT image data. The image processing device according to claim 11 .

13. The first trained model is a trained model that inputs CT image data and outputs material information, The inference means infers the second material information by inputting the second CT image data to the first trained model.

3. The image processing device according to claim 1, wherein the first and second inputs are input to the image processing device.

14. The second trained model is a trained model that inputs the first detection data and outputs the second CT image data, The first CT image data constituting the teacher data includes CT image data generated by inputting the first detection data to the second trained model.

10. The image processing device according to claim 9,

15. using a third trained model trained using data including a first sinogram and first material information based on the first detection data captured by the first X-ray CT device as training data; An image processing device comprising an inference means for inferring third material information from a second sinogram based on second detection data captured by a second X-ray CT device having a detection method different from that of the first X-ray CT device.

16. a model acquisition step of acquiring a first trained model trained using a training data set including first CT image data based on first detection data captured by a first X-ray CT device and first material information based on the first detection data; a CT image data acquisition step of acquiring second CT image data based on second detection data captured by a second X-ray CT device including an energy integral type radiation detector using a detection method different from that of the first X-ray CT device; An image processing method comprising: an inference step of inferring second material information from the second CT image data using the first trained model.

17. using a third trained model trained using data including a first sinogram and first material information based on the first detection data captured by the first X-ray CT device as training data; An image processing method comprising an inference step of inferring third material information from a second sinogram based on second detection data captured by a second X-ray CT device having a detection method different from that of the first X-ray CT device.

18. 18. A program for executing the image processing according to claim 16 or 17 on a computer.

19. a model acquisition unit that acquires a first trained model trained using a training data set including first CT image data based on first detection data captured by a first X-ray CT device and first material information based on the first detection data; a CT image data acquisition unit that acquires second CT image data based on second detection data captured by a second X-ray CT device including an energy integral type radiation detector using a detection method different from that of the first X-ray CT device; An image processing system comprising an inference means for inferring second material information from the second CT image data using the first trained model.

Images