Method and system for material decomposition in dual- or multiple-energy x-ray based imaging

Abstract

To provide a method and a system for generating material decomposition images from plural-energy X-ray based imaging.SOLUTION: A method comprises: modelling spatial relationships and spectral relationships among a plurality of images by learning features from the plurality of images in a combination and one or more of the plurality of images individually with a deep learning neural network; generating one or more basis material images employing the spatial relationships and the spectral relationships; and generating one or more material specific or material decomposition images from the basis material images. The neural network has an encoder-decoder structure and includes a plurality of encoder branches; each of one or more of the plurality of encoder branches encodes two or more images of the plurality of images in combination; and each of one or more of the plurality of encoder branches encodes each individual image of the plurality of images.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] (Cross-reference to related applications) This application claims the benefit of the filing date of the specification of Australian Patent Application Publication No. 2022204142, filed on June 14, 2022, the content of which is incorporated herein by reference in its entirety.

[0002] The present invention relates to a deep learning method and system for material decomposition in X-ray imaging with multiple (i.e., dual or multi) energies, including cold cathode X-ray CT or cold cathode X-ray imaging, dual energy CT or dual energy X-ray imaging, multi-energy CT or multi-energy X-ray imaging, and photon counting CT or X-ray imaging.

Background Art

[0003] Since its introduction, CT has been widely used in the fields of medical diagnosis and treatment. Although CT technology has made numerous advancements, its basic principle remains the same. A rotating X-ray tube and a row of detectors arranged within a gantry are used to measure the attenuation of X-rays by various tissues within the body. Compared to other imaging modalities, CT has many advantages such as high-speed scanning, high spatial resolution, and wide availability. Millions of CT examinations are performed annually, and CT has become one of the most important and widespread imaging modalities used in patient care.

[0004] Despite its remarkable success, CT technology has several limitations. One of the most significant limitations is the low contrast resolution. It is not possible to reliably distinguish substances with inherently low contrast, such as pathological tissues and healthy tissues. The low contrast resolution is due to the slightly different attenuation of X-rays between different tissues. For example, the difference in attenuation between non-calcified plaques with a high lipid content and non-calcified plaques with a low lipid content is slight, making it difficult to reliably evaluate non-calcified plaques. Also, since cartilage has low contrast with surrounding soft tissues, it is difficult to distinguish soft tissue structures such as cartilage from sensitive CT scans.

[0005] In clinical imaging, contrast agents enhance the contrast of materials in CT scans. Because contrast agents absorb external X-rays, the radiation exposure to the X-ray detector is reduced. Contrast agents, such as iodine-based contrast agents, can potentially cause kidney damage or trigger allergic reactions.

[0006] In conventional CT, the attenuation value of each voxel was a combination of the attenuation of multiple materials. Dual-energy CT uses two separate X-ray photon energy spectra instead of the single-energy technique used in conventional CT. This allows for the examination of materials with different attenuation characteristics at different energies. However, because it is limited to two energy bins, tissue identification is not yet optimal. With two or more energies and a narrow energy range, multi-energy CT can identify multiple materials simultaneously with improved accuracy.

[0007] Photon counting CT is a new technology that has made remarkable progress in the last decade. In a photon counting detector, each photon of the incident X-ray strikes a detector element, generating an electrical pulse of a height proportional to the energy deposited by each individual photon. Essentially, photon counting CT allows for dual-energy or multi-energy imaging with a single source, tube, imaging system, detector, and filter. Furthermore, user-defined energy threshold selection allows for the selection of the appropriate energy threshold for specific energy diagnostic tasks. This task-oriented energy threshold selection helps resolve different tissue types with optimal imaging settings to achieve the best image quality or the lowest radiation dose.

[0008] The basic principle of material decomposition is the same in multi-energy CT or photon-counting CT. The complete energy dependency of the decay curve in each voxel of the scan is determined. As far as X-ray decay characteristics are concerned, it is assumed that any human tissue is approximately equivalent to a combination of two or more base materials. While any material can be used as a base material, water, calcium, iodine, and fat are typically used. Therefore, material decomposition is also called base material decomposition. The general workflow is as follows: Energy-selective (or energy-specific) images are generated using multi-energy bins with multi-energy CT. A set of base material images is generated from the energy-selective images. Each base material image represents the equivalent concentration of the base material in each voxel during the scan. By linearly transforming the base images, images of human tissues such as bone, muscle, and fat can be obtained. The concentration of each base material is calculated to determine the transformation formula for a portion of the human tissue.

[0009] Methods for material decomposition have been developed. The simplest method involves inversely modifying the matrix that relates decay values ​​to material concentrations. Other methods, such as optimization using regularization, have also been developed. However, assuming certain types and numbers of base materials, material decomposition becomes a nonlinear, poorly designed problem, and inaccurate decomposition is a problem with current methods.

[0010] In recent years, machine learning, particularly deep learning methods, has been seen as promising for solving poorly configured problems such as image reconstruction, image resolution improvement, and speech recognition. This invention provides a deep learning method and system that presents a mapping between energy-selected images and material-identified images. [Overview of the Initiative] [Problems that the invention aims to solve]

[0011] The object of the present invention is to provide a method for generating material resolution images by imaging with X-rays of multiple energies. [Means for solving the problem]

[0012] According to a first aspect of the present invention, a method is provided for generating material-resolved images from a plurality of images obtained by imaging with a plurality of energies of X-rays, wherein the plurality of images correspond to each energy in the imaging with a plurality of energies of X-rays. The method comprises the steps of: modeling spatial and spectral relationships between a plurality of images by learning the features of a combination of a plurality of images and learning the features of one or more of the plurality of images individually using a deep learning neural network; generating one or more base material images using the spatial and spectral relationships; and generating one or more material-specific images or material-resolved images from the base material images (e.g., through a linear transformation of the base material images). The neural network has an encoder-decoder structure (e.g., having an encoder network and a decoder network) and includes a plurality of encoder branches. Each of the one or more encoder branches encodes two or more images from the plurality of images in combination (e.g., together, concatenated, or in series). Each of the one or more encoder branches encodes the individual images of the respective plurality of images.

[0013] Spatial relationships and spectral relationships are the relationships between spatial information (i.e., objects, substances, and structures in an image) and spectral information (i.e., the decay of different substances resulting from different photon energies), respectively.

[0014] It should be noted that images obtained from X-ray imaging with multiple energies may be composite images, meaning they may not have been obtained simultaneously or in a single scan, but rather combined from multiple scans.

[0015] One or more encoder branches that encode two or more images from a set of multiple images may receive the two or more images in a combined and concatenated state. Alternatively, one or more encoder branches that encode two or more images from a set of multiple images may receive the two or more images before encoding them, by combining, concatenating, etc.

[0016] In one embodiment, each of two or more encoder branches encodes different individual images of a group of images. In some embodiments, a first encoder branch encodes a first combination of two or more images from a plurality of images. A second encoder branch encodes a second combination of two or more images from a plurality of images. The first combination is different from the second combination (however, the first and second combinations may have a common image).

[0017] X-ray imaging with multiple energies may include, for example, cold cathode X-ray radiography, dual-energy X-ray radiography, multi-energy X-ray radiography, photon-counting X-ray radiography, cold cathode X-ray CT, dual-energy CT, multi-energy CT, and photon-counting CT.

[0018] Advantageously, in some embodiments, an encoder branch that encodes each individual image encodes all images together that are encoded together by encoder branches that encode two or more images.

[0019] However, in some other embodiments, the encoder branch encoding each individual image receives fewer images together than would be combined and encoded by the encoder branch encoding two or more images (for example, by omitting one or more low-energy images). This may be done, for example, to reduce computation time.

[0020] In yet other embodiments, an encoder branch that encodes each individual image encodes more images together than are encoded together by an encoder branch that encodes two or more images.

[0021] An encoder branch that encodes each individual image may encode only images that are not encoded by any of the encoder branches that encode two or more images. However, more advantageously, an encoder branch that encodes each individual image encodes at least one image together more than an image that is also encoded by at least one of the encoder branches that encode two or more images.

[0022] In one embodiment of the present invention, a combination of all images (each called an "energy image" because each corresponds to a respective X-ray energy bin or energy threshold) is used as an input to a first encoder branch, and each of the individual energy images is used as an input to one of each of a plurality of further branches. However, in some embodiments, not all of the energy images are used as an input to the first encoder branch and / or as an input to each of the further branches. For example, if the target basis material image (i.e., the basis material image of interest) relates only to soft tissue, the high energy images may be omitted. On the other hand, since high energy images are useful for distinguishing hard substances such as bone, in embodiments where the target basis material image relates to hard tissue, low energy images may be omitted.

[0023] Also, in these embodiments or other embodiments, it may also be advantageous (such as reducing computational overhead) to omit one or more energy images to reduce the number of encoder branches and make the neural network smaller and simpler.

[0024] Thus, in an embodiment, each of the one or more encoder branches encodes respective individual images corresponding to low X-ray energies. The material decomposition image corresponds to one or more soft tissues. In an embodiment, each of the one or more encoder branches encodes respective individual images corresponding to high X-ray energies. The material decomposition image corresponds to one or more hard tissues.

[0025] "Low" and "high" can be regarded as relative terms, but it is understood that the appropriate low-energy or high-energy subset of the full set of energy images can be easily selected by simple experiments while balancing the quality of the results (measured by resolution or completeness of material decomposition) against the computational time or computational overhead.

[0026] However, in one embodiment, the low X-ray energy images (among the n images obtained from X-ray imaging with multiple energies) include the n-1, n-2, or n-3 images with the lowest energies. In another embodiment, the low X-ray energy images include one or two images with the lowest energies.

[0027] In one embodiment, the high X-ray energy images include the n-1, n-2, or n-3 images with the highest energies. In another embodiment, the high X-ray energy images include one or two images with the highest energies.

[0028] In one embodiment, the deep learning neural network is a trained neural network trained with real or simulated training images obtained from real or simulated X-ray imaging with multiple energies and a basis material image. For example, the basis material image may include at least any one of (i) an HA (hydroxyapatite) image, (ii) a calcium image, (iii) a water image, (vi) a fat image, (v) an iodine image, and (vi) a muscle image.

[0029] In a particular embodiment, the method comprises the step of generating at least one of the following: (i) a myelolysis image, (ii) a knee cartilage breakdown image, (iii) an iodine contrast agent breakdown image, (iv) a tumor breakdown image, (v) a muscle and lipolysis image, (vi) a metal artifact reduction image, and (vii) a beam hardening reduction image.

[0030] This method may include the steps of: generating one or more bone marrow images and using one or more bone marrow images to diagnose, identify or observe a bone marrow-related disease; generating one or more knee cartilage images and using one or more bone marrow images to diagnose, identify or observe osteoarthritis or rheumatoid arthritis; generating one or more iodine contrast agent images and diagnosing, identifying or observing a tumor; and / or generating one or more muscle images and diagnosing, identifying or observing sarcopenia.

[0031] This method may include the steps of: generating at least one of (a) a bone marrow image, (b) a knee cartilage image, (c) an iodine contrast agent image, and (d) a muscle image; generating one or more metal artifact images and / or one or more beam hardening reduction images; and improving the image quality of the bone marrow image, knee cartilage image, iodine contrast agent image, and / or muscle image using the metal artifact images and / or beam hardening reduction images.

[0032] This method may include a step of training or retraining a deep learning model using a neural network. This method may include the step of combining features extracted by one or more encoder branches that encode two or more images together, using a connected layer at the end of the encoder network of a neural network or after the encoder network, with features extracted by one or more encoder branches that encode each of the individual images.

[0033] In other embodiments, the method may include the step of combining features extracted by one or more encoder branches that encode two or more images together, and features extracted by one or more encoder branches that encode each of the individual images, using one or more concatenation operations at multiple levels of the encoder network of the neural network.

[0034] In yet another embodiment, the method may include the step of combining features extracted by one or more encoder branches that encode two or more images together with features extracted by one or more encoder branches that encode individual images, using a concatenation operation that connects the encoder network and decoder network of the neural network at multiple levels.

[0035] In yet another embodiment, the method includes the step of combining features extracted by one or more encoder branches that encode two or more images together with features extracted by one or more encoder branches that encode each of the individual images. It has. However, in this method, the encoder network and decoder network of the neural network are not connected at multiple levels.

[0036] According to the first aspect, material resolution images are also provided, generated from multiple images obtained by imaging with multiple energies of X-rays according to a method of the first aspect (including any of its embodiments).

[0037] According to a second aspect of the present invention, a system is provided for generating material-resolved images from a plurality of images obtained by imaging with a plurality of energies of X-rays, wherein the plurality of images correspond to each energy in the imaging with a plurality of energies of X-rays. The system comprises a neural network having an encoder-decoder structure (e.g., having an encoder network and a decoder network) and a plurality of encoder branches. Each of the one or more encoder branches is configured to encode two or more images from the plurality of images in combination (e.g., together, concatenated, or in series). Each of the one or more encoder branches is configured to encode the individual images of each of the plurality of images. The neural network is configured to model spatial and spectral relationships between the plurality of images by learning features from the combination of the plurality of images and one or more individual images from the plurality of images, and to generate one or more base material images using the spatial and spectral relationships. The system is configured to generate one or more material-specific images or material-resolved images from the base material images.

[0038] One or more encoder branches that encode two or more images from a set of multiple images may receive the two or more images in a combined and concatenated state. Alternatively, one or more encoder branches that encode two or more images from a set of multiple images may receive the two or more images in a combined, concatenated, or otherwise combined state before encoding them.

[0039] In one embodiment, each of two or more encoder branches is configured to encode different images of a group of images. In some embodiments, a first encoder branch is configured to encode a first combination of two or more images from a plurality of images as input. A second encoder branch is configured to encode a second combination of two or more images from a plurality of images as input. The first combination is different from the second combination (however, the first and second combinations may have a common image).

[0040] X-ray imaging with multiple energies may include any of the following: cold cathode X-ray radiography, dual-energy X-ray radiography, multi-energy X-ray radiography, photon counting X-ray radiography, cold cathode X-ray CT, dual-energy CT, multi-energy CT, and photon counting CT.

[0041] Advantageously, in some embodiments, an encoder branch that encodes each individual image receives all the images together that are encoded by encoder branches configured to encode two or more images.

[0042] However, in some other embodiments, the encoder branch encoding each individual image is configured to encode fewer images together than those encoded together by the encoder branch encoding two or more images (for example, by omitting one or more low-energy images) (for example, to reduce computation time).

[0043] In yet another embodiment, the encoder branch that encodes each individual image is configured to encode more images together than would be encoded together by the encoder branches that encode two or more images.

[0044] Each individual image encoding branch may encode only images that are not encoded by any of the encoder branches encoding two or more images. However, more advantageously, each individual image encoding branch encodes at least one image together with an image that is encoded by at least one of the encoder branches encoding two or more images.

[0045] The deep learning neural network may be a trained neural network that has been trained on real or simulated images obtained from real or simulated X-ray imaging with multiple energies and base material images. For example, the base material images may include at least one of the following: (i) HA (hydroxyapatite) images, (ii) calcium images, (iii) water images, (vi) fat images, (v) iodine images, and (vi) muscle images.

[0046] The system may be configured to generate at least one of the following: (i) myelolysis images, (ii) knee cartilage analysis images, (iii) iodine contrast agent analysis images, (iv) tumor analysis images, (v) muscle and lipolysis images, (vi) metal artifact reduction images, and (vii) beam hardening reduction images.

[0047] In one embodiment, the system is configured to generate one or more bone marrow images and use one or more bone marrow images to diagnose, identify or observe bone marrow-related diseases; generate one or more knee cartilage images and use one or more bone marrow images to diagnose, identify or observe osteoarthritis or rheumatoid arthritis; generate one or more iodine contrast agent images to diagnose, identify or observe tumors; and / or generate one or more muscle images to diagnose, identify or observe sarcopenia.

[0048] This system is configured to generate at least one of (a) a bone marrow image, (b) a knee cartilage image, (c) an iodine contrast agent image, and (d) a muscle image; to generate one or more metal artifact images and / or one or more beam hardening reduction images; and to improve the image quality of the bone marrow image, knee cartilage image, iodine contrast agent image, and / or muscle image using the metal artifact images and / or beam hardening reduction images.

[0049] This system may include a deep learning model training device configured to train or retrain deep learning models using a neural network. This system may be configured to combine features extracted by one or more encoder branches that encode two or more images together, using a connected layer at the end of or after the encoder network of a neural network, with features extracted by one or more encoder branches that encode each of the individual images.

[0050] In other embodiments, the system may be configured to combine features extracted by one or more encoder branches that encode two or more images together, using one or more concatenation operations at multiple levels of the encoder network of the neural network, with features extracted by one or more encoder branches that encode each of the individual images.

[0051] In yet another embodiment, the system may be configured to combine features extracted by one or more encoder branches that encode two or more images together, with features extracted by one or more encoder branches that encode individual images, using concatenation operations that connect the encoder network and decoder network of the neural network at multiple levels.

[0052] In yet another embodiment, the system may be configured to combine features extracted by one or more encoder branches that encode two or more images together, with features extracted by one or more encoder branches that encode each individual image. However, in this system, the encoder network and decoder network of the neural network are not connected at multiple levels.

[0053] According to a third aspect of the present invention, a computer program is provided comprising program code configured to perform the method of the first aspect (or any of its embodiments) when executed by one or more computing devices. According to the third aspect, a computer-readable medium (which may be non-temporary) comprising such a computer program is also provided.

[0054] It should be noted that any of the various individual features of each aspect of the present invention described above, and any of the various individual features of the embodiments described herein, including those described in the claims, can be combined in a suitable and desired manner. [Brief explanation of the drawing]

[0055] To better understand the present invention, embodiments will be described illustratively with reference to the following drawings. [Figure 1] This is a schematic diagram of an image processing system according to one embodiment of the present invention. [Figure 2] Figure 1 is a schematic diagram illustrating the general workflow of the system. [Figure 3] Figure 3A is a schematic diagram of a deep learning neural network for generating material-resolved images from multiple energy images, according to one embodiment of the present invention. Figure 3B is a schematic diagram of a deep learning neural network for generating material-resolved images from multiple energy images, according to another embodiment of the present invention. [Figure 4]This is a schematic diagram of a deep learning neural network for generating material resolution images from multiple energy images, according to an embodiment of the present invention. [Figure 5] Figure 1 is a schematic diagram of the training of one or more deep learning models by the deep learning model training device of the system shown. [Figure 6] Figures 6A and 6B are schematic diagrams illustrating the preparation of exemplary training data. [Figure 7] This diagram illustrates the operation of the system shown in Figure 1. [Modes for carrying out the invention]

[0056] Figure 1 is a schematic diagram of an image processing system 10 (particularly for the processing of medical images) according to one embodiment of the present invention. System 10 includes an image processing controller 12 and a user interface 14 (including a GUI 16). The user interface 14 includes one or more displays (one or more of which may display the GUI 16), a keyboard and a mouse, and optionally a printer.

[0057] The image processing controller 12 includes at least one processor 18 and memory 20. Instructions and data that control the operation of the processor 18 are stored in memory 20. System 10 may be implemented, for example, as a combination of software and hardware on a computer (such as a server, personal computer, or mobile computing device), or as a dedicated image processing system. The system may be arbitrarily distributed, for example, some or all of the components of memory 20 may be located remotely from the processor 18, and the user interface 14 may be located remotely from memory 20 and / or the processor 18, and may actually have a web browser or a mobile device application.

[0058] The memory 20 is capable of data communication with the processor 18 and typically has both volatile and non-volatile memory, including RAM (Random Access Memory), ROM, and one or more mass storage devices (and may have one or more types of memory).

[0059] As will be described in more detail below, the processor 18 has an image data processor 30 having a base material image generator 32, a diagnostic / observation task image generator 34 (with a decomposer 36), and an additional task-oriented image generator 38. The processor 18 further includes outputs in the form of a deep learning model training device 40 (including one or more deep learning neural networks 42), an I / O interface 44, and a results output device 46. The deep learning model training device 40 may be omitted in this embodiment and other embodiments because it is only required when the system 10 is training a deep learning model 58 rather than accessing one or more suitable deep learning models from an external source.

[0060] Memory 20 includes program code 50, image data store 52, non-image data store 54, training data store 56, trained deep learning model(s) 58, generated base material image store 60, and generated material identification or material decomposition image store 62. The image processing controller is at least partially implemented by the processor 18 executing program code 50 from memory 20.

[0061] Broadly speaking, the I / O interface 44 is configured to read and receive image data (such as DICOM format) and non-image data related to subjects or patients, for example, into the image data store 52 and non-image data store 54 of the memory 20 for processing. The non-image data stored in the non-image data store 54 contains a wide range of information such as energy, desired substance, and desired task, and is accessible by the image generators 32, 34, and 36 for use in image generation.

[0062] The base material image generator 32 of the image data processor 12 generates one or more sets of base material images using one or more machine learning models (derived from the deep learning model 58). The diagnostic / observation task image generator 34 uses the decomposer 36 to generate one or more sets of material identification images or material decomposition images (suitable for, for example, a diagnostic or observation task) using the base material images. The additional task-oriented image generator 38 generates at least one further set of images (such as beam hardening images or metal artifact reduction images). The I / O interface 44 outputs the processing results to, for example, the results output device 46 and / or GUI 16.

[0063] System 10 employs one or more deep learning models to accurately and reproducibly generate basal material images. These basal material images are then used to generate images of different tissues and materials, particularly low-contrast tissues and materials, which may then be used for pathological or disease identification and observation (such as disease progression). For example, cartilage segmentation images from a knee joint scan may be used for the diagnosis and / or observation of osteoarthritis or rheumatoid arthritis, and bone marrow segmentation images from a musculoskeletal scan may be used for the diagnosis of related diseases and observation of related diseases or conditions. Images of pathological and normal tissues from patient scans may be used for the diagnosis and observation of tumors. Synchronized images of the degradation of multiple contrast agents from a CT scan may be used for the diagnosis or identification of renal abnormalities and for disease staging.

[0064] System 10 can also generate images for other tasks (using the additional task-oriented image generator 38). For example, System 10 can generate beam hardening or metal artifact reduction images based on the aforementioned base material image for better image quality. The beam hardening metal artifact effect occurs when a multicolor X-ray beam passes through an object, resulting in selective attenuation that primarily affects low-energy photons. As a result, higher-energy photons are supplied to the beam, either individually or in excess, increasing the average beam energy. Since material resolution takes into account all energy-dependent attenuation, it is desirable that the resolved image is free from the effects of beam hardening and metal artifacts.

[0065] Accordingly, referring to Figure 1, the system 10 is configured to receive two types of data related to the subject or patient: image data and non-image data. Image data is in the form of images of X-rays with multiple energies, such as those that can be generated in cold cathode radiography, dual-energy CT, and multi-energy CT or photon counting CT, or in the form obtained from such images. Non-image data includes information about multiple energy images, such as the energies at which multiple energy images were generated; information about desired base material images, such as the type and number of base material images; and information about desired analysis or interpretation, such as disease diagnosis / identification / observation or additional tasks (e.g., beam hardening or metal artifact reduction). The system 10 stores the image data and non-image data in the image data store 52 and the non-image data store 54, respectively.

[0066] As described above, the image data processor 30 has three components: a base material image generator 32, a diagnostic / observation task image generator 34, and an additional task-oriented image generator 38. Image data and non-image data are received by the image data processor 30 from the memory 20. Based on multiple energy images and non-image data base materials, the image data processor 30 selects one or more appropriate deep learning models 58 to generate one or more sets of base material images. In capturing task information, the image data processor 30 generates images for disease diagnosis / identification and / or observation (e.g., human tissue, contrast agent images) and images for better image quality (e.g., beam hardening images and metal artifact reduction images).

[0067] The deep learning model training device 40 pre-trains the deep learning model 58 using training data (from the training data store 56) that includes labels or annotations that constitute ground truth for machine learning. The training data is prepared to be suitable for training a deep learning model to generate a base material image from multiple energy images. The training data consists of both known multiple energy images and known base material images. Labels indicate the energy bin of each energy image (i.e., the image corresponding to a particular energy threshold or energy bin) and the material information of the base material image (e.g., material name and material density). The training data may be in the form of real clinical data, real phantom data, simulated data, or a mixture of one or more of these.

[0068] As described above, the deep learning model training device 40 is configured to train one or more deep learning models (and retrain or update trained deep learning models) using the neural network 42 and training data. However, in other embodiments, the machine learning model training device may be configured or used only to retrain or update (i.e., train again) one or more existing deep learning models.

[0069] The image data processor 30 selects one or more appropriate deep learning models from the deep learning models 58 based on multiple energy images and the target base material (identified in non-image data). The base material image generator 32 generates images of the target base material. The diagnostic / observation task image generator 34 generates images from the generated base material images according to information about diagnostic / identification and / or observation tasks (also identified in non-image data). Optionally, the additional task-oriented image generator 38 generates images from the generated base material images according to information about additional tasks (also identified in non-image data).

[0070] Base material images, diagnostic / observation images, and / or additional task-oriented images are output to the user interface 14 via the result output device 46 and the I / O interface 44. Figure 2 is a flowchart 70 of the general workflow of system 10 in Figure 1. Referring to Figure 2, in step 72, system 10 receives multiple energy images (e.g., generated by dual-energy, multi-energy, or photon-counting CT or X-ray imaging) and loads the images into image data store 52. In step 74, system 10 receives the associated non-image data and loads that data into non-image data store 54.

[0071] The memory 20 is advantageously configured to allow high-speed access to data by the system 10. For example, if the system 10 is implemented as a combination of software and hardware on a computer, it is desirable that images be loaded into the RAM of the memory 20.

[0072] In step 76, the image data processor 30 selects one or more suitable deep learning models from the trained deep learning models 58. The selection of deep learning models is based on energy information characterizing multiple energy images contained in the non-image data, and information about the target base material. Any particular model is modeled using images of a specific energy to generate a particular set of base material images. Therefore, one or more suitable models may be trained and available. Thus, multiple suitable models may be trained and available. If multiple models are selected, they are used in parallel.

[0073] For example, to generate a set of base material images, one deep learning model may be selected to be used with all loaded images. In another example, one or more deep learning models may be selected to be used with all loaded images to generate multiple sets of base material images. In yet another example, one or more deep learning models may be selected to be used with each subset of loaded images to generate one or more sets of base material images.

[0074] One or more selected deep learning models include spatial and spectral relationships learned from training data. In step 78, the base material image generator 32 uses one or more selected deep learning models and their spatial and spectral relationships to generate base material images from loaded subject or patient images in the image data store 52 and stores the generated base material images in the generated base material image store 60.

[0075] In step 80, the diagnostic / observation task image generator 34 uses the generated base material image to decompose the original subject or patient image in the image data store 52, thereby generating material identification or material decomposition images of specific, different (e.g., human) tissues suitable for disease identification, diagnosis, and / or observation, and stores these material identification or material decomposition images in the generated material identification or material decomposition image store 62.

[0076] In step 82, the image data processor 30 determines whether the additional task-oriented image generator 38 needs to generate an image according to the relevant non-image data 54 indicating the desired task. If it is determined that the additional task-oriented image generator 38 does not need to generate an image, the process is terminated. If the additional task-oriented image generator 38 needs to generate an image, in step 84, the additional task-oriented image generator 38 generates an appropriate task-oriented image, such as a beam hardening reduction image and / or a metal artifact reduction image. The process is then terminated.

[0077] Figure 3A is a schematic diagram of a deep learning neural network 90 (which may be used as the neural network 42 of system 10) for generating material resolution images from multiple energy images according to an embodiment of the present invention. The neural network 90 is shown to have inputs in the form of images 92 (where n≧2) of X-rays with n energies and outputs in the form of a base material image 94. The neural network 90 is configured to generate the base material image 94 from the images 92. That is, the function mapping between the input image 92 and the output base material image 94 is approximated by the neural network 90 configured to predict a material-specific image using the image 92 as input. Then, material resolution images can be generated from the generated base material image 94.

[0078] The neural network 90 includes an encoder network 96 and a decoder network 98. The encoder network 96 encrypts the structure of an input image (e.g., part or all of image 92) into multiple different levels of feature representations. The decoder network 98 projects the discriminative feature representations learned by the encoder network 96 into pixel / voxel space to obtain dense classification. In one example, the encoding performed by the encoder network 96 includes convolution and downsampling operations, and the decoding performed by the decoder network 98 includes convolution and upsampling operations. In another example, the encoding performed by the encoder network 96 and / or the decoding performed by the decoder network 98 include concatenation operations.

[0079] The encoder network 96 has a multi-branch structure having a first set 1001 and a second set 1002 of encoder branches (each set having one or more encoder branches). Each branch of the first set 1001 of encoder branches concatenates and encodes multiple images selected from image 92 (which may include all images 92). (Note that these multiple images selected from image 92 for processing in concatenated format may be input in either concatenated or unconcatenated format. In the latter case, the encoding network first concatenates the images.)

[0080] Each branch in the second set 1002 of the encoder branches encodes an individual image selected from image 92. The first set 1001 and the second set 1002 may have the same number of images or different numbers of images combined.

[0081] In the example in Figure 3A, the first set of encoder branches 1001 includes one encoder network branch 960 ("encoder network 0") for concatenating and encoding multiple images 92 (all of the images 92 in this example). The second set of encoder branches 1002 includes each of the individual input images 921, 922, ..., 92 m Multiple m encoder network branches 961, 962, ..., 96 for encoding each of them m (Each has "encoder network 1", "encoder network 2", ..., "encoder network m"), where m ≤ n (note that images 1, 2, ..., m do not need to be sequential and do not need to constitute the first m images of image 92). Also, encoder branch 960 may be configured to concatenate and receive multiple but not all of images 92. Individual images 921, 922, ..., 92 m Since these are generally conventional formats for each imaging style (e.g., DICOM, JPEG, TIFF, or other imaging files), they are typically two- or three-dimensional images with pixels or voxels, but they may be described as three- or four-dimensional because they have an extra dimension indicating an energy threshold or bin. Similarly, images in concatenated or combined forms 92 may also typically be described as three- or four-dimensional because they have an extra dimension (indicating an energy threshold or energy bin).

[0082] The encoder network branch 960 of the first set 1001 learns the relationships between the images 92 input to its branch and combines them effectively. The encoder network branch 960 of the second set 1002 learns the relationships between the individual images 921, 922, ..., 92 m Learn the features independently. The first set of network branches 1001 (i.e., network branch 960) and network branches 961, 962, ..., 96 m The feature representations learned by the second set 1002 are combined as input to the decoder network 98.

[0083] In one example, the first set 1001 of encoder network branch 960 and encoder network branches 961, 962, ..., 96 m The features extracted by the second set 1002 are combined using a concatenation layer (not shown) at the end of or after the encoder network 96. In another example (see embodiment in Figure 4), the features extracted from the first set 1001 and the second set 1002 of the branch are combined using one or more concatenation operations at multiple levels of the encoder network 96.

[0084] In a further example (see the embodiment in Figure 4), the encoder network 96 and the decoder network 98 are connected at multiple levels by a coupling operation. In yet another example, the encoder network 96 is not connected to the decoder network 98 at multiple levels.

[0085] As described above, all of the images 92 may be concatenated to form an input (or concatenated image) for input to the first branch 960, or input images 921, 922, ..., 92 m Only a portion (but multiple) of the images may be concatenated to form an input (or concatenated image) for input to the first set 1001 of the encoder branch (i.e., encoder branch 960). In one embodiment, all of the images 92 are input separately to the second set 1002 of the encoder branch, but in another embodiment, images 921, 922, ..., 92 n Some (i.e., one or more) of the encoder branch may not be encoded by the second set 1002. Furthermore, it should be noted that the images input to the first set 1001 and the second set 1002 of the encoder branch do not need to be the same, but are derived from the same multi-energy image 92.

[0086] Therefore, the deep learning neural network 90, which can be described as a multi-branch encoder-decoder deep learning network, generates a base material image 94 by essentially modeling the spatial and spectral relationships between multiple energy images 92.

[0087] Figure 3B is a schematic diagram of a deep learning neural network 90' (which may be adopted as the neural network 42 of system 10), and since it is equivalent to the neural network 90 in Figure 3A, the same numbers are used to indicate similar features. Thus, the neural network 90' is also suitable for generating material-resolved images from multiple energy images according to embodiments of the present invention.

[0088] The neural network 90' has an encoder network 96' which has first and second sets of encoder branches 1001' and 1002'. The neural network 90' differs from the neural network 90 in Figure 3A in that the first set of encoder branches 1001' of the neural network 90' has at least two encoder branches 960' and 961' which have encoder network 0' and encoder network 1', respectively, and each is configured to receive a plurality of concatenated images selected from image 92 (image 1021 and image 1022, respectively).

[0089] Image 1021 and Image 1022 may have the same number of images or different numbers of images, and in either case, they may constitute overlapping or non-overlapping image sets.

[0090] Figure 4 is a schematic diagram of a deep learning neural network 110 (which may be adopted as the neural network 42 of system 10) for generating a base material image from multiple energy images according to an embodiment of the present invention. The neural network 110 is shown with input in the form of multiple (four in this example) energy X-ray images 112.

[0091] The neural network 110 includes a multi-branch encoder network 114 and a decoder network 116. In this embodiment, the encoder network 114 has a first set of encoder branches, which has a single branch, a first branch 118, that receives all combinations of the four images 112 as input. In this example, the encoder network 114 has a second set of encoder branches, which has two branches: a second branch 122 that receives a first image 1121 (which is the first image among the X-ray images 112 with multiple energies) as input, and a third branch 126 that receives a third image 1123 (which is the third image among the X-ray images 112 with multiple energies) as input.

[0092] The encoder network structures of the three encoder branches 118, 122, and 126 are identical. Each encoder branch has three stages defined by the size of its feature map. Each stage includes convolution, batch normalization, and ReLU (Rectified Linear Unit) functions or operations. Thus, the first branch 118 has a first stage 1181 containing a 16-channel first feature map 1201 which has the same width and height as the original combination of image 112 (which can also be considered part of the first stage 1181 of the first branch). The second stage 1182 contains a 16-channel second feature map 1202 and a 64-channel third feature map 1203, and the third stage 1183 contains a 64-channel fourth feature map 1204 and a 128-channel fifth feature map 1205.

[0093] Similarly, the second branch 122 has a first stage 1221 having a 16-channel first feature map 1241 which has the same width and height as the first individual image 1121 (which can also be considered part of the first stage 1221 of the second branch). The second stage 1222 includes a 16-channel second feature map 1242 and a 64-channel third feature map 1243, and the third stage 1223 includes a 64-channel fourth feature map 1244 and a 128-channel fifth feature map 1245.

[0094] The third branch 126 has a first stage 1261 which includes a 16-channel first feature map 1281 that has the same width and height as the third individual image 1123 (which can also be considered part of the first stage 1261 of the third branch). The second stage 1262 includes a 16-channel second feature map 1282 and a 64-channel third feature map 1283, and the third stage 1263 includes a 64-channel fourth feature map 1284 and a 128-channel fifth feature map 1285. The feature maps 1201, 1241, and 1281 of each of the first stages 1181, 1221, and 1261, and the final feature maps 1203, 1243, and 1283 of each of the second stages 1182, 1222, and 1263, undergo maximum pooling, reducing the size of the feature maps and enabling the encoding network 114 to find the global features of each input image 112, 1121, and 1123 (note that the pooling operation is performed on the feature representations or maps between two stages; this is why there are two pooling operations for three stages).

[0095] In this embodiment, the decoder network 116 also includes three stages, and the last feature map 1205 of the first stage of the decoder network 114 also functions as the first stage of the encoder network 116. Each of the three stages 1301, 1302, and 1303 of the decoder network 116 is defined by the size of their respective feature maps. The first stage 1301 includes a 128-channel feature map 1205. The second stage 1302 includes a 64-channel feature map 1321 and a 32-channel feature map 1322. The third stage 1303 includes a 16-channel feature map 1323 and a 4-channel feature map 134 (the latter being the output base material image(s)). Each of these three stages, 1301, 1302, and 1303, involves convolution, batch normalization, and ReLU operations, and the feature maps from stages 1301, 1302, and 1303 undergo mean pooling (i.e., pooling operations are applied to the feature maps between stages 1301 and 1302, and between stages 1302 and 1303) to restore the dimensions of the feature maps to match those of the input images 112, 1121, and 1123.

[0096] In this embodiment, the feature maps of each stage of the three branches 118, 122, and 126 of the encoder network 114 are concatenated (thus feature maps 1201, 1241, and 1281; feature maps 1203, 1243, and 1283; and feature maps 1205, 1245, and 1285, respectively), and then concatenated with the feature maps of the corresponding stages of the decoder network 116 (thus feature maps 1205, 1321, and 1303, respectively). The multiple levels of connectivity between the multi-branch encoder network 114 and the decoder 116 enable the neural network 110 to learn local details of the input images 112, 1121, and 1123.

[0097] Figure 5 is a flowchart 140 of training one or more deep learning models, ultimately stored in a deep learning model 58, by a deep learning model training device 40. In step 142, training data is prepared or procured. The training data includes X-ray images at multiple energies and images of a base material. The training data may be real data, simulated data, or a combination thereof. In one example, the training data is generated using phantoms of different known materials. In another example, the training data is simulated based on known properties of known materials under different X-ray energies. In some examples, the training data has only real data or only simulated data. In another example, the training data has some real data and some simulated data. Thus, preparing the training data may include, for example, generating (or procuring) real training data (see step 144a) and / or simulating (or procuring) training data (see step 144b).

[0098] This process optionally includes step 146, which increases the training data using data augmentation. This may include adding Gaussian noise to the training data to improve the robustness of model training, and / or splitting the training data into patches to increase the amount of training data.

[0099] In step 148, the training data are labeled with appropriate and correct labels. Each “energy image” (i.e., individual images corresponding to a single energy threshold or energy bin) is labeled with the associated energy threshold or energy bin (see step 150a), and each base material image is labeled with the associated material (see step 150b). In step 152, the deep learning model training apparatus 40 trains one or more deep learning models using the correctly labeled energy images and base material images. Step 152 may involve updating (or retraining) one or more trained deep learning models if such models have been previously trained and the training data prepared or sourced in step 142 is new or additional training data.

[0100] In step 154, the trained or retrained model is stored in the machine learning model 58 and deployed for use. The process then terminates unless it includes an optional step 156 in which the deep learning model training device 40 decides whether to perform retraining or further training. Otherwise, the process terminates, but if the deep learning model training device 40 decides to perform retraining or further training, the process returns to step 142.

[0101] During use, system 10 inputs X-ray images with one or more energies into one or more currently trained deep learning models 58. The deep learning models 58 process the images and output a set of baseline images.

[0102] Figures 6A and 6B are schematic diagrams of exemplary training data preparation techniques. Figure 6A shows training data preparation using real data. For example, one or more phantoms are used, each having an insert containing a known substance (e.g., HA (hydroxyapatite), iodine, calcium, blood, or fat) or a mixture of some or all of the known substances (e.g., iodine and blood).

[0103] The composition, concentration, size, and position of each substance insert are known. The phantom is scanned using, for example, cold cathode radiography, dual-energy CT, multi-energy CT, or photon counting CT, generating X-ray images with multiple energies at two or more energy thresholds or energy bins. In this example, the goal is to generate three substrate-specific images: an HA image, an iodine image, and a fat image. Thus, each substrate-specific image is generated with the concentration, size, and insertion position of each substance known.

[0104] Figure 6B shows the preparation of training data using simulated data.170 For example, one or more phantoms are simulated172, which again have inserts containing known substances (e.g., iodine, calcium, blood, fat) and mixtures of some of the known substances. The concentration, size, and location of each substance insert are known.Multi-energy images are simulated174 based on known substances and specific energies, and are created by referencing actual scans with different substance concentrations, for example, acquired by multi-energy CT or photon counting CT. For example, actual scans can be acquired by scanning an actual phantom with inserts containing iodine at concentrations of 2, 8, and 16 mg / cc, respectively, using photon counting CT. Since photon counting CT maintains a strong linear relationship between the CT number and the substance concentration, simulated scans with 20, 25, and 30 mg / cc of iodine can be generated by applying a linear fitting to the CT number of the actual scan. In another example, energy images are created by mathematical simulations based on known reactions of known materials under different X-ray energies. Base material identification images are simulated 176 times for each material insert, considering its concentration, size, and position.

[0105] Figure 7 shows an exemplary workflow 180 of system 10 in Figure 1 when a patient injected with iodine contrast agent is scanned 182 using photon counting CT. Five images are generated using five energy thresholds (i.e., the above energies at which X-rays are counted in five corresponding energy bins), thereby generating a 25 keV threshold image 184, a 35 keV threshold image 186, a 45 keV threshold image 188, a 55 keV threshold image 190, and a 65 keV threshold image 192. From these images 184, 186, 188, 190, and 192, one or more trained deep learning models generate four base material images 194: a calcium image 196, a water image 198, a fat image 200, and an iodine image 202. From different linear combinations of the base material images 196, 198, 200, and 202, various functional images for disease diagnosis / observation and / or other tasks are generated. For example, i) a bone marrow image 204 may be generated for the diagnosis / observation of bone marrow-related diseases (using, for example, HA (hydroxyapatite) + fat + water as the base material / image); ii) a knee cartilage image 206 may be generated for the diagnosis / observation of osteoarthritis or rheumatoid arthritis (using, for example, HA + fat + water + soft issue as the base material / image); iii) an iodine contrast image 208 may be generated for tumor diagnosis / observation (using, for example, HA + fat + water + soft issue + iodine as the base material / image); and iv) a metal artifact and beam hardening reduction image 210 may be generated for better image quality (using, for example, HA + soft tissue as the base material / image).

[0106] Those skilled in the art will understand that many modifications can be made without departing from the scope of the present invention. In particular, it will be clear that certain features of the embodiments of the present invention can be adopted to form further embodiments.

[0107] Where prior art is referenced in this specification, it should be understood that such references do not constitute an acknowledgment that such prior art forms part of the general knowledge in the art in any country.

[0108] In subsequent claims and prior art descriptions, unless otherwise required by contextual explicit expression or necessary implied indication, variant words such as “equipped with” are used in a comprehensive sense, i.e., to identify the presence of the described features, but not to exclude the presence or addition of further features in various embodiments of the invention.

Claims

1. A method for generating material resolution images from multiple images obtained by imaging with X-rays of multiple energies, wherein the multiple images correspond to each of the energies in the imaging of the X-rays of the multiple energies. The steps include: modeling the spatial and spectral relationships between the multiple images by using a deep learning neural network to learn the features of the combined multiple images and to learn the features of one or more of the multiple images individually; The steps include generating one or more base material images using the spatial relationship and the spectral relationship, The steps include generating one or more material identification images or material decomposition images from the aforementioned base material image, It has, The neural network has an encoder-decoder structure and includes multiple encoder branches. Among the plurality of encoder branches, each of the one or more encoder branches encodes by combining two or more images from the plurality of images. Among the plurality of encoder branches, each of the one or more encoder branches other than the one or more encoder branches encodes the individual images of the plurality of images. method.

2. i) Each of the two or more encoder branches encodes a different individual image of the plurality of images as input, and / or, ii) The first encoder branch encodes a first combination of two or more images from the plurality of images, and the second encoder branch encodes a second combination of two or more images from the plurality of images, wherein the first combination is different from the second combination. The method according to claim 1.

3. The aforementioned imaging of X-rays with multiple energies includes any of the following: cold cathode X-ray imaging, dual-energy X-ray imaging, multi-energy X-ray imaging, photon counting X-ray imaging, cold cathode X-ray CT, dual-energy CT, multi-energy CT, and photon counting CT. The method according to claim 1.

4. a) The encoder branch that encodes each individual image receives all images together that are received by the encoder branches that encode two or more images, b) The total number of images encoded by the encoder branch that encodes individual images is less than the total number of images encoded by the encoder branch that encodes two or more images, c) Each encoder branch that encodes individual images encodes more images together than the encoder branch that encodes two or more images together encodes, It is one of the following: The method according to any one of claims 1 to 3.

5. The aforementioned deep learning neural network is a trained neural network that has been trained using actual or simulated images obtained from actual or simulated X-ray imaging with multiple energies, and from base material images. The method according to any one of claims 1 to 3.

6. The aforementioned base material image is (i) Image of hydroxyapatite, (ii) Calcium image and, (iii) Water image and, (vi) Fat images and, (v) Iodine image and, (vi) Muscle images and, Including at least one of the following: The method according to claim 5.

7. (i) Bone marrow degradation images and, (ii) Images of knee cartilage disintegration, (iii) Iodine contrast agent decomposition images and, (iv) Tumor decomposition images and, (v) Images of muscle and fat breakdown, (vi) Images with reduced metal artifacts, (vii) Beam hardening reduction image and, A step of generating at least one of the following: It further possesses, The method according to any one of claims 1 to 3.

8. The steps include generating one or more bone marrow images and using the one or more bone marrow images to diagnose, identify, or observe bone marrow-related diseases, and / or, The steps include generating one or more knee cartilage images and using the one or more knee cartilage images to diagnose, identify, or observe osteoarthritis or rheumatoid arthritis, and / or, The steps include generating one or more iodine contrast images to diagnose, identify, or observe a tumor, and / or, The steps include generating one or more muscle images and diagnosing, identifying, or observing sarcopenia, and / or, (a) generating at least one of a bone marrow image, (b) a knee cartilage image, (c) an iodine contrast agent image, and (d) a muscle image; generating one or more metal artifact images and / or one or more beam hardening reduction images; and using the metal artifact images and / or beam hardening reduction images to improve the image quality of the bone marrow image, the knee cartilage image, the iodine contrast agent image and / or the muscle image; It further possesses, The method according to any one of claims 1 to 3.

9. A system for generating material resolution images from multiple images obtained by imaging with X-rays of multiple energies, wherein the multiple images correspond to each energy in the imaging of the X-rays of the multiple energies. A neural network having an encoder-decoder structure and multiple encoder branches, Equipped with, Among the plurality of encoder branches, each of the one or more encoder branches is configured to encode by combining two or more images from the plurality of images. Among the plurality of encoder branches, each of the one or more encoder branches other than the one or more encoder branches is configured to encode the individual images of the plurality of images. The neural network is configured to model the spatial and spectral relationships between the multiple images by learning features from the combination of the multiple images and one or more individual images from the multiple images, and to generate one or more base material images using the spatial and spectral relationships. The system is configured to generate one or more material identification images or material decomposition images from the base material image. system.

10. i) Among the plurality of encoder branches, each of two or more encoder branches is configured to encode different images of the plurality of images, and / or, ii) Among the plurality of encoder branches, the first encoder branch is configured to encode a first combination of two or more images from the plurality of images, and among the plurality of encoder branches, the second encoder branch is configured to encode a second combination of two or more images from the plurality of images, and the first combination is different from the second combination. The system according to claim 9.

11. a) Each encoder branch configured to encode individual images encodes all images together that are encoded together by each encoder branch configured to encode two or more images, b) The encoder branch that encodes individual images is configured to encode fewer images together than the images that are combined and encoded by the encoder branch that encodes two or more images, c) The encoder branch that encodes individual images is configured to encode more images together than can be encoded by the encoder branch that encodes two or more images, It is one of the following: The system according to claim 9.

12. The aforementioned deep learning neural network is a trained neural network that has been trained using actual or simulated images obtained from actual or simulated X-ray imaging with multiple energies, and from base material images. The system according to any one of claims 9 to 11.

13. The aforementioned base material image is (i) Image of hydroxyapatite, (ii) Calcium image and, (iii) Water image and, (vi) Fat images and, (v) Iodine image and, (vi) Muscle images and, Including at least one of the following: The system according to claim 12.

14. (i) Bone marrow degradation images and, (ii) Images of knee cartilage disintegration, (iii) Iodine contrast agent decomposition images and, (iv) Tumor decomposition images and, (v) Images of muscle and fat breakdown, (vi) Images with reduced metal artifacts, (vii) Beam hardening reduction image and, Configured to generate at least one of the following: The system according to any one of claims 9 to 11.

15. To generate one or more bone marrow images, and to use said one or more bone marrow images to diagnose, identify, or observe bone marrow-related diseases, and / or, To generate one or more knee cartilage images, and to use the one or more knee cartilage images to diagnose, identify, or observe osteoarthritis or rheumatoid arthritis, and / or, To generate one or more iodine contrast images and diagnose, identify, or observe a tumor, and / or, To generate one or more muscle images and diagnose, identify, or observe sarcopenia, and / or, (a) generating at least one of a bone marrow image, (b) a knee cartilage image, (c) an iodine contrast agent image, and (d) a muscle image; generating one or more metal artifact images and / or one or more beam hardening reduction images; and using the metal artifact images and / or beam hardening reduction images to improve the image quality of the bone marrow image, the knee cartilage image, the iodine contrast agent image, and / or the muscle image. Configured to perform, The system according to any one of claims 9 to 11.

16. From multiple images obtained by imaging with X-rays of multiple energies, a method is used to generate a method according to any one of claims 1 to 3. Material decomposition image.

17. When executed by one or more computing devices, the system has program code configured to carry out the method described in any one of claims 1 to 3. Computer program.

18. A computer program comprising the computer program described in claim 17, A computer-readable medium.

Images