Automatic correction of metal-affected voxel representations of X-ray data using deep learning techniques

Abstract

A computer-implemented method for correcting a voxel representation of metal-affected X-ray data is described, the metal-affected X-ray data representing artifacts in the X-ray data caused by metal or metal-containing objects in a volume of tissue imaged by an X-ray imager, wherein the method comprises: a first 3D deep neural network receiving at its input an initial voxel representation of the metal-affected X-ray data and generating at its output a voxel map, the voxel map identifying voxels of the initial voxel representation belonging to a region of metal-affected voxels; and a second 3D deep neural network receiving at its input the initial voxel representation and the voxel map generated by the first 3D deep neural network, and generating a corrected voxel representation, the corrected voxel representation comprising voxel estimates for voxels identified by the voxel map as being part of the region affected by metal, the first 3D deep neural network being trained based on training data and reference data, the training data and reference data comprising voxel representations of clinical X-ray data of a predetermined body part of a patient.

Description

Technical Field

[0001] The present invention relates to automatically correcting metal-affected voxel representations of X-ray data using deep learning techniques, and in particular, but not exclusively, to methods and systems for correcting metal-affected voxel representations of X-ray data using deep learning techniques, methods for training a deep neural network system to correct metal-affected voxel representations of X-rays, and computer program products for performing such methods. Background Art

[0002] During a CT scan, different areas of tissue are exposed at different angles by an X-ray radiation beam of predetermined intensity, while an array of sensors measures the intensity of the X-ray beam traveling through the tissue. Based on the attenuated radiation beam measured by the sensors, the absorption of the X-ray radiation at different areas in the tissue can be determined. The data obtained during the scan is in the form of groups of sinusoidal signals with different amplitudes and phases, commonly called sinograms. Reconstruction algorithms can use sinograms to reconstruct a 2D or 3D image of the scanned tissue. Such reconstruction algorithms are often called backprojection.

[0003] When tissue includes materials with high absorption coefficients that deviate significantly from those of biological materials, problems may arise when attempting to reconstruct images based on the use of back projection. For example. In the dental field, metal implants and / or dental fillings are often made of relatively high density metal materials (e.g., amalgam, gold, silver, palladium, titanium, etc.) with high absorption coefficients, which may cause inconsistencies in the constructed images, such as what appear to be missing data points or incorrect data points that do not represent the actual tissue or other structure density values ​​scanned. These inconsistencies caused by high-density materials can generally be referred to as metal artifacts or simply artifacts (note that "artifact" is often used instead of "artifact" in the literature).

[0004] Artifacts can be a serious problem because they can interfere with the diagnostic process, for example, performed by a medical professional or a computer-based automated diagnostic system. The artifact problem is exacerbated by the general trend to reduce the dose in CT examinations, due to the fact that the use of short-wavelength ionizing radiation raises concerns about potential health hazards. A disadvantage of reducing the dose is an increase in noise in the reconstructed CT images. In addition, in medical fields such as oral surgery, endodontics, and orthodontics, small CT scanners such as cone-beam CT (CBCT) scanners are increasingly used. CBCT scanners are compact, low-dose scanners in which the X-ray beam has a cone shape. Although CBCT technology has many similarities to conventional CT scanning technology, there are also substantial differences. For example, because CBCT uses a 2D array of sensors and a cone-beam shaped X-ray beam, the cone-beam reconstruction algorithm is different from the typical reconstruction algorithm of a conventional CT scanner. In addition, in CBCT scans, the radiation density measured in Hounsfield units (HU) is inconsistent because different areas in the scan appear different grayscale values ​​depending on their relative position in the scanned organ. HUs measured from the same anatomical region using CBCT and medical-grade CT scanners are not identical and therefore are unreliable for determining location-specific radiographically identified measured densities. Due to this difference, the adverse effects of high-density material will appear differently in the image depending on the type of scanner and back-projection technique.

[0005] An overview of general "types" of artifacts is described, for example, in the following article: Schultze et al., Artefacts in CBCT: a review, Dentomaxillofacial Radiology, Jul 2011;40(5):265-273, which lists various types of artifacts, including: extinction artifacts, beam hardening artifacts, exponential edge gradient effects (EEGE), ring artifacts, aliasing artifacts, and motion artifacts, of which the first three are particularly important. Extinction artifacts can be considered as "missing value" artifacts. This artifact occurs when the scanned object contains highly absorbing materials (e.g., a gold crown). In this case, the high-density metal "hides" the material directly behind it. The radiation beam cannot enter these areas correctly, resulting in "black areas" next to the highly absorbing material. Beam hardening artifacts are mainly caused by the inverse relationship between high-density materials and / or high atomic numbers and the low-wavelength photons of the X-ray source. Due to the absorption of photons by high-density materials, relatively more absorption is recorded in the path behind the metallic object. This error is back-projected into the volume, resulting in black fringes in the image. In addition, exponential edge gradient effects (EEGE) occur when two adjacent structures have a high contrast in density and / or atomic number. Since the back-projection algorithm averages the measured intensities over the recorded beam, this causes fringes in the direction of the X-ray projection next to the metal.

[0006] Therefore, multiple sources may contribute to the appearance of artifacts in CT images reconstructed based on sinusoids. Aspects related to high-density materials (thickness, density, atomic number, etc.), as well as aspects related to the scanner, scanning method (e.g., scanner type, sensor type, sensor geometry, sensor quality, wavelength of the X-ray source, etc.), and reconstruction method may affect the severity and manner in which these artifacts appear in the reconstructed image. These factors make the removal of artifacts from CT data a challenging problem.

[0007] A series of methods have been proposed to reduce artifacts in CT images while retaining important details in the images. These methods are called metal artifact reduction (MAR) methods. Common MAR methods include linear interpolation, beam hardening correction methods, and normalized metal artifact reduction methods. As explained in the following article, common MAR methods are performed in the sinusoidal domain, that is, before converting the sinusoidal to a pixel-based or voxel-based image: Meyer et al., Normalized metal artifact reduction (NMAR) in computed tomography, Medical Physics, Vol. 37, No. 10, October 2010, pp. 5482-5493. These methods treat portions of the projection data affected by metal as missing data. One or more metal objects in the image are determined using a threshold method, and only the forward projection of the metal image can be used to determine the signal affected by metal in the sinusoidal. Interpolation and forward projection techniques are then applied to the sinusoidal to replace the affected data with alternative data. Such MAR techniques require access to sinogram data and knowledge of the scanning and backprojection methods used when generating the sinograms. Furthermore, the techniques use multiple forward and backward projection calculations as well as image thresholding, which requires significant computational resources. Therefore, despite improvements, current MAR techniques do not provide a general solution that can automatically remove or at least reduce artifacts in X-ray data in an efficient and reliable manner.

[0008] Recently, neural networks have been used in the MAR process. Zhang et al. (2017) in their article “Convolutional Neural Network based Metal Artifact Reduction in X-ray Computed Tomography” describe the reduction of artifacts in CT images with the help of a 2D convolutional neural network (CNN) that is trained on images processed based on conventional MAR methods to enable the CNN to learn useful information to reduce artifacts. Metal artifacts are simulated computationally. The CNN generates an image that is then processed in the sinusoidal domain together with the original uncorrected data and metal trace data to form a corrected sinusoidal map. The corrected sinusoidal map is converted to a corrected image. Artifact reduction is achieved, but the remaining artifacts are still considerable. In the study "Deep learning methods for CT image-domain metal artefact reduction" by Gjetseby et al. (SPIE, Vol. 10391, Sept. 25, 2017), a combination of state-of-the-art sinogram-domain NMAR techniques and 2D-CNNs were applied to specifically reduce metal streaks on 2D images. The image was first reconstructed by the NMAR technique, resulting in a reconstructed image with a large amount of remaining metal streak artifacts. The reconstructed image was then fed to the input of the 2D CNN CNN to determine a reconstructed CT image with reduced metal streaks.

[0009] In many practical situations, one has access only to a voxel (or pixel) representation of CT data (e.g., in DICOM format) with no or minimal technical knowledge of the system that generated the CT data. Zhang et al. (2018) applied a type of convolutional neural network called U-Net to simulated contaminated (affected by several image distortions) and reference images to obtain an operator (trained with the U-net) for predicting residual images (i.e., the difference between these contaminated and reference images), thus aiming to mimic the reference images. The data were numerically simulated by inserting phantom metal into clean CT images of the abdomen. Their results on the simulated data appear promising, however application to real-world images did not exceed conventional methods due to the conventional MAR method used to create their 2D CT image reference dataset.

[0010] Therefore, from the above, it can be seen that there is a need in the art for improved methods and systems for automatically reconstructing 3D data (e.g., voxels) affected by artifacts. In particular, it is desired that artifacts caused by high-density materials in voxel representations of tissue (generated based on sensor data from a (CB) CT scanner) be automatically identified by such improved methods and systems and replaced by voxels that provide an accurate estimate of the voxel value in the absence of the artifact, while leaving voxels of high-density materials (e.g., titanium implants or composite fillers) intact. Summary of the invention

[0011] As will be appreciated by those skilled in the art, aspects of the present invention may be embodied as systems, methods or computer program products. Therefore, aspects of the present invention may take the form of a complete hardware embodiment, a complete software embodiment (including firmware, resident software, microcode, etc.) or an embodiment combining software and hardware aspects (which may generally be referred to herein as "circuit", "module" or "system"). The functions described in this disclosure may be implemented as algorithms executed by a microprocessor of a computer. In addition, aspects of the present invention may take the form of a computer program product embodied in one or more computer-readable media having a computer-readable program code embodied therein, such as stored therein.

[0012] Any combination of one or more computer-readable media can be utilized. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, but not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or apparatuses, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer-readable storage media will include the following: electrical connections with one or more wires, portable computer floppy disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In the context of this article, a computer-readable storage medium can be any tangible medium that can contain or store programs for use by or in conjunction with an instruction execution system, device or apparatus.

[0013] Computer readable signal media may include, for example, a propagated data signal in baseband or as part of a carrier wave, having computer readable program code embodied therein. Such propagated signals may take any of a variety of forms, including but not limited to electromagnetic, optical, or any suitable combination thereof. Computer readable signal media may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in conjunction with an instruction execution system, device, or apparatus.

[0014] The program code embodied in the computer readable medium can use any suitable medium to transmit, including but not limited to wireless, wired, optical fiber, cable, RF etc., or any suitable combination mentioned above. The computer program code for performing the multifaceted operation of the present invention can be written with any combination of one or more programming languages, and the programming language includes functions such as Java (TM), Scala, C++, Python etc. or object-oriented programming languages, and conventional program programming languages ​​such as "C" programming language or similar programming languages. The program code can be executed completely on the user's computer, partly on the user's computer, executed as an independent software package, partly on the user's computer and partly on the remote computer, or completely on the remote computer, server or virtual server. In the latter case, the remote computer can be connected to the user's computer by any type of network (including local area network (LAN) or wide area network (WAN)), or can be connected to an external computer (for example, using an Internet service provider through the Internet).

[0015] Aspects of the present invention are described below with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to an embodiment of the present invention. It will be understood that each frame of the flowchart and / or block diagram and the combination of frames in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, particularly a microprocessor, or a central processing unit (CPU) or a graphics processing unit (GPU) to produce a machine, so that the instructions are executed via the processor of the computer, other programmable data processing devices or other devices to create a component for implementing the function / action specified in the flowchart and / or block diagram frame or multiple frames.

[0016] These computer program instructions may also be stored in a computer-readable medium, which may direct a computer, other programmable data processing device, or other apparatus to operate in a specific manner so that the instructions stored in the computer-readable medium produce a manufactured product, which includes instructions for implementing the functions / actions specified in the flowchart and / or block diagram box or multiple boxes.

[0017] The computer program instructions may also be loaded onto a computer, other programmable data processing device, or other apparatus to cause a series of operational steps to be performed on the computer, other programmable device, or other apparatus to produce a computer-implemented process, such that the instructions executed on the computer or other programmable device provide a process for implementing the functions / actions specified in the flowchart and / or block diagram block or multiple blocks.

[0018] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention.In this regard, each frame in the flow chart or block diagram can represent a module, segment or part of a code, which includes one or more executable instructions for realizing the specified logical function.It should also be noted that in some alternative embodiments, the function indicated in the frame may not occur in the order indicated in the figure.For example, depending on the function involved, two frames shown in succession can actually be executed substantially simultaneously, or sometimes these frames can be executed in reverse order.It should also be noted that each frame of the block diagram and / or flow chart and the combination of frames in the block diagram and / or flow chart can be realized by a system based on special-purpose hardware or a combination of special-purpose hardware and computer instructions that performs a specified function or action.

[0019] An object of embodiments in the present disclosure is to automatically correct metal-affected voxel representations of X-ray data using deep learning techniques. Preferably, the X-ray data is represented as voxels that form a 3D image and is generated using a low-dose CT scanner such as a cone-beam CT (CBCT) scanner. The 3D image typically represents biological tissue, such as the maxillofacial region or other regions of the body, such as the thorax region. Here, the metal-affected X-ray data (e.g., voxels) can represent artifacts in the X-ray data caused by metal or metal objects in the tissue volume imaged by the X-ray imager.

[0020] In one aspect, the invention relates to a computer-implemented method for correcting a voxel representation of X-ray data affected by metal, comprising: a first 3D deep neural network receiving at its input an initial voxel representation of the X-ray data affected by metal and generating at its output voxel identification information, the voxel identification information identifying at least voxels of the initial voxel representation belonging to a region of voxels affected by metal; a second 3D deep neural network receiving at its input the initial voxel representation and the voxel identification information generated by the first 3D deep neural network, and generating at its output a corrected voxel representation, the corrected voxel representation comprising voxel estimates of voxels identified by the voxel map as being part of the region affected by metal, the first 3D deep neural network being trained based on training data and reference data, the training data and reference data comprising voxel representations of clinical X-ray data of a predetermined body part of a patient. In an embodiment, the X-ray data may be 3D (CB) CT data.

[0021] Thus, the first neural network is trained to automatically distinguish between voxels represented by voxels associated with metal-affected X-ray data (artifacts) and voxels that are not artifacts, i.e., voxels representing tissue of the scanned body part. This information can be generated at the output of the first neural network in the form of voxel identification information. This information can be used by the second neural network to replace the voxel values ​​of the voxel representation of the metal-affected X-ray data that are artifacts with voxel values ​​estimated by the second neural network based on clinical training data. In this way, the method can provide an effective method for removing artifacts from X-ray images, 3D (CB) CT image data, without requiring specific knowledge of the X-ray imager (imager is considered to be another term for scanner) or the back-projection method used to generate the image. In addition, the method does not require processing of the projection data in the sinogram domain.

[0022] In an embodiment, determining the identification information by the first 3D deep neural network comprises: the first 3D deep neural network generates a voxel map, each voxel of the voxel map being associated with a voxel of an initial voxel representation of the X-ray data affected by metal and one or more probability measures, wherein a first probability measure of the one or more probability measures represents a probability that the voxel is part of a metal region and a second probability measure represents a probability that the voxel is part of a region affected by metal. In this regard, a "voxel map" may be considered as (a subset and / or a portion of) the voxel identification information. In this regard, in the case of the first measure, a region should be considered to be one or more sets of voxels that are part of a real-world 3D volume or multiple real-world 3D volumes that are metal (or metal-containing), respectively, while in the case of the second measure, a region should be considered to be one or more sets of voxels that have incorrect values ​​determined in the initial voxel representation due to being affected by metal present in the real-world situation.

[0023] In an embodiment, the method may further include classifying voxels in the initial voxel representation into voxels belonging to metal regions and voxels belonging to regions affected by metal based on the first probability measure and the second probability measure and based on one or more thresholds.

[0024] In an embodiment, the method may further include: using the first probability measure and one or more metal thresholds to determine a metal category selected from a plurality of metal categories for each voxel, preferably, the plurality of metal categories including a first metal category associated with non-metallic materials and a second category of voxels associated with metal or metal-containing materials.

[0025] In an embodiment, the first 3D deep neural network and / or the second 3D deep neural network comprises at least a first data processing path comprising at least a first set of 3D convolutional layers, preferably a first set of 3D CNN feature layers, and a second data processing path that is at least parallel to the first path, the first data processing path comprising at least a first set of 3D convolutional layers, preferably a first set of 3D CNN feature layers, the second path comprising a second set of 3D convolutional layers, preferably a second set of 3D CNN feature layers, the second set of 3D convolutional layers being configured to determine context information associated with blocks of voxels of inputs fed to the 3D convolutional layers of the first set.

[0026] In an embodiment, the training data may include voxel representations of clinical X-ray data of a patient's body part before a metal-based treatment of the body part and voxel representations of clinical X-ray data of the body part of the patient after the metal-based treatment of the body part, and wherein the reference data includes voxel representations of clinical X-ray data of the patient's body part before the metal-based treatment, wherein metal regions associated with the metal-based treatment of the body part are so identified. As described above, (subsets and / or processed versions of) these reference data may be utilized to optimize the voxel map towards accurate generation, in effect optimizing the voxel identification information.

[0027] In an embodiment, a second 3D deep neural network may be trained to minimize artifacts associated with metal-affected X-ray data in an initial voxel representation, wherein, during training, the 3D deep neural network is parameterized by first weights and biases that are selected based on a relationship derived from one or more characteristics of the training data and one or more characteristics of a reference data set to optimize a corrected voxel representation.

[0028] In an embodiment, the first 3D deep neural network and / or the second 3D deep neural network may include and / or may be part of a 3D convolutional neural network, a 3D generative adversarial neural network, or a 3D recurrent deep neural network.

[0029] In one aspect, the present invention may relate to a computer-implemented method for training a deep neural network system to correct voxel representations of metal-affected X-ray data, the method comprising: a computer receiving training data and reference data, wherein the training data comprises voxel representations of clinical X-ray data of a patient's body part before a metal-based treatment and voxel representations of the clinical X-ray data of the patient's body part after the metal-based treatment, and wherein the reference data comprises voxel representations of the clinical X-ray data of the patient's body part before the metal-based treatment, wherein metal regions associated with the metal-based treatment are so identified; the computer receiving a voxel map associated with the voxel representations of the clinical X-ray data of the patient's body part after the metal-based treatment, the voxel map identifying metal-affected voxels in the voxel representations of the clinical X-ray data of the patient's body part after the metal-based treatment; and the computer training a 3D deep neural network using the training data and the reference data to generate voxel predictions for voxels classified as belonging to metal-affected regions by the voxel map, and correcting the metal-affected voxels in the voxel representation of the metal-affected X-ray data based on the voxel predictions.

[0030] In another aspect, the present invention relates to a method for training a neural network to process a voxel representation of metal-affected X-ray data using a training data set and a reference data set to produce voxel estimates of voxels that are part of a metal-affected region, the method comprising: generating a set of training data, the training data comprising a voxel representation of X-ray data of a patient's body part before metal-based treatment, a voxel representation of metal-affected X-ray data of the patient's body part after metal-based treatment, and a voxel map identifying whether a voxel in the voxel representation of metal-affected X-ray data belongs to a metal region and / or a metal-affected region; generating a reference data set, the reference data set comprising a voxel representation of the X-ray data of the patient's body part before metal-based treatment; training a generator neural network, preferably a generator 3D deep neural network, the generator neural network by deriving one or more characteristics of the training data and one or more characteristics of at least a portion of the reference data set; wherein the generator neural network is trained to generate a corrected voxel representation comprising voxel estimates for voxels that are part of an area affected by metal identified by the voxel map, wherein training the generator 3D deep neural network comprises modifying one or more of the first weights and biases to optimize the corrected voxel representation based on a relationship derived from one or more characteristics of the training data and one or more characteristics of the reference data set; and training a discriminator neural network, preferably a discriminator 3D deep neural network, parameterized by second weights and biases derived from one or more characteristics of the corrected voxel representation and one or more characteristics of at least a portion of the reference data set, the corrected voxel representation comprising voxel estimates for voxels that are areas affected by metal, wherein the discriminator 3D deep neural network is trained to discriminate voxel representations of X-ray data affected by metal from the processed voxel representations. Information derived from the discriminator 3D deep neural network may be utilized during optimization of the generator 3D deep neural network.

[0031] In yet another aspect, the present invention relates to a method for correcting a voxel representation of X-ray data affected by metal (preferably, 3D (CB) CT data), comprising: receiving an initial voxel representation representing the X-ray data affected by metal; generating a corrected voxel representation from the initial voxel representation by using a generator 3D deep neural network, the generator 3D deep neural network being trained to minimize artifacts associated with the X-ray data affected by metal in the initial voxel representation, the generator 3D deep neural network being parameterized by first weights and biases, the first weights and biases being selected to optimize the corrected voxel representation based on a relationship derived from one or more characteristics of training data and one or more characteristics of a reference data set, information generated from a discriminator 3D deep neural network having been utilized during the optimization and training of the generator 3D deep neural network.

[0032] In an embodiment, the generator 3D deep neural network and / or the discriminator 3D deep neural network may be configured as a 3D convolutional neural network and / or a 3D deep recurrent neural network.

[0033] In another aspect, the present invention may relate to a computer system adapted to correct a voxel representation of X-ray data (preferably, 3D (CB) CT data) affected by metal, the computer system comprising: a computer-readable storage medium having computer-readable program code embodied thereon, the program code comprising a preprocessing algorithm and at least one trained first 3D deep neural network, the computer-readable program code; and a processor, preferably a microprocessor, coupled to the computer-readable storage medium, wherein in response to executing the first computer-readable program code, the processor is configured to perform executable operations including: providing a first voxel representation of the X-ray data affected by metal to an input of the first 3D deep neural network, at an output of the first 3D deep neural network receiving voxel identification information at a first voxel representation, the voxel identification information identifying at least voxels of the first voxel representation belonging to the region of voxels affected by metal; providing a second 3D deep neural network, the second 3D deep neural network receiving at its input the first voxel representation and the voxel identification information generated by the first 3D deep neural network, and generating at its output a predicted voxel value for each voxel of the first voxel representation identified by the voxel identification information, the second 3D deep neural network predicting the predicted voxel value based on training data and reference data, the training data and reference data including voxel representations of clinical X-ray data of a predetermined body part of the patient; and determining a corrected first voxel representation, the corrected first voxel representation including the predicted voxel values ​​at the voxel locations identified by the voxel map.

[0034] The invention may also relate to a computer program product comprising software code portions configured to, when run in the memory of a computer, perform the method steps according to any of the process steps described above.

[0035] The invention will be further explained with reference to the accompanying drawings, which will schematically show embodiments according to the invention. It will be understood that the invention is not in any way limited to these specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A high-level overview of a system for automatically correcting voxel representations of X-ray data affected by metals according to an embodiment of the present invention is depicted;

[0037] Figure 2 Depicted is a system configured to automatically correct a voxel representation of X-ray data affected by metals in accordance with an embodiment of the present invention;

[0038] Figure 3 Depicted is a system for determining metal-affected regions in a voxel representation in accordance with an embodiment of the present invention;

[0039] Figure 4 Depicted is a method of training a 3D deep neural network to classify voxel representations of X-ray data affected by metals in accordance with an embodiment of the present invention;

[0040] Figure 5A and 5B depicts an example of a 3D deep neural network architecture for classifying voxel representations of X-ray data affected by metals in accordance with an embodiment of the present invention;

[0041] Figure 6 A system for correcting a voxel representation of X-ray data affected by metals is depicted in accordance with an embodiment of the present invention;

[0042] Fig. 7A and 7B Depicted is a method of training a 3D deep neural network to correct a voxel representation of X-ray data affected by metals according to an embodiment of the present invention;

[0043] Fig. 8A and 8B depicts an example of a voxel representation that may be used in embodiments of the present application;

[0044] Figures 9A-9C depicts voxel representations that may be utilized or generated by embodiments of the present disclosure;

[0045] Fig.10A schematic diagram of a system for correcting voxel representation of X-ray data affected by metal based on a generative adversarial network according to an embodiment of the present invention is depicted;

[0046] Fig.11 A schematic diagram depicting a discriminator portion of a generative adversarial network according to an embodiment of the present invention;

[0047] Fig. 12A and Fig. 12B A schematic diagram of a 3D deep recurrent neural network according to an embodiment of the present invention is depicted;

[0048] Fig.13 A block diagram illustrating an exemplary data processing system that may be used with embodiments described in this disclosure is depicted. DETAILED DESCRIPTION

[0049] Figure 1 A high-level overview of a system configured to automatically correct a voxel representation of X-ray data affected by metals is depicted in accordance with an embodiment of the present invention. Figure 1 As shown, the system includes a 3D deep neural network system 102, which includes an input for receiving a voxel representation 104 of X-ray data including one or more artifacts. The voxel representation of the X-ray data can be generated by different X-ray imagers (e.g., a CT imager or a CBCT imager). Artifacts are caused by metal or metal-containing objects, such as in the dental field, implants or dental fillings present in the tissue volume imaged by the X-ray imager. When the voxel representation of the X-ray data is generated based on the back-projection process, the voxel values ​​of certain voxels will be affected by the metal or metal-containing objects. The voxel values ​​affected by the metal can represent artifacts in the voxel representation of the X-ray data, which may seriously degrade the quality of the image. The 3D deep neural network system 102 is configured to automatically detect artifacts and generate a corrected voxel representation 106 of the X-ray data, in which the artifacts are removed or at least significantly reduced. In particular, the 3D deep neural network system 104 is configured to reduce the presence of metal artifacts to a level such that the artifacts do not interfere with a user's ability to clearly perceive different tissues and corresponding shapes, while retaining key information about the metal object (e.g., a titanium implant or composite filling) intact.

[0050] Figure 2A system configured to automatically correct a voxel representation of X-ray data affected by metal is depicted in accordance with an embodiment of the present invention. As shown in the figure, a system 202 may include a first system component 208 including a first 3D deep neural network 212 for identifying metal-affected voxels in a voxel representation 204 of the X-ray data. The metal-affected voxels may form a metal-affected region in the voxel representation - an artifact. The first system component 208 may be used as a pre-processor for a second system component 210 including a second 3D deep neural network. The first 3D deep neural network may process input data, i.e., a voxel representation of the X-ray data affected by metal, and generate output data that is input to a second system component including a second 3D deep neural network configured to correct voxel values ​​that are part of a metal-affected region. The first 3D deep neural network is trained to generate identification information for identifying whether at least a portion of the voxel values ​​of the voxels of the first voxel representation are part of a metal-affected region (artifact) and / or a metal or metal-containing region (e.g., an implant or a dental filling).

[0051] In an embodiment, the identification information may include a 3D voxel map, which may be in the form of a voxel representation whose size matches the voxel representation of the metal-affected X-ray data presented at the input of the first 3D deep neural network. In this way, each voxel in the voxel map may have a corresponding voxel in the voxel representation of the X-ray data. Each voxel of the voxel map may be associated with one or more probability metrics, which may be used to determine whether the voxel value of the corresponding voxel in the first voxel representation is a metal-affected area or part of a metal area. If one or more probability metrics are above a certain threshold or within a certain range, the system can determine that the voxel belongs to a metal area or an area affected by metal. If the voxel belongs to an area affected by metal, the system can determine that the voxel value should be corrected.

[0052] Thus, the first 3D deep neural network is configured to generate identification information, for example in the form of a 3D map object that locates a volume of interest in the voxel representation of the X-ray data belonging to a region affected by metal or a metal region. Furthermore, the first 3D deep neural network can be trained to recognize volumes in the voxel representation that contain metal or metal-containing materials and assign them a "material class", e.g., titanium, gold, amalgam, composite material, etc. Figure 2 As shown, the second 3D deep neural network receives at its input the identification information and a voxel representation of the X-ray data affected by the metal, and generates a corrected voxel representation of the X-ray data in which the artifacts are removed or at least significantly reduced.

[0053] Figure 3A system for determining metal-affected regions in a voxel representation according to an embodiment of the present invention is depicted. The system can be implemented as a computer system 302 configured to receive a voxel representation 304 of metal-affected X-ray data and determine metal-affected regions in the voxel representation using a trained 3D deep neural network configured to determine metal regions and / or metal-affected regions in the voxel representation. The voxel representation 304 of metal-affected X-ray data can include a 3D image data stack of a dento-maxillofacial structure, which includes tissue structures of a body part, such as jaw structures, tooth structures, and neural structures in the dento-maxillofacial structure, and includes metal or metal-containing structures (e.g., dental fillings and / or dental implants (parts)) present in the structure of the body part. The voxel representation of the X-ray data can include voxels, i.e., 3D spatial elements associated with voxel values ​​(e.g., grayscale values ​​or color values) that (at least) represent radiation intensity or radiodensity values. Such radiodensity values ​​can be measured in Hounsfield units (HU). The voxel representation may define a voxel space, i.e., a volume of voxels of a certain size. The voxel representation may include a 3D image data stack representing (CB)CT X-ray data in a predetermined format (e.g., a DICOM format or a derivative thereof). During scanning, the presence of metal in biological tissue will affect the X-ray data and the sensing of the X-ray data by the sensor of the X-ray imager. Therefore, the voxel values ​​derived from the X-ray data may be affected by the metal structure. Therefore, the voxel representation 304 of the metal-affected X-ray data includes metal-affected voxels that form one or more artifacts, which may seriously degrade the quality of the image data. This is a substantial problem in X-ray imaging, particularly in low-dose X-ray imaging such as CBCT X-ray scanning.

[0054] Figure 3 The system can be implemented as a reference Figure 2A system component in the described system (and can be considered a preprocessor). The system has a training mode and an inference mode. In the training mode, a 3D deep neural network 316 can be trained according to a predetermined training scheme so that the trained neural network can accurately classify voxels of the voxel representation of the X-ray data affected by metal into different categories of voxels (e.g., voxels associated with different metals, areas affected by metal, etc.). The deep neural network is trained to generate a voxel map 318 as an output, which includes voxels of classified metals and / or areas affected by metal. The voxel representation of the X-ray data affected by metal can be used to determine a training set. In the inference mode, a new voxel representation of the X-ray data affected by metal can be provided to the input of the trained 3D deep neural network, and in response, the neural network will generate a voxel map that identifies voxels of metal areas and areas affected by metal in the input data.

[0055] The 3D training data may include a voxel representation of the labeled metal-affected X-ray data. In addition, the voxel representation of the metal-affected X-ray data may be segmented 312 into a voxel representation of the metal region and a voxel representation of the metal-affected region 314. These data may be generated based on a manual segmentation process or based on an automatic segmentation process using, for example, a trained neural network. An automatic segmentation system for segmenting a voxel representation of X-ray data is described in the related European patent application 17179185.8 of the same applicant (entitled “classification and 3D modelling of 3D dento-maxillofacial structures using deep learning networks”), which is hereby incorporated by reference into the present application.

[0056] In CBCT scans, radiodensity measured in Hounsfield units (HU) is inaccurate because different areas in the scan appear at different grayscale values ​​depending on their relative location in the organ being scanned. HU measured from the same anatomical region by both CBCT and medical-grade CT scanners are not the same and are therefore unreliable for determining location-specific radiographically identified bone density. In addition, CBCT systems do not employ a standardized system for scaling the grayscale levels representing the reconstructed density values. As such, the values ​​are arbitrary and therefore cannot assess bone quality. In the absence of such standardization, it is difficult to interpret the grayscale levels or even compare the values ​​produced by different machines. For example, in a CBCT voxel representation of dentofacial structures, the teeth and jaw structures have similar densities, making it difficult for the computer to distinguish between a voxel belonging to a tooth and a voxel belonging to a jaw. In addition, CBCT systems are very sensitive to artifacts known as beam hardening, which produce dark streaks between two high-attenuation objects (e.g., metal or bone) surrounded by bright streaks.

[0057] In order to make the 3D deep neural network robust to variability present in, for example, current CBCT-type voxel representations, in an embodiment, the 3D training data may also include (high-resolution) 3D models of metal objects or non-metal objects that may appear in the voxel representation 304 of the metal-affected X-ray data. The 3D model may include a 3D surface mesh of a metal object and / or a 3D surface mesh of a non-metal object (e.g., a body part (bone tissue or teeth) in the X-ray data). Such a 3D surface mesh may be generated, for example, by a known optical (laser) scanner or an intraoral optical scanner for generating a 3D mesh of teeth. In some cases, the 3D surface mesh needs to be segmented to separate related structures. The segmentation module 324 may segment the 3D surface mesh (e.g., an intraoral scan of teeth) into individual segmented objects, e.g., a plurality of 3D surface meshes, each of which represents a tooth. The division of a 3D surface mesh into separate 3D surface meshes is a technique known in the art. In another step (not shown), the segmented surface mesh may be aligned with the segmented voxel representation. The (aligned) segmented surface mesh may then be converted 326 into a voxel representation, e.g., a binary voxel representation, wherein voxels representing the surface of the object (e.g., tooth) and voxels located within the object have their voxel values ​​set to a first value, e.g., 1, while voxels outside the object are set to a second value, e.g., 0. The voxel representation 328 of the 3D model thus obtained may be used as additional training data in order to train a deep neural network to accurately determine metal regions and metal-affected regions in the voxel representation of metal-affected X-ray data.

[0058] Figure 4A method of training a 3D deep neural network to classify voxel representations of X-ray data affected by metals according to an embodiment of the present invention is described. In particular, Figure 4 The training of a 3D deep neural network is depicted to classify a voxel representation 402 of metal-affected (CB) CT data of a dento-maxillofacial complex. As shown in the figure, a first 3D deep neural network can be trained based on a labeled metal-affected voxel representation, such as the voxel representation 402 of metal-affected (CB) CT data, and a labeled voxel representation derived (segmented) from the metal-affected voxel representation representing metal objects 404 (e.g., implants and fillings) and metal-affected regions (artifacts) 406. Additionally, in some embodiments, the 3D deep neural network can also be trained based on non-metallic objects 408 (teeth and bone structures) that are labeled as non-metallic objects.

[0059] The voxels of these voxel representations are labeled with respect to the material category to which the voxel belongs. In an embodiment, the categories of materials may include: metal, non-metal, metal affected (artifact), and non-metal (non-artifact). In addition, the voxel space of all these voxel representations is the same as the voxel space of the input to the first 3D deep neural network. The target training data represents a collection of 3D voxel maps, and there is a voxel map for each voxel representation of the (CB) CT data affected by metal. The 3D voxel map has a voxel space of the same size as the voxel representation of the input fed to the neural network, so that each voxel of the 3D voxel map corresponds to a voxel of the voxel representation of the input provided to the 3D deep neural network. Each voxel in the 3D voxel map is associated with voxel information, which indicates whether the corresponding voxel of the voxel representation at the input of the first 3D deep neural network is a metal object and / or part of an object affected by metal.

[0060] In an embodiment, the additional 3D data may be used to train the first 3D deep neural network. Figure 3 As described, a 3D deep neural network can be made robust to variability present in, for example, current CBCT-type voxel representations. This can be achieved by 3D training data derived from (high-resolution) 3D models of metallic objects or non-metallic objects. In an embodiment, a high-resolution 3D surface mesh 412 of metallic objects in (CB)CT data of a dento-maxillofacial complex (e.g., crowns and / or implants) can be generated using, for example, a 3D optical scanner, which is then converted into a voxel representation 418 of the metallic objects. The voxels of the voxel representation can be labeled to identify voxels representing a certain class of metal. Similarly, a labeled voxel representation 420 of non-metallic objects in (CB)CT data of a dento-maxillofacial complex (e.g., teeth and / or jaws) can be generated based on the 3D surface mesh 414.

[0061] In some embodiments, some 3D surface meshes of metallic or non-metallic objects may be the same objects depicted in the voxel representation 402 of the metal-affected (CB) CT data of the dental-maxillofacial complex. In this case, the segmented 3D surface meshes (e.g., predetermined teeth) may be aligned (overlaid) 416 to the segmented voxel representation 408 of the same teeth in the voxel representation of the metal-affected (CB) CT data of the dental-maxillofacial complex. The labeled voxel representations of metallic and non-metallic objects derived from the 3D surface meshes may be used as training data to train a first 3D deep neural network for classifying metal-affected regions and / or metallic regions. Such alignment may be performed by a separate 3D deep neural network.

[0062] In this way, the 3D deep neural network is trained to classify the voxels represented by the voxels in the metal area and the area affected by the metal, and generate a 3D voxel map indicating the classification of each voxel in the voxel representation. For each voxel, the 3D deep neural network can generate voxel information. In an embodiment, the voxel information may include a vector, which includes one or more probability measures. The probability measure provides information about the possibility that the voxel belongs to a certain category (e.g., a metal class). The metal material category can define many different metals, such as titanium, gold, amalgam, etc. During the training process, the deep neural network can learn that the voxels representing the metal-containing objects that can be found in the jaw / bone / teeth should be classified as metals. These metal-containing objects are part of the original image (stack) and should therefore remain intact. In addition, the neural network will learn that the voxels that are classified as metal and are located "outside" the voxels representing the metal or metal-containing objects (i.e., outside the boundaries of the metal object) should be classified as voxels affected by metal (i.e., voxels that are part of the artifact).

[0063] For example, there may be situations where the voxel value determined by the imager is a value that would be associated with metal. However, the voxel may be located outside of a volume that is known from the available information to be correctly classified as a metal voxel. In this case, a trained 3D deep neural network may determine that the voxel value (e.g., radiodensity measured in Hounsfield units (HU)) is a value affected by metal or metal-containing objects located near the voxel. In this case, a first 3D deep neural network may determine that the voxel should be classified as a voxel affected by metal, which needs to be determined by a second 3D deep neural network (e.g., Figure 2 as shown) for correction.

[0064] Figure 5A and 5B An example of a 3D deep neural network architecture for classifying a voxel representation of X-ray data affected by metals according to an embodiment of the present invention is depicted. Figure 5AAs shown, the network can be implemented using a 3D convolutional neural network (3D CNN). The convolution layer can use an activation function. Multiple 3D convolution layers can be used, wherein slight changes in the number of layers and their defining parameters (e.g., different activation functions, number and size of kernels) and additional functional layers (e.g., dropout layers) can be used in the implementation without losing the essence of the 3D deep neural network design.

[0065] like Figure 5A As shown, the network may include multiple convolution paths. For example, in an embodiment, the network may include at least a first convolution path associated with a first set 504 of 3D convolution layers, a second convolution path associated with a second set 506 of 3D convolution layers, and a third convolution path associated with a third set 508 of 3D convolution layers. The outputs of the convolution paths may be merged, and the results may be processed by a fully connected layer 510, which is configured to output the probability of each class for each voxel. A voxel representation 502 of the X-ray data, 3D CT image data affected by metal may be fed to the inputs of the first, second, and third convolution paths.

[0066] exist Figure 5B The functionality of each different convolutional path is shown in more detail in . As shown in the figure, voxels representing 3D image data are fed to the input of the neural network. These voxels can define a predetermined volume, which can be referred to as an image volume 521. The image volume can be divided into a set of first blocks 5201, and each 3D convolutional layer of the first path 5221 can perform a 3D convolution operation on the first block of voxels 5011 of 3D image data. During processing, for a set of layers 524, the output of one 3D convolutional layer is the input of a subsequent 3D convolutional layer. In this way, each 3D convolutional layer can generate a 3D feature map representing each portion of the 3D image data fed to the input. Therefore, a 3D convolutional layer configured to generate such a feature map can be referred to as a 3D CNN feature layer.

[0067] like Figure 5B As shown, the convolutional layer of the second path 5222 can be configured to process a set of second voxel blocks 5202 of 3D image data, wherein the second voxel block represents a downsampled version of the associated first voxel block, wherein the first voxel block and the second voxel block have the same central origin. The downsampling factor can be any suitable value, preferably, the downsampling factor is selected between 2 and 4, preferably 3.

[0068] The first path 5221 can define a first set of 3D CNN feature layers (e.g., 5-20 layers) 524, which are configured to process input data (e.g., a first voxel block at a predetermined position in the image volume) at a voxel resolution of the target (i.e., the voxels of the classified image volume). The second path can define a second set of 3D CNN feature layers (e.g., 5-20 layers) 526, which are configured to process a second voxel block, wherein each block in the second voxel block 5202 has the same center point as its associated block from the first voxel block 5201. However, these voxels are processed at a resolution lower than the resolution of 5201. Therefore, the second voxel block represents a larger volume than the first block in real-world dimensions. The second 3D CNN feature layer processes the voxels to generate a 3D feature map that includes information about the direct neighborhood of the associated voxels processed by the first 3D CNN feature layer. In this way, the second path enables the neural network to determine contextual information, i.e., information about the context of a voxel (e.g., its surroundings) of the input 3D image data presented to the neural network. A third path 5223 with a set of 3D convolutional layers 528 may be utilized, which represents an even larger context and a more highly downsampled portion 5203 of the input data. This downsampling factor may again be set to an appropriate value selected between 5 and 15, preferably 9 from the original input resolution.

[0069] although Figure 5B Depicted is a convolution path 522 including three 1-3 , but other embodiments including even more convolution paths are also contemplated. By using three paths or more convolution paths, 3D image data (input data) and contextual information about the voxels of the 3D image data can be processed in parallel. Contextual information is important for classifying complex voxel representations (e.g., dentofacial structures), which often include closely packed and / or similar tooth structures that are difficult to distinguish without a larger context region.

[0070] Multiple 3D CNN feature layers can be trained (through their learnable parameters) to derive and pass on the best useful information that can be determined from their specific inputs, and the fully connected layer 532 can encode the parameters that will determine how the information from the three previous paths should be combined to provide the best probability of a classified voxel 534. Thereafter, the probabilities 536 can be presented in the image space 538 of the output, which can have the dimensions of the image space of the input. Thus, the output of the 3D deep neural network is the classification probability of each voxel in the image space corresponding to the image space of the voxel at the input.

[0071] By minimizing a loss function, where the loss function represents the deviation between the output of the 3D deep neural network and the target data (i.e., classified voxel data) representing the desired output for a predetermined input (i.e., the training set), an optimization method can be used to learn the optimal values ​​of the network parameters of the 3D deep neural network. When the minimization of the loss function converges to a certain value, the training process can be considered suitable for the application. The activation functions of the various layers may be different and can be set to, for example, linear, sigmoid, tangent and / or ReLu.

[0072] Figure 6 A system for correcting a voxel representation of X-ray data affected by metal is depicted according to an embodiment of the present invention. In particular, the figure depicts a computer system 602 for correcting a voxel representation of X-ray data affected by metal using a 3D deep neural network 612 that is trained to correct regions affected by metal in the voxel representation. The system has a training mode and an inference mode. In the training mode, the 3D deep neural network 612 can be trained according to a predetermined training scheme so that the trained neural network can accurately predict the voxel values ​​of voxels in the voxel representation affected by metal. The deep neural network is trained to generate a voxel representation 638 of X-ray data in which regions affected by metal are corrected. In the inference mode, a new voxel representation of X-ray data affected by metal can be provided to the input of the trained 3D deep neural network, and in response, the neural network will generate a voxel representation that is similar to the voxel representation at the input except that the voxels affected by metal are corrected. Thus, the 3D deep neural network is trained so that the output provides an accurate approximation of the original 3D image without metal-induced artifacts.

[0073] The 3D training data may include a labeled voxel representation of a patient before and after treatment, in particular, for example in the dental field, a labeled voxel representation of a patient before and after metal-based treatment (e.g., implant placement, dental restoration, orthodontic appliance, bridge placement, root canal filling, root post, bone plate and screw). In other words, it includes a first labeled voxel representation 603 of a part of a patient before treatment (i.e., before a metal object is implanted into the patient's body part) and a second labeled voxel representation 604 of the same part of the same patient after treatment (i.e., after the metal object is implanted into the patient's body part). In addition, in an embodiment, the training data may also include a voxel map 614 of the voxel representation 604 of X-ray data, the X-ray data including metal areas and metal-affected areas generated by the first deep neural network. The voxel map is used to train a second 3D deep neural network 612 to identify which voxels are associated with metal-affected areas (artifacts) that need to be corrected.

[0074] In an embodiment, an alignment process 611 may be applied to the training data, i.e. the labeled voxel representations 603, 604 and, where applicable, the voxel map 614, before the training data is fed to the input of the 3D deep neural network. In this alignment process, structures in the voxel representations may be aligned relative to each other. This may be performed manually or automatically. In the field of image registration, various methods are known that allow for automatic alignment, such as keypoint detection-based methods, intensity-based methods, etc.

[0075] The training data may also include target data, which includes a target voxel representation of X-ray data of a region of a patient, wherein the voxel representation includes metal objects (due to a metal-based treatment), but wherein there are no areas affected by metal. Such a target voxel representation may be constructed based on voxel representations of X-ray data before and after treatment and a voxel map that identifies metal areas and areas affected by metal. In particular, the voxel map may be used to identify voxels of metal areas in the post-treatment voxel representation. These voxels may be appropriately represented (inserted) in the pre-treatment voxel representation, thereby generating a realistic ground truth based on the clinical data. Reference Fig. 7A An example of generating such target data is described in more detail.

[0076] For each patient, a set of training data including the above input data and target data is generated. The training data is then used to train a 3D deep neural network to correct the metal-affected areas identified by the voxel map. An example of training a 3D deep neural network is described in more detail below with reference to FIGS. 7-10.

[0077] During training, the 3D deep neural network will learn based on the clinical training data to generate true voxel predictions for voxels in the metal-affected voxel representation. The 3D deep neural network will further learn to generate a voxel representation of the metal-affected X-ray data, wherein the metal-affected voxels are replaced with the voxel predictions. In this way, a voxel representation of the metal-affected X-ray data is generated, wherein the voxels associated with the metal-affected regions are corrected based on the voxel predictions generated by the 3D deep neural network.

[0078] Fig. 7A and 7B A method of training a 3D deep neural network to correct a voxel representation of X-ray data affected by metals according to an embodiment of the present invention is described. In particular, Fig. 7AA schematic diagram of training a 3D deep neural network to classify voxel representations of metal-affected (CB) CT data of a dental and maxillofacial complex is depicted. As shown in the figure, a first 3D deep neural network can be trained based on training data obtained from a dental and maxillofacial complex 702 of a patient (i.e., X-ray images of the patient before treatment 706 and after treatment 704), the training data including labeled voxel representations 712 of X-ray data representing dental and maxillofacial structures including metal areas and metal-affected areas, voxel representations 714 representing dental and maxillofacial structures without metal areas and metal-affected areas, and voxel maps 715 identifying voxels of metal and / or metal-affected areas in the voxel representations 712. In some embodiments, the 3D deep neural network can also be trained based on voxel representations 710 of metal-containing objects used during treatment of the patient. 3D surface meshes 708 of such metal-containing objects can be generated using a 3D optical scanner, and voxel representations are derived from each 3D mesh, wherein the voxel representations have the same dimensions as other voxel representations used as training data.

[0079] When applicable, the various representations of the 3D data, such as voxel representations and / or surface meshes, can be aligned again to properly coincide in the same voxel space. This can be done manually or automatically 711 1,2,3 .

[0080] In addition, accurate and realistic target data can be generated based on images (stacks) and voxel maps of pre- and post-treatment X-ray data. Figure 7B This process is depicted in more detail in FIG. 10 . As shown in this figure, the target voxel representation can be obtained from the aligned 721 images of the patient before and after treatment. 1,2 Voxel representation and voxel map 720 are derived. Voxel map 720 can be used to identify voxels belonging to metal regions in voxel representation 722 of metal-affected X-ray data. Metal extractor module 726 can be used to generate voxel representation of metal regions 728, which can have the same size as the voxel representation of metal-affected X-ray data. This voxel representation can be merged 730 with the voxel representation of the X-ray data of the patient before treatment into a target voxel representation of the X-ray data of the maxillofacial structure including the metal regions but not including the metal-affected voxel regions (artifacts) 732.

[0081] Training a 3D deep neural network based on clinical training data will result in a trained 3D deep neural network that can generate realistic voxel-by-voxel predictions for voxels in the voxel representation affected by metal.

[0082] Fig. 8A and 8B Depicts an example of a voxel representation that can be used in embodiments of the present application. In particular, Fig. 8AAn overview of a 3D mesh 802 of a dentofacial structure (in this case a collection of teeth) and a voxel representation 804 of a portion of a metal-treated tooth of a portion of the 3D mesh (DICOM 3D data set) is depicted, wherein some of the voxels are associated with metal-affected X-ray data. For the purposes of this figure, the voxel representation is located within the 3D mesh so that it coincides with the portion of the 3D mesh to which it belongs. Figure 8B Shown is a magnified portion of the same voxel representation 808 and the 3D mesh of the tooth 806 to which it belongs. This figure clearly shows the dimensions of the voxel representation describing the sample size (64 voxels in width, length and height in the case of this particular example) that can be used to train the neural network described in the present disclosure.

[0083] Figures 9A-9C Depicted are voxel representations that may be utilized or generated by embodiments of the present disclosure. The voxel representation may involve Fig. 8A and 8B The same parts of the dentofacial structure are depicted. In particular, Fig. 9A Depicted is a voxel representation including metal regions and regions affected by metal, which can be used as input data for a system for automatically correcting a voxel representation of X-ray data affected by metal. In this figure, a block of voxels has been removed so that the "interior" of the voxel representation can be seen. Fig. 9A In the figure, the white area 9061 represents the metal area, and the multiple black areas 9062 surrounding the (white) metal area represent the areas affected by the metal.

[0084] Fig. 9B A voxel representation 808 is depicted again with the voxel block having been omitted. For this voxel block, as shown in reference Figure 3 As described above, a voxel map has been generated by a 3D deep neural network. Fig. 9B , and includes voxels of classified metal regions 9101 and voxels of metal-affected regions 9102 that need to be corrected by another 3D deep neural network. Finally, Fig. 9C A voxel representation of metal-affected X-ray data of a patient is depicted, wherein metal-affected regions 9142 are corrected while metal regions 9141 are preserved. Correction is achieved via a 3D deep neural network trained to correct metal-affected regions as described in reference Figure 6 as described.

[0085] Different neural network architectures may be used in embodiments of the present disclosure. Fig.10A schematic diagram of a system for correcting a voxel representation of X-ray data affected by metal based on a generative adversarial neural network according to an embodiment of the present invention is depicted. As shown in the figure, a 3D deep neural network 1006 that needs to be trained to correct the voxels of the area affected by metal is called a generator network. During training, the generator network receives a voxel representation 1002 of the X-ray data affected by metal and an associated voxel map (possibly generated by a 3D deep neural network 1004, which is configured to determine the metal area and / or the area affected by metal in the voxel representation). In addition, the generator network can receive feedback 1010 from a discriminator network 1012, which either receives a target training voxel representation of X-ray data representing a dental and maxillofacial structure including a metal area, i.e., an image (stack) that conforms to a "real life" situation, which can then be considered to be the same as a "target voxel representation" 732, 1014. It is fed to a second 3D deep neural network as target training data, or it receives a voxel representation generated by the generator network 1008, i.e., a voxel representation in which the area affected by metal is corrected. The main task of the discriminator network is to distinguish whether the voxel representation it receives as input is the corrected output voxel representation of the generator network or the target voxel representation 732, 1014. In response, the discriminator receives feedback if the output it has given is correct, i.e., if the voxel representation is indeed "corrected" or "true" as indicated by the discriminator. This information helps the generator network to effectively learn to replace the voxels affected by metal with the true voxel predictions of those regions, because both the generator and the discriminator learn from the given feedback.

[0086] Fig.11 The discriminator network is depicted in more detail. Here, the discriminator network 1100 is configured to distinguish between a target voxel representation and a corrected voxel representation of the voxels of the metal-affected area. Therefore, during training, the discriminator network can be presented (randomly) with a target voxel representation or an output voxel representation of the generator network 1008. In response, the discriminator network generates an indication of whether the voxel representation at the input of the discriminator is a "real" target voxel representation or a corrected voxel representation created by the generator network. The discriminator network will then generate a "score" indicating whether the presented voxel representation is real or corrected. For example, in an embodiment, the discriminator can have a binary output 1106, where a "1" can indicate that the voxel representation is the target representation 1102, and a "0" can indicate that the voxel representation is a corrected voxel representation 1104. If the generator network generates a corrected voxel representation during training, the discriminator can distinguish it from the target voxel representation, which will be a signal to the system that such a corrected voxel representation is not "real" enough. Thus, in this way, the generator network will be motivated to produce corrected voxel representations that are similar to the training set (e.g., as in reference Fig. 7A and 7B The target voxel representation of the clinical training data).

[0087] The more realistic the voxel predictions generated by the generator network, the more challenging it is for the discriminator to distinguish the two. Therefore, the ability of the discriminator to distinguish the two is a measure of the quality of the voxel corrections generated by the generator. This information can be fed back to the discriminator network 1100 as well as the generator network 1110 via backpropagation 1108. In this way, the generator is trained to generate accurate voxel predictions of voxels in areas affected by metal. The deep neural network 1102 representing the generator network of the GAN can be any type of 3D deep neural network, including a (deep) convolutional neural network or a recurrent neural network.

[0088] Fig. 12A and 12B A schematic diagram of a 3D deep recurrent neural network according to an embodiment of the present invention is depicted. Recurrent neural networks are a class of networks that are particularly suitable for predicting pixel values ​​based on long-term dependencies between pixels in an image or between a sequence of images (of a video or 3D data stack). This is also true for the 3D case, where long-term dependencies between voxels in a voxel representation can be used to predict voxels.

[0089] Fig. 12A A deep recurrent neural network is shown, comprising a voxel input 1202, a 3D deep neural network (e.g., a 3D convolutional neural network), comprising one or more internal hidden states parameterized by weights and biases, and an output 1206, wherein layers are added that take into account the weights and biases and (a portion of) the output of one or more previous states. The number of previous states and their corresponding weights and biases are considered to calculate the output of a subsequent state, and the number of previous states and their corresponding weights and biases can indicate the depth of the recurrent neural network in terms of the number of layers, for example, Fig. 12B The deep recurrent neural network shown consists of three layers. Fig. 12B The deep recurrent neural network scheme is shown in more detail in Fig. 12B As shown, enter x i-1 , for example, the first block of input voxels surrounding the voxels for which voxel value prediction is required (e.g., the metal-affected area that needs correction), outputs x i-1 is the input to the recurrent neural network, which determines the output x based on the weights and bias of the current state and the weights and bias of the initial state 12101 i-1 (The first voxel value prediction is, for example, the value in Hounsfield units.) The next input x iThe second block of input voxels (surrounding the next voxel that needs to be predicted) can be processed by the neural network based on the weights and biases 12102 of the previous state and the first block of output voxels 12121 to generate a second block of output voxels including the second voxel that needs voxel value prediction. Thereafter, the next input x i+1 (The third block of input voxels) can be processed by the neural network based on the weights and biases 12103 of the previous state and the second block of output voxels 12122 to generate a third block of output voxels, and so on.

[0090] Fig.13 1300 is a block diagram illustrating an exemplary data processing system that can be used in the embodiments described in the present disclosure. Data processing system 1300 may include at least one processor 1302 coupled to memory element 1304 via system bus 1306. In this way, the data processing system can store program codes in memory element 1304. In addition, processor 1302 can execute program codes accessed from memory element 1304 via system bus 1306. On the one hand, the data processing system can be implemented as a computer suitable for storing and / or executing program codes. However, it should be understood that data processing system 1300 can be implemented in the form of any system including a processor and a memory capable of performing the functions described in this specification.

[0091] Memory element 1304 may include one or more physical memory devices, such as, for example, local memory 1308 and one or more mass storage devices 1310. Local memory may refer to random access memory or other non-persistent memory devices commonly used during the actual execution of program code. Mass storage devices may be implemented as hard drives or other persistent data storage devices. Data processing system 1300 may also include one or more cache memories (not shown) that provide temporary storage of at least some program code in order to reduce the number of times program code must be retrieved from mass storage device 1310 during execution.

[0092] Input / output (I / O) devices, depicted as input devices 1313 and output devices 1314, may optionally be coupled to the data processing system. Examples of input devices may include, but are not limited to, for example, a keyboard, a pointing device such as a mouse, and the like. Examples of output devices may include, but are not limited to, for example, a monitor or display, a speaker, and the like. The input devices and / or output devices may be coupled to the data processing system directly or through an intermediate I / O controller. A network adapter 1316 may also be coupled to the data processing system to enable it to be coupled to other systems, computer systems, remote network devices, and / or remote storage devices through an intermediate private or public network. The network adapter may include: a data receiver for receiving data transmitted to the data by the system, device, and / or network; and a data transmitter for transmitting data to the system, device, and / or network. Modems, cable modems, and Ethernet cards are examples of different types of network adapters that may be used with the data processing system 1350.

[0093] like Fig.13 As shown, memory element 1304 can store application 1318. It should be understood that data processing system 1300 can also execute an operating system (not shown) that can facilitate the execution of application programs. Application programs implemented in the form of executable program code can be executed by data processing system 1300, for example, by processor 1302. In response to executing the application program, the data processing system can be configured to perform one or more operations that will be further described in detail herein.

[0094] In one aspect, for example, data processing system 1300 may represent a client data processing system. In this case, application 1318 may represent a client application that, when executed, configures data processing system 1300 to perform the various functions described herein with reference to a "client." Examples of a client may include, but are not limited to, a personal computer, a portable computer, a mobile phone, and the like.

[0095] In another aspect, the data processing system may represent a server. For example, the data processing system may represent a (HTTP) server, in which case the application 1318, when executed, may configure the data processing system to perform (HTTP) server operations. In another aspect, the data processing system may represent a module, unit, or function referred to in this specification.

[0096] The terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used herein, the singular forms "one", "a (kind)" and "the (said)" are also intended to include plural forms, unless the context clearly indicates otherwise. It will be further understood that when the terms "include" and / or "comprise" are used in this specification, it is indicated that the features, wholes, steps, operations, elements and / or components are present, but it does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof.

[0097] All parts or steps in the present invention plus the corresponding structure, material, effect and equivalent of functional element are intended to include any structure, material or effect for performing functions in combination with other elements claimed for protection specifically claimed. Description of the present invention has been given for the purpose of illustration and description, but it is not intended to be exhaustive or limit the present invention to the disclosed form. Without departing from the scope and spirit of the present invention, many modifications and variations will be obvious to those of ordinary skill in the art. Selecting and describing embodiments is to best explain the principles and practical applications of the present invention, and to enable other persons of ordinary skill in the art to understand various embodiments of the present invention and various modifications suitable for intended specific purposes.

Claims

1. A computer-implemented method for correcting a voxel representation of metal-affected X-ray data, the metal-affected X-ray data representing artifacts in the X-ray data caused by metal or metal-containing objects in a volume of tissue imaged by an X-ray imager, the method comprising: a first 3D deep neural network receiving at its input an initial voxel representation of X-ray data affected by metal and generating at its output a voxel map identifying voxels of the initial voxel representation belonging to a region of voxels affected by metal; as well as A second 3D deep neural network receives at its input the initial voxel representation and the voxel map generated by the first 3D deep neural network, and generates a corrected voxel representation comprising voxel estimates for voxels that are part of the area identified by the voxel map as being affected by metal, the first 3D deep neural network being trained based on training data and reference data comprising voxel representations of clinical X-ray data of a predetermined body part of a patient.

2. The method according to claim 1, wherein: The identification information determined by the first 3D deep neural network includes: The first 3D deep neural network generates a voxel map, each voxel of the voxel map is associated with a voxel of an initial voxel representation of the metal-affected X-ray data and a plurality of probability measures, wherein a first probability measure of the plurality of probability measures represents a probability that the voxel is part of a metal region and a second probability measure represents a probability that the voxel is part of a region affected by metal.

3. The method according to claim 2, further comprising: Based on the first probability measure and the second probability measure and based on one or more thresholds, voxels in the initial voxel representation are classified into voxels belonging to metal regions and voxels belonging to regions affected by metal.

4. The method according to claim 2 or 3, further comprising: A metal class selected from a plurality of metal classes is determined for each voxel using the first probability measure and one or more metal thresholds. 5 . The method of claim 4 , wherein the plurality of metal classes comprises a first metal class associated with non-metallic materials and a second class of voxels associated with metal or metal-containing materials.

6. The method according to claim 1, wherein: The first 3D deep neural network and / or the second 3D deep neural network comprises a first data processing path and a second data processing path parallel to the first data processing path, wherein the first data processing path comprises a first set of 3D convolutional layers, and the second data processing path comprises a second set of 3D convolutional layers, wherein the second set of 3D convolutional layers are configured to determine context information associated with blocks of voxels fed to inputs of the first set of 3D convolutional layers.

7. The method according to claim 1, wherein: The training data includes voxel representations of clinical X-ray data of a patient's body part before metal-based treatment of the body part and voxel representations of clinical X-ray data of the patient's body part after metal-based treatment of the body part, and wherein the reference data includes voxel representations of clinical X-ray data of the patient's body part before metal-based treatment, and wherein metal regions associated with the metal-based treatment of the body part are identified in the voxel representations included in the reference data.

8. The method according to claim 1, wherein: The second 3D deep neural network is trained to minimize artifacts associated with metal-affected X-ray data in the initial voxel representation, wherein, during training, the second 3D deep neural network is parameterized by first weights and biases, and the first weights and the biases are selected to optimize a corrected voxel representation based on a relationship derived from one or more characteristics of the training data and one or more characteristics of a reference data set.

9. The method according to claim 1, wherein: At least one of the first 3D deep neural network and the second 3D deep neural network includes a portion of a 3D convolutional neural network, a 3D generative adversarial neural network, or a 3D recursive deep neural network, or at least one of the first 3D deep neural network and the second 3D deep neural network is a portion of a 3D convolutional neural network, a 3D generative adversarial neural network, or a 3D recursive deep neural network.

10. A computer-implemented method for training a deep neural network system to correct a voxel representation of metal-affected X-ray data, the metal-affected X-ray data representing artifacts in the X-ray data caused by metal or metal-containing objects in a volume of tissue imaged by an X-ray imager, the method comprising: The computer receives training data and reference data, wherein the training data includes a voxel representation of clinical X-ray data of a patient's body part before a metal-based treatment and a voxel representation of the clinical X-ray data of the patient's body part after the metal-based treatment, and wherein the reference data includes a voxel representation of the clinical X-ray data of the patient's body part before the metal-based treatment, wherein metal regions associated with the metal-based treatment are identified in the voxel representation included in the reference data; The computer receives one or more voxel maps associated with the voxel representation of clinical X-ray data of the patient's body part after the metal-based treatment, the voxel maps identifying metal-affected voxels in the voxel representation of the clinical X-ray data of the patient's body part after the metal-based treatment; and The computer trains a 3D deep neural network using the training data and the reference data to generate voxel predictions for voxels classified by the voxel map as belonging to a metal-affected region, and corrects the metal-affected voxels in the voxel representation of the metal-affected X-ray data based on the voxel predictions.

11. A method of training a neural network to process a voxel representation of X-ray data affected by metal, the method comprising: generating training data comprising one or more voxel representations of X-ray data of a patient's body part before metal-based treatment, one or more voxel representations of metal-affected X-ray data after metal-based treatment of the patient's body part, and one or more voxel maps, each of the one or more voxel maps identifying whether a voxel in the voxel representation of the metal-affected X-ray data belongs to a metal region and / or a region affected by metal; generating reference data comprising a voxel representation of X-ray data of a body portion of a patient prior to metal-based treatment; training a generator neural network parameterized by first weights and biases derived from one or more characteristics of the training data and one or more characteristics of the reference data, wherein the generator neural network is trained to receive at its input an initial voxel representation of the metal-affected X-ray data and to generate at its output a corrected voxel representation comprising voxel estimates for voxels in the initial voxel representation of voxels identified by the voxel map as being part of a region affected by metal, wherein training the generator neural network comprises: modifying one or more of the first weights and biases to optimize the corrected voxel representation based on a relationship derived from the one or more characteristics of the training data and the one or more characteristics of the reference data; and training a discriminator neural network parameterized by second weights and biases derived from one or more characteristics of the corrected voxel representation and one or more characteristics of the reference data, the corrected voxel representation comprising voxel estimates for voxels that are part of a region affected by metal, wherein the discriminator neural network is trained to distinguish between a voxel representation of x-ray data representing a body part of a patient including a metal region and the corrected voxel representation generated by the generator neural network; and Information generated from the discriminator neural network is utilized during training of the generator neural network.

12. A method of correcting a voxel representation of X-ray data for metal effects, comprising: receiving an initial voxel representation representing metal-affected X-ray data representing artifacts in the X-ray data caused by metal or metal-containing objects in a volume of tissue imaged by an X-ray imager; receiving a voxel map, the voxel map identifying voxels represented by the initial voxel that belong to a region of voxels affected by metal; generating a corrected voxel representation based on the initial voxel representation and the voxel map using a generator 3D deep neural network trained to minimize artifacts associated with metal-affected X-ray data in the initial voxel representation based on training data and reference data, the training data and the reference data comprising a voxel representation of clinical X-ray data of a predetermined body part of a patient and one or more voxel maps, each of the one or more voxel maps identifying whether a voxel in the voxel representation of the metal-affected X-ray data belongs to a metal region and / or a region affected by metal, wherein, during training, the generator 3D deep neural network is parameterized by first weights and biases, the first weights and biases being selected based on a relationship derived from one or more characteristics of the training data and one or more characteristics of the reference data and based on information generated from a discriminator 3D deep neural network during training of the generator 3D deep neural network to optimize a corrected voxel representation, the discriminator 3D deep neural network being trained to distinguish between a voxel representation including metal regions of X-ray data representing a body part of a patient and a corrected voxel representation generated by the generator 3D deep neural network.

13. The method according to claim 12, wherein: At least one of the generator 3D deep neural network and the discriminator 3D deep neural network is configured as at least one of a 3D convolutional neural network and a 3D deep recurrent neural network.

14. A computer system adapted to correct a voxel representation of metal-affected X-ray data, the metal-affected X-ray data representing artifacts in the X-ray data caused by metal or metal-containing objects in a volume of tissue imaged by an X-ray imager, the system comprising: A computer-readable storage medium containing a computer-readable program code, wherein the program code includes a preprocessing algorithm and a trained first 3D deep neural network; as well as a processor coupled to the computer-readable storage medium, wherein in response to executing the computer-readable program code, the processor is configured to perform executable operations including: providing an initial voxel representation of the X-ray data affected by metal to an input of a first 3D deep neural network trained to generate a voxel map at its output, the voxel map identifying voxels of the initial voxel representation that belong to a region of voxels affected by metal; The initial voxel representation and the voxel map generated by the first 3D deep neural network are provided to inputs of a second 3D deep neural network, the second 3D deep neural network being trained to generate a corrected voxel representation comprising voxel estimates for voxels that are part of an area identified by the voxel map as being affected by metal, the first 3D deep neural network being trained based on training data and reference data comprising voxel representations of clinical X-ray data of a predetermined body part of a patient.

15. A computer-readable storage medium comprising software code portions configured to, when executed in a memory of a computer, perform the method steps according to any one of claims 1 to 13.

Images