Network Architecture for 3D Image Processing

Abstract

A mechanism for processing input 3D image data. In a first phase, the input 3D image data is processed separately using one or more neural networks to generate one or more modified 3D image data. In a second phase, the input 3D image data and the modified 3D image data are processed using neural networks to generate an output. The neural networks that generate the modified 3D image data process slices or sub-volumes of the input 3D image data to generate the modified 3D image data.

Description

[Technical field]

[0001] The present invention relates to the field of neural network architectures, and in particular to the processing of 3D image data using neural network architectures. [Background technology]

[0002] There is growing interest in the automated processing of images, particularly 3D images. There are many different industries that could benefit from improved 3D image processing, but one area of ​​interest is in the healthcare industry. 3D medical images are becoming more prevalent as they are generated by 3D ultrasound, CT scans, or MRI scans, which are becoming more widely used. Automated processing of such images can assist clinicians in evaluating and / or diagnosing the imaged subject, performing clinical support tasks, and the like.

[0003] Examples of image processing tasks that may be performed by appropriately configured neural network architectures processing 3D (medical) images include object detection, object segmentation, noise removal, artifact removal, super-resolution, and / or synthesis or prediction of other imaging modality results.

[0004] Various neural network architectures have been developed for the analysis of (medical) images. One of the most successful architectures is the U-Net architecture. Summary of the Invention [Problem to be solved by the invention]

[0005] There is a continuing desire to improve the performance of neural network architectures that process 3D images.

[0006] US Patent Application Publication No. 2020 / 058126 discloses a method for image segmentation of 3D images using a fully convolutional neural network operating on a series of 2D slices. [Means for solving the problem]

[0007] The invention is defined by the claims.

[0008] According to an embodiment of an aspect of the present invention, a computer-implemented method for processing 3D image data using a neural network architecture is provided.

[0009] The computer-implemented method includes steps of obtaining input 3D image data; performing a modified 3D image data generation process including, for each of a plurality of 3D sub-volumes of the input 3D image data, processing the 3D sub-volume using a first neural network to generate first modified 3D image data, and for each 2D image slice along a first dimension of the input 3D image data, processing the 2D image slice using a second neural network to generate second modified 3D image data; and processing the input 3D image data together with at least the first modified 3D image data and the second modified 3D image data using an output neural network to generate an output of the neural network architecture.

[0010] The proposed approach provides a mechanism for improving the analysis or processing of 3D image data, i.e., volumetric image processing. The proposed approach facilitates the use of more contextual information to perform 3D image analysis by incorporating a wider range of data that is processed virtually simultaneously, thereby circumventing the natural limitations on the size of the receptive field for neural network processing.

[0011] It is recognized that many neural networks in use today were originally designed for 2D image processing. When moving to 3D, these neural networks begin to suffer from an inherent "curse of dimensionality" phenomenon. In particular, in the processing of 3D images, this is manifested by a limitation in the size of the network's possible receptive field, which places an upper limit on the amount of context that can be utilized when processing different parts of a 3D image. This limited context limits the capabilities of the model, especially in processing systems with limited memory and processing power.

[0012] The proposed approach improves the amount of context available when processing input 3D image data using a neural network (here, the output neural network), thereby improving the performance of the image processing tasks performed by the neural network architecture.

[0013] The input 3D image data may comprise medical 3D image data. Due to at least the increasing prevalence and clinical reliance of medical 3D image data, embodiments are particularly advantageous when used to process medical 3D image data.

[0014] Preferably, the resolution of the input 3D image data and any modified 3D image data processed by the output neural network is the same.

[0015] The step of processing the input 3D image data together with the second modified 3D image data and the first modified 3D image data includes the steps of: for each 2D image slice of the input 3D image data along a second dimension, different from the first dimension, obtaining a corresponding 2D image slice from the first modified 3D image data and the second modified 3D imaging data; and processing the 2D image slice of the input 3D image data together with at least the corresponding 2D image slice from the first modified 3D image data and the corresponding slice from the second modified 3D imaging data using an output neural network to generate processed 2D image slices, wherein the combination of the processed 2D image slices forms an output of the neural network architecture.

[0016] In this approach, when processing a slice along a particular dimension (the second dimension), we provide context that is available from at least one other dimension, in particular data that would otherwise only be available to other slices along the particular dimension.

[0017] Thus, for each slice of a 3D image of a particular dimension, information from other slices of the particular dimension can contribute to the processing of that slice, as this information influences or contributes to corresponding slices from the first and second modified 3D image data, thereby increasing the effective receptive field of the processed 2D image slice, thereby improving the performance of the overall neural network architecture.

[0018] The input 3D image data may be a medical image of a subject, with each 2D image slice along a first dimension being a slice in a coronal or sagittal plane of the subject, and each 2D image slice along a second dimension being a slice in an axial plane of the subject, although embodiments are not limited in this respect.

[0019] Each of the first, second, and output neural networks may have a U-net architecture, which is recognized as offering better or superior image processing capabilities (over other architectures), particularly in the medical field.

[0020] In some embodiments, the modified 3D image data generation process includes, for each 2D image slice along a first dimension of the input 3D image data, processing the 2D image slice using a second neural network to generate second modified 3D image data, and, for each of a plurality of 3D sub-volumes of the input 3D image data, processing the 3D sub-volume using a first neural network to generate first modified 3D image data.

[0021] Correspondingly, the step of processing the input 3D image data together with at least the first modified 3D image data and the second modified 3D image data may include a step of processing the input 3D image data together with at least the first modified 3D image data and the second modified 3D image data using an output neural network to generate an output of the neural network architecture.

[0022] In some embodiments, the modified 3D image data generation process further comprises, for each 2D image slice along a third dimension of the input 3D image data, processing the 2D image slice along the third dimension using a third neural network to generate third modified 3D image data, the third dimension being different from the first dimension, and processing the input 3D image data together with at least the first modified 3D image data and the second modified 3D image data comprises processing at least the input 3D image data, the second modified 3D image data, and the third modified 3D image data using an output neural network to generate an output of the neural network architecture.

[0023] This approach provides more context information when processing the input 3D image data, effectively providing context information from a third dimension, which may be different from the second dimension if the step of processing the input 3D image data together with at least the first modified 3D image data and the second modified 3D image data uses the second dimension.

[0024] Optionally, the step of processing the input 3D image data together with at least the first modified 3D image data and the second modified 3D image data comprises, for each 2D image slice along the second dimension of the input 3D image data, obtaining a corresponding 2D image slice from the second modified 3D image data and obtaining a corresponding 2D image slice from the third modified 3D image data, and processing the 2D image slices of the input 3D image data together with at least the corresponding 2D image slice from the second modified 3D image data and the corresponding slice from the third modified 3D image data using an output neural network to generate processed 2D image slices, wherein the combination of the processed 2D image slices forms an output of the neural network architecture.

[0025] The step of processing the input 3D image data together with at least the first modified 3D image data and / or the second modified 3D image data may include processing the input 3D image data, the first modified 3D image data, the second modified 3D image data, and the third modified 3D image data using an output neural network to generate an output of the neural network architecture.

[0026] In some embodiments, the step of processing the input 3D image data together with at least the first modified 3D image data and the second modified 3D image data comprises, for each 2D image slice along the second dimension of the input 3D image data, obtaining a corresponding 2D image slice from the first modified 3D image data, obtaining a corresponding 2D image slice from the second modified 3D image data, and obtaining a corresponding 2D image slice from the third modified 3D image data, and using an output neural network to process the 2D image slices of the input 3D image data together with at least the corresponding 2D image slice from the first modified 3D image data, the corresponding slice from the second modified 3D imaging data, and the corresponding slice from the third modified 3D imaging data to generate processed 2D image slices, wherein the combination of the processed 2D image slices forms an output of the neural network architecture.

[0027] Optionally, the input 3D image data is a medical image of a subject, and each 2D image slice along a first dimension is a slice in a coronal plane of the subject, each 2D image slice along a second dimension is a slice in an axial plane of the subject, and each 2D image slice along a third dimension is a slice in a sagittal plane of the subject.

[0028] The second neural network and the third neural network may be the same, or the second neural network and the third neural network may be different.

[0029] Optionally, a size of each of the plurality of sub-volumes is less than or equal to 1 / 16 of the size of the input 3D image data.

[0030] The method may further include providing an output, which may include controlling a user interface to provide a user perceptible output (such as a visual or audio representation) of the output, or providing the output to a further processing system for further processing. It will be apparent that the output of the neural network architecture may provide information useful for performing one or more further processing tasks, such as, for example, a diagnostic task or a patient assessment task.

[0031] The output may be an output for use in a clinical support task.

[0032] The input 3D image data may consist of medical 3D image data. The neural network architecture may have been trained to perform a clinical support task, such as identifying or predicting the presence or absence of a particular pathology in the medical 3D image. Other suitable clinical support tasks (segmentation, noise removal, etc.) will be apparent to those skilled in the art.

[0033] Also proposed is a computer program product comprising computer program code means which, when executed on a computing device having a processing system, causes the processing system to perform all the steps of any of the methods described herein.

[0034] Also proposed is a processing system for processing 3D image data using a neural network architecture, comprising: obtaining input 3D image data, performing a modified 3D image data generation process including, for each of a plurality of 3D sub-volumes of the input 3D image data, processing the 3D sub-volume using a first neural network to generate first modified 3D image data and / or, for each 2D image slice along a first dimension of the input 3D image data, processing the 2D image slice using a second neural network to generate second modified 3D image data, and processing the input 3D image data together with at least the first modified 3D image data and the second modified 3D image data using an output neural network to generate an output of the neural network architecture.

[0035] The processing system is adapted to carry out the approach provided by any of the methods or mechanisms described herein.

[0036] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. [Brief description of the drawings]

[0037] For a better understanding of the present invention and to show more clearly how it may be carried into effect, reference will now be made, by way of example only, to the accompanying drawings in which:

[0038] [Figure 1] FIG. 1 illustrates a neural network architecture according to various embodiments. [Diagram 2] Figure 2 shows the U-net neural network architecture. [Diagram 3] FIG. 3 provides example images illustrating the effect of the embodiment. [Figure 4] FIG. 4 is a flow chart illustrating a method according to an embodiment. [Diagram 5]FIG. 5 illustrates a processing system according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0039] The present invention will now be described with reference to the drawings.

[0040] It should be understood that the detailed description and specific examples, while indicating exemplary embodiments of the devices, systems, and methods, are for purposes of illustration only and are not intended to limit the scope of the invention. These and other features, aspects, and advantages of the devices, systems, and methods of the present invention will become better understood from the following description, the appended claims, and the accompanying drawings. It should be understood that the figures are schematic representations only and are not drawn to scale. It should also be understood that the same reference numerals are used throughout the figures to indicate the same or similar parts.

[0041] The present invention provides a mechanism for processing input 3D image data. In a first phase, the input 3D image data is processed separately using one or more neural networks to generate one or more modified 3D image data. In a second phase, the input 3D image data and the modified 3D image data are processed using another neural network to generate an output. The one or more neural networks that generate the modified 3D image data process slices or sub-volumes of the input 3D image data to generate the modified 3D image data.

[0042] The embodiments are based on the recognition that existing approaches for processing 3D image data using neural networks are either too inefficient to run efficiently on existing hardware or lose potentially valuable information (e.g., contextualization) across the dimensions of the 3D image data. Using a two-phase approach, this contextual information can be included or provided as an input to the neural network that generates the output.

[0043] The approach disclosed by the embodiments can be employed in any scenario where processing of 3D image data is desired. However, the embodiments are particularly advantageous in the medical field, such as for processing medical 3D image data, where large amounts of 3D image data are generated and it would be beneficial to improve the efficiency of processing such large amounts of possible data.

[0044] In the context of the present invention, an image slice is a two-dimensional cross-section of three-dimensional or volumetric image data.

[0045] It will be appreciated that image slices can be extracted with respect to three different dimensions of the three-dimensional image data. An image slice "along a dimension" is an image slice at a particular location in that dimension. Thus, if the 3D image data has dimensions [x, y, z], then an image slice along dimension z will have dimensions [x, y].

[0046] In the context of this disclosure, image slices correspond to one another if they contain or represent the same scene, i.e., they represent the same imaging area.

[0047] Figure 1 conceptually illustrates the approach taken in embodiments of the present invention. In particular, Figure 1 illustrates a neural network architecture 100 illustrating various components that may be used in embodiments of the present invention.

[0048] The network architecture 100 processes input 3D image data 191 and generates output 192 .

[0049] The output 192 itself comprises 3D image data, such as a denoised version of the input 3D image data 191, an upsampled or higher resolution version of the input 3D image data 191, or synthetic image data. Alternatively, the output 192 may comprise other possible neural network outputs of a 3D image, such as a segmentation mask, a prediction indicator, etc. Other suitable implementations will be apparent to those skilled in the art.

[0050] The network architecture 100 is divided into two phases.

[0051] In a first phase 101, input 3D image data 191 is processed individually by one or more first phase neural networks 110, 120, 130. Each first phase neural network produces modified 3D image data 111, 121, 131.

[0052] One suitable example of a first phase neural network is a neural network that processes chunks or blocks of 3D image data, such as sub-volumes of the input 3D image data. Such a neural network is named a 3D sub-volume neural network. As an example, the neural network architecture may be configured such that the first neural network 110 processes different sub-volumes of the input 3D image data to generate the first modified 3D image data 111.

[0053] The size of each sub-volume can be up to 1 / 16th the size of the input 3D image data, thereby reducing the processing power required to process the input 3D image data using the 3D sub-volume neural network.

[0054] Another suitable example of a first phase neural network is a neural network that processes 2D image data or image slices. Such a neural network is named a 2D image slice neural network. Each image slice along a particular dimension of the input 3D image data is processed using such a 2D image slice neural network to generate modified 3D image data. Processing the image slices may include generating processed or modified image slices.

[0055] By way of example only, the second neural network 120 processes image slices along a first dimension of the input 3D image data 191 to generate the second modified 3D image data 121. In a further example, the third neural network 130 processes image slices along a dimension different from the first dimension of the input 3D image data 191 to generate the third modified 3D image data 131.

[0056] Although not specifically described, the second and third neural networks may be identical (i.e., identical), which may reduce the size of the neural network architecture and increase the training data. In other examples, the second and third neural networks may be independent, e.g., trained with different training data sets.

[0057] As an example, if the input 3D image data is image data of a subject, each slice in the first dimension is a slice in either a sagittal, coronal, or axial plane of the subject.

[0058] In the second phase 102 , the input 3D image data 191 and at least one of the modified 3D image data 111 , 121 , 131 are processed by the second phase neural network 150 or the output neural network to generate an output 192 .

[0059] The second phase neural network (or "output neural network") iteratively processes sets of slices of the image data. Each slice set includes an image slice of the input 3D image data and a corresponding image slice from one or more of the modified 3D image data. Each set of slices corresponds to a different position in a second dimension. The second dimension is different from the first dimension.

[0060] The second neural network 120 processes image slices along a first dimension of the input 3D image data 191 to generate second modified 3D image data 121. The output neural network processes image slices along the second dimension of the input 3D image data (together with corresponding slices of the one or more modified 3D image data) to generate output, and the third neural network 130 processes image slices along a third dimension of the input 3D image data 191 to generate third modified 3D image data 131. The first, second and third dimensions may all be different.

[0061] If the input 3D image data is image data of a subject, each slice in the first dimension is a slice in either the sagittal, coronal, or axial plane of the subject. Each slice in the second dimension is a slice in a different one of the sagittal, coronal, or axial planes of the object. If present, each slice in the third dimension is a slice in the last of the sagittal, coronal, or axial planes of the object.

[0062] In particular, if the input 3D image data is image data of a subject, each slice in the first dimension is a slice in a sagittal or coronal plane of the subject, each slice in the second dimension is a slice in an axial plane of the object, and if present, each slice in the third dimension is a slice in the other of the sagittal and coronal planes of the object.

[0063] Thus, in the second phase 102, two, three or four input channels of the 2D image are defined. The first channel is for a slice of the input 3D image data. The remaining channels are for corresponding slices from each of the first and second (and optionally third) modified image data generated by the first phase neural network in the first phase 101.

[0064] Stated differently, in the second phase 102, for each 2D image slice along the second dimension of the input 3D image data 191, a corresponding 2D image slice from the first modified 3D imaging data 111 and the second modified 3D image data 121 (and optionally the third modified 3D image data 131) is obtained. The 2D image slices of the input 3D image data, together with any obtained corresponding 2D image slices, are then processed using the second phase neural network 150 (also referred to as the output neural network) to generate a processed 2D image slice. The combination of the processed 2D image slices (generated by processing each 2D image slice of the input 3D image data) forms the output of the neural network architecture 100.

[0065] Thus, the output of the second phase neural network 150 comprises one channel of the resulting image data.

[0066] However, it will be appreciated that the exact processing performed by each neural network in the neural network architecture will vary depending on the purpose or goal, i.e., output 192, of the neural network architecture 100.

[0067] In some embodiments, each neural network is trained for the same purpose, e.g., with the intent of generating or predicting the same output.

[0068] For example, if the goal of the neural network architecture 100 is to synthesize 3D image data of a different modality from the input 3D image data, each neural network is trained to synthesize 3D image data of a different modality from the input 3D image data.

[0069] As another example, if the goal of the neural network architecture 100 is to denoise 3D image data, then each neural network is trained to denoise the input 3D image data 191.

[0070] However, embodiments are not limited to this limitation.

[0071] For example, a first phase neural network is trained for use in generating modified 3D image data (such as denoised or contrast enhanced 3D image data), and a second phase neural network is trained for use in predicting or identifying the presence of a particular pathology within the input 3D image data.

[0072] As another example, the first phase neural networks 110, 120, 130 are trained for use in generating modified 3D image data (such as denoised or contrast enhanced 3D image data), and the second phase neural networks are trained for use in segmenting the input 3D image data.

[0073] As yet another example, the first phase neural networks 110, 120, 130 are trained for use in generating modified 3D image data (such as denoised or contrast enhanced 3D image data), and the second phase neural network 150 is trained for use in synthesising the input 3D image data with 3D image data of a different modality.

[0074] Other suitable implementations will be apparent to those skilled in the art.

[0075] An embodiment uses a neural network to process image data (such as 2D image slices or chunks of 3D image data) to generate an output.

[0076] The structure of a neural network is inspired by the human brain. A neural network is composed of layers, each layer containing multiple neurons. Each neuron contains a mathematical operation. In particular, each neuron contains a different weighted combination of a single type of transformation (e.g., the same type of transformation, such as sigmoid, but with different weights). In the process of processing input data, the mathematical operation of each neuron is performed on the input data to generate a numerical output, and the output of each layer of the neural network is fed to the next layer in turn. The final layer provides the output.

[0077] Methods for training neural networks are well known. Typically, such methods involve obtaining a training data set that includes training input data entries and corresponding training output data entries. An initialized neural network is applied to each input data entry to generate a predicted output data entry. The error between the predicted output data entry and the corresponding training output data entry is used to correct the neural network. This process is repeated until the error converges, i.e., until the predicted output data entries are sufficiently similar to the training output data entries (e.g., ±1%). This is commonly known as a supervised learning approach.

[0078] For example, the weights of the mathematical operations of each neuron are modified until the error converges. Known methods for modifying neural networks include the gradient descent algorithm, the backpropagation algorithm, etc.

[0079] In a particularly advantageous embodiment, each neural network of the neural network architecture has a U-net architecture: U-Net and its 3D version, 3D U-Net, are believed to provide some of the standard and most successful CNN architectures for image processing tasks.

[0080] 2 shows a neural network 200 with a basic U-Net architecture formed of two paths: a contraction path 210 and an expansion path 220. The neural network processes an input 201 to generate an output 202.

[0081] The contraction path is composed of a series of levels / blocks formed of layers such as convolutional layers 211 and pooling layers 212. The expansion path is composed of a series of levels / blocks formed of layers such as convolutional layers 221 and ascending convolution / upsampling layers 222. The number of levels / blocks in the expansion path is the same as the number of levels / blocks in the contraction path. The number of pooling layers in the contraction path 210 (and therefore the number of ascending convolutional layers in the expansion path 220) defines the number of levels / blocks in each path, and therefore the depth of the U-Net. In the example shown, the depth of the neural network is 4.

[0082] As shown, there may be one or more feedforward links 250 between the same levels of the contracting and expanding paths.

[0083] Studies of the relationship between network depth and performance have shown that performance typically improves with network depth, typically up to a depth of 4.

[0084] A more complete description of the basic (2D) U-Net architecture can be found in Ronneberger, Olaf, Philipp Fischer, and Thomas Brox, “U-Net: Convolutional networks for biomedical image segmentation,” International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, 2015.

[0085] A description of a suitable 3D U-Net architecture can be found in "3D U-Net: Learning dense volumetric segmentation from sparse annotation" by Ozgun Cicek, Ahmed Abdulkadir, Soeren S. Lienkamp, ​​Thomas Brox, and Olaf Ronneberger, in Proceedings of the International Conferene on Medical Image Computing and Computer-Assisted Intervention, Athens, Greece, October 17-21, 2016.

[0086] As an example, we have evaluated the proposed network architecture on CT volumetric data, where each neural network in the neural network architecture is trained to synthesize monochromatic 50 KeV results from conventional CT volumetric data.

[0087] In this example, the training data set for each neural network consists of 14 separate clinical studies containing a total of 6,552 slices. All studies are abdominal scans reconstructed using a filtered backprojection algorithm, a slice thickness of 1 mm, a slice increment of 0.9 mm or 1 mm, and a field of view of 350 mm to 425 mm.

[0088] Six neural network architectures were trained to generate the outputs.

[0089] The first neural network architecture uses a first 2D neural network, i.e., a neural network that processes only image slices. Each image slice along a first dimension of the input 3D image data is processed using this first 2D neural network to generate first output 3D image data. The first 2D neural network employs a depth 4 U-Net architecture.

[0090] The second neural network architecture uses a single 3D neural network to process the input 3D image data. Specifically, the single 3D neural network processes chunks / blocks / subvolumes of the input 3D image data and processes the entire input 3D image data (with padding). Thus, the input 3D image data is divided into chunks / subvolumes, each of which is processed by a single 3D neural network process to generate the second output image data.

[0091] The third neural network architecture uses a second 2D neural network, i.e., a neural network that processes only image slices. Each image slice along the second and / or third dimension of the input 3D image data is processed using the second 2D neural network to generate output 3D image data. Processing the image slices along the second dimension generates a third output image data. Processing the image slices along the third dimension generates a fourth output image data. The second 2D neural network employs a depth 4 U-Net architecture.

[0092] The fourth neural network architecture uses a third two-dimensional neural network. The inputs to the third two-dimensional neural network are image slices from the input 3D image data and image slices from the second output image data (the output of the second neural network architecture). Each image slice is an image slice acquired along a first dimension. Thus, the third two-dimensional neural network receives a two-channel input: 1) image slices from the conventional input image data, and 2) image slices from the output of the second neural network architecture. A plurality of image slice sets (each set including a slice from each channel) are individually processed by the third two-dimensional neural network to generate a fifth output image data. Each image slice set corresponds to a different position along a particular dimension (e.g., different axial, coronal, or sagittal images if the input image data includes medical imaging data). The depth of the third two-dimensional neural network is four.

[0093] The fifth neural network architecture uses a fourth two-dimensional neural network. The inputs to the fourth two-dimensional neural network are image slices from the input 3D image data, image slices from the third output image data, and image slices from the fourth output image data. Each image slice is an image slice acquired along a first dimension. Thus, the fourth two-dimensional neural network receives three channel inputs: 1) image slices from the conventional input image data, 2) image slices from the third output image data, and 3) image slices from the fourth output image data. A plurality of image slice sets (each set including a slice from each channel) are individually processed by the fourth two-dimensional neural network to generate the sixth output image data. Each image slice set corresponds to a different position along a particular dimension (e.g., different axial, coronal, or sagittal images if the input image data includes medical imaging data).

[0094] The sixth neural network architecture uses a fifth two-dimensional neural network. The inputs to the fifth two-dimensional neural network are image slices from the input 3D image data, image slices from the second output image data, image slices from the third output image data, and image slices from the fourth output image data. Each image slice is an image slice acquired along a first dimension. Thus, the fifth two-dimensional neural network receives a four-channel input: 1) an image slice from the conventional input image data, 2) an image slice from the second output image data, 3) an image slice from the third output image data, and 4) an image slice from the fourth output image data. A plurality of image slice sets (each set including a slice from each channel) are individually processed by the fifth two-dimensional neural network to generate the seventh output image data. Each image slice set corresponds to a different position along a particular dimension (e.g., different axial, coronal, or sagittal images if the input image data includes medical imaging data).

[0095] For purposes of this evaluation, an image slice in the first dimension is an image in or parallel to the axial plane, an image slice in the second dimension is an image in or parallel to the coronal plane, and an image slice in the third dimension is an image in or parallel to the sagittal plane.

[0096] The evaluation results are summarized in Table 1 below. [Table 1]

[0097] In Table 1, L1 represents the value of the first loss function L1(.), which is the sum of the absolute differences between the true and predicted values ​​of each pixel (over all pixels of the output image data generated by the associated network architecture). L2 represents the value of the second loss function L2(.), which is the sum of the squared differences between the true and predicted values ​​of each pixel (over all pixels of the output image data generated by the associated network architecture).

[0098] The true value of the pixel is contained in the ground truth image data, which is image data acquired by a CT scanner operating in monochromatic 50 keV mode, and the predicted value of the pixel is contained in the output image data provided by the associated network architecture, which attempts to predict the content of the ground truth image data.

[0099] It can be seen that the proposed approach improves the results, in particular the proposed approach (embodied by the fourth, fifth and sixth neural network architectures) shows a significant improvement over the existing available neural network architectures (represented by the first and second neural network architectures).

[0100] Figure 3 further illustrates the advantages of the proposed approach.

[0101] In FIG. 3, there are four representative image slices, each showing the same scene, ie, the same portion of the subject's abdomen.

[0102] The first image slice 310 is an image slice derived from conventional CT 3D image data. The second slice 320 is an image slice derived from 3D image data acquired with a CT scanner operating in monochromatic 50 keV mode. The third slice 330 is an image slice derived from output image data generated by processing the conventional CT 3D image data using a first neural network architecture. The fourth slice 340 is an image slice derived from output image data generated by processing the conventional CT 3D image data using a sixth neural network architecture.

[0103] The second slice 320 effectively represents the "ground truth" image data that the neural network architecture is attempting to reconstruct from the conventional CT 3D image data.

[0104] It can be seen that the fourth slice is closer to the second slice compared to the third slice, i.e. the difference between the fourth slice 340 and the second slice 320 is smaller than the difference between the third slice 330 and the second slice 320. This difference is perhaps most evident in the marked region 390.

[0105] 4 illustrates a computer-implemented method 400 according to an embodiment for processing 3D image data using a neural network architecture.

[0106] The method 400 includes a step 410 of acquiring input 3D image data. The input 3D image data is, for example, a medical image of a subject. However, other suitable examples of 3D image data will be apparent to those skilled in the art.

[0107] Step 410 may include receiving the input 3D image data from a processing system, such as an imaging system that generates a medical 3D image. In other examples, step 410 may include retrieving the input 3D image data from a memory or storage unit.

[0108] Method 400 further includes a process 420 for performing a modified 3D image data generation process, in which one or more modified image data items are generated, each of which is a modified version of the input 3D image data.

[0109] In particular, process 420 includes performing steps 421 and 422 .

[0110] Step 421 includes, for each of a plurality of 3D sub-volumes of the input 3D image data, processing the 3D sub-volume using a first neural network to generate first modified 3D image data.

[0111] Step 422 includes, for each 2D image slice along the first dimension of the input 3D image data, processing the 2D image slice using a second neural network to generate second modified 3D image data.

[0112] The first and second neural networks are embodied as first phase neural networks as described above.

[0113] After performing step 420, the method 400 performs step 430 of processing the input 3D image data, together with at least the first modified 3D image data and the second modified 3D image data, using an output neural network to generate an output of the neural network architecture. The output neural network is functionally equivalent to the neural network of the second phase described above.

[0114] Thus, step 430 involves processing the input 3D image data and any modified 3D image data generated by process 420 to generate an output. The output itself includes 3D image data, such as a denoised version of the input 3D image data, an upsampled or higher resolution version of the input 3D image data, or synthetic image data. Alternatively, the output may include other possible neural network outputs of the 3D image, such as a segmentation mask or a prediction indicator. Other suitable implementations will be apparent to those skilled in the art.

[0115] In some examples, step 430 includes, for each 2D image slice along a second dimension of the input 3D image data that is different from the first dimension (if used), obtaining a corresponding 2D image slice from the first modified 3D image data and the second modified 3D imaging data, and processing the 2D image slice of the input 3D image data together with at least the corresponding 2D image slice from the first modified 3D image data and the corresponding slice from the second modified 3D imaging data using a third neural network to generate a processed 2D image slice. The combination of the processed 2D image slices forms an output of the neural network architecture.

[0116] In some embodiments, when process 420 includes performing step 421, the process may also include performing step 423, which includes, for each 2D image slice along a third dimension of the input 3D image data, processing the 2D image slice along the third dimension using a third neural network to generate third modified 3D image data, where the third dimension is different from the first dimension and the second dimension.

[0117] Step 430 is suitably adapted to process at least the input 3D image data, the first modified 3D image data, and the third modified 3D image data using a third neural network to generate an output of the neural network architecture.

[0118] Of course, step 430 may include processing the input 3D image data, the first modified 3D image data, the second modified 3D image data, and the third modified 3D image data using a third neural network to generate an output of the neural network architecture.

[0119] Method 400 further includes a step 440 of providing an output. Step 440 may include, for example, controlling a user interface to provide a user-perceivable output (such as a visual or audio representation) of the output generated in step 430. As another example, step 440 may include providing the output to a further processing system for further processing. It will be apparent that the output of the neural network architecture may provide useful information for performing one or more further processing tasks.

[0120] As a further example, Figure 5 illustrates an example of a processing system 500 in which one or more portions of the embodiments may be used. The various operations described above utilize the functionality of the processing system 500. For example, one or more portions of the system for processing 3D image data using a neural network architecture may be incorporated into any of the elements, modules, applications, and / or components described herein. In this regard, it should be understood that the functional blocks of the system may operate on a single computer or may be distributed across several computers or locations (e.g., connected via the Internet).

[0121] The processing system 500 may include, but is not limited to, a PC, a workstation, a laptop, a PDA, a palm device, a server, storage, etc. In general, with respect to a hardware architecture, the processing system 500 includes one or more processors 501, a memory 502, and one or more I / O devices 507 communicatively coupled via a local interface (not shown). The local interface may be, for example, but not limited to, one or more buses or other wired or wireless connections as known in the art. The local interface may have controllers, buffers (caches), drivers, repeaters, and receivers to enable communication. Additionally, the local interface may include address, control, and / or data connections to enable appropriate communication between the aforementioned components.

[0122] Processor 501 is a hardware device that executes software that may be stored in memory 502. Processor 501 may be virtually any custom or commercially available processor, central processing unit (CPU), digital signal processor (DSP), or coprocessor of multiple processors associated with processing system 500, and processor 501 may be a semiconductor-based microprocessor (in the form of a microchip) or microprocessor.

[0123] The memory 502 may include any one or combination of volatile memory elements (e.g., random access memory (RAM), such as dynamic random access memory (DRAM), static random access memory (SRAM), and non-volatile memory elements (e.g., ROM, erasable programmable read only memory (EPROM), electronically erasable programmable read only memory EEPROM), programmable read only memory (PROM), tape, compact disk read only memory (CD-ROM), disk, diskette, cartridge, cassette, etc.). Additionally, the memory 502 may incorporate electronic, magnetic, optical, and / or other types of storage media. It is noted that the memory 502 may have a distributed architecture in which various components are remote from one another but accessible by the processor 501.

[0124] The software in memory 502 may include one or more separate programs, each including an ordered list of executable instructions for implementing logical functions. The software in memory 502 includes a suitable operating system (O / S) 505, a compiler 504, source code 503, and one or more applications 506, according to an exemplary embodiment. As shown, the applications 506 include a number of functional components for implementing the features and operations of the exemplary embodiments. The applications 506 of the processing system 500 may represent various applications, computational units, logic, functional units, processes, operations, virtual entities, and / or modules, according to an exemplary embodiment, although the applications 506 are not intended to be limiting.

[0125] Operating system 505 controls the execution of other computer programs and provides scheduling, input / output control, file and data management, memory management, and communication control and related services. The inventors contemplate that application 506 for implementing the exemplary embodiment is applicable to any commercially available operating system.

[0126] The application 506 may be a source program, an executable program (object code), a script, or any other entity that includes a set of instructions to be executed. In the case of a source program, the program is typically translated via a compiler (such as compiler 504), assembler, interpreter, etc., which may or may not be included in memory 502, so that the program operates properly in conjunction with the O / S 505. Additionally, the application 506 may be written as an object-oriented programming language with classes of data and methods, or a procedural programming language (e.g., but not limited to, C, C++, C#, Pascal, BASIC, API calls, HTML, XHTML, XML, ASP script, JavaScript, FORTRAN, COBOL, Perl, Java, ADA, .NET, etc.) with routines, subroutines, and / or functions.

[0127] The I / O devices 507 include input devices such as, but not limited to, a mouse, keyboard, scanner, microphone, camera, etc. The I / O devices 507 also include output devices such as, but not limited to, a printer, display, etc. Finally, the I / O devices 507 also include devices that communicate both input and output such as, but not limited to, a NIC or modulator / demodulator (for accessing remote devices, other files, devices, systems, or networks), radio frequency (RF) or other transceivers, telephone interfaces, bridges, routers, etc. The I / O devices 507 also include components for communicating over various networks such as the Internet or an intranet.

[0128] If the processing system 500 is a PC, workstation, intelligent device, etc., the software in memory 502 may further include a Basic Input / Output System (BIOS) (omitted for simplicity). The BIOS is a set of essential software routines that initializes and tests the hardware at power-up, starts the O / S 505, and supports data transfers between hardware devices. The BIOS is stored in some type of read-only memory, such as a ROM, PROM, EPROM, EEPROM, etc., for execution when the processing system 500 is powered up.

[0129] During operation of processing system 500, processor 501 executes software stored in memory 502, transfers data to and from memory 502, and generally controls the operation of processing system 500 in accordance with the software. Applications 506 and O / S 505, in whole or in part, are read by processor 501 and, in some cases, buffered within processor 501 before being executed.

[0130] It should be noted that if the application 506 is implemented in software, the application 506 may be stored on virtually any computer-readable medium for use with or in connection with any computer-related system or method. In the context of this document, a computer-readable medium may be an electronic, magnetic, optical, or other physical device or means that can store or retain a computer program for use with or in connection with a computer-related system or method.

[0131] The application 506 may be embodied in any computer-readable medium for use with or in connection with an instruction execution system, apparatus, or device (such as a computer-based system, processor-based system, or other system that can fetch instructions from and execute instructions from an instruction execution system, apparatus, or device). In the context of this document, a "computer-readable medium" may be any means that can store, communicate, propagate, or transfer a program for use with or in connection with an instruction execution system, apparatus, or device. Computer-readable media include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, devices, or propagation media.

[0132] Ordinal numbers ("first," "second," etc.) are used for clarity only to distinguish between different elements, and a reference to a non-"first" (e.g., "second" or "third") element does not require that a "first" element be present. One of ordinary skill in the art would be able to re-label such elements appropriately (e.g., re-label a "second" element as a "first" element if only a second element is present, etc.).

[0133] Variations of the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure and the appended claims. In the claims, the term "comprising" does not exclude other elements or steps, and the singular elements do not exclude a plurality. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0134] A single processor or other unit may fulfill the functions of several items recited in the claims. The computer program can be stored / distributed on any suitable medium, such as an optical storage medium or a solid-state medium, supplied together with or as part of other hardware, but can also be distributed in other forms, such as via the Internet or other wired or wireless communication systems.

[0135] It should be noted that when the term "adapted to" is used in the claims or description, it is intended to be equivalent to the term "configured to." Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. 1. A computer-implemented method for processing 3D image data using a neural network architecture, comprising: obtaining input 3D image data; for each of a plurality of 3D sub-volumes of the input 3D image data, processing the 3D sub-volumes using a first neural network to generate first modified 3D image data; performing a modified 3D image data generation process, the process including, for each 2D image slice along a first dimension of the input 3D image data, processing the 2D image slice using a second neural network to generate second modified 3D image data; The computer-implemented method comprising: The computer-implemented method further comprises:

23. A computer-implemented method comprising: processing the input 3D image data, together with at least the first modified 3D image data and the second modified 3D image data, using an output neural network to generate an output of the neural network architecture.

2. Processing the input 3D image data together with the second modified 3D image data and the first modified 3D image data comprises: For each 2D image slice of the input 3D image data along a second dimension different from the first dimension, obtaining corresponding 2D image slices from the first modified 3D image data and the second modified 3D image data; processing the 2D image slices of the input 3D image data together with at least the corresponding 2D image slices from the first modified 3D image data and the corresponding slices from the second modified 3D imaging data using the output neural network to generate processed 2D image slices; Including, The computer-implemented method of claim 1 , wherein a combination of the processed 2D image slices forms the output of the neural network architecture.

3. 3. The computer-implemented method of claim 2, wherein the input 3D image data is a medical image of a subject, and each 2D image slice along the first dimension is a slice in a coronal or sagittal plane of the subject, and each 2D image slice along the second dimension is a slice in an axial plane of the subject.

4. The computer-implemented method of claim 1 , wherein each of the first neural network, the second neural network, and the output neural network has a U-net architecture.

5. The modified 3D image data generation process includes: for each 2D image slice along a first dimension of the input 3D image data, processing the 2D image slice using a second neural network to generate second modified 3D image data; for each of a plurality of 3D sub-volumes of the input 3D image data, processing the 3D sub-volumes using a first neural network to generate first modified 3D image data; Including, 5. The computer-implemented method of claim 1, wherein processing the input 3D image data together with at least the first modified 3D image data and the second modified 3D image data comprises using the output neural network to process the input 3D image data together with at least the first modified 3D image data and the second modified 3D image data to generate the output of the neural network architecture.

6. the modified 3D image data generation process further comprises, for each 2D image slice along a third dimension of the input 3D image data, processing the 2D image slice along the third dimension using a third neural network to generate third modified 3D image data; the third dimension is different from the first dimension; 6. The computer-implemented method of claim 1, wherein processing the input 3D image data together with at least the first modified 3D image data and the second modified 3D image data comprises using the output neural network to process at least the input 3D image data, the second modified 3D image data, and the third modified 3D image data to generate the output of the neural network architecture.

7. Processing the input 3D image data together with at least the first modified 3D image data and the second modified 3D image data comprises: For each 2D image slice along the second dimension of the input 3D image data: obtaining corresponding 2D image slices from the second modified 3D image data; obtaining corresponding 2D image slices from the third modified 3D image data; processing the 2D image slices of the input 3D image data together with at least the corresponding 2D image slices from the second modified 3D image data and the corresponding slices from the third modified 3D image data using the output neural network to generate processed 2D image slices; Including, 7. The computer-implemented method of claim 6 when dependent on claim 2, wherein a combination of the processed 2D image slices forms the output of the neural network architecture.

8. 8. The computer-implemented method of claim 6 or 7, wherein processing the input 3D image data together with at least the first modified 3D image data and the second modified 3D image data comprises processing the input 3D image data, the first modified 3D image data, the second modified 3D image data, and the third modified 3D image data using the output neural network to generate the output of the neural network architecture.

9. said step of processing said input 3D image data together with at least said first modified 3D image data and said second modified 3D image data further comprising: For each 2D image slice along the second dimension of the input 3D image data: obtaining a corresponding 2D image slice from the first modified 3D image data; obtaining corresponding 2D image slices from the second modified 3D image data; obtaining corresponding 2D image slices from the third modified 3D image data; processing the 2D image slices of the input 3D image data together with at least the corresponding 2D image slices from the first modified 3D image data, the corresponding slices from the second modified 3D imaging data, and the corresponding slices from the third modified 3D imaging data using the output neural network to generate processed 2D image slices; Including, 9. The computer-implemented method of claim 8 when dependent on claim 2, wherein a combination of the processed 2D image slices forms the output of the neural network architecture.

10. the input 3D image data is a medical image of a subject; each 2D image slice along the first dimension is in a coronal plane of the subject; each 2D image slice along the second dimension is a slice in an axial plane of the object; 10. The computer-implemented method of claim 6, wherein each 2D image slice along the third dimension is a slice in a sagittal plane of the subject.

11. 11. The computer-implemented method of claim 6, wherein the second neural network and the third neural network are identical.

12. 11. The computer-implemented method of claim 6, wherein the second neural network and the third neural network are different.

13. 13. The computer-implemented method of claim 1, wherein a size of each of the plurality of sub-volumes is less than or equal to one-sixteenth of a size of the input 3D image data.

14. 14. A computer program comprising computer program code means which, when executed on a computing device having a processing system, causes the processing system to perform all the steps of the method of any one of claims 1 to 13.

15. 1. A processing system for processing 3D image data using a neural network architecture, the processing system comprising: Obtaining input 3D image data; for each of a plurality of 3D sub-volumes of the input 3D image data, processing the 3D sub-volumes using a first neural network to generate first modified 3D image data; 13. A processing system for performing a modified 3D image data generation process, comprising, for each 2D image slice along a first dimension of the input 3D image data, processing the 2D image slice using a second neural network to generate second modified 3D image data, The processing system further comprises:

22. A processing system comprising: a processor for processing the input 3D image data, together with at least the first modified 3D image data and the second modified 3D image data, using an output neural network to generate an output of the neural network architecture.

Images