Network architecture for 3D image processing

Abstract

A mechanism for processing input 3D image data. In a first stage, the input 3D image data is processed separately using one or more neural networks to produce one or more modified 3D image data. In a second stage, the input 3D image data and the modified 3D image data are processed using a neural network to produce an output. The neural network that produces the modified 3D image data is configured to process a slice or sub-volume of the input 3D image data to produce the modified 3D image data.

Description

Technical Field

[0001] The present invention relates to the field of neural network architectures, and in particular to using neural network architectures to process 3D image data. Background Art

[0002] There is a growing interest in the automatic processing of images, especially 3D images. While many industries can benefit from improvements in 3D image processing, the medical industry is a particular area of ​​interest. The popularity of 3D medical images is increasing as they are generated by the increasingly widespread 3D ultrasound, CT scans or MRI scans. Automatic processing of these images would be useful to assist clinicians in the assessment and / or diagnosis of imaged objects, for example, for performing clinical assistance tasks.

[0003] Example image processing tasks that may be performed by appropriately configured neural network architectures include processing 3D (medical) images, including object detection, object segmentation, noise removal, artifact removal, super-resolution, and / or synthesizing or predicting results from other imaging modalities.

[0004] A variety of neural network architectures have been developed to analyze (medical) images. One of the most successful is the U-net architecture.

[0005] There has been a long-standing desire to improve the performance of neural network architectures for processing 3D images.

[0006] U.S. patent application US2020 / 058126 discloses a method for image segmentation of 3D images using a fully convolutional neural network operating on a series of 2D slices. Summary of the invention

[0007] The invention is defined by the claims.

[0008] According to an example of one aspect of the invention, a computer-implemented method for processing 3D image data using a neural network architecture is provided.

[0009] The computer-implemented method comprises: obtaining input 3D image data; performing a modified 3D image data generation process comprising: for each of a plurality of 3D sub-volumes of the input 3D image data, processing the 3D sub-volume using a first neural network to produce first modified 3D image data; and for each 2D image slice along a first dimension of the input 3D image data, processing the 2D image slice using a second neural network to produce second modified 3D image data; and processing the input 3D image data and at least the first modified 3D image data and the second modified 3D image data using an output neural network to produce an output of the neural network architecture.

[0010] The proposed method provides an improved mechanism for analyzing or processing 3D image data, i.e., improved volumetric image processing. The proposed method facilitates the use of a greater amount of contextual information for performing 3D image analysis, avoiding the natural limitation of the receptive field size for neural network processing by simultaneously incorporating a wider range of data for efficient processing.

[0011] It is recognized that many of the neural networks in use today were originally designed for 2D image processing. When moving to 3D, these neural networks begin to suffer more and more from the inherent phenomenon of "curse of dimensionality". Particularly for the processing of 3D images, this manifests as a limit on the size of the network's possible receptive field, which imposes an upper limit on the amount of context available when processing different parts of a 3D image. This restricted contextual bound limits the functionality of the model, especially in processing systems with limited memory or processing power.

[0012] The proposed method increases the amount of context available when processing input 3D image data using a neural network (here; the output neural network). This improves the performance of image processing tasks performed by the neural network architecture.

[0013] The input 3D image data may include medical 3D image data.Due to the increasing prevalence of and clinical reliance on medical 3D image data, embodiments are particularly advantageous when used to process medical 3D image data.

[0014] Preferably, the input 3D image data and any modified 3D image data processed by the output neural network are of the same resolution.

[0015] The step of processing the input 3D image data and the second modified 3D image data and the first modified 3D image data may comprise: for each 2D image slice along a second dimension of the input 3D image data different from the first dimension: obtaining a corresponding 2D image slice from the first modified 3D image data and the second modified 3D imaging data; and processing the 2D image slice of the input 3D image data, and at least the corresponding 2D image slice from the first modified 3D image data and the corresponding slice from the second modified 3D imaging data using the output neural network to produce a processed 2D image slice, wherein a combination of the processed two-dimensional image slices forms an output of the neural network architecture.

[0016] This approach, when processing slices along a particular dimension (the second dimension), makes context from at least one other dimension available, particularly data that would otherwise be available only in other slices along the particular dimension.

[0017] Thus, for each slice of the 3D image in a particular dimension, information from other slices in the particular dimension may assist in the processing of the slice because the information affects or contributes to the corresponding slice from the first and second modified 3D image data. This increases the effective receptive field of the processed two-dimensional image slice, thereby improving the performance of the overall neural network architecture.

[0018] The input 3D image data may be a medical image of an object, and each 2D image slice along the first dimension may be a slice located in a coronal plane or a sagittal plane of the object, and each 2D image slice along the second dimension may be a slice located in an axial plane of the object. However, the embodiment is not limited thereto.

[0019] The first, second and output neural networks may all have a U-net architecture. The U-net architecture is believed to provide good or excellent (to other architectures) image processing capabilities, particularly in the medical field.

[0020] In some examples, the modified 3D image data generation process includes: for each 2D image slice along a first dimension of the input 3D image data, processing the 2D image slice using a second neural network to generate second modified 3D image data; and for each of a plurality of 3D sub-volumes of the input 3D image data, processing the 3D sub-volume using a first neural network to generate first modified 3D image data.

[0021] Accordingly, processing the input 3D image data and at least the first modified 3D image data and the second modified 3D image data may include processing the input 3D image data and at least the first modified 3D image data and the second modified 3D image data using the output neural network to produce the output of the neural network architecture.

[0022] In some embodiments, the modified 3D image data generation process further comprises, for each 2D image slice along a third dimension of the input 3D image data, processing the 2D image slice along the third dimension using a third neural network to produce third modified 3D image data; the third dimension is different from the first dimension; and processing the input 3D image data, and at least the first modified 3D image data and the second modified 3D image data, comprises processing at least the input 3D image data, the second modified 3D image data and the third modified 3D image data using the output neural network to produce the output of the neural network architecture.

[0023] The method provides more context information available when processing the input 3D image data, effectively providing context information from a third dimension. When the steps of processing the input 3D image data and at least the first modified 3D image data and the second modified 3D image data utilize a second dimension, the third dimension may be different from the second dimension.

[0024] Optionally, the step of processing the input 3D image data and at least the first modified 3D image data and the second modified 3D image data comprises: for each 2D image slice along a second dimension of the input 3D image data: obtaining a corresponding 2D image slice from the second modified 3D image data; obtaining a corresponding 2D image slice from the third modified 3D image data, and using the output neural network to process the 2D image slices of the input 3D image data and at least the corresponding 2D image slices from the second modified 3D image data and the corresponding slices from the third modified 3D imaging data to produce processed 2D image slices, wherein the combination of the processed two-dimensional image slices forms the output of the neural network architecture.

[0025] The step of processing the input 3D image data and at least the first modified 3D image data and / or the second modified 3D image data may include using the output neural network to process the input 3D image data, the first modified 3D image data, the second modified 3D image data and the third modified 3D image data to produce an output of the neural network architecture.

[0026] In some examples, the step of processing the input 3D image data and at least the first modified 3D image data and the second modified 3D image data includes: for each 2D image slice along a second dimension of the input 3D image data: obtaining a corresponding 2D image slice from the first modified 3D image data; obtaining a corresponding 2D image slice from the second modified 3D image data; obtaining a corresponding 2D image slice from the third modified 3D image data, and processing the 2D image slices of the input 3D image data, and at least the corresponding 2D image slices from the first modified 3D image data, the corresponding slices from the second modified 3D imaging data, and the corresponding slices from the third modified 3D imaging data using the output neural network to produce processed 2D image slices, wherein a combination of the processed 2D image slices forms an output of the neural network architecture.

[0027] Optionally, the input 3D image data is a medical image of an object; each 2D image slice along the first dimension is a slice located in the coronal plane of the object; each 2D image slice along the second dimension is a slice located in the axial plane of the object; and each 2D image slice along the third dimension is a slice located in the sagittal plane of the object.

[0028] The second neural network and the third neural network may be the same. Alternatively, the second neural network and the third neural network may be different.

[0029] Optionally, a size of each of the plurality of sub-volumes is no greater than one sixteenth of a size of the input 3D image data.

[0030] The method may also include the step of providing an output. This may include controlling a user interface to provide a user-perceivable output (e.g., a visual representation or an audio representation) and / or providing the output to a further processing system for further processing. Obviously, the output of the neural network architecture may provide useful information to perform one or more further processing tasks, such as a diagnostic task or a patient assessment task.

[0031] The output may be an output for use in a clinical assistance task.

[0032] The input 3D image data may include medical 3D image data, and the neural network architecture may be configured or trained to perform a clinical auxiliary task, such as identifying or predicting the presence or absence of a specific pathology in the medical 3D image data. Other suitable clinical auxiliary tasks (e.g., segmentation, denoising, etc.) will be apparent to the skilled person.

[0033] Also proposed is a computer program product comprising computer program code means which, when executed on a computing device having a processing system, causes the processing system to perform all the steps of any method described herein.

[0034] A processing system for processing 3D image data using a neural network architecture is also proposed. The processing system is configured to: obtain input 3D image data; perform a modified 3D image data generation process, including: for each of a plurality of 3D sub-volumes of the input 3D image data, use a first neural network to process the 3D sub-volume to generate first modified 3D image data; and / or for each 2D image slice along a first dimension of the input 3D image data, use a second neural network to process the 2D image slice to generate second modified 3D image data; and use an output neural network to process the input 3D image data and at least the first modified 3D image data and the second modified 3D image data to generate an output of the neural network architecture.

[0035] The processing system may be configured or adapted to perform the methods provided by any of the methods or mechanisms described herein, or vice versa.

[0036] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] For a better understanding of the invention, and to show more clearly how it may be put into practice, reference will now be made, by way of example only, to the accompanying drawings, in which:

[0038] Figure 1 shows a neural network architecture according to various embodiments;

[0039] Figure 2 The U-net neural network architecture is shown;

[0040] Figure 3 Example images illustrating the effects of the embodiments are provided;

[0041] Figure 4 is a flowchart illustrating a method according to an embodiment; and

[0042] Figure 5 A processing system according to an embodiment is shown. DETAILED DESCRIPTION

[0043] The present invention will be described with reference to the accompanying drawings.

[0044] It should be understood that the detailed description and specific examples, although indicating exemplary embodiments of the device, system and method, are intended only for illustrative purposes and are not intended to limit the scope of the invention. These and other features, aspects and advantages of the device, system and method of the present invention will be better understood based on the following description, the appended claims and the accompanying drawings. It should be understood that the drawings are only schematic and are not drawn to scale. It should also be understood that throughout the drawings, the same reference numerals are used to represent the same or similar parts.

[0045] The present invention provides a mechanism for processing input 3D image data. In a first stage, the input 3D image data is processed separately using one or more neural networks to generate one or more modified 3D image data. In a second stage, the input 3D image data and the modified 3D image data are processed using another neural network to generate an output. The one or more neural networks that generate the modified 3D image data are configured to process slices or sub-volumes of the input 3D image data to generate the modified 3D image data.

[0046] Embodiments are based on the recognition that existing methods for processing 3D image data using neural networks are either too inefficient to be effectively performed on existing hardware or lose potentially valuable (e.g., contextualized) information in the dimensions of the 3D image data. By using a two-stage approach, this contextual information can be incorporated and / or provided as input to the neural network that produces the output.

[0047] The methods disclosed in the embodiments can be applied to scenarios where 3D image data needs to be processed. However, the embodiments are particularly advantageous in the medical field, for example, for processing medical 3D image data. In this field, a large amount of 3D image data is generated, and improving the efficiency of processing such a large amount of data will be very useful.

[0048] In the context of the present invention, an image slice is a two-dimensional portion of three-dimensional or volumetric image data.

[0049] It will be appreciated that image slices may be extracted according to the three different dimensions of the 3D image data. An image slice "along a dimension" is an image slice located at a specific position in that dimension. Thus, if the 3D image data has dimensions [x, y, z], an image slice along dimension z will have dimensions [x, y].

[0050] In the context of the present disclosure, image slices correspond to each other if they contain or represent the same scene, ie represent the same imaged area.

[0051] Figure 1 The method used in the embodiment of the present invention is conceptually illustrated. Specifically, Figure 1A neural network architecture 100 is shown in which various components that may be used in embodiments of the present invention are shown.

[0052] The network architecture 100 is configured to process input 3D image data 191 to produce output 192 .

[0053] The output 192 itself may include 3D image data, such as a denoised version of the input 3D image data 191, an upsampled or higher resolution version of the input 3D image data 191, or synthesized image data. Alternatively, the output 192 may include some other possible neural network output of a 3D image, such as a segmentation mask or prediction indicator. Other suitable examples will be apparent to those skilled in the art.

[0054] The network architecture 100 is divided into two phases.

[0055] In the first stage 101, input 3D image data 191 is processed by one or more first stage neural networks 110, 120, 130, respectively. Each first stage neural network produces modified 3D image data 111, 121, 131.

[0056] A suitable example of a first stage neural network is a neural network that processes blocks of 3D image data (e.g., sub-volumes of input 3D image data). Such a neural network may be labeled a 3D sub-volume neural network. For example, the neural network architecture may be configured such that the first neural network 110 processes different sub-volumes of the input 3D image data to generate first modified 3D image data 111.

[0057] The size of each sub-volume may be no larger than one sixteenth of the size of the input 3D image data. This reduces the processing power required to process the input 3D image data using the 3D sub-volume neural network.

[0058] Another suitable example of a first stage neural network is a neural network that processes 2D image data or image slices. Such a neural network may be labeled a 2D image slice neural network. Each image slice in a specific dimension of the input 3D image data is processed using such a 2D image slice neural network to generate modified 3D image data. Processing the image slices may include generating processed or modified image slices.

[0059] By way of example only, the second neural network 120 may process the image slice along a first dimension of the input 3D image data 191 to generate the second modified 3D image data 121. In another example, the third neural network 130 may process the image slice along a dimension different from the first dimension of the input 3D image data 191 to generate the third modified 3D image data 131.

[0060] Although not specifically illustrated, the second and third neural networks can be the same (i.e., identical). This allows both a reduction in the size of the neural network architecture and enhanced training data. In other examples, the second and third neural networks can be independent, for example, trained using different training data sets.

[0061] For example, when the input 3D image data is image data of an object, each slice in the first dimension may be a slice located in one of a sagittal plane, a coronal plane, or an axial plane of the object.

[0062] In the second stage 102 , at least one of the input 3D image data 191 and the modified 3D image data 111 , 121 , 131 is processed by the second stage neural network 150 or the output neural network to produce an output 192 .

[0063] The second stage neural network (or "output neural network") can be configured to iteratively process a set of slices of the image data. Each set of slices includes an image slice of the input 3D image data and a corresponding image slice from one or more modified 3D image data. Each set of slices corresponds to a different position in a second dimension. The second dimension is different from the first dimension.

[0064] The second neural network 120 may process image slices along a first dimension of the input 3D image data 191 to produce second modified 3D image data 121. The output neural network may process image slices along a second dimension of the input 3D image data (and corresponding slices from one or more modified 3D image data) to produce an output; and the third neural network 130 may process image slices along a third dimension of the input 3D image data 191 to produce third modified 3D image data 131. The first, second, and third dimensions may all be different.

[0065] When the input 3D image data is image data of an object, each slice in the first dimension may be a slice located in one of the sagittal, coronal or axial planes of the object. Each slice in the second dimension may be a slice located in a different plane among the sagittal, coronal or axial planes of the object. Each slice in the third dimension may be a slice located in the last of the sagittal, coronal or axial planes of the object, if any.

[0066] Specifically, when the input 3D image data is image data of an object, each slice in the first dimension may be a slice located in a sagittal plane or a coronal plane of the object. Each slice in the second dimension may be a slice located in an axial plane of the object. If present, each slice in the third dimension may be a slice located in the other of the sagittal plane and the coronal plane of the object.

[0067] Thus, in the second stage 102, two, three or four 2D image input channels may be defined. The first channel is for a slice of the input 3D image data. The remaining channels are for corresponding slices of the first and second (and optionally third) modified image data generated by the first stage neural network(s) in the first stage 101, respectively.

[0068] In other words, in the second stage 102, for each 2D image slice along the second dimension of the input 3D image data 191, a corresponding 2D image slice from the first modified 3D imaging data 111 and the second modified 3D image data 121 (and optionally the third modified 3D image data 131) is obtained. The 2D image slices of the input 3D image data and any obtained corresponding 2D image slices are then processed using the second stage neural network 150 (also referred to as the output neural network) to produce processed 2D image slices. The combination of the processed 2D image slices (produced by each 2D image slice of the processed input 3D image data) forms the output of the neural network architecture 100.

[0069] Therefore, the output of the second stage neural network 150 may include single channel result image data.

[0070] However, it should be understood that the precise processing performed by each neural network in the neural network architecture may vary depending on the intended purpose or goal of the neural network architecture 100 (i.e., output 192).

[0071] In some embodiments, each neural network is trained or configured for the same purpose, e.g., to produce or predict the same output.

[0072] For example, if the purpose of the neural network architecture 100 is to synthesize 3D image data of a different modality than the input 3D image data, each neural network may be trained or configured to synthesize 3D image data of a different modality than the input 3D image data.

[0073] As another example, if the purpose of the neural network architecture 100 is to denoise 3D image data, each neural network can be trained or configured to denoise the input 3D image data 191.

[0074] However, the embodiments are not limited thereto.

[0075] For example, a first stage neural network may be trained or configured to generate modified 3D image data (e.g., denoised 3D image data or contrast-enhanced 3D image data), while a second stage neural network may be trained or configured to predict or identify the presence of a specific pathology in the input 3D image data.

[0076] As another example, the first stage neural network 110, 120, 130 may be trained or configured to generate modified 3D image data (e.g., denoised 3D image data or contrast-enhanced 3D image data), and the second stage neural network may be trained or configured to segment the input 3D image data.

[0077] As another example, the first stage neural networks 110, 120, 130 may be trained or configured to generate modified 3D image data (e.g., denoised 3D image data or contrast-enhanced 3D image data), and the second stage neural network 150 may be trained or configured to synthesize 3D image data of a different modality than the input 3D image data.

[0078] Other suitable examples will be apparent to those skilled in the art.

[0079] Embodiments utilize a neural network to process image data (eg, a 2D image slice or a 3D image data block) to produce an output.

[0080] The structure of a neural network is inspired by the human brain. A neural network consists of layers, each containing multiple neurons. Each neuron includes a mathematical operation. Specifically, each neuron can include a different weighted combination of a single type of transformation (e.g., the same type of transformation, sigmoid, etc., but with different weights). In the process of processing input data, each neuron performs a mathematical operation on the input data and produces a numerical output. The output of each layer in the neural network is sequentially fed into the next layer. The last layer provides the output.

[0081] Methods for training neural networks are well known. Typically, such methods include obtaining a training data set, the training data set comprising training input data entries and corresponding training output data entries. An initialized neural network is applied to each input data entry to generate a predicted output data entry. The error between the predicted output data entry and the corresponding training output data entry is used to modify the neural network. The process can be repeated until the error converges and the predicted output data entry is sufficiently similar to the training output data entry (e.g., ±1%). This is generally referred to as a supervised learning technique.

[0082] For example, the weight of the mathematical operation of each neuron can be modified until the error converges. Known methods for modifying neural networks include gradient descent, back propagation algorithms, etc.

[0083] In a particularly advantageous embodiment, each neural network of the neural network architecture has a U-net architecture. U-Net and its 3D version 3D U-Net are considered to provide some of the standard and most successful CNN architectures for image processing tasks.

[0084] Figure 2 A neural network 200 is illustrated having a basic U-net architecture, formed of two paths: a contracting path 210 and an expanding path 220. The neural network processes an input 201 to produce an output 202.

[0085] The contracting path consists of a series of levels / blocks formed by layers, such as convolutional layers 211 and pooling layers 212. The dilating path consists of a series of levels / blocks formed by layers, such as convolutional layers 221 and upconvolutional / upsampling layers 222. The number of levels / blocks in the dilating path is the same as the number of levels / blocks in the contracting path. The number of pooling layers in the contracting path 210 (and therefore the number of upconvolutional layers in the dilating path 220) defines the number of levels / blocks in each path, and therefore the depth of the U-net. In the illustrated example, the depth of the neural network is 4.

[0086] As shown, there may be one or more feed-forward links 250 between the same levels on the contracting and expanding paths.

[0087] Studies of the relationship between network depth and performance have shown that performance generally improves as the network depth increases, typically up to a depth of four layers.

[0088] A more complete explanation of the basic (2D) U-net architecture is given in Ronneberger, Olaf, Philipp Fischer, and Thomas Brox, “U-net: Convolutional networks for biomedical image segmentation.” International Conference on Medical image computing and computer-assisted intervention. Springer, Cham (2015).

[0089] Ahmed Abdulkadir, Soeren S. Lienkamp, ​​Thomas Brox, Olaf Ronneberger. 3D U-Net: Learning dense volumetric segmentation from sparse annotation. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Athens, Greece, (October 17–21, 2016). A suitable 3D U-net architecture is explained.

[0090] As a working example, the proposed network architecture is evaluated on CT volume data. In this working example, each neural network of the neural network architecture is trained to synthesize monochromatic 50keV results from conventional CT volume data.

[0091] For this working example, the training dataset for each neural network consisted of 14 independent clinical studies with a total of 6552 slices. All studies were abdominal scans reconstructed using the filtered back-projection algorithm with a slice thickness of 1 mm, slice increments of 0.9 mm or 1 mm, and a field of view between 350 mm and 425 mm.

[0092] Six neural network architectures were trained to generate the outputs.

[0093] The first neural network architecture uses a first two-dimensional neural network, that is, a neural network that processes only image slices. Each image slice along the first dimension of the input 3D image data is processed using the first two-dimensional neural network to generate first output 3D image data. The first two-dimensional neural network uses a U-net architecture with a depth of four.

[0094] The second neural network architecture uses a single three-dimensional neural network to process the input 3D image data. More specifically, the single three-dimensional neural network processes a block / block / sub-volume of the input 3D image data, and the processing is performed on the entire input 3D image data (with padding). Therefore, the input 3D image data is divided into blocks / sub-volumes, each block / sub-volume is processed by a single three-dimensional neural network process to produce the second output image data.

[0095] The third neural network architecture uses a second two-dimensional neural network, i.e., a neural network that processes only image slices. Each image slice along the second and / or third dimension of the input 3D image data is processed using the second two-dimensional neural network to generate output 3D image data. Processing the image slice along the second dimension generates third output image data. Processing the image slice along the third dimension generates fourth output image data. The second two-dimensional neural network uses a U-net architecture with a depth of four.

[0096] The fourth neural network architecture uses a third two-dimensional neural network. The inputs to the third two-dimensional neural network are image slices from the input 3D image data and image slices from the second output image data (the output of the second neural network architecture). Each image slice is an image slice obtained along the first dimension. Therefore, the third two-dimensional neural network receives a dual-channel input: 1) image slices from the regular input image data, and 2) image slices from the output of the second neural network architecture. Multiple sets of image slices (each set containing one slice for each channel) are processed separately by the third two-dimensional neural network to produce fifth output image data. Each set of image slices corresponds to a different position on a particular dimension (for example, the input image data contains medical image data - different axial, coronal or sagittal images). The depth of the third two-dimensional neural network is four.

[0097] The fifth neural network architecture uses a fourth two-dimensional neural network. The inputs to the fourth two-dimensional neural network are image slices from the input 3D image data, image slices from the third output image data, and image slices from the fourth output image data. Each image slice is an image slice obtained along the first dimension. Therefore, the fourth two-dimensional neural network receives a three-channel input: 1) an image slice from the conventional input image data; 2) an image slice from the third output image data; and 3) an image slice from the fourth output image data. Multiple sets of image slices (each set containing one slice for each channel) are processed separately by the fourth two-dimensional neural network to produce the sixth output image data. Each set of image slices corresponds to a different position on a particular dimension (e.g., the input image data contains medical image data - different axial, coronal or sagittal images).

[0098] The sixth neural network architecture uses a fifth two-dimensional neural network. The inputs to the fifth two-dimensional neural network are image slices from the input 3D image data, image slices from the second output image data; image slices from the third output image data; and image slices from the fourth output image data. Each image slice is an image slice obtained along the first dimension. Therefore, the fifth two-dimensional neural network receives four-channel inputs: 1) image slices from regular input image data; 2) image slices from the second output image data; 3) image slices from the third output image data; and 4) image slices from the fourth output image data. Multiple sets of image slices (each set containing one slice for each channel) are processed separately by the fifth two-dimensional neural network to produce the seventh output image data. Each set of image slices corresponds to a different position on a particular dimension (e.g., the input image data contains medical image data - different axial, coronal or sagittal images).

[0099] For the purposes of this review: image slices in the first dimension were images that were in the axial plane or parallel to the axial plane; image slices in the second dimension were images that were in the coronal plane or parallel to the coronal plane; and image slices in the third dimension were images that were in the sagittal plane or parallel to the sagittal plane.

[0100] The evaluation results are summarized in Table 1 below.

[0101]

[0102] Table 1

[0103] In Table 1, L1 represents the value of the first loss function L1(.), which is the sum of the absolute differences between the true value and the predicted value for each pixel (across all pixels of the output image data generated by the relevant network architecture). L2 represents the value of the second loss function L2(.), which is the sum of the squared differences between the true value and the predicted value for each pixel (across all pixels of the output image data generated by the relevant network architecture).

[0104] The true value of the pixel is contained in the ground truth image data, which is obtained by a CT scanner operating in a monochrome 50keV mode. The predicted value of the pixel is contained in the output image data provided by the associated network architecture. The output image data is configured to attempt to predict the content of the ground truth image data.

[0105] It can be seen that the proposed method improves the results. Specifically, the proposed method (embodied in the fourth, fifth and sixth neural network structures) is significantly improved over the currently existing and available neural network structures (embodied in the first and second neural network structures).

[0106] Figure 3 The advantages of the proposed method are further demonstrated.

[0107] Figure 3 Four representative image slices are provided, each showing the same scene, ie, the same portion of the abdomen of a subject.

[0108] The first image slice 310 is an image slice taken from conventional CT 3D image data. The second slice 320 is an image slice acquired from 3D image data acquired by a CT scanner operating in a monochrome 50keV mode. The third slice 330 is an image slice taken from output image data generated by processing the conventional CT 3D image data using the first neural network architecture. The fourth slice 340 is an image slice taken from output image data generated by processing the conventional CT 3D image data using the sixth neural network architecture.

[0109] The second slice 320 effectively represents the “ground truth” image data that the neural network architecture attempts to reconstruct from the conventional CT 3D image data.

[0110] It can be seen that the fourth slice is more similar to the second slice than the third slice. That is, the difference between the fourth slice 340 and the second slice 320 is smaller than the difference between the third slice 330 and the second slice 320. This difference may be most obvious in the marked area 390.

[0111] Figure 4 A computer-implemented method 400 is shown according to an embodiment. The method 400 is for processing 3D image data using a neural network architecture.

[0112] The method 400 comprises a step 410 of obtaining input 3D image data. The input 3D image data may for example be a medical image of an object. However, other suitable examples of 3D image data will be clear to those skilled in the art.

[0113] Step 410 may include receiving input 3D image data from a processing system (eg, an imaging system generating a medical 3D image). In other examples, step 410 may include retrieving the input 3D image data from a memory or storage unit.

[0114] The method 400 also includes a process 420 of executing a modified 3D image data generation process. The modified 3D image data generation process generates one or more modified image data, each modified image data being a modified version of the input 3D image data.

[0115] Specifically, process 420 includes executing step 421 and step 422 .

[0116] Step 421 comprises: for each of a plurality of 3D sub-volumes of the input 3D image data, processing the 3D sub-volume using a first neural network to generate first modified 3D image data;

[0117] Step 422 includes: for each 2D image slice along a first dimension of the input 3D image data, processing the 2D image slice using a second neural network to generate second modified 3D image data;

[0118] The first and second neural networks may be embodied as the first stage neural network as described above.

[0119] After performing step 420, method 400 performs step 430, i.e., processing the input 3D image data and at least the first modified 3D image data and the second modified 3D image data using an output neural network to produce an output of the neural network architecture. The output neural network is functionally equivalent to the second stage neural network described above.

[0120] Step 430 thus includes processing the input 3D image data and any modified 3D image data generated by process 420 to produce an output. The output itself may include 3D image data, such as a denoised version of the input 3D image data, an upsampled or higher resolution version of the input 3D image data, or synthesized image data. Alternatively, the output may include some other possible neural network output of the 3D image, such as a segmentation mask or prediction indicator. Other suitable examples will be apparent to those skilled in the art.

[0121] In some examples, step 430 includes, for each 2D image slice along a second dimension of the input 3D image data, the second dimension being different from the first dimension (if used): obtaining a corresponding 2D image slice from the first modified 3D image data and the second modified 3D imaging data; processing the 2D image slice of the input 3D image data, and at least a corresponding 2D image slice from the first modified 3D image data and a corresponding slice from the second modified 3D imaging data using the sensitive neural network to produce a processed 2D image slice, the combination of the processed 2D image slices forming an output of the neural network architecture.

[0122] In some embodiments, when process 420 includes performing step 421, the process may further include performing step 423. Step 423 includes, for each 2D image slice along a third dimension of the input 3D image data, processing the 2D image slice along the third dimension using a third neural network to generate third modified 3D image data. The third dimension is different from the first dimension and the second dimension.

[0123] Step 430 may be suitably adapted to include processing at least the input 3D image data, the first modified 3D image data and the third modified 3D image data using a third neural network to produce an output of the neural network architecture.

[0124] Of course, step 430 may include processing the input 3D image data, the first modified 3D image data, the second modified 3D image data, and the third modified 3D image data using a third neural network to produce an output of the neural network architecture.

[0125] Method 400 may also include a step 440 of providing an output. Step 440 may include, for example, controlling a user interface to provide a user-perceivable output (e.g., a visual representation or an audio representation) of the output generated in step 430. As another example, step 440 may include providing the output to a further processing system for further processing. Clearly, the output of the neural network architecture may provide useful information for performing one or more further processing tasks.

[0126] As another example, Figure 5 An example of a processing system 500 is illustrated in which one or more portions of the embodiments may be employed. The various operations discussed above may utilize the capabilities of the processing system 500. For example, one or more portions of a system for processing 3D image data using a neural network architecture may be incorporated into any of the elements, modules, applications, and / or components discussed herein. In this regard, it should be understood that the system functional blocks may be run on a single computer or may be distributed across multiple computers and locations (e.g., connected via the Internet).

[0127] The processing system 500 includes, but is not limited to, a PC, a workstation, a laptop computer, a PDA, a handheld device, a server, a memory, and the like. Typically, in terms of hardware architecture, the processing system 500 may include one or more processors 501, a memory 502, and one or more I / O devices 570 communicatively coupled via a local interface (not shown). The local interface may be, for example but not limited to, one or more buses or other wired or wireless connections, as known in the art. The local interface may have additional elements, such as controllers, buffers (caches), drivers, repeaters, and receivers, to enable communication. In addition, the local interface may include addressing, control, and / or data connections to enable appropriate communication between the above components.

[0128] Processor 501 is a hardware device for running software that may be stored in memory 502. Processor 501 may be virtually any custom or commercial processor, a central processing unit (CPU), a digital signal processor (DSP), or an auxiliary processor among several processors associated with processing system 500, and processor 501 may be a semiconductor-based microprocessor (in the form of a microchip) or a microprocessor.

[0129] The memory 502 may include any one or a combination of volatile memory elements (e.g., random access memory (RAM), such as dynamic random access memory (DRAM), static random access memory (SRAM), etc.) and non-volatile memory elements (e.g., ROM, erasable programmable read-only memory (EPROM), electronically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic tape, compact disk read-only memory (CD-ROM), magnetic disk, floppy disk, card, cassette, etc.). In addition, the memory 502 may contain electronic, magnetic, optical and / or other types of storage media. Note that the memory 502 may have a distributed architecture in which various components are remote from each other but can be accessed by the processor 501.

[0130] The software in the memory 502 may include one or more separate programs, each of which includes an ordered list of executable instructions for implementing logical functions. According to an exemplary embodiment, the software in the memory 502 includes a suitable operating system (O / S) 505, a compiler 504, a source code 503, and one or more application programs 506. As shown, the application program 506 includes numerous functional components for implementing the features and operations of the exemplary embodiment. According to an exemplary embodiment, the application program 506 of the processing system 500 may represent various application programs, computing units, logics, functional units, processes, operations, virtual entities, and / or modules, but the application program 506 is not intended to be limiting.

[0131] The operating system 505 controls the execution of other computer programs and provides scheduling, input-output control, file and data management, memory management, and communication control and related services. The inventors expect that the application program 506 used to implement the exemplary embodiments can be applied to all commercially available operating systems.

[0132] Application 506 may be a source program, an executable program (object code), a script, or any other entity including a set of instructions to be executed. In the case of a source program, the program is typically translated by a compiler (e.g., compiler 504), an assembler, an interpreter, etc., which may or may not be included in memory 502, so as to operate correctly with operating O / S 505. In addition, application 506 may be written in an object-oriented programming language having classes of data and methods, or a procedural programming language having routines, subroutines, and / or functions, such as, but not limited to, C, C++, C#, Pascal, BASIC, API calls, HTML, XHTML, XML, ASP scripts, JavaScript, FORTRAN, COBOL, Perl, Java, ADA, .NET, etc.

[0133] I / O devices 507 may include input devices such as, but not limited to, a mouse, keyboard, scanner, microphone, camera, etc. In addition, I / O devices 507 may also include output devices such as, but not limited to, a printer, display, etc. Finally, I / O devices 507 may also include devices that transmit both input and output such as, but not limited to, a NIC or modulator / demodulator (for accessing remote devices, other files, devices, systems, or networks), a radio frequency (RF) or other transceiver, a telephone interface, a bridge, a router, etc. I / O devices 507 also include components for communicating over various networks such as the Internet or an intranet.

[0134] If the processing system 500 is a PC, workstation, intelligent device, etc., the software in the memory 502 may also include a basic input and output system (BIOS) (omitted for simplicity). The BIOS is a set of basic software routines that initialize and test the hardware at startup, start the O / S 505, and support data transfer between hardware devices. The BIOS is stored in some kind of read-only memory, such as ROM, PROM, EPROM, EEPROM, etc., so that the BIOS is run when the processing system 500 is started.

[0135] When the processing system 500 is in operation, the processor 501 is configured to execute software stored in the memory 502, to transfer data to and from the memory 502, and to generally control the operation of the processing system 500 according to the software. The application programs 506 and the operating O / S 505 are read in whole or in part by the processor 501, possibly buffered within the processor 501, and then executed.

[0136] When the application 506 is implemented in software, it should be noted that the application 506 can be stored on virtually any computer-readable medium for use by or in conjunction with any computer-related system or method. In the context of this document, a computer-readable medium can be an electronic, magnetic, optical, or other physical device or means that can contain or store a computer program for use by or in conjunction with a computer-related system or method.

[0137] Application 506 may be implemented in various computer-readable media for use with or in conjunction with an instruction execution system, device, or apparatus, such as a computer-based system, a system containing a processor, or other system capable of retrieving instructions from an instruction execution system, device, or apparatus and executing the instructions. In the context of this document, a "computer-readable medium" may be any means by which a program may be stored, communicated, propagated, or transmitted for use with or in conjunction with an instruction execution system, device, or apparatus. A computer-readable medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium.

[0138] Ordinal numbers (e.g., "first," "second," etc.) are used only to distinguish different elements from each other for clarity, and reference to a non-"first" (e.g., "second" or "third") element does not necessarily mean that there is a "first" element. Those skilled in the art are able to relabel any such element as needed (e.g., if there is only a second element, relabel the "second" element as a "first" element).

[0139] Those skilled in the art will be able to understand and implement variations to the disclosed embodiments in practicing the claimed invention by studying the drawings, the disclosure and the claims. In the claims, the word "comprising" does not exclude other elements or steps, and the word "a" or "an" does not exclude a plurality. Although certain measures are recited in mutually different dependent claims, this does not indicate that a combination of these measures cannot be used to advantage.

[0140] A single processor or other unit may implement the functions of several items recited in the claims. The computer program may be stored / distributed on a suitable medium such as an optical storage medium or a solid-state medium provided together with or as part of other hardware, but may also be distributed in other forms such as via the Internet or other wired or wireless telecommunications systems.

[0141] If the term “suitable for” is used in the claims or the specification, it should be noted that the term “suitable for” is intended to be equivalent to the term “configured to.” Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. A computer-implemented method (400) for processing 3D image data using a neural network architecture (100), the computer-implemented method comprising: Obtaining input 3D image data (191); Executing a modified 3D image data generation process (420) includes: for each of a plurality of 3D sub-volumes of the input 3D image data, processing the 3D sub-volume using a first neural network (110) to generate first modified 3D image data (111); and for each 2D image slice along a first dimension of the input 3D image data, processing the 2D image slice using a second neural network (120) to generate second modified 3D image data (121); Characterized in that the method further comprises: The input 3D image data (191) and at least the first modified 3D image data (111) and the second modified 3D image data (121) are processed using an output neural network (150) to produce an output of the neural network architecture.

2. The computer-implemented method of claim 1 , wherein: The step of processing the input 3D image data and the second modified 3D image data and the first modified 3D image data comprises: For each 2D image slice along a second dimension of the input 3D image data different from the first dimension: obtaining corresponding 2D image slices from the first modified 3D image data and the second modified 3D imaging data; and processing the 2D image slice of the input 3D image data and at least a corresponding 2D image slice from the first modified 3D image data and a corresponding slice from the second modified 3D imaging data using the output neural network to produce a processed 2D image slice, Wherein, the combination of the processed 2D image slices forms the output of the neural network architecture.

3. The computer-implemented method of claim 2, wherein: The input 3D image data is a medical image of an object, and each 2D image slice along the first dimension is a slice located in a coronal plane or a sagittal plane of the object, and each 2D image slice along the second dimension is a slice located in an axial plane of the object.

4. A computer-implemented method according to any one of claims 1 to 3, wherein: The first neural network, the second neural network and the output neural network all have a U-net architecture.

5. A computer-implemented method according to any one of claims 1 to 3, in, The modified 3D image data generation process comprises: for each 2D image slice along a first dimension of the input 3D image data, processing the 2D image slice using a second neural network to generate second modified 3D image data; and for each 3D sub-volume of a plurality of 3D sub-volumes of the input 3D image data, processing the 3D sub-volume using a first neural network to generate first modified 3D image data; and wherein processing the input 3D image data and at least the first modified 3D image data and the second modified 3D image data comprises processing the input 3D image data and at least the first modified 3D image data and the second modified 3D image data using the output neural network to produce the output of the neural network architecture.

6. The computer-implemented method of claim 5, wherein: The modified 3D image data generation process (420) further comprises: for each 2D image slice along a third dimension of the input 3D image data, processing the 2D image slice along the third dimension using a third neural network (130) to generate third modified 3D image data (131); The third dimension is different from the first dimension; and Processing the input 3D image data and at least the first modified 3D image data and the second modified 3D image data includes using the output neural network to process at least the input 3D image data, the second modified 3D image data and the third modified 3D image data to produce the output of the neural network architecture.

7. The computer-implemented method of claim 6, wherein: Processing the input 3D image data and at least the first modified 3D image data and the second modified 3D image data includes using the output neural network to process the input 3D image data, the first modified 3D image data, the second modified 3D image data and the third modified 3D image data to produce the output of the neural network architecture.

8. The computer-implemented method of claim 6, wherein: The input 3D image data is a medical image of an object; Each 2D image slice along the first dimension is a slice located in a coronal plane of the object; Each 2D image slice along the second dimension is a slice located in an axial plane of the object; and Each 2D image slice along the third dimension is a slice located in a sagittal plane of the object.

9. The computer-implemented method of claim 6, wherein: The second neural network and the third neural network are the same neural network.

10. The computer-implemented method of claim 6, wherein: The second neural network and the third neural network are different.

11. The computer-implemented method of any one of claims 1 to 3, wherein: A size of each of the plurality of sub-volumes is no greater than one sixteenth of a size of the input 3D image data.

12. A computer-implemented method (400) for processing 3D image data using a neural network architecture (100), the computer-implemented method comprising: Obtaining input 3D image data (191); Executing a modified 3D image data generation process (420) includes: for each of a plurality of 3D sub-volumes of the input 3D image data, processing the 3D sub-volume using a first neural network (110) to generate first modified 3D image data (111); and for each 2D image slice along a first dimension of the input 3D image data, processing the 2D image slice using a second neural network (120) to generate second modified 3D image data (121); Characterized in that the method further comprises: processing the input 3D image data (191) and at least the first modified 3D image data (111) and the second modified 3D image data (121) using an output neural network (150) to produce an output of the neural network architecture, The step of processing the input 3D image data and the second modified 3D image data and the first modified 3D image data comprises: For each 2D image slice along a second dimension of the input 3D image data different from the first dimension: obtaining corresponding 2D image slices from the first modified 3D image data and the second modified 3D imaging data; and processing the 2D image slice of the input 3D image data and at least a corresponding 2D image slice from the first modified 3D image data and a corresponding slice from the second modified 3D imaging data using the output neural network to produce a processed 2D image slice, wherein the combination of the processed 2D image slices forms the output of the neural network architecture, The modified 3D image data generation process includes: for each 2D image slice along a first dimension of the input 3D image data, processing the 2D image slice using a second neural network to generate second modified 3D image data; and for each 3D sub-volume of a plurality of 3D sub-volumes of the input 3D image data, processing the 3D sub-volume using a first neural network to generate first modified 3D image data; and wherein processing the input 3D image data and at least the first modified 3D image data and the second modified 3D image data comprises processing the input 3D image data and at least the first modified 3D image data and the second modified 3D image data using the output neural network to produce the output of the neural network architecture, The modified 3D image data generation process (420) further comprises: for each 2D image slice along a third dimension of the input 3D image data, processing the 2D image slice along the third dimension using a third neural network (130) to generate third modified 3D image data (131); The third dimension is different from the first dimension; and processing the input 3D image data and at least the first modified 3D image data and the second modified 3D image data comprises processing at least the input 3D image data, the second modified 3D image data and the third modified 3D image data using the output neural network to produce the output of the neural network architecture, The step of processing the input 3D image data and at least the first modified 3D image data and the second modified 3D image data comprises: For each 2D image slice along the second dimension of the input 3D image data: obtaining corresponding 2D image slices from the second modified 3D image data; obtaining corresponding 2D image slices from the third modified 3D image data; and processing the 2D image slice of the input 3D image data and at least a corresponding 2D image slice from the second modified 3D image data and a corresponding slice from the third modified 3D imaging data using the output neural network to produce a processed 2D image slice, Wherein, the combination of the processed 2D image slices forms the output of the neural network architecture.

13. A computer-implemented method (400) for processing 3D image data using a neural network architecture (100), the computer-implemented method comprising: Obtaining input 3D image data (191); Executing a modified 3D image data generation process (420) includes: for each of a plurality of 3D sub-volumes of the input 3D image data, processing the 3D sub-volume using a first neural network (110) to generate first modified 3D image data (111); and for each 2D image slice along a first dimension of the input 3D image data, processing the 2D image slice using a second neural network (120) to generate second modified 3D image data (121); Characterized in that the method further comprises: processing the input 3D image data (191) and at least the first modified 3D image data (111) and the second modified 3D image data (121) using an output neural network (150) to produce an output of the neural network architecture, The step of processing the input 3D image data and the second modified 3D image data and the first modified 3D image data comprises: For each 2D image slice along a second dimension of the input 3D image data different from the first dimension: obtaining corresponding 2D image slices from the first modified 3D image data and the second modified 3D imaging data; and processing the 2D image slice of the input 3D image data and at least a corresponding 2D image slice from the first modified 3D image data and a corresponding slice from the second modified 3D imaging data using the output neural network to produce a processed 2D image slice, wherein the combination of the processed 2D image slices forms the output of the neural network architecture, The modified 3D image data generation process includes: for each 2D image slice along a first dimension of the input 3D image data, processing the 2D image slice using a second neural network to generate second modified 3D image data; and for each 3D sub-volume of a plurality of 3D sub-volumes of the input 3D image data, processing the 3D sub-volume using a first neural network to generate first modified 3D image data; and wherein processing the input 3D image data and at least the first modified 3D image data and the second modified 3D image data comprises processing the input 3D image data and at least the first modified 3D image data and the second modified 3D image data using the output neural network to produce the output of the neural network architecture, The modified 3D image data generation process (420) further comprises: for each 2D image slice along a third dimension of the input 3D image data, processing the 2D image slice along the third dimension using a third neural network (130) to generate third modified 3D image data (131); The third dimension is different from the first dimension; and processing the input 3D image data and at least the first modified 3D image data and the second modified 3D image data comprises processing at least the input 3D image data, the second modified 3D image data and the third modified 3D image data using the output neural network to produce the output of the neural network architecture, wherein processing the input 3D image data and at least the first modified 3D image data and the second modified 3D image data comprises processing the input 3D image data, the first modified 3D image data, the second modified 3D image data and the third modified 3D image data using the output neural network to produce the output of the neural network architecture, The step of processing the input 3D image data and at least the first modified 3D image data and the second modified 3D image data comprises: For each 2D image slice along the second dimension of the input 3D image data: obtaining corresponding 2D image slices from said first modified 3D image data; obtaining corresponding 2D image slices from the second modified 3D image data; obtaining corresponding 2D image slices from the third modified 3D image data; and processing the 2D image slice of the input 3D image data and at least a corresponding 2D image slice from the first modified 3D image data and a corresponding slice from the second modified 3D image data and a corresponding slice from the third modified 3D image data using the output neural network to produce a processed 2D image slice, Wherein, the combination of the processed 2D image slices forms the output of the neural network architecture.

14. A computer program product comprising computer program code means which, when executed on a computing device having a processing system, causes the processing system to perform all the steps of the method according to any one of claims 1 to 11.

15. A processing system (500) for processing 3D image data using a neural network architecture (100), wherein: The processing system comprises a processor (501) and a memory (502), wherein the processor (501) is configured to run software stored in the memory (502) to cause the processing system to: Obtaining input 3D image data (191); Executing a modified 3D image data generation process (420) includes: for each of a plurality of 3D sub-volumes of the input 3D image data, processing the 3D sub-volume using a first neural network (110) to generate first modified 3D image data (111); and for each 2D image slice along a first dimension of the input 3D image data, processing the 2D image slice using a second neural network (120) to generate second modified 3D image data (121); Characterized in that the processor (501) is further configured to run the software to cause the processing system to: The input 3D image data (191) and at least the first modified 3D image data (111) and the second modified 3D image data (121) are processed using an output neural network (150) to produce an output of the neural network architecture.

Images