Image processing device and image processing method

Abstract

An image processing device generates a measurement image in a measurement space that differs from real space, based on measurement data of a subject acquired by a tomography device. The device inputs the measurement image into a CNN, which then generates a tomographic image in real space. The CNN is based on a U-Net structure comprising an encoder unit 21, a decoder unit 22, and a bottleneck unit 23, and further comprising a projection link unit 24. The projection link unit 24 performs projection processing from a sinogram space (the measurement space) to real space on an arbitrary feature map in the encoder unit 21 and, after the projection processing is complete, connects the feature map to the feature map of the same size in the decoder unit 22.In this way, an image processing device is realized that is able to suppress an increase in the amount of data required for training a CNN and to improve the performance of generating a tomographic image by the CNN.

Description

Technical field

[0001] The present invention relates to a device and a method for generating a tomographic image of a subject based on measurement data of the subject acquired by a measurement using a tomography device. State of the art

[0002] Examples of a tomography device that can acquire measurement data to produce a tomographic image of a subject include a positron emission tomography (PET) device, a single-photon emission computed tomography (SPECT) device, an X-ray computed tomography (CT) device, and a magnetic resonance imaging (MRI) device.

[0003] The measurement data of the subject acquired using the aforementioned tomography devices do not directly represent the tomographic image of the subject in real space without alteration. To generate the tomographic image of the subject in real space based on the measurement data acquired by the tomography device, a measurement image is first generated in a measurement space that differs from real space, based on the measurement data. Subsequently, the tomographic image in real space is generated based on this measurement image. Based on the aforementioned tomographic image, the subject's health status can be diagnosed.

[0004] In the case of the PET device, the SPECT device and the X-ray CT device, the measuring space is set to a sinogram space and the measured image to a sinogram.

[0005] In the case of a three-dimensional PET device, the sinogram is expressed as a histogram representing a frequency (a generation frequency of coincidence events) at which coincidence detection information is acquired in the sinogram space, which is represented by four variables: r, θ, z, and δ. The variable r represents a position in the radial direction of a coincidence detection line (a line connecting two detectors that perform coincidence detection of a photon pair). The variable θ represents an azimuth angle of the coincidence detection line. The variable z represents a position in the mid-axis direction of a center point of the coincidence detection line. Furthermore, the variable δ represents a polar angle of the coincidence detection line.

[0006] In the case of the MRI device, the measurement space is defined as a k-space, which is represented with a spatial frequency as a variable, and the measurement image is formed by k-space data.

[0007] When generating the tomographic image in real space based on the measurement image in the measurement space, image reconstruction processing is performed. Several techniques are known for image reconstruction from the sinogram to the tomographic image. For example, maximum likelihood expectation maximization (ML-EM), ordered subset expectation maximization (OSEM), row action maximum likelihood algorithm (RAMLA), and dynamic RAMLA (DRAMA), among others, are known.

[0008] In the image reconstruction methods mentioned above, the forward projection from real space into sinogram space and the back projection from sinogram space into real space are performed alternately and repeatedly, so that the reconstruction of the tomographic image is iterative.

[0009] Furthermore, a technique has been proposed that uses a neural network to generate the tomographic image in real space from the measurement image in the measurement space. The aforementioned technique is expected to be capable of generating the tomographic image with higher performance. A technique described in non-patent document 1 aims to input the sinogram into a convolutional neural network (CNN) and generate the tomographic image using the CNN. Reference list: Non-patented literature

[0010] Non-patent document 1: I. Haggstrom et al., “DeepPET: A deep encoder-decoder network for directly solving the PET image reconstruction inverse problem,” Med. Image Anal. Vol. 54, pp. 253-262, 2019 Summary of the invention: Technical problem

[0011] The method described in Non-Patent Document 1 requires that the CNN be pre-trained using many pairs of sinograms and tomographic images of the subjects. The input image for the CNN is the sinogram in sinogram space, and the output image from the CNN is the tomographic image in real space. The input image (the sinogram) and the output image (the tomographic image) are images of different types in different spaces and differ significantly in their appearance. Therefore, generating the tomographic image based on the sinogram using the CNN is a difficult task. Accordingly, the method described in Non-Patent Document 1 requires large amounts of data (pairs of sinograms and tomographic images) for training the CNN.

[0012] In cases where the amount of data used for training is small, the CNN cannot learn an algorithm to generate the tomographic image from the sinogram and only stores the pairs of sinograms and tomographic images used for training. Furthermore, if an unknown sinogram is input, the CNN outputs the tomographic image that most closely matches the sinogram from the sinograms used for training.

[0013] In the method described in non-patent document 1, the performance of generating the tomographic image using the CNN as described above is low when the amount of data used to train the CNN is small. To improve the performance of generating the tomographic image using the CNN, a large amount of data is required for training the CNN.

[0014] The problems in generating the tomographic image using the CNN described above occur not only when generating the tomographic image from the sinogram in the PET device, but also when generating the tomographic image in real space from the measurement image in the measurement space in each of the SPECT devices, the X-ray CT device and the MRI device.

[0015] One objective of the present invention is to provide an image processing device and an image processing method with which the amount of data required for training a CNN can be reduced and the performance of the creation of a tomographic image by the CNN can be improved. Solution to the problem

[0016] One embodiment of the present invention is an image processing device. The image processing device is an image processing device for generating a tomographic image of a subject in real space based on measurement data of the subject acquired by a measurement using a tomography device, and comprises (1) a measurement image generation unit for generating a measurement image in a measurement space that differs from real space, based on the measurement data;and (2) a CNN processing unit for inputting the measurement image into a convolutional neural network and for generating the tomographic image by the convolutional neural network, wherein the convolutional neural network comprises (a) an encoder unit for inputting the measurement image, updating a feature map by means of a convolution operation performed by a plurality of convolution layers, and gradually decreasing a spatial size of the feature map; (b) a decoder unit provided in a subsequent stage of the encoder unit, which updates the feature map by means of a convolution operation, each performed by a plurality of convolution layers, and gradually increases the spatial size of the feature map;and (c) a projection linking unit for performing projection processing from the measurement space to real space on any feature map in the encoder unit and for connecting the feature map after performing the projection processing with the feature map of the same size in the decoder unit.;

[0017] One embodiment of the present invention is a tomography system. The tomography system comprises a tomography device for acquiring measurement data to generate a tomographic image of a subject; and the image processing device of the above-mentioned configuration for generating the tomographic image based on the measurement data.

[0018] One embodiment of the present invention is an image processing method. The image processing method is an image processing method for generating a tomographic image of a subject in real space based on measurement data of the subject acquired by a measurement using a tomography device, and it comprises (1) a measurement image generation step for generating a measurement image in a measurement space that differs from real space, based on the measurement data;and (2) a CNN processing step for inputting the measurement image into a convolutional neural network and for generating the tomographic image by the convolutional neural network, wherein the convolutional neural network comprises (a) an encoder unit for inputting the measurement image, updating a feature map by means of a convolution operation performed by a plurality of convolution layers, and gradually decreasing a spatial size of the feature map; (b) a decoder unit provided in a subsequent stage of the encoder unit, which updates the feature map by means of a convolution operation performed by a plurality of convolution layers, and gradually increases the spatial size of the feature map;and (c) a projection linking unit for performing projection processing from the measurement space to real space on any feature map in the encoder unit and for connecting the feature map after performing the projection processing with the feature map of the same size in the decoder unit. Advantageous effects of the invention

[0019] According to the embodiments of the present invention, it is possible to suppress an increase in the amount of data required for training a CNN and to improve the performance of generating a tomographic image by the CNN. Brief description of the drawings Fig. Figure 1 is a diagram representing a configuration of a tomography system 1. Fig. Figure 2 is a diagram that illustrates a configuration example of a CNN. Fig. Figure 3 is a diagram describing the projection processing between a sinogram space (a measurement space) and a real space. Fig. Figure 4 is a diagram that shows another configuration example of the CNN. Fig. Figure 5 contains diagrams (a) and (b) showing tomographic images in the case where a noise-free sample 1 is used in a comparison example. Fig. 6 contains (a), (b) diagrams showing tomographic images in the case where a noise-free sample 19 is used in the comparison example. Fig. 7 contains (a), (b) diagrams showing tomographic images in the case where sample 1 is used without noise in an example. Fig. Figure 8 contains (a), (b) diagrams showing tomographic images when sample 19 is used without noise in the example. Fig. Figure 9 contains (a), (b) diagrams showing tomographic images when sample 1 is used with noise in the comparison example. Fig. 10 contains (a), (b) diagrams showing tomographic images in which sample 19 is used with noise in the comparison example. Fig. 11 contains (a), (b) diagrams showing tomographic images in which sample 1 with noise is used in the example. Fig. 12 contains (a), (b) diagrams showing tomographic images in the case where sample 19 is used with noise in the example. Description of embodiments

[0020] The following are detailed descriptions of embodiments of an image processing device and an image processing method with reference to the accompanying drawings. In the description of the drawings, identical elements are designated with the same reference numerals, and no redundant descriptions are used. The present invention is not limited to these examples. The claims, their equivalents, and all amendments are to be considered as falling within the scope of the present invention.

[0021] Fig. Figure 1 is a diagram representing a configuration of a tomography system 1. The tomography system 1 is a system for generating a tomographic image of a subject and comprises a tomography device 2 and an image processing device 10. The tomography device 2 is a device for acquiring measurement data used to generate the tomographic image of the subject and is, for example, a PET device, a SPECT device, an X-ray CT device, or an MRI device.

[0022] The image processing device 10 comprises a storage unit 11, a measurement image generation unit 12, a correction unit 13, a CNN processing unit 14 and a CNN training unit 15. An image processing method comprises a storage step, a measurement image generation step, a correction step, a CNN processing step and a CNN training step.

[0023] The image processing device 10 generates the tomographic image of the subject in real space based on the subject's measurement data acquired by a measurement using the tomography device 2. The image processing device 10 comprises a processing unit with a CPU or GPU for performing various types of processing; a storage unit with a hard disk, RAM, and the like for storing a program and data for processing; a display unit, such as a liquid crystal display, for showing processing status, processing results, and the like; and an input unit with a keyboard, mouse, and the like for inputting processing status information and the like. The image processing device 10 can be configured by a computer.

[0024] In addition to the storage unit 11 for storing the program and data, the image processing device 10, as the operational processing unit for performing the various types of processing, includes the measurement image generation unit 12, the correction unit 13, the CNN processing unit 14, and the CNN training unit 15. In the storage step, the storage unit 11 inputs the measurement data of the subject, which were acquired by the measurement performed with the tomography device 2, and stores the measurement data.

[0025] In the image generation step, the image generation unit 12 generates an image in a measurement space different from real space, based on the measurement data stored by the storage unit 11. In the correction step, the correction unit 13 performs various corrections (e.g., sensitivity correction, absorption correction, and scattering correction) on the image generated by the image generation unit 12.

[0026] In the CNN processing step, the CNN processing unit 14 inputs the measurement image, corrected by the correction unit 13, into a convolutional neural network (CNN) and uses the CNN to generate the tomographic image in real space. In the CNN training step, the CNN training unit 15 trains the CNN using the measurement image and the tomographic image of each of a multitude of subjects. The image processing device 10 following the CNN training step may not include the CNN training unit 15.

[0027] In the case where the tomography device 2 is the PET device, the SPECT device, or the X-ray CT device, the measurement space is set to a sinogram space and the measurement image is set to a sinogram. In the case where the tomography device 2 is the MRI device, the measurement space is set to a k-space and the measurement image is formed using k-space data.

[0028] In the following description, it is assumed that the tomography device 2 is a PET device. The PET device is a radiation tomography device capable of acquiring a tomographic image of the subject (a living body). The PET device has a detection unit comprising a large number of small radiation detectors arranged around a measurement chamber in which the subject is placed.

[0029] The PET device detects a γ-photon pair with an energy of 511 keV, generated by electron-positron annihilation in the subject into which a positron-emitting isotope (an RI source) is injected, using a coincidence method with the detection unit and collects the coincidence detection information. This coincidence detection information includes positional information for each of the two radiation detectors performing the photon pair coincidence detection. From this positional information, a coincidence detection line (a line connecting the two radiation detectors performing the photon pair coincidence detection) can be derived. The storage unit 11 stores the coincidence detection information collected using the PET device.

[0030] The image generation unit 12 generates the sinogram based on the collected coincidence detection information, which is stored by the storage unit 11. The sinogram (the image) is displayed as a histogram representing a frequency (a generation frequency of coincidence events) at which the coincidence detection information is acquired in the sinogram space (the measurement space), where the position, direction, and other parameters of the coincidence detection line are defined as variables. The CNN processing unit 14 inputs the sinogram (the image) in the sinogram space (the measurement space) into the CNN and generates the tomographic image in real space using the CNN.

[0031] Fig. Figure 2 is a diagram illustrating a configuration example of the CNN. The CNN shown in this diagram is based on a U-Net structure comprising an encoder unit 21, a decoder unit 22, and a bottleneck unit 23, and further comprising a projection link unit 24.

[0032] In this diagram, the feature maps after the respective processing steps of a convolution layer (Conv 3 × 3, Stride 1), a normalization layer (Batch Normalization), a nonlinear activation function (Leaky ReLU), a downsampling (Conv 4 × 4, Stride 2), and an upsampling are represented by different shades of gray. The diagram also shows the sizes of the respective feature maps, where the number of pixels of each sinogram 31 input to the CNN and each tomographic image 32 output from the CNN is set to 128 × 128.

[0033] The convolution layer shifts a filter relative to the previously created feature map (or the input image), performs a convolution operation on the feature map and the filter at each position, and generates a new feature map with the same spatial size. The normalization layer normalizes the feature for each minibatch of the feature map. The nonlinear activation function inputs a weighted sum z of each value of the feature map into an activation function σ(z) = max{0, z} and outputs a function value σ(z).

[0034] Downward sampling performs the convolution of the feature map and filter, reduces the spatial size of the feature map by 1 / 2 × 1 / 2, and doubles the number of feature map channels. Upward sampling increases the spatial size of the feature map by a factor of 2 × 2 using linear interpolation, performs the convolution of the feature map and filter, and reduces the number of feature map channels by 1 / 2.

[0035] Encoder unit 21 comprises the convolution layer, the normalization layer, the nonlinear activation function, and downsampling. Encoder unit 21 inputs the sinogram (the measurement image) into the sinogram space (the measurement space) and updates the feature map through the convolution layer, the normalization layer, and the nonlinear activation function. After the processing steps through the convolution layer, the normalization layer, and the nonlinear activation function have been performed several times, encoder unit 21 reduces the spatial size of the feature map by 1 / 2 × 1 / 2 and doubles the number of feature map channels through downsampling. In this diagram, the spatial size of the feature map is first set to 128 × 128 and then reduced to 64 × 64, 32 × 32, and 16 × 16.

[0036] Decoder unit 22 comprises the convolution layer, the normalization layer, the nonlinear activation function, and the upsampling function. Decoder unit 22 is located in the subsequent stage of encoder unit 21 and, through upsampling, doubles the spatial size of the feature map and reduces the number of feature map channels by 1 / 2 × 1 / 2. In this diagram, the spatial size of the feature map increases to 16 × 16, 32 × 32, 64 × 64, and 128 × 128. After the upsampling processing step, decoder unit 22 updates the feature map through the convolution layer, the normalization layer, and the nonlinear activation function.

[0037] The bottleneck unit 23 is located between the encoder unit 21 and the decoder unit 22. Bottleneck unit 23 comprises the convolution layer, the normalization layer, and the nonlinear activation function, and updates the feature map through these components. Within bottleneck unit 23, the number of channels is doubled while maintaining the spatial size of the feature map.

[0038] The encoder unit 21 inputs the sinogram (the measurement image) into the sinogram space (the measurement space) and extracts the feature of the input sinogram as a feature map. The decoder unit 22 then reconstructs the tomographic image step by step in real space based on the feature map, which specifies the extracted feature of the sinogram.

[0039] The projection connection unit 24 performs projection processing from the sinogram space (the measurement space) to real space on an arbitrary feature map in the encoder unit 21 and, after the projection processing is complete, connects the feature map to the feature map of the same size in the decoder unit 22. The projection connection unit 24 can perform the projection processing for any feature map in the encoder unit 21. It is also preferred to perform the projection processing for the feature map with the maximum size in the encoder unit 21. It is even more preferred to perform the projection processing for feature maps of all sizes in the encoder unit 21.

[0040] In any case, the spatial size of the feature map is not changed by the projection processing, and the feature map is linked to the feature map of the same spatial size in decoder unit 22 after the projection processing has been carried out. The linking of the feature map to the feature map in the decoder unit after the projection processing can be done by adding for each channel or by stacking channels. For the feature map of spatial size to which the projection link is not applied, the unchanged feature map can be linked to the feature map of the same spatial size in decoder unit 22 (a skip link).

[0041] In the configuration of Fig. 2 The projection connection unit 24 performs the projection processing for the feature cards of all sizes in the encoder unit 21 and connects the feature cards to the feature cards of the same size in the decoder unit 22 after the projection processing has been carried out.

[0042] This means that the projection processing is performed on feature card 41, measuring 128 × 128, in encoder unit 21, and feature card 51 is connected to the feature card measuring 128 × 128 in decoder unit 22 after the projection processing is completed. Projection processing is also performed on feature card 42, measuring 64 × 64, in encoder unit 21, and feature card 52 is connected to the feature card measuring 64 × 64 in decoder unit 22 after the projection processing is completed.

[0043] Projection processing is performed on feature card 43, measuring 32 × 32, in encoder unit 21, and feature card 53, after projection processing, is connected to the feature card measuring 32 × 32 in decoder unit 22. Projection processing is also performed on feature card 44, measuring 16 × 16, in encoder unit 21, and feature card 54, after projection processing, is connected to the feature card measuring 16 × 16 in decoder unit 22.

[0044] Fig. Figure 3 is a diagram describing the projection processing between the sinogram space (the measurement space) and real space. This diagram illustrates the relationship between the sinogram 31 and the tomographic image 32 in the two-dimensional PET scanner. In the iterative image reconstruction process, the projection from real space to the sinogram space is called forward projection, and the projection from the sinogram space to real space is called back projection. The iterative image reconstruction process alternately and repeatedly performs the forward and back projections to iteratively reconstruct the tomographic image.

[0045] Forward projection is a procedure for converting the tomographic image 32 in real space into the sinogram 31 and is represented by the following formula (1). p is the sinogram and f is the tomographic image. A is a system matrix representing a forward projection operation. According to the representation, forward projection is the process by which the line of response (LOR) of each position x r and in each direction ϕ a sum p(x r , ϕ) the pixel values ​​of the pixels through which the coincidence detection line passes in the tomographic image 32. [Formula 1] p=Af

[0046] The forward projection is analytically represented by the following formula (2). In this case, there is a relationship between a coordinate (x, y) and a coordinate (x) that is given by the following formula (3). r , y r) is represented. ϕ is an azimuth angle, representing a direction of the projection. x r and y r are an x-axis and a y-axis in the case where the tomographic image is rotated by -ϕ. After rotating the image by -ϕ degrees, the forward projection becomes the one-dimensional projection p(x). r , ϕ) in the direction of ϕ degrees by integrating the pixel values ​​along the y r -axis. By repeatedly performing the above procedure for 0 ≤ ϕ < π, the sinogram 31 is obtained. [Formula 2] p(xr,ϕ)=∫−∞∞dyrf(x,y) [Formula 3] [xy]=[cos ϕ−sin ϕsin ϕcos ϕ][xryr]

[0047] Backprojection is a process for projecting sinogram 31 onto real space using a transposed matrix A. Tof matrix A and is represented by the following formula (4). b is a back-projection image of the sinogram p. The back-projection image b is not the same as the tomographic image f. Formula 4] b=ATp

[0048] The projection processing performed by the projection link unit 24 (the projection processing from sinogram space to real space on an arbitrary feature map in the encoder unit 21) corresponds to the backprojection described above. Providing the aforementioned projection link unit 24 is expected to facilitate learning the sinogram processing in the encoder unit 21 and the tomographic image processing in the decoder unit 22. Furthermore, it is expected to promote the learning of the algorithm for generating the tomographic image from the sinogram, without being limited to storing the pairs of sinograms and tomographic images used for training.

[0049] Fig. 4 is a diagram that illustrates another configuration example of the CNN. Compared to the one in Fig. The CNN shown in this diagram differs from the CNN shown in diagram 2 in that it additionally includes a projection unit 25 and an addition unit 26. The projection unit 25 performs the projection processing (backprojection) from the sinogram space to real space on the sinogram (the measurement image) 31, which is input to the encoder unit 21, and generates a projection image 33, which is an image after the projection has been performed. The addition unit 26 adds the projection image 33 to the output image from the decoder unit 22 to generate the tomographic image 32.

[0050] Using the configuration described above, when the CNN training unit 15 trains the CNN using the sinogram 31 and the tomographic image 32 of each of the multiple subjects, it is caused to learn (residual learning) a difference between the projection image 33 obtained by backprojecting the sinogram 31 and the tomographic image 32. The difference for each pixel between the projection image 33 and the tomographic image 32 can be positive or negative and has a distribution that can be approximated by a normal distribution with a mean of 0. Accordingly, training the CNN using a least-squares method becomes simpler.

[0051] Next, simulation results obtained by generating simulation data using numerical simulation with a digital brain phantom image are described. As an example, a case of using the [missing information] was presented. Fig. The CNNs shown in Figure 2 were selected. As a comparison example, a technique for generating the tomographic image from the histogram using the CNN, as described in Non-Patent Document 1, was used. In the configuration of the comparison example, the CNN does not provide a unit corresponding to the projection connection unit 24.

[0052] MRI brain segmentation images of 20 samples were downloaded from BrainWeb (https: / / brainweb.bic.mni.mcgill.ca / brainweb / ), and brain phantom images were generated using PET simulation of 18F-FDG (18F-fluorodeoxyglucose) as a drug with a gray matter : white matter : cerebrospinal fluid (CSF) contrast ratio of 1 : 0.25 : 0.05. The matrix size of each phantom image was set to 128 (X) × 128 (Y) × 70 (Z). A sinogram was generated by forward projection onto each phantom image. The matrix size of each sinogram was set to 128 (X) × 128 (Y) × 70 (Z). r ) × 128 (ϕ) × 70 (Z).

[0053] For each sample, two types were prepared: one without noise and one with noise. A sinogram with noise was generated by adding Poisson noise, corresponding to a case where the total number of samples is set to 10 M. Of the sinogram-to-tomographic image pairs for the 20 samples (70 slices per sample), 18 samples were used as training data (a total of 1,260 slices) and 2 samples as test data (a total of 140 slices).

[0054] In both cases, after training the CNN using the training data (samples 1 to 18), the sinograms of the test data (samples 19 and 20) were fed into the CNN to generate tomographic images. The generated tomographic images were then compared to the original tomographic images. The tomographic image generated by the CNN was evaluated based on its peak signal-to-noise ratio (PSNR). The PSNR indicates image quality in decibels (dB), with a higher value signifying better image quality.

[0055] Fig. 5 and Fig. Figure 6 contains diagrams showing the tomographic images in the case where the samples are used without noise in the comparison example. (a) in Fig. Figure 5 is a diagram showing the original tomographic image of sample 1. (b) in Fig. Figure 5 is a diagram showing the tomographic image generated by the CNN based on the sinogram of sample 1. (a) in Fig. Figure 6 is a diagram showing the original tomographic image of sample 19. (b) in Fig. Figure 6 is a diagram showing the tomographic image generated by the CNN based on the sinogram of sample 19.

[0056] The PSNR of the tomographic image ((b) in Fig. 5), which was generated by the CNN based on the sinogram of the training data (of sample 1), is 35.30 dB. The PSNR of the tomographic image ((b) in Fig. 6), which was generated by the CNN based on the sinogram of the test data (of sample 19), is 22.63 dB.

[0057] Fig. 7 and Fig. Figure 8 contains diagrams showing the tomographic images in the case where the samples are used without noise in the example. (a) in Fig. Figure 7 is a diagram showing the original tomographic image of sample 1. (b) in Fig. Figure 7 is a diagram showing the tomographic image generated by the CNN based on the sinogram of sample 1. (a) in Fig. Figure 8 is a diagram showing the original tomographic image of sample 19. (b) in Fig. Figure 8 is a diagram showing the tomographic image generated by the CNN based on the sinogram of sample 19.

[0058] The PSNR of the tomographic image ((b) in Fig. 7), which was generated by the CNN based on the sinogram of the training data (of sample 1), is 35.08 dB. The PSNR of the tomographic image ((b) in Fig. 8), which was generated by the CNN based on the sinogram of the test data (of sample 19), is 35.05 dB.

[0059] Fig. 9 and Fig. Figure 10 contains diagrams showing the tomographic images in the case where the samples with noise are used in the comparison example. (a) in Fig. Figure 9 is a diagram showing the original tomographic image of sample 1. (b) in Fig. Figure 9 is a diagram showing the tomographic image generated by the CNN based on the sinogram of sample 1. (a) in Fig. Figure 10 is a diagram showing the original tomographic image of sample 19. (b) in Fig. Figure 10 is a diagram showing the tomographic image generated by the CNN based on the sinogram of sample 19.

[0060] The PSNR of the tomographic image ((b) in Fig. 9), which was generated by the CNN based on the sinogram of the training data (of sample 1), is 34.70 dB. The PSNR of the tomographic image ((b) in Fig. 10), which was generated by the CNN based on the sinogram of the test data (of sample 19), is 22.58 dB.

[0061] Fig. 11 and Fig. Figure 12 contains diagrams showing the tomographic images in the case where the samples with noise are used in the example. (a) in Fig. Figure 11 is a diagram showing the original tomographic image of sample 1. (b) in Fig. Figure 11 is a diagram showing the tomographic image generated by the CNN based on the sinogram of sample 1. (a) in Fig. Figure 12 is a diagram showing the original tomographic image of sample 19. (b) in Fig. Figure 12 is a diagram showing the tomographic image generated by the CNN based on the sinogram of sample 19.

[0062] The PSNR of the tomographic image ((b) in Fig. 11), which was generated by the CNN based on the sinogram of the training data (of sample 1), is 30.34 dB. The PSNR of the tomographic image ((b) in Fig. 12), which was generated by the CNN based on the sinogram of the test data (of sample 19), is 28.26 dB.

[0063] By comparing the PSNR values ​​of the respective tomographic images, which are in Fig. 5 to Fig.As shown in Figure 12, the following can be understood. In the case where the tomographic image is generated by the CNN based on the sinogram of the training data, the PSNR of the tomographic image generated in the comparison example is the same as the PSNR of the tomographic image generated in the example, regardless of the presence or absence of noise. However, if the tomographic image is generated by the CNN based on the sinogram of the test data, the PSNR of the tomographic image generated in the comparison example is low, regardless of the presence or absence of noise. This is due to the following reason.

[0064] It is assumed that the CNN used in the comparison example only stores the pairs of sinograms and tomographic images from the training data, and that when the unknown sinogram from the test data is input, the CNN outputs the tomographic image that most closely matches the input sinogram from the training data. Therefore, it is assumed that no generalization to unknown test data can be achieved in the comparison example, and the tomographic image of a different person is output.

[0065] On the other hand, in this example, the PSNR of the tomographic image generated by the CNN is high not only when the tomographic image is generated by the CNN based on the sinogram of the training data, but also when the tomographic image is generated by the CNN based on the sinogram of the test data, regardless of the presence or absence of noise. In this example, very good generalization to unknown test data is achieved. This is attributed to the following reason.

[0066] In the example, the CNN is provided with the projection link unit 24, and therefore it is assumed that the CNN is able to learn the algorithm to generate the tomographic image from the sinogram without merely storing the pairs of sinograms and tomographic images used for training.

[0067] According to the above description, it is possible according to the present embodiment to suppress an increase in the amount of data required for training the CNN and to improve the performance of the generation of the tomographic image by the CNN.

[0068] The embodiments and examples described above mainly describe the case in which the tomography device is set up on the PET scanner. However, the present invention can be applied similarly to other tomography devices.

[0069] In the case where the tomography device is a SPECT or X-ray CT scanner, as with a PET scanner, the measurement space is set to sinogram space, the measurement image to sinogram, and the projection processing from measurement space to real space to back projection. If the tomography device is an MRI scanner, the measurement space is set to k-space, the measurement image to the image configured by the k-space data, and the projection processing from measurement space to real space to the inverse Fourier transform.

[0070] The image processing device and the image processing method are not limited to the embodiments and configuration examples described above, but various modifications are possible.

[0071] The image processing device of a first aspect according to the above embodiment is an image processing device for generating a tomographic image of a subject in real space based on measurement data of the subject acquired by a measurement using a tomography device, and it comprises (1) a measurement image generation unit for generating a measurement image in a measurement space that differs from real space based on the measurement data; and (2) a CNN processing unit for inputting the measurement image into a convolutional neural network and for generating the tomographic image by the convolutional neural network, wherein the convolutional neural network comprises (a) an encoder unit for inputting the measurement image, for updating a feature map by a convolution operation performed by each of a plurality of convolution layers, and for gradually decreasing a spatial size of the feature map;(b) a decoder unit provided at a subsequent stage of the encoder unit, which updates the feature map by a convolution operation performed by each of a plurality of convolution layers, incrementally increasing the spatial size of the feature map; and (c) a projection linking unit for performing projection processing from measurement space to real space on any feature map in the encoder unit and for linking the feature map, after performing the projection processing, to the feature map of the same size in the decoder unit.

[0072] In the image processing device of a second aspect, the device in the configuration of the first aspect may further include a CNN training unit for training the convolutional neural network using the measurement image and the tomographic image of each of a multitude of subjects.

[0073] In the image processing device of a third aspect, in the configuration of the first or second aspect, the CNN processing unit can perform the projection processing from the measurement space to the real space onto the measurement image entered into the encoder unit and, after performing the projection processing, add an image to an output image from the decoder unit.

[0074] In the image processing device of a fourth aspect, in the configuration of any of the first to third aspects, the projection linking unit can perform the projection processing from the measuring space to the real space on the feature map with maximum size in the encoder unit and connect the feature map with the feature map of the same size in the decoder unit after performing the projection processing.

[0075] In the image processing device of a fifth aspect, the projection linking unit, in the configuration of any of the first to third aspects, can perform the projection processing from the measuring space to the real space on each of the feature cards of all sizes in the encoder unit and, after performing the projection processing, link the feature card with the feature card of the same size in the decoder unit.

[0076] The tomography system according to the above embodiment comprises a tomography device for acquiring measurement data to generate a tomographic image of a subject and the image processing device of the above configuration for generating the tomographic image based on the measurement data.

[0077] The image processing method of a first aspect according to the above embodiment is an image processing method for generating a tomographic image of a subject in real space based on measurement data of the subject acquired by a measurement using a tomography device, and it comprises (1) a measurement image generation step for generating a measurement image in a measurement space that is distinct from real space, based on the measurement data; and (2) a CNN processing step for inputting the measurement image into a convolutional neural network and for generating the tomographic image by the convolutional neural network, wherein the convolutional neural network comprises (a) an encoder unit for inputting the measurement image, for updating a feature map by a convolution operation performed by each of a plurality of convolutional layers, and for gradually decreasing a spatial size of the feature map;(b) a decoder unit provided at a subsequent stage of the encoder unit, which updates the feature map by a convolution operation performed by each of a plurality of convolution layers, incrementally increasing the spatial size of the feature map; and (c) a projection linking unit for performing projection processing from measurement space to real space on any feature map in the encoder unit and for linking the feature map, after performing the projection processing, to the feature map of the same size in the decoder unit;

[0078] In the image processing procedure of a second aspect, the procedure in the configuration of the first aspect may further include a CNN training step to train the convolutional neural network using the measurement image and the tomographic image of each of a multitude of subjects.

[0079] In the image processing method of a third aspect, in the configuration of the first or second aspect in the CNN processing step, the projection processing from the measurement space to the real space can be carried out on the measurement image entered into the encoder unit, and an image after the projection processing can be added to an output image from the decoder unit.

[0080] In the image processing method of a fourth aspect, in the configuration of any of the first to third aspects, the projection connection unit can perform the projection processing from the measuring space to the real space on the feature map with maximum size in the encoder unit and connect the feature map with the feature map of the same size in the decoder unit after performing the projection processing.

[0081] In the image processing method of a fifth aspect, the projection connection unit, in the configuration of any of the first to third aspects, can perform the projection processing from the measurement space to the real space on each of the feature cards of all sizes in the encoder unit and, after performing the projection processing, connect the feature card with the feature card of the same size in the decoder unit. Industrial applicability

[0082] The present invention can be used as an image processing device and image processing method that are able to suppress an increase in the amount of data required for training a CNN and to improve the performance of generating a tomographic image by the CNN. List of reference symbols

[0083] 1 - Tomography system, 2 - Tomography device, 10 - Image processing device, 11 - Storage unit, 12 - Measurement image generation unit, 13 - Correction unit, 14 - CNN processing unit, 15 - CNN training unit, 21 - Encoder unit, 22 - Decoder unit, 23 - Bottleneck unit, 24 - Projection link unit, 25 - Projection unit, 26 - Addition unit, 31 - Sinogram, 32 - Tomographic image, 33 - Projection image, 41-44 - Feature map, 51-54 - Feature map. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited non-patent literature

[0000] I. Haggstrom et al., „DeepPET: A deep encoder-decoder network for directly solving the PET image reconstruction inverse problem“, Med. Image Anal. Vol. 54, S. 253-262, 2019

[0010]

Claims

[1] Image processing device for generating a tomographic image of a subject in real space based on measurement data of the subject acquired by a measurement using a tomography device, wherein the image processing device comprises: a measurement image generation unit for generating a measurement image in a measurement space that differs from real space, based on the measurement data; and a CNN processing unit for inputting the measurement image into a convolutional neural network and for generating the tomographic image through the convolutional neural network, wherein the convolutional neural network includes: an encoder unit for inputting the measurement image, for updating a feature map by means of a convolution operation performed by a multitude of convolution layers, and for gradually reducing a spatial size of the feature map; a decoder unit, provided in a subsequent stage of the encoder unit, which updates the feature map by a convolution operation, performed by one of a plurality of convolution layers at a time, and gradually increases the spatial size of the feature map; and a projection connection unit for performing projection processing from the measuring space to the real space on any feature map in the encoder unit and for connecting the feature map after performing the projection processing with the feature map of the same size in the decoder unit. [2] Image processing device according to claim 1, further comprising: a CNN training unit to train the convolutional neural network using the measurement image and the tomographic image of each from a variety of subjects. [3] Image processing device according to claim 1 or 2, wherein the CNN processing unit performs the projection processing from the measurement space to the real space on the measurement image input into the encoder unit and, after performing the projection processing, adds an image to an output image from the decoder unit. [4] Image processing device according to one of claims 1 to 3, wherein the projection linking unit performs the projection processing from the measuring space to the real space on the feature map with maximum size in the encoder unit and connects the feature map to the feature map of the same size in the decoder unit after the projection processing has been carried out. [5] Image processing device according to one of claims 1 to 3, wherein the projection linking unit performs the projection processing from the measuring space to the real space on the feature map of all sizes in the encoder unit and connects the feature map with the feature map of the same size in the decoder unit after the projection processing has been carried out. [6] Tomography system, including: a tomography device for acquiring measurement data to generate a tomographic image of a subject; and the image processing device according to one of claims 1 to 5 for generating the tomographic image based on the measurement data. [7] Image processing method for generating a tomographic image of a subject in real space based on measurement data of the subject acquired by a measurement using a tomography device, wherein the image processing method comprises: a measurement image generation step for generating a measurement image in a measurement space that differs from real space, based on the measurement data; and a CNN processing step for inputting the measurement image into a convolutional neural network and for generating the tomographic image through the convolutional neural network, wherein the convolutional neural network includes: an encoder unit for inputting the measurement image, for updating a feature map by means of a convolution operation, which is performed by a plurality of convolution layers, and for gradually reducing a spatial size of the feature map; a decoder unit, provided in a subsequent stage of the encoder unit, which updates the feature map through a convolution operation, performed by a plurality of convolution layers, gradually increasing the spatial size of the feature map; and a projection connection unit for performing projection processing from the measuring space to the real space on any feature map in the encoder unit and for connecting the feature map after performing the projection processing with the feature map of the same size in the decoder unit. [8] Image processing method according to claim 7, further comprising: a CNN training step to train the convolutional neural network using the measurement image and the tomographic image, each from a large number of subjects. [9] Image processing method according to claim 7 or 8, wherein in the CNN processing step the projection processing from the measurement space to the real space is carried out on the measurement image input into the encoder unit and an image is added to an output image from the decoder unit after the projection processing has been carried out. [10] Image processing method according to one of claims 7 to 9, wherein the projection linking unit performs the projection processing from the measuring space to the real space on the feature map with maximum size in the encoder unit and connects the feature map with the feature map of the same size in the decoder unit after the projection processing has been carried out. [11] Image processing method according to one of claims 7 to 9, wherein the projection linking unit performs the projection processing from the measuring space to the real space on each of the feature cards of all sizes in the encoder unit and connects the feature card with the feature card of the same size in the decoder unit after the projection processing has been carried out.

Images