Image processing device, image processing system, image processing method and storage medium

Abstract

An image processing device, an image processing system, an image processing method and a storage medium are disclosed. The image processing device according to the present invention includes: a first classification unit, which is configured to classify a plurality of pixels in a two-dimensional image data constituting a first three-dimensional image data including an object into a first category group by using a trained classifier; and a second classification unit, which is configured to classify a plurality of pixels in a second three-dimensional image data including the object into a second category group based on the classification result of the first classification unit, the second category group including at least one category of the first category group. According to the image processing device of the present invention, the burden of the user to give pixel information can be reduced, and an area can be extracted with high precision.

Description

Technical Field

[0001] The invention relates to an image processing device, an image processing system, an image processing method and a storage medium. Background Art

[0002] Segmentation is one of the image processing techniques. Segmentation refers to processing for distinguishing a region of interest included in an image from regions other than the region of interest, and is also called region extraction, region division, image division, etc. Many segmentation techniques have been discussed so far. In recent years, a method of performing segmentation with high accuracy by giving pixel information about the region of interest and the region other than the region of interest, such as discussed in Non-Patent Literature 1, has been discussed.

[0003] Reference List

[0004] Non-patent literature

[0005] Non-patent literature 1: Y.Boykov, and G.Funka-Lea, "Graph Cuts and Efficient N-DImage Segmentation," International Journal of Computer Vision 70(2),109-131,2006 Summary of the invention

[0006] Technical issues

[0007] However, if segmentation is performed using the technique discussed in Non-Patent Document 1, the user needs to give the aforementioned pixel information in advance, which increases the burden on the user. The present invention is intended to provide an image processing apparatus that can reduce the burden on the user to give pixel information and extract a region with high accuracy.

[0008] Technical solutions to the problem

[0009] According to the present invention, the image processing device includes: a first classification unit, which is constructed to classify multiple pixels in two-dimensional image data constituting first three-dimensional image data including an object into a first category group by using a trained classifier; and a second classification unit, which is constructed to classify multiple pixels in second three-dimensional image data including the object into a second category group based on the classification result of the first classification unit, wherein the second category group includes at least one category of the first category group.

[0010] Advantageous Effects of the Invention

[0011] According to the image processing device of the present invention, the burden of the user to give pixel information can be reduced, and a region can be extracted with high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a diagram illustrating an example of a configuration of an image processing apparatus according to the first exemplary embodiment.

[0013] Figure 2 is a diagram illustrating an example of a hardware configuration of an image processing apparatus according to the first exemplary embodiment.

[0014] Figure 3 is a flowchart illustrating an example of a processing procedure of the image processing apparatus according to the first exemplary embodiment.

[0015] Figure 4A is a diagram for describing an example of an image according to the first exemplary embodiment.

[0016] Figure 4B is a diagram for describing an example of an image according to the first exemplary embodiment.

[0017] Figure 4C is a diagram for describing an example of an image according to the first exemplary embodiment.

[0018] Figure 4D is a diagram for describing an example of an image according to the first exemplary embodiment.

[0019] Figure 5 is a diagram for describing an example of processing by a first classification unit according to the first exemplary embodiment.

[0020] Fig. 6A is a diagram for describing an example of training data for a first classification unit according to the first exemplary embodiment.

[0021] Figure 6B is a diagram for describing an example of training data for a first classification unit according to the first exemplary embodiment.

[0022] Fig. 7A is a diagram for describing an example of an output of the first classification unit according to the first exemplary embodiment.

[0023] Figure 7B is a diagram for describing an example of an output of the first classification unit according to the first exemplary embodiment.

[0024] Figure 8 is a flowchart illustrating an example of a processing procedure of the image processing apparatus according to the second exemplary embodiment.

[0025] Fig.9A is a diagram for describing an example of an image according to the second exemplary embodiment.

[0026] Fig. 9Bis a diagram for describing an example of an image according to the second exemplary embodiment.

[0027] Fig. 9C is a diagram for describing an example of an image according to the second exemplary embodiment.

[0028] Fig.9D is a diagram for describing an example of an image according to the second exemplary embodiment. DETAILED DESCRIPTION

[0029] An exemplary embodiment of the image processing device disclosed in this specification will be described below with reference to the accompanying drawings. The target image data may be data of an image captured by any medical device (such as (MRI) magnetic resonance imaging equipment, ultrasonic diagnostic equipment, or X-ray computed tomography equipment) that outputs three-dimensional image data. An example of processing data of a medical image captured by an X-ray computed tomography (X-ray CT) device is processed by an image processing device according to the following exemplary embodiment.

[0030] [First exemplary embodiment]

[0031] The image processing device according to the present exemplary embodiment uses a two-dimensional segmentation method to widely extract (roughly extract) a region of interest from each (slice) two-dimensional image data constituting a spatial three-dimensional image data (three-dimensional tomographic image) including an object. An example of an object is a subject. The spatial three-dimensional image data is referred to as three-dimensional image data below. The two-dimensional segmentation method obtains a two-dimensional roughly segmented image of the region of interest corresponding to the input two-dimensional image data. Then, the two-dimensional roughly segmented image is stacked or interpolated or integrated to obtain a three-dimensional roughly segmented image. As used herein, stacking refers to a process for combining two or more roughly segmented images into a continuous image. Integration processing refers to a process for integrating the overlapping area between two or more roughly segmented images into one area. Then, based on the three-dimensional image data and the three-dimensional roughly segmented image obtained by the two-dimensional segmentation method, a three-dimensional segmentation method is used to extract a more accurate region of interest. As used herein, extracting a region refers to classifying pixels in an image into one of a predetermined category group. Classification can be performed so that the position of the segmentation target can be identified, and classification can be used to distinguish whether a pixel is located inside a segmentation target (such as an organ or a lesion), or to distinguish whether a pixel is located on the contour of the segmentation target.

[0032] In the following example, an abdominal CT image of a human body taken by an X-ray computed tomography (X-ray CT) device will be described as an example of three-dimensional image data. Here, the area of ​​interest in the two-dimensional segmentation method is the liver and the right kidney. In other words, the processing here is a classification problem for classifying pixels into one of the three categories of the category group (hereinafter referred to as the first category group) including "liver", "right kidney" and "area other than liver and right kidney". The area of ​​interest in the three-dimensional segmentation method is the liver. In other words, the processing here is a problem for classifying pixels into one of the two categories of the category group (hereinafter referred to as the second category group) including "liver" and "area other than liver". The construction of this processing is intended to reduce the error (wrong segmentation) of classifying the right kidney area as the liver area by assuming that both the liver area and the right kidney area that may be erroneously extracted as the liver area are the area of ​​interest of the two-dimensional segmentation method. It is expected to improve the segmentation accuracy of the liver area by stacking the segmentation results obtained by two-dimensional segmentation and using the stacked segmentation results as the input of the three-dimensional segmentation method. For convenience, the area of ​​interest of the first classifier (two-dimensional segmentation method) is referred to as the two-dimensional area of ​​interest below. The region of interest of the second classifier (three-dimensional segmentation method) is referred to as a three-dimensional region of interest.

[0033] In the present exemplary embodiment, a technique in which the user does not necessarily need to pre-given pixel information is used as a two-dimensional segmentation method. As used herein, pixel information refers to at least one of the position (foreground) information about the segmentation target or the position (background) information other than the segmentation target. One of the segmentation methods that does not necessarily require pixel information in advance is a segmentation method based on machine learning. In machine learning, the machine itself learns features from given data. For example, this means that classification can be performed without the user pre-setting classification conditions in the classification problem. In the present exemplary embodiment, CNN (convolutional neural network) is used as a two-dimensional segmentation method among the segmentation methods based on machine learning. As a CNN network structure, FCN [J. Long et al., "Fully Convolutional Networks for Semantic Segmentation," In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3431-3440, 2015.] can be used as a three-dimensional segmentation method using any technique for segmentation using given pixel information, such as a region growing method, a level set method, a graph cut method, or a snake method. In the present exemplary embodiment, a graph cut method is used.

[0034] When using a machine learning based classifier such as CNN, the appearance of the region of interest in the target image is preferably roughly the same even in different blocks of image data. In particular, if there is not much training data available for training the classifier and the appearance of the region of interest varies, the classifier may not be able to learn the changes in the appearance of the region of interest, and the classification accuracy may decrease.

[0035] In the case of three-dimensional image data (X-ray CT image) according to the present exemplary embodiment, the appearance of the region of interest in the body axis direction changes due to the nature of the human body as the subject. The reason for the change in the appearance of the region of interest includes, for example, drawing the region of interest at different positions. On the other hand, the axial image (axial cross-sectional image) is characterized in that the region of interest is drawn at roughly the same position with approximately the same resolution even in different three-dimensional image data. In addition, when classifying the three-dimensional image data, since different three-dimensional image data have different resolutions in the body axis direction, the accuracy of the classifier may be low depending on the image. In contrast, the axial image is less likely to be affected by the resolution difference in the body axis direction. Compared with the three-dimensional image data, the classification of the two-dimensional image represented by the axial image is performed by using only the two-dimensional information in the three-dimensional information about the three-dimensional image data. If the image data based on the two-dimensional information is processed, the difference in the appearance of the segmentation target in the image during the training and during the classification is small even between different three-dimensional image data. Therefore, in particular, the 2D-CNN (CNN with two-dimensional image data as input) using the two-dimensional image or axial image data (an example of the two-dimensional image data constituting the three-dimensional image data) as input can reduce the burden of spatial normalization for unifying the appearance of the region drawn in the image.

[0036] If 3D-CNN (CNN with three-dimensional image data as input) is used instead of 2D-CNN, spatial normalization is required, such as cutting out the three-dimensional space region where the region of interest is located and aligning the position of the region of interest. If the resolution in the body axis direction is different, resolution matching processing is also performed. Generally, such spatial normalization operations are difficult to apply to unknown images, and the use of 2D-CNN makes image processing easier. The same applies to classifiers based on non-CNN machine learning, such as SVM (support vector machine).

[0037] The image processing apparatus according to this exemplary embodiment combines a 2D-FCN (fully convolutional network) (an example of a 2D-CNN) using axial image data as an input with a graph cut method as a three-dimensional segmentation method. This construction can reduce the trouble of normalizing the input image space and enable segmentation while taking into account three-dimensional connectivity. Note that although the segmentation method for which the user does not necessarily need to give pixel information in advance is described as a two-dimensional segmentation method, the two-dimensional segmentation method may be a method that appropriately uses pixel information given by the user.

[0038] (Module structure)

[0039] The following will refer to Figure 1 The functional configuration of the image processing apparatus according to the present exemplary embodiment is described. As shown in the figure, the image processing apparatus 100 according to the present exemplary embodiment includes an obtaining unit 101, a first classification unit 102, and a second classification unit 103. The image processing system according to the present exemplary embodiment also includes a storage device 70 outside the image processing apparatus 100.

[0040] The storage device 70 is an example of a computer-readable storage medium. The storage device 70 is a large-capacity information storage device represented by a hard disk drive (HDD) and a solid state drive (SSD). The storage device 70 stores at least one or more three-dimensional image data.

[0041] The obtaining unit 101 obtains three-dimensional image data from the storage device 70. Then, the obtaining unit 101 sends the obtained three-dimensional image data to the first classification unit 102 as first three-dimensional image data, and to the second classification unit 103 as second three-dimensional image data.

[0042] The first classification unit 102 is input with two-dimensional image data constituting the three-dimensional image data (first three-dimensional image data) obtained from the acquisition unit 101. Then, the first classification unit 102 performs two-dimensional segmentation on each of the two-dimensional image data constituting the first three-dimensional image data, thereby obtaining a two-dimensional roughly segmented image corresponding to the first category group. The first classification unit also generates a three-dimensional roughly segmented image by performing at least one of stacking, interpolation processing, and integration processing on the two-dimensional roughly segmented images corresponding to the first category group by category, and sends the three-dimensional roughly segmented image to the second classification unit 103. The two-dimensional roughly segmented image or the three-dimensional roughly segmented image obtained by stacking, interpolation processing, or integration processing on the two-dimensional roughly segmented image will be referred to as the classification result of the first classification unit 102. The classification result corresponding to the first category group is a likelihood map indicating the likelihood of each pixel in a category whose pixel value is greater than or equal to 0 and less than or equal to 1. For example, each roughly segmented image in the classification result represents a pixel that is likely to be in the category by a value close to 1, and represents a pixel that is unlikely to be in the category by a value close to 0. The two-dimensional roughly segmented image has the same image size as the axial image. The three-dimensional roughly segmented image has the same size as the three-dimensional image data. The pixel values ​​in the roughly segmented image as the classification result can be represented by any value that can represent the possibility in those categories. For example, a binary value can be given as the pixel value. Different ranges of values ​​can be given category by category. For example, if different ranges of values ​​are given category by category, a value greater than or equal to 0 and less than 1 is assigned to the first category, and a value greater than or equal to 1 and less than 2 is assigned to the second category. The result of the classification can be an output about which category each pixel belongs to the category group, or an output about the likelihood of each category in the category group. The two-dimensional roughly segmented image can have the same image size as the axial image as described above, or have a different image size. Similarly, the three-dimensional roughly segmented image can have the same image size as the three-dimensional image as described above, or have a different image size.

[0043] In this exemplary embodiment, the first classification unit 102 is a trained 2D-FCN. Fig. 6A and Figure 6B A method for training a 2D-FCN is described. The 2D-FCN is one of the supervised training models in the machine learning model. The machine trained in advance using two-dimensional ground truth images and two-dimensional training images associated with each other exerts classification capabilities. Here, the ground truth image including a plurality of two-dimensional ground truth images and the training image including a plurality of two-dimensional training images are collectively referred to as training data.

[0044] Fig. 6A and Figure 6B The training data when training 2D-FCN is shown. Fig. 6A A training image 610 is shown, which is a set of two-dimensional axial images. In the present exemplary embodiment, the training image 610 is a set of abdominal CT images taken by an X-ray CT device. The training image 610 includes an axial image 611a and an axial image 611b. Figure 6B A set of two-dimensional ground truth images corresponding to each category in the first category group of training image 610 is shown. In the present exemplary embodiment, the liver region, the right kidney region, and the region other than the liver and the right kidney are classified. The ground truth images corresponding to each category include a plurality of two-dimensional ground truth images, respectively. Specifically, the ground truth image includes a ground truth image 630 of the liver region, a ground truth image 640 of the right kidney region, and a ground truth image 650 of the region other than the liver and the right kidney. Each ground truth image (e.g., the ground truth image 630 of the liver region) includes a two-dimensional ground truth image 631a of the liver region corresponding to the axial image 611a, and a two-dimensional ground truth image 631b of the liver region corresponding to the axial image 611b. Similarly, the ground truth image of the right kidney region includes a two-dimensional ground truth image 641a of the right kidney region corresponding to the axial image 611a, and a two-dimensional ground truth image 641b of the right kidney region corresponding to the axial image 611b. The ground truth images of the region other than the liver and the right kidney include a two-dimensional ground truth image 651a and a two-dimensional ground truth image 651b.

[0045] The ground truth images corresponding to each category of the first category group include pixels indicating whether the pixel is in the category by a binary value, and are characterized in that there is no overlapping area between the ground truth images corresponding to each category. The 2D-FCN is trained by using a training data set including one or more of the above-mentioned training data. For example, back propagation, which is a typical technique in CNN training, is used as a training method. Thus, the 2D-FCN is trained using features for classifying the region of interest, and the 2D-FCN can even extract the region of interest from an unknown image. The ground truth image can be an image indicating whether a pixel is in a category by a binary value as described above, or an image indicating the possibility of a pixel being in a category by a continuous value. If the ground truth image is an image indicating the possibility of being in a category by a continuous value, the first classification unit 102 is constructed to solve the regression problem of each pixel.

[0046] The second classification unit 103 is input with the three-dimensional image data (second three-dimensional image data) obtained from the obtaining unit 101 and the three-dimensional roughly segmented image corresponding to each class obtained from the first classification unit 102, and outputs a three-dimensional focus image corresponding to the second class group. The three-dimensional focus image is an image indicating the possibility of being a region to be extracted, and its pixel values ​​are expressed as in the aforementioned three-dimensional roughly segmented image.

[0047] Notice, Figure 1 At least some components of the image processing apparatus 100 shown may be implemented as independent devices. Alternatively, such components may be implemented as software for implementing the respective functions. In the present exemplary embodiment, the components are implemented by the respective software.

[0048] (Hardware Structure)

[0049] Figure 2 2 is a diagram showing an example of the hardware configuration of the image processing apparatus 100. A CPU (Central Processing Unit) 201 mainly controls the operation of the components. A main memory 202 stores a control program to be executed by the CPU 201, and provides a work area when the CPU 201 executes the program. A disk 203 stores an OS (Operating System), device drivers for peripheral devices, and programs for implementing various types of application software (including programs for performing the following processing, etc.). The CPU 201 executes the programs stored in the main memory 202 and the disk 203, thereby implementing Figure 1 The functions (software) of the image processing apparatus 100 shown in the figure and the processing of the flowchart described below.

[0050] The display memory 204 temporarily stores display data. The monitor 205 is, for example, a CRT monitor, a liquid crystal monitor, etc. The monitor 205 displays images, texts, etc. based on the data from the display memory 204. The mouse 206 and the keyboard 207 perform pointing input and character input by the user. These components are connected via a common bus 208 in a manner that allows them to communicate with each other.

[0051] The CPU 201 corresponds to an example of a processor. The image processing apparatus 100 may include at least a GPU (Graphics Processing Unit) or an FPGA (Field Programmable Gate Array) in addition to the CPU 201. The image processing apparatus 100 may include at least a GPU or an FPGA in place of the CPU 201. The main memory 202 and the disk 203 correspond to examples of memories.

[0052] (Processing Flow)

[0053] Next, we will refer to Figure 3 A processing procedure of the image processing apparatus 100 according to the present exemplary embodiment is described.

[0054] <Step S310: Step of obtaining three-dimensional image data>

[0055] In step S310 , the obtaining unit 101 obtains three-dimensional image data from the storage device 70 .

[0056] <Step S320: Classification step based on 2D-FCN>

[0057] In step S320, the first classification unit 102 extracts a two-dimensional region of interest from each two-dimensional axial image constituting the three-dimensional image data by using a two-dimensional segmentation method. The two-dimensional region of interest is output as a two-dimensional roughly segmented image corresponding to each category of the first category group. In the present exemplary embodiment, the first classification unit 102 is a trained 2D-FCN for classifying pixels into three categories, namely, liver, right kidney, and other than liver and right kidney.

[0058] Will refer to FIG. 4A to FIG. 4D Describe the input and output of 2D-FCN. Figure 4A 4 shows the three-dimensional image data 410 obtained in step S310. Figure 4A , three-dimensional image data 410 includes a plurality of axial images, such as axial image 411a and axial image 411b. Each of such axial images is input to the 2D-FCN. Figure 4C The three-dimensional roughly segmented images corresponding to the respective categories are shown. A three-dimensional roughly segmented image 430 of the liver region, a three-dimensional roughly segmented image 440 of the right kidney region, and a three-dimensional roughly segmented image 450 of the region other than the liver and the right kidney are shown in FIG. Figure 4C In Figure 4C , the regions of each category are shown in light colors, and the regions other than the regions of each category are shown in dark colors. Each three-dimensional roughly segmented image includes a two-dimensional roughly segmented image corresponding to each axial image as an input of the 2D-FCN. For example, in the three-dimensional roughly segmented image 430 of the liver region, the two-dimensional roughly segmented image 431a of the liver region is a two-dimensional roughly segmented image corresponding to the axial image 411a, and the two-dimensional roughly segmented image 431b of the liver region is a two-dimensional roughly segmented image corresponding to the axial image 411b. If an axial image is input through the following processing, the 2D-FCN outputs a two-dimensional roughly segmented image corresponding to each category. The roughly segmented image 441a and the roughly segmented image 441b of the right kidney region correspond to the axial image 411a and the axial image 411b of the three-dimensional image data 410, respectively. Similarly, the roughly segmented image 451a and the roughly segmented image 451b of the region other than the liver and the right kidney correspond to the axial image 411a and the axial image 411b, respectively.

[0059] Will refer to Figure 5 The processing of 2D-FCN is described. If the input image 510 (axial image) is input to the 2D-FCN, multiple convolution processes, pooling processes, and upsampling processes are performed in the intermediate layer 530. Then in the output layer 540, convolution processes are performed, and the output values ​​of each pixel are normalized by softmax processing to obtain an output image 520, which is a two-dimensional image containing classification information about the pixels.

[0060] Here, the convolution process extracts features while maintaining the shape of the image. The pooling process reduces the spatial size (i.e., the width and height of the image) to expand the receptive field. Upsampling uses pooling information to obtain detailed resolution. The output image 520 includes a two-dimensional roughly segmented image with the same number as the number of categories to be classified. In the present exemplary embodiment, the output image 520 includes a two-dimensional roughly segmented image 521 of the liver region, a two-dimensional roughly segmented image 522 of the right kidney region, and a two-dimensional roughly segmented image 523 of the region other than the liver and the right kidney. The model represented by FCN as a network structure including convolution, pooling (encoder) and upsampling (decoder) is called an encoder-decoder model. The encoder retains global information while allowing some displacement of pixels. The decoder restores resolution while maintaining the feature quantity generated by the encoder. Even if the position of the segmentation target changes between the two-dimensional image data constituting different three-dimensional image data, it is expected that this model provides a certain accuracy. The network structure is not limited to this structure, as long as the network structure has an architecture that can process image information in a multi-scale manner.

[0061] Now, the softmax process will be described by using the equation. Assume that the pixel value before the softmax process is a i，j，k , the pixel value p after softmax processing is calculated based on the following equation (1): i，j，k .

[0062] [Equation 1]

[0063]

[0064] Here, i and j represent pixel numbers for identifying pixels in the two-dimensional roughly segmented image. k represents the number corresponding to each category of the roughly segmented image. K represents the number of categories to be classified. Since the present exemplary embodiment deals with three category problems of the liver region, the right kidney region, and the region other than the liver and the right kidney, K=3. This processing makes the sum of the pixel values ​​in the classification results corresponding to each category 1. In other words, each two-dimensional roughly segmented image corresponding to each category is a two-dimensional likelihood map representing the likelihood of the region.

[0065] (Changes in the first taxon)

[0066] The roughly segmented image as the output of the first classification unit 102 need not be an image that represents each pixel according to the likelihood indicating the possibility of being in the category as described above. For example, the value of each pixel can be represented by a binary value or by a different range of values ​​on a category by category basis. The value obtained from the first classification unit can be used directly as the value of the pixel. The numerical value can be converted by using one or more thresholds.

[0067] The first classification unit 102 may be a CNN different from the aforementioned 2D-FCN. The first classification unit 102 is not limited to CNN, and may be a classifier based on any two-dimensional segmentation method, as long as the classifier extracts the region of interest from each axial image. For example, a classifier based on non-CNN machine learning (SVM, k-means, enhancement, and random forest) or a classifier that does not require training (e.g., a threshold classifier) ​​may be used. The first classification unit may include multiple classifiers. The multiple classifiers may be used in parallel or hierarchically.

[0068] The first classification unit 102 is also not limited to classifiers that classify into three or more categories at the same time as described above. For example, the first classification unit 102 can be a classifier for classifying into two categories (liver region and region other than liver region). A plurality of classifiers for two-category classification can be prepared to obtain a roughly segmented image from each classifier. In a specific example, by preparing a classifier for classifying the liver region and other regions and a classifier for classifying the right kidney region and other regions, the same result as the above process can be obtained.

[0069] <Step S330: Processing end determination step>

[0070] In step S330, the first classification unit 102 determines whether there is unprocessed two-dimensional image data to be processed in the three-dimensional image data. If there is unprocessed two-dimensional image data, the process proceeds to step S320. In step S320, the first classification unit 102 roughly segments the region of interest. If there is no unprocessed two-dimensional image data, the process proceeds to the next step.

[0071] <Step S340: 3D roughly segmented image generation step>

[0072] In step S340, the first classification unit 102 serving as a three-dimensional data generation unit generates a three-dimensional roughly segmented image by stacking or interpolating or integrating the two-dimensional roughly segmented images corresponding to the categories obtained by the processing up to step S330. Then, the first classification unit 102 outputs the generated three-dimensional roughly segmented image to the second classification unit 103. The three-dimensional roughly segmented image corresponds to the three-dimensional classification result. Note that the function of the three-dimensional data generation unit for generating a three-dimensional roughly segmented image can be implemented by the second classification unit 103 or by a computing unit other than the first classification unit 102 and the second classification unit 103.

[0073] In the present exemplary embodiment, the classification technique performed by the second classification unit 103 described below is a graph cut method for classifying pixels into two categories (liver and other than liver). The graph cut method constructs a graph based on pixel information about the area to be extracted (foreground) and the area outside the area to be extracted (background), and extracts the area to be extracted so that the designed energy function is minimized (or maximized). The energy function E of the graph cut method is generally defined as the following equation (2).

[0074] [Equation 2]

[0075]

[0076] In the aforementioned equation (2), i and j represent different pixel numbers in the image, respectively. λ is a constant parameter for adjusting the contribution of the data item E1 and the smoothing item E2.

[0077] <Step S350: Pixel information setting step>

[0078] The second classification unit 103 sets pixel information indicating the position of a pixel corresponding to at least the foreground or the background based on the three-dimensional roughly segmented image. Fig. 7A and Figure 7B Here, how to give pixel information about the liver region as the foreground and the region other than the liver region as the background when extracting the liver region will be described. For ease of description, a two-dimensional image is used for illustration. Fig. 7A A rough segmentation image 720 of the liver region, a rough segmentation image 730 of the right kidney region, and a rough segmentation image 740 of the region other than the liver and the right kidney are shown, and each rough segmentation image is a rough segmentation image with five pixels on each side. The pixel value of the rough segmentation image according to the present exemplary embodiment changes continuously, because the pixel value indicates the likelihood of the possibility in each category. Therefore, in the present exemplary embodiment, the rough segmentation images corresponding to the three categories are compared pixel by pixel, and the pixels whose likelihood of the liver is higher than the likelihood of other categories among the likelihood of the liver, the likelihood of the right kidney, and the likelihood other than the liver and the right kidney are set as foreground pixels (pixels determined as foreground). Similarly, the pixels whose likelihood of the liver is lower than the likelihood of other categories are set as background pixels (pixels determined as background). The remaining pixels (such as the pixels whose likelihood of the liver is the second highest) are set as intermediate pixels (it has not yet been determined whether the pixel is foreground or background). Figure 7BA seed image 750 obtained by integrating the roughly segmented image using the above method is shown. The seed image 750 includes a foreground pixel 751, a background pixel 752, and an intermediate pixel 753. In this exemplary embodiment, an energy function represented by equation 2 is defined based on the seed image 750. Specifically, the data item E1 sets the energy of each pixel based on the distance value of the foreground pixel 751 and the background pixel 752 in the seed image 750. The energy is set to give a sufficiently high cost to the corresponding edge (t-link) of the pixel in the figure, so that even after segmentation by the graph cut method, the foreground pixel 751 and the background pixel 752 still remain the foreground pixel and the background pixel (remain unchanged). The smoothing item E2 sets the energy based on the difference in the voxel value between adjacent pixels in the three-dimensional image data 410. By defining the energy function in this way, the intermediate pixel 753 is classified into the foreground or the background. Then, the second classification unit 103 divides the area to be extracted and the area other than the area to be extracted by assuming that the pixels classified as the foreground are in the area to be extracted and the pixels classified as the background are in the area other than the area to be extracted.

[0079] Any method for determining at least foreground pixels or background pixels based on the roughly segmented image obtained from the first classification unit 102 can be used as a method for setting pixel information. For example, the second classification unit can determine one (foreground or background) pixel based on the roughly segmented image, and determine another (background or foreground) pixel based on the pixel value of the three-dimensional image data. In the aforementioned method, when determining whether a pixel is in the foreground or in the background based on the roughly segmented image, the determination is made based on the magnitude relationship between the likelihood of the liver and the likelihood other than the liver. However, this is not restrictive. For example, the foreground pixel can be determined first, and the pixels that are a predetermined distance (distance value) or farther from the determined foreground pixel can be determined as background pixels. Alternatively, the foreground (or background) pixel can be determined by threshold processing with reference only to the roughly segmented image of the liver. Thus, even if the first classification unit is a classifier for two-category classification, the second classification unit can thereby determine the foreground pixel and the background pixel, and the remaining pixels can also be determined as intermediate pixels.

[0080] Although in the aforementioned method, the energy of each pixel is set based on the distance value, the energy of each pixel can be set based on the likelihood in the roughly segmented image. In the aforementioned method, the energy is set to give a sufficiently high cost to the corresponding edge (t-link) of the pixel in the image so that the foreground pixel and the background pixel will not be changed by the graph cut method. However, this is not restrictive. For example, the size of the likelihood can be set to the energy that the foreground pixel or the background pixel can change. The energy can be set based on the likelihood and the distance value.

[0081] In addition, the foreground pixels, background pixels, and intermediate pixels in the seed image can be set based on the size of the likelihood. For example, in the aforementioned method for setting the foreground pixels and the background pixels, it is described that the roughly segmented images corresponding to each category are first compared pixel by pixel. If the likelihood of the liver is the highest, the target pixel is set to the foreground, and if the likelihood of the liver is the lowest, the target pixel is set to the background. However, if the likelihoods of the categories are similar, only comparing the likelihoods of the pixels and at least determining the foreground or background may not provide sufficient accuracy. In view of this, for example, a predetermined threshold value can be set for the likelihood. If the predetermined threshold value is met, the foreground or background can be set. Pixels that do not meet the predetermined threshold value can be set as intermediate pixels. The predetermined threshold value of the likelihood can be combined with the aforementioned pixel-by-pixel comparison results of the roughly segmented images corresponding to each category.

[0082] If the likelihood does not meet the predetermined threshold, or if the pixel-by-pixel comparison between the categories shows that the likelihood of the liver is neither higher nor lower than the likelihood of other pixels, the pixel is set as an intermediate pixel. Then, information about the foreground and background is assigned to the data item E1 in the graph cut energy function and segmented. Here, pixels other than foreground pixels or background pixels are considered intermediate pixels and classified by graph cuts.

[0083] At least foreground pixels, background pixels and intermediate pixels are set based on the size of likelihood or pixel-by-pixel likelihood comparison. A sufficiently high energy is set for the corresponding edge (t-link) of the pixel so that the foreground pixel and the background pixel will not be changed by the graph cut method. Here, the likelihood of the intermediate pixel is further assigned to the graph cut energy function. Such a construction enables the intermediate pixel to be classified by the graph cut to add information based on the features learned by the classifier, and it is expected to improve the segmentation accuracy. More specifically, foreground pixels, background pixels and intermediate pixels are set, and the first energy and the second energy, both of which are sufficiently high, are set for the edge (t-link) of the foreground pixel and the background pixel. A third energy corresponding to the likelihood of the output from the classification unit 102 is set for the intermediate area.

[0084] <Step S360: Step of extracting the region of interest by graph cut method>

[0085] The second classification unit 103 extracts the region to be extracted from the three-dimensional image data by a three-dimensional segmentation method based on the pixel information obtained in step S350. Then, the second classification unit 103 uses the information about the extraction region to be extracted as Figure 4B The three-dimensional image of interest 420 shown is output to the disk 203 and the external storage device 70. The image of interest 420 includes an image of interest 421a and an image of interest 421b.

[0086] Now, refer to FIG. 4A to FIG. 4D The difference between the three-dimensional roughly segmented image 430 of the liver region which is one of the outputs of the first classification unit 102 and the three-dimensional focus image 420 which is the output of the second classification unit 103 is described. Figure 4D As shown in (enlarged view of the roughly segmented image), the two-dimensional roughly segmented image 431b of the liver region and the two-dimensional roughly segmented image 441b of the right kidney region include high-likelihood regions 432 and 442, respectively, near the boundary between the liver region and the right kidney region. Here, region 432 and region 442 are regions representing the same region in the three-dimensional image data. Assume that in this region, among the three three-dimensional roughly segmented images that are the output of the first classification unit 102, the likelihood indicating the possibility of being the right kidney is the highest (light color), while the likelihood indicating the possibility of being the liver is the second highest. In this case, the region corresponds to the middle pixel. Therefore, the region is the region to be classified as the foreground or background by the graph cut method. Since the graph cut method tends to divide regions along nearby edges, region 432 ( Figure 4B ).

[0087] Note that if the second classification unit uses the graph cut method, the second classification unit can be constructed to perform classification into the second category group by using the first energy (foreground), second energy (background) and third energy (middle pixels) set in step S350.

[0088] (Changes in the second taxon)

[0089] The second classification unit 103 can use any three-dimensional segmentation method based on pixel information about at least one of the foreground or the background. Examples of other segmentation methods may include a region growing method, a snake method, and a level set method. In the region growing method, the pixel position of the region to be extracted needs to be given. To this end, for example, the position where the foreground pixel 751 is located can be given as the pixel position of the region to be extracted. In the case of the snake method and the level set method, the coordinate data on the outline of the region to be extracted needs to be given as an initial value. To this end, for example, Figure 7B The boundary pixel of the foreground pixel 751 in is given as the initial value. In order to extract the contour, a classifier trained with and without the contour can be used for extraction. Although in the present exemplary embodiment, the second classification unit 103 classifies the pixels into two categories using the three-dimensional segmentation method, a method for classifying the pixels into three or more categories can be used.

[0090] (Effects of the First Exemplary Embodiment)

[0091] As described above, the image processing device 100 according to the first exemplary embodiment roughly extracts the region of interest from each two-dimensional axial image data constituting the three-dimensional image data by using the two-dimensional segmentation method of machine learning. Then, the image processing device 100 extracts the three-dimensional region of interest by the three-dimensional segmentation method based on the three-dimensional image data and the three-dimensional roughly segmented image. Since such a configuration can automatically give pixel information about the region of interest and the region other than the region of interest to the three-dimensional segmentation method, the burden of the user inputting pixel information can be reduced. In addition, since an image in which the region of interest is located at a position roughly the same as the position of the region of interest in the training image can be input, the region of interest can be extracted with high accuracy.

[0092] (Variation Example)

[0093] In the image processing device 100 according to the aforementioned first exemplary embodiment, the three-dimensional image data identical to the original three-dimensional image data is input to the first classification unit 102 and the second classification unit 103. However, three-dimensional image data different from the original three-dimensional image data may be input. To this end, the acquisition unit 101 may include a calculation unit. The calculation unit may be included in a unit other than the acquisition unit, or may include a plurality of calculation units. The image processing device 100 may further include a calculation unit in addition to the acquisition unit 101. Before the different three-dimensional image data are input to the first classification unit and the second classification unit, the acquisition unit 101 may, for example, perform noise reduction, normalization of voxel values, spatial normalization, and / or image scaling on the original three-dimensional image data. Such processing may be common between the classification units, or different processing may be performed on the inputs of each classification unit. In the former case, the first three-dimensional image data and the second three-dimensional image data are the same image. In the latter case, the first three-dimensional image data and the second three-dimensional image data are different images. For example, if different spatial normalization is performed on the input image of the classification unit, a modification unit for normalizing the output of the first classification unit 102 to the input space of the second classification unit 103 is required. Since this configuration can input three-dimensional image data based on the characteristics of each classifier, it is expected to improve the segmentation accuracy of the region of interest. If the first classification unit 102 is classified based on machine learning, it is preferably processed to reduce the resolution according to limitations such as calculation time, memory capacity and image size during training. In this case, the first three-dimensional image data is constructed to have a resolution lower than that of the second three-dimensional image data. Note that the aforementioned processing can be performed only on the input of any classification unit. If the object is the same subject, additional information in the past can be referred to, and three-dimensional image data taken at different times can be used. For example, an image of the subject taken at a certain time phase is input to the first classifier, and an image taken at another time phase is input to the second classifier. In other words, the first three-dimensional image data and the second three-dimensional image data are three-dimensional image data associated with the same subject and having different imaging times. For example, such a configuration is expected to improve segmentation accuracy by referring to different time phases compared to a case where a region of interest is extracted only from information on the result of a single imaging operation.

[0094] As another variation, the calculation unit or the modification unit may apply a process for removing components other than the maximum connected component to a roughly segmented image of the region of interest as an output of the first classification unit 102 or a three-dimensional image of interest as an output of the second classification unit 103. The maximum connected component refers to the largest region among regions where pixels are continuous. In addition, small independent regions may be deleted by applying an opening process or a closing process or by applying a process for deleting regions other than the maximum connected component. If the number of regions of interest to be drawn in the three-dimensional image data is 1, the segmentation accuracy of the region of interest is improved because unnecessary regions can be removed thereby. Other pre-processing and post-processing may be used in combination.

[0095] The segmentation target can be any organ other than the liver, and can be a cyst, a tumor, a lymph node, their contours, etc., as long as the segmentation target is an area represented on the image. In addition, the training data of the first classification unit and the category of the segmentation target do not need to be the liver, the right kidney, and other than the liver and the right kidney as in the present exemplary embodiment. For example, the category can include organs that are close to the segmentation target and have similar CT values, such as the liver and the right kidney. The category can include non-organ objects, such as bones.

[0096] For example, if the second classifier is a classifier for classifying into the liver and other than the liver, the number of categories (liver, right kidney, and other than the liver and right kidney) constituting the category group of the first classifier is constructed so as to classify more categories than the number of categories constituting the category group of the second classifier. Since the classification result of the first classifier provides classification based on features of a larger number of categories, it is expected that this construction improves the segmentation accuracy of the liver region.

[0097] In addition, the training data and the segmentation target of the first classification unit 102 can be changed according to the configuration and segmentation target of the second classification unit 103. For example, if the second classification unit 103 uses the level set method or the snake method, the boundary pixels of the focus area can be used as the training data of the first classification unit. In this case, the first classification unit roughly extracts the boundary pixels, and the second classification unit 103 extracts an image based on the roughly segmented image.

[0098] If the segmentation accuracy of the first classification unit is insufficient due to lack of training data, the training data can be expanded. As described above, machine learning includes learning features from training data and exerting classification capabilities based on these features. On the other hand, for example, if the organ changes shape or has different voxel values ​​due to lesions, the amount of training data may not be enough for high-precision classification. In this case, the data can be expanded by data expansion (such as shifting voxel values ​​and performing rotation and translation) or using GAN (Generative Adversarial Network) to provide a sufficient amount of training data. The classifier is trained using the obtained training data on lesions with different shapes and voxel values, so that highly versatile segmentation can be performed. Training data with different shapes and voxel values ​​can be used to train the same classifier as a classifier trained with images of typical organs, etc., or to train different classifiers respectively. In the case of different classifiers, the image processing device 100 may include an additional processing unit for determining whether to use a classifier trained with images of ordinary organs, etc. or a classifier trained with training data of different shapes and voxel values ​​for classification. Alternatively, the image processing device 100 may have a hierarchical structure to use different classifiers based on the classification results of the classifiers trained using images of common organs, etc. The image processing device 100 may have a structure to use classifiers trained using images of common organs, etc. based on the classification results of different classifiers. For example, if the classification target of the training device is a liver and a normal liver with different shapes or voxel values, the ground truth labels may be different or the same. The labels may be changed based on the presence or characteristics of other categories. If multiple classifiers are used for segmentation, the classification is not limited to the classification including the same organ. For example, if multiple classifiers are used for classification hierarchically, the liver may be extracted first, and then the right kidney may be extracted based on the segmentation result of the liver.

[0099] [Second exemplary embodiment]

[0100] In an image processing device according to a first exemplary embodiment, a first classification unit uses a two-dimensional segmentation method of inputting individual two-dimensional image data constituting three-dimensional image data. However, if the region of interest is drawn as a small area on the two-dimensional image data, or if another area near the region of interest has a voxel value similar to that of the region of interest, extracting the region of interest from the two-dimensional image data reduces the segmentation accuracy of the region of interest. In addition, if the region of interest has an abnormal shape, using a two-dimensional segmentation method based on machine learning reduces the segmentation accuracy. In contrast, the advantage of three-dimensional image data is that voxels containing information about the connection between two-dimensional images can be used. However, if the region of interest in the target image changes appearance, or if the resolution is different, classifying only the three-dimensional image data may not provide sufficient segmentation accuracy. In addition, the number of images of medical image data is often insufficient.

[0101] Then, the image processing apparatus according to the second exemplary embodiment uses a three-dimensional space region with a predetermined size as training data for the classification unit. As used herein, the predetermined size refers to a three-dimensional space region including two or more two-dimensional image data in the three-dimensional image data. Note that the two-dimensional image data constituting the three-dimensional space region with a predetermined size does not need to include continuous slices. For example, the two-dimensional slices constituting the three-dimensional space region with a higher resolution among the three-dimensional image data with different resolutions can be thinned out in a predetermined number, and the resulting slices can be used as training data. It is expected that the training data with a predetermined size provides the following effect: while maintaining the number of training data, the time and effort for processing (spatial normalization) for matching the image size during training with the image size during classification is reduced. The first classification unit first inputs each three-dimensional space region with a predetermined size in the three-dimensional image data (an example of the first three-dimensional space region and the second three-dimensional space region), and roughly extracts the region of interest. Here, the first classifier after training is expected to match the size of the training data to be input. To this end, for example, the image processing apparatus may include a storage device for storing at least one of the number of pixels, the number of voxels, and the number of slices indicating the input size of the image used during training. The size of the input image of the classification unit is then determined based on the input size stored during training. For example, if the size of the image to be classified during training is larger than the input size, the image can be extracted to be used as the input image. On the other hand, if the size of the image to be classified during training is smaller than the input size, an additional interpolation process is performed to obtain the input image.

[0102] In the present exemplary embodiment, a roughly segmented image corresponding to each three-dimensional spatial region is obtained. Therefore, the roughly segmented images are further stacked or interpolated or integrated to obtain a three-dimensional roughly segmented image. Like the first exemplary embodiment, based on the three-dimensional image data and the three-dimensional roughly segmented image obtained by the first classification unit, the region to be extracted is extracted by a three-dimensional segmentation method. In the present exemplary embodiment, since the input to the first classification unit is a three-dimensional spatial region, a 3D-FCN for processing three-dimensional image data is used. The differences from the first exemplary embodiment will now be described.

[0103] (Module structure)

[0104] The image processing apparatus according to the present exemplary embodiment has a configuration similar to that of the image processing apparatus 100 according to the first exemplary embodiment. Figure 1 A functional configuration of the image processing apparatus according to the present exemplary embodiment is described, with portions overlapping with the image processing apparatus according to the first exemplary embodiment omitted.

[0105] The components of the image processing apparatus 100 will be described below.

[0106] The obtaining unit 101 performs processing similar to that of the obtaining unit 101 according to the first exemplary embodiment.

[0107] The first classification unit 102 performs three-dimensional processing on each three-dimensional space region having a predetermined size constituting the three-dimensional image data obtained from the obtaining unit 101, thereby obtaining a roughly segmented image of the three-dimensional space region corresponding to each category. Then, the first classification unit generates a three-dimensional roughly segmented image corresponding to each category by stacking the roughly segmented images of the three-dimensional space region corresponding to each category category by category, and sends the three-dimensional roughly segmented image to the second classification unit.

[0108] The second classification unit 103 performs processing similar to that of the second classification unit 103 according to the first exemplary embodiment.

[0109] (Processing Flow)

[0110] Next, we will refer to Figure 8 A processing procedure of the image processing apparatus 100 according to the present exemplary embodiment is described.

[0111] <Step S810: 3D image data acquisition step>

[0112] The process of step S810 is substantially the same as the process of step S310 according to the first exemplary embodiment, and therefore a description thereof will be omitted.

[0113] <Step S820: Classification step based on 3D-FCN>

[0114] In step S820, the first classification unit 102 divides the three-dimensional image data into a plurality of three-dimensional spatial regions of a predetermined size, and roughly extracts the region of interest from each of the divided three-dimensional spatial regions through three-dimensional processing. An example of a three-dimensional spatial region of a predetermined size is a set of a predetermined number of continuous axial images in the three-dimensional image data. The result of the rough segmentation of the region of interest by the first classification unit 102 is output as a three-dimensional roughly segmented image of the three-dimensional spatial region corresponding to each category in the first category group. Note that the output of the first classification unit 102 may be a two-dimensional roughly segmented image.

[0115] In the present exemplary embodiment, the first classification unit 102 is a trained 3D-FCN for classifying pixels into three categories: liver, right kidney, and other than liver and right kidney. In the present exemplary embodiment, the three-dimensional spatial region having a predetermined size in the three-dimensional image data is an image obtained by stacking three (an example of the predetermined size) continuous axial images in the three-dimensional image data. The roughly segmented image of the three-dimensional spatial region corresponding to each category is a three-dimensional roughly segmented image having the same image size as the three-dimensional spatial region having the predetermined size.

[0116] Will refer to 9A to 9D Describe the input and output of 3D-FCN. 9A to 9D The three-dimensional image data 410 in represents the three-dimensional image data 410 obtained in step S810. For example, according to this exemplary embodiment, Fig.9A The three-dimensional spatial regions shown in different shades in FIG. 1 are input to the 3D-FCN. In the figure, the first three-dimensional spatial region 911 and the second three-dimensional spatial region 912 each include three consecutive axial images. Here, the first three-dimensional spatial region 911 and the second three-dimensional spatial region 912 do not include mutually overlapping regions. The 3D-FCN performs Figure 5 The processing shown in the figure is performed, and the first three-dimensional roughly segmented image 921 and the second three-dimensional roughly segmented image 922 corresponding to the three-dimensional space regions corresponding to the respective categories are output. Here, the first three-dimensional roughly segmented image 921 corresponding to the first three-dimensional space region 911 and the second three-dimensional roughly segmented image 922 corresponding to the second three-dimensional space region 912 are stacked as a continuous three-dimensional roughly segmented image. Although in the foregoing description, the three-dimensional space regions of a predetermined size into which the three-dimensional image data is divided are described as not including regions that overlap with each other, overlapping regions may be included. Fig. 9BA first three-dimensional spatial region 911 and a second three-dimensional spatial region 913 having an overlapping region are shown. The overlapping region includes two of the three consecutive axial images. If there is an overlapping region, the output of the first classification unit can be obtained, for example, by integrating the overlapping parts of the roughly segmented images. The integration processing of the overlapping region can be performed, for example, before the softmax processing, or can be performed after the softmax processing by the following method (described in step S840). If the integration processing is performed before the softmax processing, for example, convolution processing or pixel-by-pixel pooling processing is performed for integration.

[0117] <Step S830: Step of determining an unprocessed three-dimensional space area>

[0118] In step S830, the first classification unit 102 determines whether there is an unprocessed three-dimensional space area that has not been processed in the three-dimensional image data to be processed. If it is determined that there is an unprocessed three-dimensional space area, the process of step S820 is performed on the area. If it is determined that there is no unprocessed three-dimensional space area, the process proceeds to step S840.

[0119] <Step S840: Generation of a 3D Rough Segmentation Image Corresponding to 3D Segmentation>

[0120] In step S840, the first classification unit 102 performs at least one of stacking processing, interpolation processing, and integration processing on the three-dimensional roughly segmented images corresponding to each category of the first category group. Through such processing, the first classification unit 102 generates a three-dimensional roughly segmented image corresponding to each category for the purpose of three-dimensional segmentation, and sends the three-dimensional roughly segmented image to the second classification unit 103. In other words, the first classification unit generates a three-dimensional roughly segmented image when inputting to the second classification unit 103 based on multiple classification results corresponding to the three-dimensional space area of ​​the first classification unit 102. There are several variations as follows, and these variations are used to achieve the generation of three-dimensional roughly segmented images corresponding to each category.

[0121] For example, the variation of the input to the first classification unit 102 may include a case where there is no overlapping area (e.g. Fig.9A and Fig. 9C As shown) and when there are overlapping areas (such as Fig. 9B and Fig.9D In addition, the case where there is no overlapping area may include a case where the first three-dimensional space area 911 and the second three-dimensional space area 912 are continuous, and a case where the first three-dimensional space area 911 and the second three-dimensional space area 912 are discontinuous.

[0122] Variations of the output of the first classification unit 102 may include: generating a three-dimensional roughly segmented image based on all roughly segmented images output from the 3D-FCN (e.g. Fig.9A and Fig. 9B ), and a case where a three-dimensional coarsely segmented image is generated by using each single coarsely segmented image selected from the coarsely segmented images output by the 3D-FCN (e.g., from the coarsely segmented images corresponding to the three-dimensional spatial region as input) (as shown in Fig. 9C and Fig.9D More specifically, there are cases where a three-dimensional roughly segmented image is output from an image corresponding to a three-dimensional spatial region from an input image, and a case where fewer roughly segmented results are output than two-dimensional images constituting the three-dimensional spatial region. Note that there is no particular limitation on the number of slices of a selected single roughly segmented image as long as the number of slices is less than the number of slices of a roughly segmented image as an output from 3D-FCN. Alternatively, a roughly segmented image generated by performing integration processing based on all roughly segmented images corresponding to the three-dimensional spatial region may be used.

[0123] First, refer to Fig.9A The following case is described: the first three-dimensional spatial region and the second three-dimensional spatial region as input are continuous regions without overlapping regions, and a three-dimensional roughly segmented image is generated by using all roughly segmented images as outputs of 3D-FCN. In this case, all slices of the first three-dimensional roughly segmented image 921 and the second three-dimensional roughly segmented image 922 are simply stacked to generate a three-dimensional roughly segmented image as an input to the second classification unit 103.

[0124] Next, we will refer to Fig. 9CA case of a first two-dimensional roughly segmented image 925 corresponding to a first three-dimensional space region 911 and a second two-dimensional roughly segmented image 926 corresponding to a second three-dimensional space region 912 is described. A three-dimensional roughly segmented image is generated by using a predetermined number of roughly segmented images among the roughly segmented images corresponding to each three-dimensional space region having a predetermined size classified by the first classification unit 102. A method for generating a three-dimensional roughly segmented image by using a single roughly segmented image selected from the roughly segmented images of each input three-dimensional space region or obtained by integrating the roughly segmented images of each input three-dimensional space region will now be described. The single roughly segmented image may be a roughly segmented image at the center of all roughly segmented images corresponding to the three-dimensional space region having a predetermined size, or a roughly segmented image obtained by averaging and integrating the pixel values ​​of all roughly segmented images corresponding to the three-dimensional space region. Alternatively, a single roughly segmented image may be generated by performing maximum value integration using the maximum value among the pixel values ​​of all roughly segmented images corresponding to the three-dimensional space region. The average integration or maximum integration does not need to target all slices of the roughly segmented images constituting the three-dimensional space region having a predetermined size. For example, multiple roughly segmented images among all the roughly segmented images corresponding to the three-dimensional space region can be integrated. Multiple roughly segmented images can be integrated differently. Each of the multiple roughly segmented images can be integrated multiple times. The generated first two-dimensional roughly segmented image 925 and the generated second two-dimensional roughly segmented image 926 are stacked, interpolated, and integrated to generate a three-dimensional roughly segmented image.

[0125] Next, the following case will be described: the first three-dimensional spatial region and the second three-dimensional spatial region as inputs to the first classification unit 102 are discontinuous regions without overlapping regions, and the output is a three-dimensional roughly segmented image corresponding to each category or a two-dimensional roughly segmented image corresponding to a three-dimensional spatial region having a predetermined size. If all roughly segmented images extracted by 3D-FCN are used, at least one of a stacking process, an interpolation process, and an integration process is performed on both the roughly segmented image corresponding to the first three-dimensional spatial region and the roughly segmented image corresponding to the second three-dimensional spatial region, thereby generating a three-dimensional roughly segmented image. The step of generating a three-dimensional roughly segmented image from a two-dimensional roughly segmented image corresponding to a three-dimensional spatial region having a predetermined size is similar to the above.

[0126] Finally, the case where the input to the first classification unit 102 includes overlapping regions will be described. Fig. 9B and Fig.9D shown. Fig. 9BThe roughly segmented image 921 corresponding to the first three-dimensional space region 911 having a predetermined size and the roughly segmented image 923 corresponding to the second three-dimensional space region 913 having a predetermined size are included. The roughly segmented image 921 and the roughly segmented image 923 each include an overlapping region 924. In this case, average integration or maximum integration may be applied to the overlapping region 924. At least one of stacking processing, interpolation processing, and integration processing may be performed between the roughly segmented images to generate a three-dimensional roughly segmented image. Fig.9D In the embodiment, the first three-dimensional space region 911 and the second three-dimensional space region 913 have an overlapping area. Here, the first three-dimensional space region 911 corresponds to a two-dimensional roughly segmented image 927, and the second three-dimensional space region 913 corresponds to a two-dimensional roughly segmented image 928. A roughly segmented image is generated based on each roughly segmented image corresponding to a three-dimensional space region having a predetermined size. The number of roughly segmented images generated based on each roughly segmented image corresponding to a three-dimensional space region does not have to be one, as long as the number is less than the number of slices constituting a three-dimensional space region having a predetermined size. A single roughly segmented image to be generated can be obtained by interpolating, stacking, or integrating the roughly segmented images corresponding to a three-dimensional space region having a predetermined size. The steps of generating a three-dimensional roughly segmented image based on a two-dimensional roughly segmented image corresponding to a three-dimensional space region having a predetermined size are similar to those described above. In addition, in the present configuration, the connectivity between the three-dimensional space regions can be considered by adjusting the number of slices generated based on each three-dimensional space region having a predetermined size and the number of non-overlapping slices.

[0127] <Step S850: Pixel information setting step>

[0128] The process of step S850 is substantially the same as the process of step S350 according to the first exemplary embodiment, and thus a description thereof will be omitted.

[0129] <Step S860: Step of extracting the region of interest by graph cut method>

[0130] The process of step S860 is substantially the same as the process of step S360 according to the first exemplary embodiment, and thus a description thereof will be omitted.

[0131] (Effects of the Second Exemplary Embodiment)

[0132] As described above, the image processing apparatus 100 according to the second exemplary embodiment divides the three-dimensional image data into a plurality of three-dimensional space regions having a predetermined size, and roughly extracts the region of interest from each three-dimensional space region. Since the first classification unit 102 can consider the three-dimensional connectivity, this configuration improves the accuracy of the result of the rough segmentation of the region of interest. This effect helps to improve the accuracy of the second classification unit 103 in segmenting the region of interest.

[0133] (Variation Example)

[0134] In the aforementioned image processing device 100 according to the second exemplary embodiment, the first classification unit 102 uses only a three-dimensional space region of a predetermined size in the three-dimensional image data as its input. However, other information about the region of interest may be input at the same time. Here, the input refers to the training data when training the classifier and the three-dimensional image data input to the first classification unit. For example, the result of segmenting the region of interest from the adjacent three-dimensional space region may be input at the same time. The existence probability map of the region of interest estimated by other methods and / or the bounding box including the region of interest may be input. Such a configuration improves the accuracy of the result of roughly segmenting the region of interest by the first classification unit, because the first classification unit 102 can further use the region of interest information about other regions and the information about the region of interest estimated by other methods. This effect helps to improve the accuracy of the second classification unit 103 segmenting the region of interest. Such additional information may also be used for the second classification unit 103. The additional information for the second classification unit is given to the pixel information to generate the seed image, which is expected to achieve a more accurate segmentation of the region of interest. The additional information may be given only to the first classification unit, only to the second classification unit, or to both classification units. This modification is not limited to the second exemplary embodiment, and is also effective for the first exemplary embodiment.

[0135] (Other Exemplary Embodiments)

[0136] The present invention can also be implemented by performing the following processing. The processing includes: providing software (program) for implementing the functions of the aforementioned exemplary embodiments to a system or device via a network or various storage media, and having a computer (or CPU, MPU, etc.) of the system or device read and execute the program.

[0137] The present invention is not limited to the above-described embodiments, and various changes and modifications can be made within the spirit and scope of the present invention. Therefore, in order to disclose the scope of the present invention, claims are made.

[0138] This application claims priority from Japanese Patent Application No. 2018-247775 filed on December 28, 2018, and Japanese Patent Application No. 2019-183345 filed on October 3, 2019, the entire contents of which are incorporated herein by reference.

Claims

1. An image processing system, comprising: a first classification unit configured to classify a plurality of pixels in each of a plurality of two-dimensional axial image data corresponding to a slice of the object constituting first three-dimensional image data including image data of the object into a first category group by inputting the two-dimensional axial image data corresponding to the slice of the object into a trained classifier, wherein the trained classifier is a classifier trained by using the two-dimensional axial image data as training data; and A second classification unit is constructed to classify multiple pixels in second three-dimensional image data constituting a three-dimensional image of an object including image data of the object into a second category group by a three-dimensional segmentation method, using the classification results of the multiple two-dimensional image data corresponding to the multiple slices of the object respectively performed by the first classification unit. The second category group includes at least one category of the first category group.

2. The image processing system according to claim 1, in, The first classification unit further includes a three-dimensional data generating unit configured to generate a three-dimensional classification result by classifying the plurality of pixels in each of the plurality of two-dimensional axial image data corresponding to the slices of the object constituting the first three-dimensional image data into the first category group, and The second classification unit classifies the plurality of pixels in the second three-dimensional image data constituting the three-dimensional image of the object into the second category group based on the three-dimensional classification result.

3. The image processing system according to claim 2, wherein: The three-dimensional data generating unit generates the three-dimensional classification result by performing at least one of stacking processing, interpolation processing, and integration processing on the classification results of each of the plurality of two-dimensional axial image data by the first classification unit.

4. The image processing system according to claim 1, wherein: The first three-dimensional image data and the second three-dimensional image data are medical image data.

5. The image processing system according to claim 1, wherein: The first group of categories includes a category representing a liver and a category representing a kidney.

6. The image processing system according to claim 1, wherein: The second group of categories includes a category representing the liver.

7. The image processing system according to claim 1, wherein: The second classification unit classifies the plurality of pixels in the second three-dimensional image data into the second class group by a graph cut method.

8. The image processing system according to claim 1, wherein: The classification result of the first classification unit includes the likelihood of the categories constituting the first category group.

9. The image processing system according to claim 8, further comprising: an obtaining unit configured to obtain, based on the likelihood, region information indicating a foreground, a background, and an intermediate region corresponding to neither the foreground nor the background, Wherein, the second classification unit is based on the likelihood and the region information: setting a first energy for pixels corresponding to the foreground, setting a second energy for pixels corresponding to the background, A third energy is set for a pixel corresponding to the middle area, wherein the third energy corresponds to the likelihood at the pixel, and The plurality of pixels in the second three-dimensional image data constituting a three-dimensional image of an object are classified into the second class group by a graph cut method using the first energy, the second energy, and the third energy.

10. The image processing system according to claim 1, wherein: The first three-dimensional image data and the second three-dimensional image data are the same three-dimensional image data.

11. The image processing system according to claim 1, wherein: The first three-dimensional image data and the second three-dimensional image data are different three-dimensional image data.

12. The image processing system according to claim 11, wherein: The first three-dimensional image data has a lower resolution than the second three-dimensional image data.

13. The image processing system according to claim 11 or 12, wherein: The first three-dimensional image data and the second three-dimensional image data are three-dimensional image data associated with the same subject and having different imaging times.

14. The image processing system according to claim 1, wherein: The trained classifier includes at least one of a CNN, a SVM, and a k-means model.

15. The image processing system according to claim 1, wherein: The trained classifier is a CNN having a network structure using an encoder and a decoder.

16. The image processing system according to claim 1, wherein: The second classification unit classifies the plurality of pixels in the second three-dimensional image data into the second class group by at least one of a graph cut method, a level set method, a region growing method, and a snake method.

17. An image processing system, comprising: a first classification unit, which is constructed to classify each of a plurality of voxels corresponding to a first three-dimensional spatial region having a predetermined size corresponding to two or more slices in first three-dimensional image data including image data of the object into a first category group by inputting three-dimensional data corresponding to the first three-dimensional spatial region into a trained classifier, wherein the three-dimensional data corresponding to the first three-dimensional spatial region includes two or more axial image data, and to classify each of a plurality of voxels corresponding to a second three-dimensional spatial region having the predetermined size corresponding to the two or more slices into the first category group by inputting three-dimensional data corresponding to the second three-dimensional spatial region into the trained classifier, wherein the classifier is a classifier trained using the three-dimensional image data including the two or more axial image data corresponding to the three-dimensional spatial region having a predetermined size constituting the three-dimensional image data as training data; and A second classification unit is constructed to classify the multiple voxels included in the second three-dimensional image data having a predetermined size corresponding to two or more slices of the object into a second category group through a three-dimensional segmentation method, using the classification results of the multiple voxels corresponding to the first three-dimensional spatial area and the classification results of the multiple voxels corresponding to the second three-dimensional spatial area, wherein the second category group includes at least one category of the first category group.

18. The image processing system according to claim 17, wherein: The classification result of the first classification unit on a region different from the first three-dimensional space region or the second three-dimensional space region is also input to the first classification unit.

19. The image processing system according to claim 17 or 18, wherein: The first three-dimensional spatial region and the second three-dimensional spatial region do not overlap each other.

20. The image processing system according to claim 17, further comprising: a three-dimensional data generating unit configured to generate a three-dimensional classification result by performing at least one of stacking processing, interpolation processing, and integration processing on the classification results of the plurality of voxels corresponding to the first three-dimensional space region and the classification results of the plurality of voxels corresponding to the second three-dimensional space region, and The second classification unit classifies the at least one voxel included in the second three-dimensional image data into the second category group by a three-dimensional segmentation method based on the three-dimensional classification result.

21. The image processing system according to claim 17 or 18, wherein: The first three-dimensional space region and the second three-dimensional space region include regions overlapping each other.

22. The image processing system according to claim 21, further comprising: a three-dimensional data generating unit configured to generate a three-dimensional classification result by integrating the classification results corresponding to the overlapping area by the first classification unit, The second classification unit classifies the at least one voxel included in the second three-dimensional image data into the second category group based on the three-dimensional classification result.

23. The image processing system according to claim 17, wherein: The first three-dimensional spatial region and the second three-dimensional spatial region respectively include a plurality of axial image data.

24. The image processing system according to claim 17, wherein: The first three-dimensional image data and the second three-dimensional image data are medical image data.

25. The image processing system according to claim 17, wherein: The first group of categories includes a category representing a liver and a category representing a kidney.

26. The image processing system according to claim 17, wherein: The second group of categories includes a category representing the liver.

27. The image processing system according to claim 17, wherein: The second classification unit classifies the plurality of voxels in the second three-dimensional image data into the second class group by a graph cut method.

28. The image processing system according to claim 17, wherein: The classification result of the first classification unit includes the likelihood of the categories constituting the first category group.

29. The image processing system according to claim 28, further comprising: an obtaining unit configured to obtain, based on the likelihood, region information indicating a foreground, a background, and an intermediate region corresponding to neither the foreground nor the background, Wherein, the second classification unit is based on the likelihood and the region information: setting a first energy for pixels corresponding to the foreground, setting a second energy for pixels corresponding to the background, A third energy is set for a pixel corresponding to the middle area, wherein the third energy corresponds to the likelihood at the pixel, and The plurality of voxels in the second three-dimensional image data are classified into the second class group by a graph cut method using the first energy, the second energy, and the third energy.

30. The image processing system according to claim 17, wherein: The first three-dimensional image data and the second three-dimensional image data are the same three-dimensional image data.

31. The image processing system according to claim 17, wherein: The first three-dimensional image data and the second three-dimensional image data are different three-dimensional image data.

32. The image processing system according to claim 31, wherein: The first three-dimensional image data has a lower resolution than the second three-dimensional image data.

33. The image processing system according to claim 31 or 32, wherein: The first three-dimensional image data and the second three-dimensional image data are three-dimensional image data associated with the same subject and having different imaging times.

34. The image processing system according to claim 17, wherein: The trained classifier includes at least one of a CNN, a SVM, and a k-means model.

35. The image processing system according to claim 17, wherein: The trained classifier is a CNN having a network structure using an encoder and a decoder.

36. The image processing system according to claim 17, wherein: The second classification unit classifies the plurality of voxels in the second three-dimensional image data into the second class group by at least one of a graph cut method, a level set method, a region growing method, and a snake method.

37. An image processing method, comprising: classifying a plurality of pixels in each of a plurality of two-dimensional axial image data corresponding to the slice of the object constituting first three-dimensional image data including image data of the object into a first category group by inputting the two-dimensional axial image data corresponding to the slice of the object into a trained classifier, wherein the trained classifier is a classifier trained by using the two-dimensional axial image data as training data; and Using the classification results of classifying multiple two-dimensional image data corresponding to multiple slices of the object respectively into the first category group, multiple pixels in second three-dimensional image data constituting a three-dimensional image of the object including the image data of the object are classified into a second category group through a three-dimensional segmentation method, and the second category group includes at least one category of the first category group.

38. An image processing method, comprising: as a first classification, classifying each of a plurality of voxels corresponding to a first three-dimensional spatial region having a predetermined size corresponding to two or more slices in first three-dimensional image data including image data of the object into a first category group by inputting the three-dimensional data corresponding to the first three-dimensional spatial region into a trained classifier, wherein the three-dimensional data corresponding to the first three-dimensional spatial region includes two or more axial image data; as a first classification, classifying each of a plurality of voxels corresponding to the second three-dimensional spatial region having the predetermined size into the first category group by inputting three-dimensional data corresponding to the second three-dimensional spatial region into the trained classifier, wherein the classifier is a classifier trained using three-dimensional image data including two or more axial image data corresponding to the three-dimensional spatial region having the predetermined size constituting the three-dimensional image data as training data; and As a second classification, using the classification results of the multiple voxels corresponding to the first three-dimensional spatial region and the classification results of the multiple voxels corresponding to the second three-dimensional spatial region, the multiple voxels included in the second three-dimensional image data with a predetermined size corresponding to two or more slices of the object are classified into a second category group through a three-dimensional segmentation method, and the second category group includes at least one category of the first category group.

39. A computer-readable storage medium storing a program for causing a computer to perform the image processing method according to claim 37 or 38.

Images