Neural network, neural network learning method, program, and image processing device

Abstract

To use for learning even training data in which correct answer data corresponding to some classes is missing.SOLUTION: A neural network includes: a common network that is common to a plurality of classes and extracts a first feature from an input image; and a plurality of single-class networks corresponding to the plurality of classes, and extracting each of a plurality of second features based on the first feature. Each of the plurality of single-class networks has a single-class output layer for individually identifying a class corresponding to each of the networks.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The disclosure of this specification relates to a neural network, a neural network learning method, a program, and an image processing device. [Background technology]

[0002] In recent years, in the field of image processing, classifiers based on deep learning have been attracting attention. Among them, a method called supervised learning trains a classifier by using teaching data consisting of training images and correct answer data. Here, the training image is an image obtained by photographing an object of interest existing in real space with an imaging device, and an image of the object of interest is depicted. The correct answer data is data that exists corresponding to the training image. Taking segmentation as an example, the correct answer data is an image in which a correct answer region is depicted, and indicates at which position in the corresponding training image the object of interest is depicted. When there are multiple objects of interest, multiple correct answer data corresponding to the classes representing each of the objects of interest are required.

[0003] Non-Patent Document 1 discloses a technology for constructing a classifier that simultaneously segments the regions of multiple objects of interest by using training data consisting of a training image and multiple pieces of ground truth data corresponding to classes that respectively represent the objects of interest. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Jonathan Long, et al. “Fully Convolutional Networks for Semantic Segmentation”, CVPR, 2015. Summary of the Invention [Problem to be solved by the invention]

[0005] The technology disclosed in Non-Patent Document 1 cannot use training data in which correct answer data corresponding to some classes are missing among correct answer data corresponding to classes representing a plurality of target objects for learning. This is not limited to classifiers that perform segmentation, but is an issue that occurs in all methods that belong to supervised learning.

[0006] The present invention aims to provide a technology that can be used for learning even when teaching data is missing correct answer data corresponding to some classes among the correct answer data corresponding to classes representing each of a plurality of target objects. [Means for solving the problem]

[0008] The present invention one The embodiment of the present invention is as follows: A method for training a neural network including a common network common to a plurality of classes, and a plurality of single-class networks having a single-class output layer connected to the common network for individually identifying each of the plurality of classes, comprising: An acquisition step of acquiring training data including correct answer data corresponding to one or more classes among the plurality of classes; a learning step of learning the neural network using the teaching data; Equipped with The learning step includes, when learning is performed using teaching data in which correct answer data corresponding to one or more classes among the plurality of classes is missing, learning a single-class network corresponding to a class in which correct answer data is not missing, and learning a single-class network corresponding to a class in which correct answer data is missing. does not train a single-class network that corresponds to the The neural network learning method is characterized by the above.

[0009] The present invention two The embodiment of the present invention is as follows: An acquisition unit that acquires an image; A classification unit that performs classification using a trained neural network; An image processing device comprising: the trained neural network comprises a common network common to a plurality of classes, and a plurality of single-class networks connected to the common network and having a single-class output layer for individually identifying each of the plurality of classes, the trained neural network being trained using teaching data including correct answer data corresponding to one or more of the plurality of classes; the trained neural network is a neural network in which a single-class network corresponding to a class in which no correct answer data is missing is trained when training using teaching data in which correct answer data corresponding to one or more classes among the plurality of classes is missing, The image processing device is characterized in that

[0010] The present invention three The embodiment of the present invention is as follows: An acquisition step of acquiring an image; A classification step of performing classification using a trained neural network; An image processing method comprising: the trained neural network comprises a common network common to a plurality of classes, and a plurality of single-class networks connected to the common network and having a single-class output layer for individually identifying each of the plurality of classes, the trained neural network being trained using teaching data including correct answer data corresponding to one or more of the plurality of classes; the trained neural network is a neural network in which a single-class network corresponding to a class in which no correct answer data is missing is trained when training using teaching data in which correct answer data corresponding to one or more classes among the plurality of classes is missing, The image processing method is characterized in that: Effect of the Invention

[0012] According to the disclosure of this specification, even teaching data in which correct answer data corresponding to some classes among the correct answer data corresponding to each of multiple target objects is missing can be used for learning. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a block diagram showing an example of the functional configuration of an image processing apparatus 100 according to a first embodiment. [Diagram 2] FIG. 1 is a block diagram showing an example of a hardware configuration of an image processing apparatus according to a first embodiment. [Diagram 3] FIG. 2 is a diagram showing an example of a processing procedure of the image processing apparatus 100 according to the first embodiment. [Figure 4] FIG. 1 is a diagram for explaining the arrangement of a CNN according to a first embodiment. [Diagram 5] FIG. 2 is a block diagram showing an example of the functional configuration of an image processing apparatus 500 according to the first embodiment. [Figure 6] FIG. 2 is a diagram showing an example of a processing procedure of the image processing apparatus 500 according to the first embodiment. [Figure 7] FIG. 4 is a view for explaining an example of teaching data according to the first embodiment; [Figure 8] FIG. 11 is a diagram for explaining the arrangement of a CNN according to a second embodiment. [Figure 9] FIG. 13 is a diagram for explaining the arrangement of a CNN according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, preferred embodiments of the image processing device disclosed in this specification will be described with reference to the drawings. The same or equivalent components, members, and processes shown in each drawing will be given the same reference numerals, and duplicated descriptions will be omitted as appropriate. In addition, some of the components, members, and processes will be omitted as appropriate in each drawing.

[0015] In the following, the technology disclosed in this specification will be described using a convolutional neural network (CNN), which is one type of neural network, as an example of a classifier based on deep learning.

[0016] <First embodiment> (Overview) In this embodiment, a CNN that segments organ regions from an abdominal CT image (three-dimensional tomographic image) captured by an X-ray computed tomography (X-ray CT) device will be described as an example. The organs (objects of interest) to be segmented are the liver, kidneys, and gallbladder.

[0017] In this embodiment, the inference process is realized using the image processing device 100, and the learning process is realized using the image processing device 500.

[0018] The image processing device 100 uses CNN, which is one aspect of the invention disclosed in this specification, to segment the liver, kidney, and gallbladder regions from an abdominal CT image. The image processing device 100 first constructs a CNN based on parameters obtained by learning. The CNN used in this embodiment belongs to 2D-CNN, which inputs a two-dimensional image and outputs a classification result for the two-dimensional image. The CNN is constructed by connecting a network corresponding to the liver, a network corresponding to the kidney, and a network corresponding to the gallbladder (an example of a single-class network) to a network (an example of a common network) common to the liver, kidney, and gallbladder. More generally, the CNN includes a common network common to all classes and a plurality of single-class networks corresponding to each of the plurality of classes, which are connected so that the output of the common network is input to each of the single-class networks. The image processing device 100 then acquires an abdominal CT image to be subjected to segmentation processing, and sequentially inputs each of a plurality of two-dimensional tomographic images constituting the abdominal CT image to the CNN. After receiving the two-dimensional tomographic image as an input, the CNN executes a forward propagation process and outputs the classification result for the two-dimensional tomographic image to the image processing device 100. Then, by processing all the two-dimensional tomographic images with the CNN, the image processing device 100 obtains the classification result for the abdominal CT image. The classification result by the CNN is a segmentation result representing the areas of the liver, kidneys, and gallbladder.

[0019] The image processing device 500 generates parameters for the CNN used in the image processing device 100. First, the image processing device 500 constructs a CNN equivalent to the CNN used in the image processing device 100. The parameters of the network that constitutes the CNN are initialized by a predetermined method. Next, the image processing device 500 acquires training data. Here, the training data is composed of an abdominal CT image, which is a learning image, and correct answer data that exists in correspondence with the learning image. The correct answer data is , an image in which the correct answer region is drawn (correct answer image), and indicates at which position of the corresponding learning image the target object is drawn. Here, the image in which the correct answer region of the liver is drawn (correct answer image of the liver), the image in which the correct answer region of the kidney is drawn (correct answer image of the kidney), and the image in which the correct answer region of the gallbladder is drawn (correct answer image of the gallbladder) are correct answer data. The teaching data here is intended to be those in which at least one correct answer image exists among the correct answer image of the liver, the correct answer image of the kidney, and the correct answer image of the gallbladder for each teaching data. Then, the image processing device 500 uses the acquired teaching data to train the CNN. In the training of the CNN in the present invention, in the case of teaching data in which a correct answer image of the liver exists, the parameters of the common network and the network corresponding to the liver are updated. Similarly, in the case of teaching data in which a correct answer image of the kidney exists and teaching data in which a correct answer image of the gallbladder exists, the parameters of the common network constituting the CNN and the networks corresponding to each organ are updated. In this way, when the image processing device 500 learns using teaching data in which supervised answer data corresponding to one or more classes is missing, it learns (updates parameters) a single-class network corresponding to a class that is not missing. On the other hand, the image processing device 500 does not learn a single-class network corresponding to a class in which supervised answer data is missing. The image processing device 500 may also learn a common network regardless of whether supervised answer data is missing in the teaching data. With the above-mentioned configuration, the CNN in the present invention can use teaching data in which supervised answer data corresponding to some classes is missing among the supervised answer data corresponding to the classes representing each of a plurality of target objects for learning.

[0020] In this specification, the process of segmenting an organ region from an abdominal CT image to be processed by the image processing device 100 is referred to as an inference process, and the process of learning CNN by the image processing device 500 is referred to as a learning process. In this embodiment, the inference process and the learning process are respectively realized by the image processing device 100 and the image processing device 500, but both the inference process and the learning process may be realized by one image processing device.

[0021] In the following, training data in which all correct answer data corresponding to classes representing each of a plurality of target objects are available is referred to as complete training data. Training data in which correct answer data corresponding to some classes among the correct answer data corresponding to classes representing each of a plurality of target objects is missing is referred to as incomplete training data. Furthermore, network parameters are simply referred to as parameters.

[0022] [Inference process] First, the configuration of an image processing device 100 according to this embodiment will be described.

[0023] (Functional configuration) The functional configuration of an image processing device 100 according to this embodiment will be described with reference to Fig. 1. As shown in the figure, the image processing device 100 includes a first data acquisition unit 110, a first network construction unit 120, and an identification unit 130. In addition, the image processing device 100 according to this embodiment is communicably connected to an external storage device 70.

[0024] The storage device 70 is an example of a computer-readable storage medium, and is a large-capacity information storage device such as a hard disk drive (HDD) or a solid-state drive (SSD). The storage device 70 stores at least one abdominal CT image to be processed and parameters of the CNN in the classification unit 130.

[0025] The first data acquisition unit 110 acquires CNN parameters in the classification unit 130 from the storage device 70. Then, the first data acquisition unit 110 transmits the acquired CNN parameters to the first network construction unit 120. The first data acquisition unit 110 also acquires an abdominal CT image to be processed as an input image from the storage device 70. Then, the first The data acquisition unit 110 transmits the acquired input image to the classification unit 130 .

[0026] The first network construction unit 120 receives the CNN parameters from the first data acquisition unit 110. Next, the first network construction unit 120 constructs a CNN based on the CNN parameters. The configuration of the CNN will be described later. Then, the constructed CNN is transmitted to the identification unit 130.

[0027] The classification unit 130 receives the input image from the first data acquisition unit 110, and receives the CNN from the first network construction unit 120. The classification unit 130 then inputs the input image to the CNN and obtains a segmentation result.

[0028] At least some of the units of the image processing device 100 shown in Fig. 1 may be realized as independent devices. Also, each unit of the image processing device 100 may be realized as software that realizes each function, or may be realized by dedicated hardware. In this embodiment, each unit is realized by software.

[0029] (Hardware configuration) 2 is a diagram showing an example of the hardware configuration of the image processing device 100. The image processing device 100 has the configuration of a known computer (information processing device). The image processing device 100 includes, as its hardware configuration, a CPU 201, a main memory 202, a magnetic disk 203, a display memory 204, a monitor 205, a mouse 206, and a keyboard 207.

[0030] A CPU (Central Processing Unit) 201 mainly controls the operation of each component. A main memory 202 stores a control program executed by the CPU 201 and provides a working area when the CPU 201 executes the program. A magnetic disk 203 stores programs for implementing various application software including an OS (Operating System), device drivers for peripheral devices, and programs for performing processes described below. The CPU 201 executes programs stored in the main memory 202, the magnetic disk 203, etc., thereby implementing the functions (software) of the image processing device 100 shown in FIG. 1 and processes in flowcharts described below.

[0031] The display memory 204 temporarily stores display data. The monitor 205 is, for example, a CRT monitor or a liquid crystal monitor, and displays images, text, and the like based on the data from the display memory 204. The mouse 206 and keyboard 207 allow the user to input pointing and characters, respectively. The above components are connected to each other via a common bus 208 so that they can communicate with each other.

[0032] The CPU 201 corresponds to an example of a processor or a control unit. In addition to the CPU 201, the image processing device 100 may have at least one of a GPU (Graphics Processing Unit) and an FPGA (Field-Programmable Gate Array). Also, instead of the CPU 201, the image processing device 100 may have at least one of a GPU and an FPGA. The main memory 202 and the magnetic disk 203 correspond to an example of a memory or a storage device.

[0033] (Processing flow) The processing procedure of the image processing device 100 in this embodiment will be described with reference to FIG.

[0034] (Step S3100) In this step, the first data acquisition unit 110 acquires parameters of the CNN in the classification unit 130 from the storage device 70. Then, the first data acquisition unit 110 acquires the acquired parameters. The obtained CNN parameters are then transmitted to the first network construction unit 120.

[0035] (Step S3200) In this step, the first network constructing unit 120 constructs a CNN based on the CNN parameters received from the first data acquiring unit 110. Then, the first network constructing unit 120 transmits the constructed CNN to the identifying unit .

[0036] The configuration of the CNN in this embodiment will be described with reference to Fig. 4. The CNN 4000 according to this embodiment is configured by connecting a common network 4100 to a plurality of single-class networks 4200.

[0037] The common network 4100 is a network common to multiple classes. The common network 4100 is composed of an input layer that receives input and multiple convolution layers, and plays a role in extracting a first feature from an input image. In this embodiment, the common network 4100 is configured using a pooling layer and an upsampling layer, but may further include other layers such as a batch normalization layer. The convolution layer is configured to apply an activation function to a feature map that is a result after the convolution process. In this embodiment, the activation function in the common network 4100 is ReLU (Rectified Linear Unit).

[0038] The multiple single class networks 4200 are multiple networks corresponding to multiple classes. In this embodiment, there are three single class networks: a liver network 4210 corresponding to the liver, a kidney network 4220 corresponding to the kidney, and a gallbladder network 4230 corresponding to the gallbladder. Each of the multiple single class networks 4200 is composed of multiple convolution layers, and plays a role of extracting each of the second features based on the first feature extracted by the common network 4100. At the end of each of the multiple single class networks 4200, there is a single class output layer for individually identifying the corresponding class. Specifically, at the end of the liver network 4210, there is a single class output layer 4211 for identifying the liver class. Similarly, at the ends of the kidney network 4220 and the gallbladder network 4230, there are single class output layers 4221 and 4231 for identifying the respective classes. These single class output layers are convolution layers, and a Sigmoid function is set as the activation function. The Sigmoid function is a function that normalizes an input value to a range from 0 to 1. Therefore, each of the multiple single class networks 4200 outputs an image in which the pixel value of each pixel is between 0 and 1.

[0039] The CNN 4000 used in this embodiment is, for example, a 2D-CNN. The 2D-CNN receives a two-dimensional image as input and outputs a classification result for the two-dimensional image. Therefore, the 2D-CNN receives a two-dimensional tomographic image (a type of two-dimensional image) constituting an abdominal CT image (a three-dimensional tomographic image) by an input layer, and two-dimensional processing is performed by other layers constituting the CNN to output a classification result for the two-dimensional tomographic image.

[0040] The CNN 4000 configured in this manner has a plurality of parameters. For example, the weights in the convolution layer correspond to the parameters. In this step S3200, the parameters obtained in the learning process described below are set in the CNN to construct a CNN that segments the liver, kidney, and gallbladder regions. As a result, the CNN according to this embodiment can output each likelihood map as the classification result of the corresponding class from the single-class output layer present at the end of each of the multiple single-class networks 4200. For example, the liver network 4210 outputs a likelihood map of the liver in which the pixel value of each pixel indicates the likelihood of it being liver from 0 to 1. Similarly, the kidney network 4220 and the gallbladder network 4230 also output likelihood maps corresponding to their respective classes.

[0041] The CNN4000 according to this embodiment can use even teaching data (incomplete teaching data) that does not contain one or more correct images among the correct image of the liver, the correct image of the kidney, and the correct image of the gallbladder for learning by using a learning method described later. In other words, even teaching data that is missing correct answer data corresponding to some classes among the correct answer data corresponding to the classes representing each of a plurality of target objects can be used for learning.

[0042] Here, the CNN4000 in this embodiment will be compared with a method of preparing CNNs for identifying single classes in the number of classes representing each of the objects of interest. Comparing the two CNNs, the former (CNN4000 in this embodiment) has a portion of the network shared by the common network 4100, and therefore has a lower calculation cost during inference and learning than the latter. In addition, the common network 4100 can learn using all incomplete teaching data, such as incomplete teaching data in which only the correct answer image of the liver exists, incomplete teaching data in which only the correct answer image of the kidney exists, and incomplete teaching data in which only the correct answer image of the gallbladder exists. This allows the common network 4100 to learn more features, and therefore improvement in the classification accuracy by CNN can be expected.

[0043] Note that the CNN 4000 according to this embodiment receives a two-dimensional image as an input and outputs a classification result for the two-dimensional image (a type of 2D-CNN), but is not limited to this. For example, the CNN according to this embodiment may be a CNN belonging to 3D-CNN that receives a three-dimensional image as an input and outputs a classification result for the three-dimensional image. In this case, the CNN is composed of layers that perform three-dimensional processing, etc. In addition, the number of dimensions of the input may be any number, and there are no restrictions on the layers that constitute the classifier.

[0044] In the CNN4000 according to the present embodiment, only one common network exists that is common to all classes. However, one or more common networks may exist that are common to some and more than one of all classes. For example, the common network may include a first common network that is common to the first class and the second class, and a second common network that is common to the first class and the third class. In this case, the first common network and the second common network are connected to a single class network corresponding to the first class. For example, the CNN4000 may include a common network that is common to the liver class and the kidney class, and a common network that is common to the liver class and the gallbladder class. In this case, the common network that is common to the liver class and the kidney class is connected to the liver network and the kidney network, and similarly, the common network that is common to the liver class and the gallbladder class is connected to the liver network and the kidney network. At this time, two feature maps (first features) extracted from each of the two common networks are input to the liver network. Therefore, for example, the two first features are combined by a concatenation layer that combines two feature maps, or the two feature maps are added together and integrated, and then input to the liver network. In the human body, the liver and kidneys, and the liver and gallbladder may be adjacent to each other, but the kidneys and gallbladder are almost never adjacent to each other. Therefore, as described above, by separating the kidneys and gallbladder into separate common networks and avoiding the learning of unnecessary features, it is expected that the classification accuracy will improve.

[0045] Although each of the single-class networks 4200 in the CNN 4000 according to the present embodiment is configured with multiple convolution layers, it may be configured with only a single-class output layer. Also, each of the single-class networks 4200 may have the same layer configuration as each other, or may have different layer configurations.

[0046] (Step S3250) In this step, the first data acquisition unit 110 acquires the input image from the storage device 70 and transmits it to the classification unit 130 .

[0047] (Step S3300) In this step, the classification unit 130 executes an inference process by inputting the input image received from the first data acquisition unit 110 into the CNN constructed by the first network construction unit 120 and executing a forward propagation process. In this embodiment, as a result of the classification process by the CNN, a likelihood map of the liver, a likelihood map of the kidney, and a likelihood map of the gallbladder for the input image are obtained.

[0048] In this step, first, an input image is input to the CNN 4000. The CNN 4000 receives the input image at an input layer in the common network 4100. Then, forward propagation is performed to the subsequent layers while applying the processing of each layer. In the subsequent layers, the abdominal CT image received at the input layer and the feature map which is the processing result of the previous layer are input, and the feature map is output by applying each processing. When the processing of the common network 4100 is completed, the feature map which is the output of the common network 4100 is input to each of the multiple single class networks 4200. Each of the multiple single class networks 4200 executes forward propagation processing in the same manner as the common network 4100. At this time, the forward propagation processing of the liver network 4210, the kidney network 4220, and the gallbladder network 4230 may be realized by parallel processing or sequential processing. After the forward propagation process of each single-class network is completed, the liver network outputs a likelihood map of the liver, the kidney network outputs a likelihood map of the kidney, and the gallbladder network outputs a likelihood map of the gallbladder.

[0049] In the above, the forward propagation process is performed on all single class networks, but a configuration in which only some of the single class networks are processed may be used. For example, if only the liver identification result is required, the forward propagation process may be performed on only the common network 4100 and the liver network 4210. Alternatively, in step S3200, the first network construction unit 120 may construct a network consisting of, for example, only the common network 4100 and the liver network 4210. In this case, the kidney network 4220 and the gallbladder network 4230 are used only in the learning process. In this way, it is possible to obtain only the desired identification result while reducing the calculation cost.

[0050] (Step S3400) In this step, the classification unit 130 performs post-processing on each of the likelihood maps of the liver, kidneys, and gallbladder, and outputs the respective segmentation results.

[0051] In this embodiment, threshold processing and maximum connected component acquisition processing are applied as post-processing. The discrimination unit 130 first performs threshold processing on each likelihood map using a threshold value t=0.5. In the threshold processing, the pixel value (likelihood) of each pixel in each likelihood map is compared with the threshold value t, and the pixel value is converted into a binary image by replacing it with 0 or 1. The discrimination unit 130 then performs maximum connected component acquisition processing on the binary image obtained by performing threshold processing. Since the liver and gallbladder are a single organ, it is possible to eliminate isolated noise regions by acquiring the maximum connected component. Since the kidneys are a pair of right and left kidneys, the binary image of the kidney is divided into left and right, and the maximum connected component is acquired for each of the left and right images, thereby eliminating similarly isolated noise regions. The discrimination unit 130 outputs each binary image after applying the above post-processing as a segmentation result.

[0052] In the above, threshold processing and maximum connected component acquisition processing are performed as post-processing, but the present invention is not limited to this. For example, segmentation using a graph cut method may be applied. In this case, the data terms of the energy function may be designed based on each likelihood map. In addition, in order to eliminate overlapping regions, the maximum likelihood may be obtained at the same coordinates in each likelihood map, and a process of converting the images into a binary image may be performed based on this. Any other post-processing may be applied.

[0053] Note that step S3400 is not essential and may be omitted. In that case, the classification unit 130 outputs a likelihood map of the liver, a likelihood map of the kidney, and a likelihood map of the gallbladder as segmentation results of the liver, kidney, and gallbladder.

[0054] According to the above procedure, the image processing device 100 according to this embodiment obtains segmentation results of the liver, kidneys, and gallbladder from the abdominal CT image, and outputs them to the storage device 70 and the monitor 205.

[0055] When there are multiple input images to be processed by the classification unit 130, the processing from step S3250 to step S3400 may be repeatedly performed for each input image. In this embodiment, the classification unit 130 repeatedly performs the processing from step S3250 to step S3400 for each of the two-dimensional tomographic images constituting the abdominal CT image.

[0056] [Learning process] Next, the configuration of the image processing device 500 according to this embodiment will be described.

[0057] (Functional configuration) The functional configuration of an image processing device 500 according to this embodiment will be described with reference to Fig. 5. As shown in the figure, the image processing device 500 includes a second data acquisition unit 510 that acquires training data, a second network construction unit 520 that constructs a CNN, and a learning unit 530 that learns the CNN using the training data. In addition, the image processing device 500 according to this embodiment is connected to an external storage device 70 so as to be able to communicate with it.

[0058] The storage device 70 is an example of a computer-readable storage medium. The storage device 70 holds a plurality of pieces of teaching data. The teaching data will be described in detail later.

[0059] The second data acquisition unit 510 acquires the teaching data from the storage device 70. Then, the second data acquisition unit 510 transmits the acquired teaching data to the learning unit 530.

[0060] The second network construction unit 520 constructs a CNN and initializes parameters. Then, the constructed CNN is transmitted to the learning unit 530.

[0061] The learning unit 530 receives a plurality of pieces of teaching data from the second data acquisition unit 510, and receives the CNN from the second network construction unit 520. Next, the learning unit 530 learns the CNN by a learning method described below, and stores the parameters of the CNN after learning in the storage device 70.

[0062] At least some of the units of the image processing device 500 shown in Fig. 5 may be realized as independent devices. Also, each unit of the image processing device 500 may be realized as software that realizes each function, or may be realized by dedicated hardware. In this embodiment, each unit is realized by software.

[0063] (Hardware configuration) The hardware configuration of the image processing device 500 is similar to that of the image processing device 100 shown in FIG. 2, and therefore a description thereof will be omitted.

[0064] (Processing Procedure) The processing procedure of the image processing device 500 in this embodiment will be described with reference to FIG.

[0065] (Step S6100) In this step, the second data acquiring unit 510 acquires a plurality of pieces of teaching data from the storage device 70. Then, the second data acquiring unit 510 transmits the acquired plurality of pieces of teaching data to the learning unit 530.

[0066] The teaching data will be described with reference to FIG. 7. In this embodiment, the teaching data is composed of a training image 710 and correct answer data corresponding to the training image 710. The training image 710 is an image having the same attributes as an image to be processed by CNN, and is an abdominal CT image in this embodiment. In addition, the correct answer data in this embodiment are a correct answer image 711 of the liver, a correct answer image 712 of the kidney, and a correct answer image 713 of the gallbladder. The correct answer images 711, 712, and 713 are data indicating the positions or regions of the liver, kidney, and gallbladder, respectively, in the training image 710. The teaching data may also be referred to as learning data, teacher data, training data, etc. The correct answer image may also be referred to as a correct answer label image.

[0067] FIG. 7 shows four types of teaching data. FIG. 7(a) shows teaching data in which a correct answer image 711 of the liver, a correct answer image 712 of the kidney, and a correct answer image 713 of the gallbladder are all present for a learning image 710. This is called complete teaching data. FIG. 7(b) shows teaching data in which only a correct answer image 711 of the liver is present for a learning image 710. In other words, this is teaching data in which a correct answer image of the kidney and a correct answer image of the gallbladder are not present (missing). This is called incomplete teaching data. Similarly, FIG. 7(c) shows incomplete teaching data in which only a correct answer image 712 of the kidney is present for a learning image, and FIG. 7(d) shows incomplete teaching data in which only a correct answer image 713 of the gallbladder is present for a learning image.

[0068] In this step S6100, the second data acquisition unit 510 arbitrarily acquires the above four types of teaching data. That is, the second data acquisition unit 510 may acquire only the incomplete teaching data, or may acquire both the incomplete teaching data and the complete teaching data. Of course, the second data acquisition unit 510 may acquire only the complete teaching data.

[0069] For the sake of simplicity, the four types of teaching data are described using the same learning image 710 as an example. However, the four types of teaching data may or may not include the same learning image. If the same learning image is not included, it means that there is no overlapping learning image among the four types of teaching data. In addition, in FIG. 7, as the incomplete teaching data, teaching data in which only a correct answer image corresponding to any one of a plurality of classes exists is shown, but teaching data in which any number of correct answer images exists may be used. For example, incomplete teaching data in which a correct answer image of the liver and a correct answer image of the kidney exist, but a correct answer image of the gallbladder does not exist may be included.

[0070] (Step S6200) In this step, the second network construction unit 520 constructs a CNN and initializes parameters. Then, the constructed CNN is transmitted to the learning unit 530.

[0071] In this embodiment, the second network construction unit 520 uses the same CNN as the CNN used in the image processing device 100. The parameters of the CNN are determined by the initial value determination method proposed by He et al. [Kaiming He, et al., “Delving Deep Learning” The parameters are initialized by a known method, such as the method described in [Integrated Rectifiers: Surpassing Human-Level Performance on ImageNet Classification,” ICCV, 2015] or a method of setting a constant value or a random value as the initial value. Alternatively, the initial values ​​of the CNN parameters may be obtained from the storage device 70 and the parameters may be initialized. For example, when transferring parameters obtained by previously learning (transfer learning), the parameters are initialized by this method.

[0072] The CNN constructed by the second network construction unit 520 may not have the same configuration as the CNN used in the image processing device 100. For example, a Dropout layer may be added to improve generalization performance by learning. Even if the gallbladder network 4230 (one of the single-class networks) is not used in the classification process, the gallbladder network 4230 may be connected to the common network 4100 in the learning process to construct a CNN. This allows the common network to learn more features, and improvement in generalization performance can be expected, since teaching data containing only the correct image of the gallbladder can also be used for learning.

[0073] (Step S6300) In this step, the learning unit 530 learns the CNN received from the second network construction unit 520 by a learning method described later, using the multiple teaching data received from the second data acquisition unit 510. Then, the parameters of the CNN after learning are stored in the storage device 70.

[0074] The learning method will now be described in detail with reference to FIG.

[0075] (Step S6310) In this step, the learning unit 530 judges whether the progress of learning in the CNN satisfies the learning termination condition. As the learning termination condition, a known method such as a condition related to the number of epochs or a condition related to the convergence of the loss value is used. If the progress of learning in the CNN does not satisfy the learning termination condition, the process proceeds to step S6320. If the learning termination condition is satisfied, the process proceeds to step S6370.

[0076] (Step S6320) In this step, the learning unit 530 selects one set of arbitrary teaching data (a set of a learning image and correct answer data) from among a plurality of teaching data.

[0077] In this embodiment, the learning unit 530 selects the attributes of the teaching data as one group in order for each group. The attribute of the teaching data is information on which class of correct answer data (correct answer image) the teaching data has. That is, the learning unit 530 groups the teaching data according to which class of correct answer data the teaching data has, and selects teaching data for each group within one epoch to perform learning. In this embodiment, the storage device 70 holds information (attribute information) on the attributes of each teaching data. The learning unit 530 acquires the attributes of each teaching data by referring to the attribute information, and executes processing according to the attributes. For example, the learning unit 530 first selects complete teaching data from among a plurality of teaching data, and executes step S6300 sequentially. When there is no unprocessed complete teaching data, next, the incomplete teaching data of the liver is selected, and similarly, step S6300 is executed sequentially. Then, step S6300 is executed sequentially for the incomplete teaching data of the kidney and the incomplete teaching data of the gallbladder. This is called one epoch. After step S6300 has been executed for all groups of training data, the training data is processed again in order, starting from the complete training data. In this way, by having the CNN learn the attributes of the training data as one group in order, it becomes less likely to fall into a limit cycle and the training becomes stable.

[0078] The order in which the teaching data are selected may be any order. In the above-mentioned method, the influence of the group selected immediately before the end of learning remains relatively strong on the parameters. For example, in step S6310, when the number of epochs is set as the end condition, incomplete teaching data in which only the correct image of the gallbladder exists is learned last in the above-mentioned method. In this case, the accuracy of identifying the gallbladder may be high, and the accuracy of identifying the liver and kidneys, which are not included as correct images in the teaching data, may be low. To avoid this, for example, a method of randomly selecting teaching data or a method of selecting the complete teaching data to be learned last in one epoch may be adopted.

[0079] (Step S6330) In this step, the learning unit 530 determines the layer in which the parameters are to be updated based on the attributes of the teaching data selected in step S6320. That is, in this step, the learning unit 530 determines the layer in which the parameters are to be updated when learning using the teaching data, depending on which class of correct answer data the selected teaching data has.

[0080] If complete teaching data such as that shown in FIG. 7(a) is selected in step S6320, in this step, the learning unit 530 determines the layers included in the common network 4100 and all of the multiple single-class networks 4200 as the layers whose parameters will be updated.

[0081] On the other hand, when incomplete teaching data in which only a correct answer image of the liver is included as shown in Fig. 7(b) is selected, in this step, the learning unit 530 determines layers included in the common network 4100 and the liver network 4210 as layers for updating parameters. In the case of incomplete teaching data in which only a correct answer image of the kidney is included as shown in Fig. 7(c) or incomplete teaching data in which only a correct answer image of the gallbladder is included as shown in Fig. 7(d), the learning unit 530 similarly determines layers for updating parameters. That is, when incomplete teaching data is selected in step S6320, the learning unit 530 determines layers included in the common network 4100 and the single class network corresponding to the class in which the correct answer image is included as layers for updating parameters.

[0082] Note that the layers whose parameters are not to be updated may be determined arbitrarily. For example, in the above description, it is determined that the parameters of the layers included in the common network 4100 are updated in all cases, but the parameters of a predetermined part of layers may be fixed. This allows the transferred parameters to be used as they are when the previously learned parameters are transferred in step S6200, for example.

[0083] (Step S6340) In this step, the learning unit 530 inputs the learning image of the teaching data selected in step S6320 to the CNN and executes a forward propagation process, thereby obtaining a provisional classification result (likelihood map) based on the current parameters.

[0084] Here, the range of the network on which the forward propagation process is performed may always be the entire network regardless of the attributes of the teaching data, but unnecessary calculations may be eliminated by performing the process only on the range required for the processing in step S6350 depending on the attributes. That is, when complete teaching data as shown in FIG. 7(a) is selected in step S6320, in this step the learning unit 530 performs the forward propagation process on the common network 4100 and all of the multiple single class networks 4200. As a result, a provisional identification result of the liver is obtained from the liver network 4210. Similarly, a provisional identification result of the kidney is obtained from the kidney network 4220, and a provisional identification result of the gallbladder is obtained from the gallbladder network 4230. .

[0085] On the other hand, when incomplete teaching data in which only a correct answer image of the liver is present as shown in FIG. 7(b) is selected in step S6320, the learning unit 530 performs forward propagation processing on the common network 4100 and the liver network 4210 in this step. As a result, a provisional identification result of the liver is obtained from the liver network 4210. At this time, the provisional identification result of the kidney and the provisional identification result of the gallbladder are not used in the subsequent steps, so that the kidney network 4220 and the gallbladder network 4230 do not perform forward propagation processing. For incomplete teaching data in which only a correct answer image of the kidney is present as shown in FIG. 7(c) and incomplete teaching data in which only a correct answer image of the gallbladder is present as shown in FIG. 7(d), the forward propagation processing is similarly performed to obtain a provisional identification result. That is, when incomplete teaching data is selected in step S6320, the forward propagation processing is performed on the common network 4100 and the single class network corresponding to the class in which the correct answer image is present, and a provisional identification result is obtained.

[0086] (Step S6350) In this step, learning unit 530 calculates a loss value based on the correct image of the training data selected in step S6320 and the provisional classification result obtained in step S6340.

[0087] When complete training data as shown in FIG. 7(a) is selected in step S6320, a provisional classification result for the liver, a provisional classification result for the kidney, and a provisional classification result for the gallbladder are obtained in step S6340. Therefore, in this step, the learning unit 530 calculates the loss value for each class based on each of the correct images of the training data and each of the provisional classification results output by the multiple single class networks 4200. More specifically, the correct image t liver and the provisional liver identification result y output by the liver network 4210. liver and the loss function L based on liver (y liver ,t liver ) to calculate the loss value for the provisional liver identification result. Similarly, the loss function L kidney (y kidney ,t kidney ) to calculate the loss value for the provisional kidney classification result, and the loss function L gallbladder (y gallbladder ,t gallbladder ) to calculate the loss value for the provisional identification result of the gallbladder.

[0088] On the other hand, when incomplete teaching data in which only a correct answer image of the liver is included as shown in FIG. 7(b) is selected in step S6320, a provisional liver identification result is obtained in step S6340. Therefore, in this step, the learning unit 530 calculates a loss function L based on the correct answer image of the liver of the teaching data and the provisional liver identification result output by the liver network 4210. liver (y liver ,t liver ) to calculate a loss value for the provisional classification result of the liver. A loss value is calculated similarly for incomplete teaching data containing only the correct image of the kidney as shown in Fig. 7(c) and for incomplete teaching data containing only the correct image of the gallbladder as shown in Fig. 7(d). That is, when incomplete teaching data is selected in step S6320, a loss value for each provisional classification result is calculated based on each of the correct images and each of the provisional classification results corresponding to the class in which the correct image exists.

[0089] It should be noted that any known loss function may be used. For example, MSE (Mean Squared Error) or Cross Entropy may be used. Furthermore, for each output of the single-class output layer, which is a plurality of provisional classification results, a loss value may be calculated using the same loss function, or different loss functions may be used to calculate the loss value. For example, the loss value for the provisional result of the liver may be calculated using MSE, and the loss values ​​for the provisional results of the kidneys and the gallbladder may be calculated using Cross Entropy. Furthermore, weights may be assigned to each loss value. For example, if it is desired to place more emphasis on the liver during learning, a large weight may be assigned to the loss value for the provisional result of the liver, and a large weight may be assigned to the loss value for the provisional results of the kidneys and the gallbladder. A small weight may be assigned to the loss value.

[0090] (Step S6360) In this step, learning unit 530 updates the parameters based on the information on the layer for which the parameters are to be updated determined in step S6330 and the loss value for each of the provisional results calculated in step S6350.

[0091] When complete teaching data as shown in FIG. 7(a) is selected in step S6320, a loss value for the provisional identification result of the liver, a loss value for the provisional identification result of the kidney, and a loss value for the identification result of the gallbladder are obtained in step S6350. Therefore, in this step, the learning unit 530 updates the parameters using a known method such as the backpropagation method based on each loss value. Here, the backpropagation method is used as an example. The loss value for the provisional identification result of the liver is a loss value for the output of the liver network 4210. Therefore, the learning unit 530 calculates the gradient of the loss value for the provisional identification result of the liver by performing backpropagation processing from the single class output layer 4211 for identifying the liver class present at the end of the liver network 4210. Similarly, the gradient of the loss value for the provisional identification result of the kidney is calculated by performing backpropagation processing from the single class output layer 4221 for identifying the kidney class. The loss value for the provisional classification result of the gallbladder is calculated by performing backpropagation processing from the single class output layer 4231 for classifying the gallbladder class to calculate the gradient. The common network 4100 calculates the gradient by performing backpropagation processing, taking into account the values ​​backpropagated from each of the multiple single class networks 4200. Then, the learning unit 530 updates the parameters of the layer for updating the parameters determined in step 6330 based on the gradient.

[0092] On the other hand, when incomplete teaching data in which only a correct answer image of the liver is included as shown in FIG. 7(b) is selected in step S6320, a loss value for the provisional identification result of the liver is obtained in step S6350. Therefore, in this step, the learning unit 530 performs backpropagation processing to calculate a gradient of the loss value for the provisional identification result of the liver from the single class output layer 4211 for identifying the liver class present at the end of the liver network 4210. The common network 4100 performs backpropagation processing to calculate a gradient, taking into account the value backpropagated from the liver network 4210. Then, the learning unit 530 updates the parameters of the layer for updating the parameters determined in step 6330 based on the gradient. The parameters are updated in the same manner in incomplete teaching data in which only a correct answer image of the kidney is included as shown in FIG. 7(c) and incomplete teaching data in which only a correct answer image of the gallbladder is included as shown in FIG. 7(d).

[0093] (Step S6370) In this step, the learning unit 530 stores the CNN parameters in the storage device 70, and ends step S6300.

[0094] According to the above procedure, the image processing device 500 according to this embodiment generates parameters for the CNN in the image processing device 100.

[0095] (Effects of this embodiment) As described above, the CNN in the image processing device 100 and the image processing device 500 according to the present embodiment can learn even from teaching data in which some correct answer images are missing among the correct answer images of the liver, the kidney, and the gallbladder. In other words, even teaching data in which some correct answer data corresponding to classes representing each of a plurality of target objects is missing can be used for learning.

[0096] <Second embodiment> (Overview) The CNN in the first embodiment is configured to be able to learn even with incomplete teaching data by connecting single-class networks corresponding to each of a plurality of classes to a common network. However, since the CNN in the first embodiment outputs only the classification results of the corresponding classes at the respective ends of the single-class networks, it is difficult to obtain an output that takes into account the arrangement relationship of each class. For example, the liver network updates parameters only based on the loss value for the provisional classification result of the liver, so it is trained to classify into either the liver or other than the liver. That is, in the liver network, the kidney and the gallbladder are trained as one of the objects included in the objects other than the liver, so that the arrangement relationship between them is difficult to learn. The same is true for the kidney network and the gallbladder network.

[0097] For example, in the case of organs in a human body, where the positional relationship between organs is determined to some extent, learning the positional relationship between each organ (class) is expected to improve the classification accuracy. Therefore, in this embodiment, the configuration of a CNN equipped with a multi-class network having a multi-class output layer and a learning method thereof will be described so that an output that takes into account the mutual positional relationship can be obtained.

[0098] In this embodiment, the invention disclosed in this specification is realized, similarly to the first embodiment, by using an image processing device 100 and an image processing device 500. The image processing device 100 executes the inference process of the CNN according to this embodiment, and the image processing device 500 executes the learning process.

[0099] [Inference process] (functional configuration, hardware configuration) The functional configuration and hardware configuration of the image processing device 100 according to this embodiment are the same as those of the image processing device 100 according to the first embodiment, and therefore a description thereof will be omitted.

[0100] (Processing flow) A processing procedure of the image processing device 100 in this embodiment will be described with reference to Fig. 3. In the following, the description will be omitted to avoid overlapping parts with the description of the image processing device 100 according to the first embodiment.

[0101] (Step S3100) The processing of this step is the same as step S3100 in the first embodiment, and therefore a description thereof will be omitted.

[0102] (Step S3200) In this step, the first network constructing unit 120 constructs a CNN based on the CNN parameters received from the first data acquiring unit 110. Then, the first network constructing unit 120 transmits the constructed CNN to the identifying unit .

[0103] The configuration of the CNN in this embodiment will be described with reference to Fig. 8. The CNN 8000 according to this embodiment is configured by further connecting a multi-class network 8300 to the CNN 4000 according to the first embodiment. That is, the CNN 8000 according to this embodiment has a common network 4100, a plurality of single-class networks 4200, and a multi-class network 8300. The common network 4100 and the plurality of single-class networks 4200 are the same as the CNN according to the first embodiment, and therefore description thereof will be omitted.

[0104] The multi-class network 8300 is connected to the single-class networks 4200, and receives a plurality of second features (features related to each single class) extracted by each of the single-class networks 4200. The multi-class network 8300 has a multi-class output layer for identifying a plurality of classes, and Based on the feature, a third feature (a feature related to multi-class) is extracted. Then, the multi-class network 8300 simultaneously outputs the classification results of the multiple classes based on the third feature. Specifically, the classification results of the multiple classes are likelihood maps of the liver, kidney, gallbladder, and other regions. Here, the other regions refer to all regions other than the liver, kidney, and gallbladder depicted in the abdominal CT image. The multi-class network 8300 has a concatenation layer for combining the multiple second features. In addition, at the end, there is a multi-class output layer 8301 for simultaneously outputting the likelihood maps of the liver, kidney, gallbladder, and other regions. The multi-class output layer 8301 is a convolution layer, and a Softmax function is set as an activation function.

[0105] The CNN 8000 configured in this manner has a plurality of parameters. Therefore, in this step S3200, the parameters obtained in the learning process described later are set in the CNN 8000 to construct a CNN that segments the liver, kidney, and gallbladder regions. As a result, the CNN 8000 according to this embodiment can output each likelihood map as a classification result of the corresponding class from the single-class output layer present at the end of each of the multiple single-class networks 4200, as in the CNN according to the first embodiment. Furthermore, it becomes possible to output each likelihood map as a classification result of the liver, kidney, and gallbladder from the multi-class output layer present at the end of the multi-class network 8300. Hereinafter, the classification results output from the single-class output layer present at the end of each of the multiple single-class networks 4200 are referred to as the first likelihood map of the liver, the first likelihood map of the kidney, and the first likelihood map of the gallbladder. These three likelihood maps are collectively referred to as the first likelihood map. Moreover, the classification results output from the multi-class output layer 8301 present at the end of the multi-class network 8300 are called the second likelihood map of the liver, the second likelihood map of the kidney, and the second likelihood map of the gallbladder. These three likelihood maps are collectively called the second likelihood map.

[0106] The CNN8000 according to this embodiment can learn even from teaching data (incomplete teaching data) that does not contain one or more correct images among the correct image of the liver, the correct image of the kidney, and the correct image of the gallbladder, by using a learning method described later. In other words, even teaching data that is missing correct answer data corresponding to some classes among the correct answer data corresponding to the classes representing each of a plurality of target objects can be used for learning.

[0107] The multi-class network 8300 in the CNN according to the present embodiment is configured to receive the multiple second features extracted by each of the multiple single-class networks 4200 as input, but is not limited thereto. For example, the first feature extracted by the common network 4100 may be configured to receive the input of the multi-class network 8300. That is, the multi-class network 8300 may be connected to the common network 4100. In this case, the multiple single-class networks 4200 and the multi-class network 8300 can be processed in parallel. Also, the first feature and the multiple second features may be configured to receive the input of the multi-class network 8300. That is, the multi-class network 8300 may be connected to the common network 4100 and the multiple single-class networks 4200. The first feature is extracted in a layer closer to the input layer than the multiple second features, and therefore is considered to hold details of the abdominal CT image that is the input. On the other hand, the multiple second features are extracted in a layer farther from the input layer than the first feature, and therefore are considered to hold higher-order information. Therefore, by inputting a first feature and a plurality of second features, an output that takes into account the characteristics of both can be obtained, and therefore improvement in classification accuracy can be expected. In addition, in the CNN according to this embodiment, one multi-class network that connects to all of the plurality of single-class networks is used, but one or more multi-class networks that connect to some of the single-class networks may also be used. For example, if the liver is the organ that requires the most attention, a liver network 4210 and the other two The CNN 8000 may be configured using two multi-class networks connected to one of the networks 4210 and 4220. More specifically, the first multi-class network is connected to the liver network 4210 and the kidney network 4220, and the second multi-class network is connected to the liver network 4210 and the gallbladder network 4230. In this way, the first multi-class network can learn the relationship between the liver and the kidneys more reliably, and similarly, the second multi-class network can learn the relationship between the liver and the gallbladder more reliably.

[0108] (Step S3250) The processing of this step is the same as step S3250 in the first embodiment, and therefore a description thereof will be omitted.

[0109] (Step S3300) In this step, the classification unit 130 executes inference processing by inputting the input image received from the first data acquisition unit 110 into the CNN received from the first network construction unit 120 and executing forward propagation processing. In this embodiment, a first likelihood map is acquired from a plurality of single-class networks 4200, and a second likelihood map is acquired from the multi-class network 8300. Details of the processing in this step are the same as those in the first embodiment, and therefore will not be described.

[0110] In this embodiment, the identification unit 130 acquires both the first likelihood map and the second likelihood map, but it may acquire only one of them. For example, when acquiring only the second likelihood map, in the multiple single-class networks 4200, forward propagation is not performed to the single-class output layer present at the end of each, but is performed to the multi-class network 8300. Alternatively, in step S3200, when constructing the CNN, the single-class output layer may be removed in advance. In this case, the single-class output layer is used only in the learning process.

[0111] (Step S3400) In this step, the classification unit 130 performs post-processing on the first likelihood map and the second likelihood map to obtain a segmentation result, and outputs the segmentation result to the storage device 70 or the monitor 205.

[0112] In this step, the identification unit 130 integrates likelihood maps relating to the same class in the first likelihood map and the second likelihood map to generate a new likelihood map (third likelihood map) of the class. Specifically, the identification unit 130 averages pixel values ​​of pixels included in the first likelihood map of the liver and pixel values ​​of pixels included in the second likelihood map of the liver to generate a third likelihood map of the liver. Similarly, the identification unit 130 generates a third likelihood map of the kidney and a third likelihood map of the gallbladder. Then, the identification unit 130 outputs the third likelihood map of the liver, the third likelihood map of the kidney, and the third likelihood map of the gallbladder as segmentation results.

[0113] In the above-mentioned method, the first likelihood map and the second likelihood map are average value integrated, but the integration method is not limited to this. For example, when integrating the average values, weights may be assigned to the first likelihood map and the second likelihood map and then average value integration may be performed. In addition, the pixel values ​​of the pixels included in the first likelihood map and the second likelihood map may be integrated by maximum value integration or minimum value integration, or the likelihood maps may be converted into binary images and then integrated by a method such as logical sum integration or logical product integration. For example, when the liver region is prone to be underextracted in both the first likelihood map of the liver and the second likelihood map of the liver, the underextraction of the liver region can be suppressed by performing maximum value integration or logical sum integration. In addition, the first likelihood map of the liver and the second likelihood map of the liver may be integrated by a method such as maximum value integration or logical sum integration. In both cases, if regions other than the liver are likely to be erroneously extracted, minimum value integration or logical product integration can be performed to suppress erroneous extraction of regions other than the liver.

[0114] In the above, the average value integration is an integration method in which the average value of corresponding pixels in the first likelihood map and the second likelihood map is used as the pixel value of the integrated likelihood map. Similarly, the maximum value integration, the minimum value integration, the logical sum integration, and the logical product integration are integration methods in which the maximum value, the minimum value, the logical sum, and the logical product are used instead of the average value as the pixel value after integration.

[0115] In the above, the integration process of the first likelihood map and the second likelihood map is performed as post-processing, but the post-processing described in step S3400 in the first embodiment may be applied. In addition, any post-processing may be applied to the first likelihood map and the second likelihood map.

[0116] This step is not essential and may be omitted. In this case, the classification unit 130 outputs, as the segmentation results, a first likelihood map of the liver, a first likelihood map of the kidneys, a first likelihood map of the gallbladder, and a second likelihood map of the liver, a second likelihood map of the kidneys, and a second likelihood map of the gallbladder.

[0117] According to the above procedure, the image processing device 100 according to this embodiment outputs the segmentation results of the liver, kidneys, and gallbladder from the abdominal CT image.

[0118] [Learning process] (functional configuration, hardware configuration) The functional configuration and hardware configuration of the image processing device 500 according to this embodiment are the same as those of the image processing device 500 according to the first embodiment, and therefore a description thereof will be omitted.

[0119] (Processing flow) A processing procedure of the image processing device 500 in this embodiment will be described with reference to Fig. 6. In the following, the description will be omitted to avoid overlapping parts with the description of the image processing device 500 according to the first embodiment.

[0120] (Step S6100) In this step, the second data acquiring unit 510 acquires a plurality of pieces of teaching data from the storage device 70. Then, the second data acquiring unit 510 transmits the acquired plurality of pieces of teaching data to the learning unit 530.

[0121] The CNN 8000 according to this embodiment includes a multi-class network 8300 having a multi-class output layer 8301. In order to train this multi-class network 8300, training data including all correct images of the liver, kidney, and gallbladder (complete training data) is required. This is because, in order to calculate a correct loss value for each of the multiple provisional classification results (provisional classification result of the liver, provisional classification result of the kidney, and provisional classification result of the gallbladder) output by the multi-class output layer 8301, each correct image is required. Therefore, the second data acquisition unit 510 acquires training data such that it includes at least one complete training data.

[0122] (Step S6200) The processing of this step is the same as step S6200 in the first embodiment, and therefore a description thereof will be omitted.

[0123] (Step S6300) In this step, the learning unit 530 receives the plurality of data from the second data acquisition unit 510. Using the teaching data, the CNN received from the second network constructor 520 is trained by a training method described below. Then, the parameters of the trained CNN are stored in the storage device 70.

[0124] The learning method will be described in detail with reference to Fig. 6(b). Explanation of steps similar to those in the first embodiment will be omitted, and only the differences will be described.

[0125] (Step S6310) The processing of this step is the same as step S6310 in the first embodiment, so a description thereof will be omitted.

[0126] (Step S6320) The processing of this step is the same as step S6320 in the first embodiment, and therefore a description thereof will be omitted.

[0127] (Step S6330) In this step, the learning unit 530 determines the layer for updating parameters based on the attributes of the teaching data selected in step S6320.

[0128] 7(a) is selected in step S6320, the learning unit 530 determines in this step the layers included in all the networks as the layers whose parameters are to be updated. That is, the learning unit 530 determines the layers included in the common network 4100, the multiple single-class networks 4200, and the multi-class network 8300 as the layers whose parameters are to be updated.

[0129] On the other hand, when incomplete teaching data as shown in Figures 7(b) to 7(d) is selected in step S6320, the process described in step S6330 in the first embodiment is executed. That is, the learning unit 530 determines layers included in the common network 4100 and the single-class network corresponding to the class in which the correct image exists as layers for updating parameters.

[0130] In this manner, in this embodiment, when training is performed using complete teaching data, the multi-class network 8300 is trained, and when training is performed using incomplete teaching data, the multi-class network 8300 is not trained.

[0131] (Step S6340) In this step, the learning unit 530 inputs the learning image of the teaching data selected in step S6320 to the CNN and executes a forward propagation process, thereby obtaining a provisional classification result (likelihood map) based on the current parameters.

[0132] When complete teaching data as shown in Fig. 7(a) is selected in step S6320, in this step, forward propagation processing is performed on the common network 4100, the multiple single-class networks 4200, and the multi-class network 8300. As a result, a first tentative identification result of the liver, a first tentative identification result of the kidney, and a first tentative identification result of the gallbladder are obtained from each of the multiple single-class networks 4200. These are collectively referred to as the first tentative identification result. In addition, a second tentative identification result of the liver, a second tentative identification result of the kidney, a second tentative identification result of the gallbladder, and tentative identification results of other regions are obtained from the multi-class network 8300. These are collectively referred to as the second tentative identification result.

[0133] On the other hand, when incomplete teaching data as shown in Fig. 7(b) to Fig. 7(d) is selected in step S6320, the process described in step S6340 in the first embodiment is executed. That is, the learning unit 530 acquires a first provisional classification result corresponding to a class in which a correct image exists or to all classes.

[0134] (Step S6350) In this step, learning unit 530 calculates a loss value based on the correct image of the training data selected in step S6320 and the provisional classification result obtained in step S6340.

[0135] When complete training data as shown in FIG. 7(a) is selected in step S6320, a first tentative classification result and a second tentative classification result are obtained in step S6340. Therefore, in this step, the learning unit 530 calculates a plurality of loss values ​​for each of the first tentative classification results based on each of the correct images of the training data and each of the first tentative classification results. The learning unit 530 also calculates a loss value for the second tentative classification result based on each of the correct images of the training data and each of the second tentative classification results. The loss value for the second tentative classification result is calculated by multiplying the loss value by the number of correct images t multi-class =[t liver ,t kidney ,tgallbladder ,t others ] and the second provisional identification result y multi-class =[y liver ,y kidney ,y gallbladder ,y others ] based on the loss function L multi-class (y multi-class ,t multi-class ) where t others is the correct image for the other regions, and y others represents the tentative classification results for the other regions. The calculation of the multiple loss values ​​for each of the first tentative classification results is as described in step S6350 in the first embodiment.

[0136] On the other hand, if incomplete teaching data as shown in Figures 7(b) to 7(d) is selected in step S6320, the process of step S6350 in the first embodiment is executed. That is, the learning unit 530 calculates a loss value for the first tentative classification result obtained in step S6340.

[0137] It should be noted that the loss function may be a publicly known one. The loss values ​​of the first provisional classification result and the second provisional classification result may be calculated using the same loss function, or different loss functions may be used to calculate the loss values, or weights may be assigned to each loss value. When training a CNN having only a single-class output layer and a CNN having only a multi-class output layer, the loss functions that provide good training results may differ. For example, in a CNN having only a single-class output layer, the best training results may be obtained using MSE, whereas in a CNN having only a multi-class output layer, the best training results may be obtained using Cross Entropy. In such a case, the same phenomenon may occur in the CNN according to this embodiment. Therefore, by calculating the loss value for the first provisional classification result using MSE and the loss value for the second provisional classification result using Cross Entropy, it is expected that the classification accuracy of both will be improved.

[0138] (Step S6360) In this step, learning unit 530 updates the parameters based on the information on the layer for which the parameters are to be updated determined in step S6330 and the loss value for each of the provisional results calculated in step S6350.

[0139] When complete training data as shown in FIG. 7(a) is selected in step S6320, multiple loss values ​​for each of the first tentative classification results and a loss value for the second tentative classification result are obtained in step S6350. Therefore, in this step, the learning unit 530 uses a known method such as backpropagation based on each loss value to calculate the parameters. Here, the loss value for the second provisional classification result is the loss value for the output of the multi-class network 8300. Therefore, the loss value for the second provisional classification result is back-propagated from the multi-class output layer 8301 present at the end of the multi-class network 8300 to calculate the gradient. Similarly, the learning unit 530 back-propagates the multiple loss values ​​for the first provisional classification result from the single-class output layers present at the ends of each of the multiple single-class networks 4200 to calculate the gradient. At this time, since the multi-class network 8300 is connected to each of the multiple single-class networks 4200, the back-propagation process is performed taking into account the value back-propagated from the multi-class network 8300 to calculate the gradient. Similarly, the common network 4100 performs the back-propagation process taking into account the value back-propagated from each of the multiple single-class networks 4200 to calculate the gradient. Then, the learning unit 530 updates the parameters of the layer for updating the parameters determined in step 6330 based on the gradient.

[0140] On the other hand, if incomplete teaching data as shown in FIGS. 7(b) to 7(d) is selected in step S6320, the process described in step S6360 in the first embodiment is executed.

[0141] (Step S6370) The processing of this step is the same as step S6370 in the first embodiment, so a description thereof will be omitted.

[0142] According to the above procedure, the image processing device 500 according to this embodiment generates parameters for the CNN in the image processing device 100.

[0143] (Effects of this embodiment) As described above, the CNN in the image processing device 100 and the image processing device 500 according to the present embodiment can learn even from teaching data in which some correct answer images are missing among the correct answer images of the liver, the kidney, and the gallbladder. In other words, even teaching data in which some correct answer data corresponding to classes representing each of a plurality of target objects is missing can be used for learning.

[0144] <Third embodiment> The invention disclosed in this specification is not limited to segmentation. For example, it can be applied to object recognition and object detection. Object recognition is a task of identifying which class (category) an object of interest depicted in an image belongs to. Object detection is a task of identifying the position (or position and class) of an object of interest depicted in an image.

[0145] FIG. 9 illustrates a CNN 9000 for object recognition. The illustrated CNN 9000 includes a common network 9100, a plurality of single-class networks 9200, and a multi-class network 9300. This CNN 9000 includes a full connection layer that connects all neurons between layers, and is configured to apply an activation function to the output result of the full connection layer. A single-class output layer exists at the end of each of the plurality of single-class networks 9200. The single-class output layer is a full connection layer, and a Sigmoid function is set as the activation function. A multi-class output layer 9301 exists at the end of the multi-class network 9300. The multi-class output layer is a full connection layer, and a Softmax function is set as the activation function. This CNN can use even incomplete teaching data for learning due to the single-class output layer existing at the end of the single-class network 9200, and can obtain the effects of the invention disclosed in this specification.

[0146] Furthermore, the invention disclosed in this specification has no restrictions on the data to be processed. In the above, a CT image taken by an X-ray CT device has been described as an example, but other medical images taken by a magnetic resonance imaging (MRI) device, an ultrasonic imaging device, or the like can also be processed. In addition, images taken by a camera, three-dimensional data acquired by a depth sensor, audio signals, text data, and the like can also be processed. The invention disclosed in this specification can be applied to any other data.

[0147] Other Examples The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions. [Explanation of symbols]

[0148] 4000: Neural Network (CNN) 4100: Common network 4210, 4220, 4230: Single class networks 4211, 4221, 4231: Single class output layer

Claims

1. A method for training a neural network comprising: a common network common to a plurality of classes; and a plurality of single-class networks connected to the common network and having a single-class output layer for individually identifying each of the plurality of classes, the method comprising: an acquisition step of acquiring training data including correct answer data corresponding to one or more classes among the plurality of classes; a learning step of learning the neural network using the teaching data; Equipped with the learning step, when learning using teaching data in which supervised answer data corresponding to one or more classes among the plurality of classes is missing, learns a single-class network corresponding to a class in which supervised answer data is not missing, and does not learn a single-class network corresponding to a class in which supervised answer data is missing; 13. A neural network learning method comprising:

2. The learning step learns the common network regardless of whether or not the teaching data includes missing correct answer data.

2. The neural network training method according to claim 1, wherein the training step is performed based on the training result.

3. The learning step calculates a loss value using a different loss function for each output of the single-class output layer.

3. The neural network learning method according to claim 1 or 2.

4. The neural network further includes a multi-class network having a multi-class output layer for identifying two or more classes among the plurality of classes; the learning step includes: learning the multi-class network when learning using teaching data including correct answer data corresponding to all of the plurality of classes; and not learning the multi-class network when learning using teaching data in which correct answer data corresponding to one or more classes is missing; The neural network according to any one of claims 1 to 3. How to learn q.

5. The learning step calculates a loss value using different loss functions for the output of the single-class output layer and the output of the multi-class output layer.

5. The neural network training method according to claim 4, wherein the neural network training method further comprises:

6. The learning step determines a layer for updating parameters when learning using the teaching data according to which class of correct answer data the teaching data has.

6. The neural network learning method according to claim 1, wherein the training step is performed by performing a step of training the neural network.

7. The learning step includes grouping the teaching data according to which class of correct answer data the teaching data has, and selecting teaching data for each group within one epoch to perform learning.

7. The neural network learning method according to claim 1, wherein the training step is performed based on the training result of the neural network.

8. The learning step selects a group of teaching data having correct answer data of all classes at the end of one epoch and performs learning.

7. The neural network training method according to claim 6, wherein the training step is performed based on the training result.

9. 1. An image processing device that trains a neural network including a common network common to a plurality of classes, and a plurality of single-class networks having a single-class output layer connected to the common network for individually identifying each of the plurality of classes, an acquisition unit that acquires training data including correct answer data corresponding to one or more classes among the plurality of classes; a learning unit that learns the neural network using the teaching data; Equipped with the learning unit, when learning using teaching data in which supervised answer data corresponding to one or more classes among the plurality of classes is missing, learns a single-class network corresponding to a class in which supervised answer data is not missing, and does not learn a single-class network corresponding to a class in which supervised answer data is missing; 13. An image processing device comprising:

10. An acquisition unit that acquires an image; A classification unit that performs classification using a trained neural network; An image processing device comprising: the trained neural network comprises a common network common to a plurality of classes, and a plurality of single-class networks connected to the common network and having a single-class output layer for individually identifying each of the plurality of classes, the trained neural network being trained using teaching data including correct answer data corresponding to one or more of the plurality of classes; the trained neural network is a neural network in which a single-class network corresponding to a class in which no correct answer data is missing is trained when training using teaching data in which correct answer data corresponding to one or more classes among the plurality of classes is missing, 13. An image processing device comprising:

11. The trained neural network is adapted to one or more of the plurality of classes. When training using teaching data in which corresponding correct answer data is missing, a single-class network corresponding to a class in which correct answer data is missing is not trained as a neural network. The image processing device according to claim 10 .

12. An acquisition step of acquiring an image; A classification step of performing classification using a trained neural network; An image processing method comprising: the trained neural network comprises a common network common to a plurality of classes, and a plurality of single-class networks connected to the common network and having a single-class output layer for individually identifying each of the plurality of classes, the trained neural network being trained using teaching data including correct answer data corresponding to one or more of the plurality of classes; the trained neural network is a neural network in which a single-class network corresponding to a class in which no correct answer data is missing is trained when training using teaching data in which correct answer data corresponding to one or more classes among the plurality of classes is missing, 13. An image processing method comprising:

13. The trained neural network is a neural network in which a single-class network corresponding to a class lacking correct answer data is not trained when the trained neural network is trained using teaching data lacking correct answer data corresponding to one or more of the multiple classes.

13. The image processing method according to claim 12.

14. A program for causing a computer to execute each step of the method according to any one of claims 1 to 7, 12 and 13.

Images