Information processing device, information processing method, and program

Abstract

To efficiently and accurately generate correct answer image data indicating an area of an object.SOLUTION: An information processing apparatus 100 of the present invention comprises: an acquisition unit 110 that acquires first information including information on the position and size of an object in learning image data, and second information including information on constraints on an area of the object; and a generation unit 120 that generates a likelihood map indicating the likelihood of the object based on the first information and the second information as correct answer image data for the learning image data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This disclosure relates to an information processing device, an information processing method, and a program for generating ground truth image data. [Background technology]

[0002] The accuracy of machine learning-based segmentation depends on the amount of training data, which consists of training image data and ground truth image data; therefore, it is desirable to prepare a large amount of training data. Since creating ground truth image data is a laborious task, techniques for efficiently creating ground truth image data are important. For example, Non-Patent Document 1 discloses a technique for generating ground truth image data (ground truth likelihood map) that indicates the likelihood of an object being represented from a bounding box, which shows the rectangular outline surrounding the object. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] M. Najibi et al., Towards the Success Rate of One: Real-time Unconstrained Salient Object Detection, arXiv:1708.00079v2,2017. [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] However, the ground truth image data generated by the technology disclosed in Non-Patent Document 1 is created based on the width and height of the bounding box, so in some cases the approximation accuracy to the actual area of ​​the object may be low depending on the ground truth image data.

[0005] The present invention aims to provide an information processing device that can efficiently and accurately generate ground truth image data indicating the region of an object. [Means for solving the problem]

[0006] The information processing apparatus according to the present invention includes an acquisition unit that acquires first information including information on the position and size of an object in training image data and second information including information on constraints on the region of the object, and a generation unit that generates a likelihood map indicating the object's resemblance as ground truth image data for the training image data, based on the first information and the second information. A training data acquisition unit acquires training data consisting of the ground truth image data, which is a likelihood map generated by the generation unit based on the first information and the second information, and the training image data; and a learning unit performs training on the learning model based on the loss value obtained based on the estimation result obtained by processing the training image data with the learning model and the ground truth image data. It is equipped with. Furthermore, another information processing device of the present invention includes an acquisition unit that acquires first information including information on the position and size of an object in training image data, and second information including information on constraints regarding the region of the object, and a generation unit that generates a likelihood map indicating the likelihood of the object based on the first information and the second information as ground truth image data for the training image data, wherein the second information is image data indicating the region of an object adjacent to the object. [Effects of the Invention]

[0007] According to the disclosures herein, ground truth image data indicating the region of an object can be generated efficiently and with high accuracy. [Brief explanation of the drawing]

[0008] [Figure 1] A diagram showing an example of the functional configuration of an information processing device according to the first embodiment. [Figure 2] A diagram showing an example of the hardware configuration of an information processing device according to the first embodiment. [Figure 3] A diagram showing an example of the processing procedure of an information processing device according to the first embodiment. [Figure 4] A diagram showing an example of the processing procedure of an information processing device according to the first embodiment. [Figure 5] A diagram illustrating an image according to the first embodiment. [Figure 6] A diagram showing an example of the processing procedure of an information processing device according to the first embodiment. [Figure 7] A diagram showing an example of the processing procedure of an information processing device according to the second embodiment. [Modes for carrying out the invention]

[0009] Hereinafter, a preferred embodiment of the information processing apparatus disclosed in this specification will be described with reference to the drawings. The same or equivalent components, members, and processes shown in each drawing are denoted by the same reference numerals, and redundant descriptions will be omitted as appropriate. Also, in each drawing, some of the components, members, and processes are omitted as appropriate for display.

[0010] Hereinafter, the present invention will be described by taking, as an example, a liver tumor depicted in abdominal CT image data obtained by an X-ray computed tomography (X-ray CT) apparatus. However, the present invention is applicable not only to liver tumors but also to other lesions (e.g., lung nodules, lymph nodes, bone metastases, etc.) and any other structures. Further, it is applicable to tomographic image data obtained by a magnetic resonance imaging (MRI) apparatus, a positron emission tomography (PET) apparatus, an ultrasonic imaging apparatus, and general image data obtained by a camera. Also, the present invention is applicable to both two-dimensional image data and three-dimensional image data. Furthermore, the present invention is applicable to image data obtained by photographing general objects. Note that the embodiments of the present invention are not limited to the following embodiments.

Example

[0011] <First Embodiment> In this embodiment, the object is a liver tumor, the liver tumor is approximated by an ellipsoid, and a method for generating a likelihood map (hereinafter referred to as the likelihood map of the liver tumor) indicating the likelihood of the region of the liver tumor as the correct answer image data for the learning image data will be described. The information processing apparatus 100 according to this embodiment first obtains, as first information representing the position and size of the liver tumor in the learning image data, the center coordinates of the ellipsoid approximating the liver tumor and the radii with respect to three orthogonal axes. Also, the information processing apparatus 100 obtains, as second information including information on constraints regarding the region of the liver tumor, mask image data (hereinafter referred to as the liver mask image data) indicating the region of the liver including the liver tumor. Then, the information processing apparatus 100 generates a likelihood map of the liver tumor based on the first information and the second information. Here, the likelihood map is a likelihood map that satisfies the constraints regarding the region of the object based on the second information.

[0012] The information processing apparatus 100 generates a likelihood map of a liver tumor based on the central coordinates of the liver tumor, which is the first information, the radii with respect to three orthogonal axes, and the mask image data of the liver, which is the second information including information on constraints regarding the region of the liver tumor.

[0013] Specifically, the information processing apparatus 100 uses, as parameters of an ellipsoid, the central coordinates of the liver tumor, which is the first information, and the radii with respect to three orthogonal axes, and generates an initial likelihood map that spreads in an ellipsoidal shape (such that the likelihood changes concentrically). At this time, since the liver is a region that includes the liver tumor, it is considered that there is no liver tumor outside the region of the liver. Therefore, the information processing apparatus 100 uses the mask image data of the liver, which is the second information, to generate a likelihood map of the liver tumor such that the likelihood of the liver tumor in voxels outside the region of the liver becomes 0. That is, the information processing apparatus 100 corrects the initial likelihood map so as to satisfy the constraint regarding the region of the object that there is no liver tumor outside the region of the liver, and generates a likelihood map indicating the likelihood of the region of the liver tumor.

[0014] Furthermore, the information processing apparatus 100 according to the present embodiment uses learning image data and the likelihood map indicating the likelihood of the region of the liver tumor generated for the learning image data as correct answer image data, and learns a learning model by a method based on machine learning. That is, the information processing apparatus 100 uses abdominal CT image data as learning image data and learns a learning model using teacher data in which the likelihood map of the liver tumor corresponding to the abdominal CT image data is used as correct answer image data. In the present embodiment, among deep learning techniques, 3D U-Net, which is one of Convolutional Neural Networks (CNNs), is used as a learning model. By learning 3D U-Net using the above-described teacher data, it becomes possible to infer a region that looks like a liver tumor.

[0015] The functional configuration of the information processing device 100 according to this embodiment will be described below with reference to Figure 1. As shown in the figure, the information processing device 100 consists of a ground truth image data generation processing unit 101, which includes an acquisition unit 110 and a generation unit 120, and a learning processing unit 102, which includes a training data acquisition unit 130 and a learning unit 140. The information processing device 100 according to this embodiment also includes a storage device 70. Note that the ground truth image data generation processing unit 101 and the learning processing unit 102 may function as an information processing system composed of different devices.

[0016] The storage device 70 is an example of a computer read storage medium, and is a large-capacity storage device such as a hard disk drive (HDD) or solid-state drive (SSD). The storage device 70 holds training image data in which the object is depicted, first information including information on the position and size of the object in the training image data, and second information including information on constraints regarding the region of the object. In this embodiment, the object is described as a liver tumor and the training image data as abdominal CT image data. Furthermore, the first information includes information on the radii of the three axes orthogonal to the central coordinates of the liver tumor, and the second information includes information on mask image data of the liver, which is the area where the liver tumor is located.

[0017] In the first information stored in the memory device 70, the central coordinates of the liver tumor are, for example, coordinate values ​​in the image coordinate system corresponding to the abdominal CT image, and coordinate values ​​for the X, Y, and Z axes (c x ,c y ,c z It is expressed as follows. Furthermore, the radii for the three orthogonal axes that constitute the first piece of information are, for example, the radii of the liver tumor in the X, Y, and Z axes, respectively, with respect to the central coordinates of the liver tumor, and (r x ,r y ,r zIt is expressed as shown above. The values ​​such as the central coordinates of the liver tumor and the radii relative to the three orthogonal axes that constitute the first piece of information are values ​​annotated by, for example, a physician or radiologist. The above-mentioned representation of the central coordinates of the liver tumor and the radii relative to the three orthogonal axes is just one example, and any format that can express the characteristics of each is acceptable. For example, the first piece of information may be the vertex coordinates that define a rectangular prism circumscribing an ellipsoid that approximates the shape of the liver tumor.

[0018] In the second piece of information stored in the storage device 70, the liver mask image data has the same image size as the abdominal CT image data. The liver mask image data is binary image data in which the values ​​of voxels corresponding to the voxels depicting the liver on the abdominal CT image data are represented as 1, and the values ​​of all other voxels are represented as 0. The above representation format of the liver mask image is just one example, and any format capable of representing the liver region is acceptable. For example, the second piece of information may be an image that represents the liver-likeness of each voxel in a multi-level representation. Note that the storage device 70 may be configured as a different device from the information processing device 100 described later. The functions of each part constituting the information processing device 100 will be described in detail below.

[0019] The acquisition unit 110 acquires the training image data and the first and second pieces of information contained in the training image data from the storage device 70, and transmits the first and second pieces of information to the generation unit 120. Specifically, the acquisition unit 110 acquires the first piece of information from the storage device 70, which is the radius of the three axes orthogonal to the central coordinates of the liver tumor, and the second piece of information, which is the mask image data of the liver, and transmits them to the generation unit 120.

[0020] The generation unit 120 receives first and second information from the acquisition unit 110 regarding the training image data. Based on the first information, the generation unit 120 generates an initial likelihood map showing the general area of ​​the object, and then applies corrections to the initial likelihood map based on the second information to generate a likelihood map indicating the object's resemblance. This likelihood map indicating the object's resemblance is the ground truth image data for the training image data. The generation unit 120 stores the generated likelihood map of the object in the storage device 70 as the ground truth image data for the training image data.

[0021] In other words, the generation unit 120 generates a likelihood map indicating the likelihood of a liver tumor region based on the first information, which is the radius relative to the three axes orthogonal to the central coordinates of the liver tumor, and satisfies the constraints based on the second information, which is the mask image data of the liver. The generation unit 120 then stores the likelihood map indicating the likelihood of a liver tumor region in the storage device 70 as the ground truth image data for the training image data. The likelihood map of the liver tumor is a continuous image data with the same image size as the abdominal CT image data, where voxels with a high likelihood of being a liver tumor are represented by values ​​close to 1, and voxels with a low likelihood of being a liver tumor are represented by values ​​close to 0. The above representation format of the likelihood map of the liver tumor is just one example, and any format that can represent the high or low likelihood of being a liver tumor is acceptable.

[0022] The training data acquisition unit 130 receives multiple training image data and ground truth image data, which are likelihood maps of the target object, from the storage device 70, and transmits them to the learning unit 140. Here, the likelihood map of the target object is the ground truth image data generated by the generation unit 120. That is, the training data acquisition unit 130 acquires a training dataset from the storage device 70 consisting of multiple abdominal CT image data as training image data and multiple likelihood maps of liver tumors corresponding to each of the multiple abdominal CT images as ground truth image data, and transmits them to the learning unit 140. Note that the training data acquired here may be processed data that includes training data containing ground truth image data generated by the ground truth image data generation processing unit 101 described above.

[0023] The learning unit 140 receives multiple training image data and ground truth image data, which are likelihood maps of multiple objects corresponding to each of the training image data, from the training data acquisition unit 130. Next, the learning unit 140 initializes the parameters of the learning model. Subsequently, the learning unit 140 uses the multiple training image data and the ground truth image data, which are likelihood maps of multiple objects corresponding to each of the multiple target images, as a training dataset to train the learning model using a machine learning-based method. The parameters of the trained learning model are then stored in the storage device 70. In this embodiment, the learning model is a 3D U-Net, which is a type of CNN. That is, the learning unit 140 trains the 3D U-Net using multiple abdominal CT image data and likelihood maps of multiple liver tumors corresponding to each of the CT image data as a training dataset. The learning model trained by the learning unit 140 may also be used for inference processing. Furthermore, the trained learning model can be applied to inference processing in other information processing devices.

[0024] At least a portion of each part of the information processing device 100 shown in Figure 1 may be implemented as an independent device. Alternatively, each function may be implemented as software. In this embodiment, each part is assumed to be implemented by software.

[0025] Figure 2 shows an example of the hardware configuration of the information processing device 100. The information processing device 100 has the configuration of a known computer (information processing device). The information processing device 100 includes, as its hardware configuration, a CPU 201, main memory 202, magnetic disk 203, display memory 204, monitor 205, mouse 206, and keyboard 207.

[0026] The CPU (Central Processing Unit) 201 primarily controls the operation of each component. The main memory 202 stores control programs executed by the CPU 201 and provides a workspace for program execution by the CPU 201. The magnetic disk 203 stores programs for implementing various application software, including the OS (Operating System), device drivers for peripheral devices, and programs for processing described later. By executing programs stored in the main memory 202, magnetic disk 203, etc., the CPU 201 realizes the functions (software) of the information processing device 100 shown in Figure 1 and the processing shown in the flowchart described later.

[0027] The display memory 204 temporarily stores display data. The monitor 205 is, for example, a CRT monitor or an LCD monitor, and displays images, text, etc., based on the data from the display memory 204. The mouse 206 and keyboard 207 accept pointing input and character input, respectively, from the user. Each of the above components is connected to each other via a common bus 208 so that they can communicate with one another.

[0028] The CPU 201 corresponds to an example of a processor or control unit. In addition to the CPU 201, the information processing device 100 may have at least one of a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array). Alternatively, the CPU 201 may be replaced with at least one of a GPU or an FPGA. The main memory 202 and magnetic disk 203 correspond to an example of memory or storage device.

[0029] Next, the processing procedure of the information processing device 100 in this embodiment will be explained using Figures 3 and 6. First, the processing flow of the ground truth image data generation processing unit 101 in the information processing device 100 will be explained with reference to Figure 3.

[0030] (Step S3100) In step S3100, the acquisition unit 110 acquires from the storage device 70 the first piece of information in the learning image data, which is the radius of the three axes orthogonal to the central coordinates of the liver tumor, and the second piece of information, which is the mask image data of the liver, and transmits the acquired information to the generation unit 120.

[0031] (Step S3200) In step S3200, the generation unit 120 receives, as first information, the radius of the three axes orthogonal to the central coordinates of the liver tumor, and as second information, image data, the mask image data of the liver. Next, the generation unit 120 generates mask image data showing the approximate region of the liver tumor based on the first information, the radius of the three axes orthogonal to the central coordinates of the liver tumor. Subsequently, the generation unit 120 generates an initial likelihood map showing the approximate region of the liver tumor from the mask image data showing the approximate region of the liver tumor. Then, the initial likelihood map showing the approximate region of the liver tumor is corrected to satisfy the constraints based on the second information, the mask image data of the liver, to generate a likelihood map showing the likelihood of it being the liver tumor region.

[0032] The process in step S3200 will be explained in detail with reference to Figures 4 and 5.

[0033] Here, Figure 5(a) is the abdominal CT image data 510, which is the training image data depicting the liver tumor 511, and Figure 5(b) is the second piece of information, the liver mask image data 520. The liver mask image data 520 contains the liver region 521, and as an example, the voxel value of the liver region 521 is represented as 1 (light color), and the voxel values ​​of other regions are represented as 0 (dark color).

[0034] (Step S3210) In step S3210, the generation unit 120 generates mask image data indicating a rough region of the liver tumor based on the first information. In the present embodiment, the initial likelihood map of the liver tumor is generated by approximating the liver tumor as an ellipsoid. First, the generation unit 120 uses, as parameters of the ellipsoid, the radii (not shown) with respect to three axes orthogonal to the center coordinates of the liver tumor, which is the first information, to generate ellipsoidal mask image data 530 indicating a rough region of the liver tumor. Specifically, voxels corresponding to coordinates (x, y, z) that satisfy the following conditions are regarded as voxels belonging to the ellipsoid indicating the rough region of the liver tumor.

[0035] [Number]

[0036] Here, (c x , c y , c z ) are the center coordinates of the liver tumor in the image coordinate system, and (r x , r y , r z ) represent the radii in the respective axial directions with respect to the center coordinates of the liver tumor. The ellipsoidal mask image data 530 holds an ellipsoidal region 531 indicating a rough region of the liver tumor. As an example, assume that the voxel value of the ellipsoidal region 531 is 1 (light color) and the voxel values of the others are 0 (dark color). That is, in the ellipsoidal mask image data 530, the value of voxels that satisfy the condition shown in Equation 1 is 1, and the values of the other voxels are 0.

[0037] (Step S3220) In step S3220, the generation unit 120 generates an initial likelihood map showing the approximate region of the liver tumor based on the mask image data showing the approximate region of the liver tumor generated in step S3210. More specifically, for the ellipsoid region 531 generated in step S3210, the generation unit 120 generates a likelihood map such that the likelihood of it being a liver tumor decreases as it moves away from the vicinity of the center coordinates of the liver tumor. In this embodiment, the generation unit 120 generates ellipsoid mask image data 530 showing the approximate region of the liver tumor, and generates an initial likelihood map 540 showing the approximate region of the liver tumor by applying a Gaussian filter to the mask image data to blur the ellipsoid region 531. Note that the blurring method is not limited to a Gaussian filter, but any known method of smoothing is acceptable. That is, the generation unit 120 generates mask image data showing the approximate region of the object based on the first information, and generates an initial likelihood map showing the approximate region of the object by smoothing the mask image data.

[0038] The initial likelihood map 540, which shows the approximate area of ​​the liver tumor, is, for example, represented by voxels with a high likelihood of liver tumors having values ​​close to 1 (light color) and voxels with a low likelihood having values ​​close to 0 (dark color). Alternatively, the generation unit 120 may directly generate the initial likelihood map showing the approximate area of ​​the liver tumor from the first piece of information, which is the radius relative to the three axes orthogonal to the central coordinates of the liver tumor, without using a mask image showing the approximate area of ​​the liver tumor.

[0039] (Step 3230) In step S3230, the generation unit 120 corrects the initial likelihood map showing the general area of ​​the liver tumor generated in step S3220 to satisfy the constraints based on the second information, thereby generating a likelihood map indicating the object's resemblance. In this embodiment, the second information is mask image data of the liver. Therefore, the generation unit 120 uses the second information, which is constraint information regarding the object's area, that "the liver is the area that encompasses the liver tumor, therefore, there is no liver tumor outside the liver area," to correct the initial likelihood map of the liver tumor. In other words, the image data in the second information is image data that encompasses the area of ​​the object that contains the object. That is, the initial likelihood map 540 showing the general area of ​​the liver tumor is corrected so that the likelihood of the liver tumor in the voxels outside the liver area becomes 0, thereby generating a likelihood map 550 of the liver tumor.

[0040] Specifically, the generation unit 120 generates a likelihood map indicating the likelihood of a liver tumor region using the following mathematical formula.

[0041]

number

[0042] Here,

[0043]

number

[0044] These are the voxel value (likelihood) at voxel i in the initial likelihood map 540, which shows the general area of ​​the liver tumor, and the voxel value (likelihood) in the likelihood map 550 of the liver tumor, respectively. iis the voxel value of the liver mask image 520, which is the second piece of information. As shown in Figure 5(e), the likelihood map 550, which indicates the likelihood of a liver tumor region generated by the generation unit 120 using the method described above, is a likelihood map that appears to be cut out by the liver region 521, and is a likelihood map with a shape and extent that is closer to that of the liver tumor 511 in the abdominal CT image 510. In the above example, the initial likelihood map of the liver tumor was corrected so that the likelihood of the liver tumor in voxels outside the liver region becomes 0, but other methods may be used to correct the likelihood of the liver tumor in voxels outside the liver region to be small. For example, the likelihood of the liver tumor in voxels outside the liver region may be corrected to a constant close to 0, or the likelihood p, which indicates the general area of ​​the liver tumor, may be used. i Alternatively, this can be corrected by multiplying it by a constant less than 1. After completing the above process, step S3200 is terminated. Next, we return to Figure 3 and explain the subsequent processes.

[0045] (Step S3300) In step S3300, the generation unit 120 outputs a likelihood map indicating the likelihood of a liver tumor area and stores it in the storage device 70.

[0046] (Step S3400) In step S3400, the information processing device 100 determines whether or not there is training image data (abdominal CT image data) to be processed. If there is, it returns to step S3100 and executes the ground truth image generation process for the remaining training image data. On the other hand, if there is no training image data to be processed, the ground truth image generation process by the ground truth image data generation processing unit 101 is terminated. Following the above procedure, the information processing device 100 according to this embodiment generates a likelihood map of a liver tumor as ground truth image data corresponding to the training image data using the ground truth image data generation processing unit 101. That is, the information processing device 100 includes an acquisition unit 110 that acquires first information including information on the position and size of an object in the training image data and second information including information on constraints regarding the region of the object, and a generation unit 120 that generates a likelihood map indicating the likelihood of an object based on the first information and the second information as ground truth image data for the training image data. By having this configuration, the information processing device 100 can generate ground truth image data efficiently and with high accuracy.

[0047] Next, the processing procedure of the learning processing unit 102 in the information processing device 100 will be explained with reference to Figure 6. Note that the learning process of the learning model by the learning processing unit 102 may be implemented by a different information processing device.

[0048] Here, the learning processing unit 102 uses the likelihood map of liver tumors generated by the ground truth image data generation processing unit 101 as the ground truth image data for the training image data, and trains the learning model using a machine learning-based method. For convenience, the likelihood map of liver tumors generated by the ground truth image data generation processing unit 101 will be referred to as the ground truth liver tumor likelihood map from now on.

[0049] (Step S6100) In step S6100, the training data acquisition unit 130 acquires multiple abdominal CT image data, which are training image data, and likelihood maps of liver tumors, which are ground truth image data corresponding to each of the multiple abdominal CT images, as training data sets, and transmits them to the learning unit 140. In other words, the training data acquisition unit 130 considers the abdominal CT image 510 and the ground truth likelihood map of liver tumors 550 as a set of training data, acquires multiple training data sets, and transmits them to the learning unit 140.

[0050] (Step S6200) In step S6200, the learning unit 140 initializes the parameters of the 3D U-Net, which is a learning model acquired from the storage device 70, etc. More specifically, the learning unit 140 initializes the weights of the convolutional layer kernel using a known method. In this embodiment, the convolutional layer is initialized using the method for determining initial values ​​based on a normal distribution proposed by He (Kaiming He, et al., “Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification,” ICCV, 2015) et al. The initialization of the convolutional layer is not limited to this, and any known method such as constant values ​​or random values ​​may be used. Furthermore, if the 3D U-Net includes layers with parameters such as batch normalization layers and dropout layers, the parameters of each layer are initialized using a known method.

[0051] (Step S6300~S6700) In steps S6300 to S6700, the learning unit 140 receives the training dataset from the training data acquisition unit 130 and trains the 3D U-Net, which is the learning model, using the training dataset. The following describes how to train the 3D U-Net.

[0052] In step S6300, the learning unit 140 selects multiple training data (batches) from the training dataset to update the parameters of the 3D U-Net. Each training data consists of a pair of training image data, which is an abdominal CT image, and a ground truth image data corresponding to the abdominal CT image, which is a likelihood map of a ground truth liver tumor. In this step, one or more training data sets are selected.

[0053] In step S6400, the learning unit 140 inputs abdominal CT image data included in the multiple training data selected in step S6300 into the 3D U-Net and performs forward propagation processing. By performing forward propagation processing, the learning unit 140 obtains multiple estimated likelihood maps of liver tumors for each of the multiple abdominal CT image data input into the learning model.

[0054] In step S6500, the learning unit 140 calculates a loss value based on the likelihood maps of multiple estimated liver tumors obtained in step S6300 and the corresponding likelihood maps of the correct liver tumors. The loss function used is, for example, Mean Squared Error (MSE). The loss function is not limited to MSE; other known methods such as Mean Absolute Error (MAE) or Huber may also be used.

[0055] In step S6600, the learning unit 140 calculates the gradient using backpropagation based on the loss value calculated in step S6500 and updates the parameters of the 3D U-Net. As an example of an optimization method, the Stochastic Gradient Descent (SGD) method is used. The optimization method is not limited to the SGD method; other known methods such as the Adam method or AdaGrad method may also be used.

[0056] In step S6700, the learning unit 140 determines whether the learning termination condition has been met and decides on the next step. One example of a learning termination condition is reaching a predetermined maximum number of epochs. That is, if the predetermined maximum number of epochs has been reached, the process proceeds to step S6800; otherwise, it returns to step S6300. Other conditions may be used for the learning termination condition. For example, the condition may be that the difference between the loss value of the previous epoch and the loss value of the current epoch satisfies a specified condition.

[0057] The above describes a typical training method for 3D U-Net, a learning model for inferring likelihood maps of liver tumors.

[0058] (Step S6800) In step S6800, the learning unit 140 outputs the parameters of the learning model and stores them in the storage device 70.

[0059] In accordance with the above procedure, the information processing device 100 according to this embodiment uses the likelihood map of liver tumors generated by the ground truth image data generation processing device 101 as ground truth image data in the learning processing unit 102 to learn a learning model for inferring the likelihood map of liver tumors using a machine learning-based method.

[0060] As described above, the information processing device 100 according to this embodiment generates a likelihood map (ground truth image data) of an object based on first information regarding the position and size of the object, so as to satisfy constraints based on second information regarding the region of the object. This makes it possible to efficiently and accurately generate ground truth image data that can be used for training a machine learning-based learning model. Furthermore, the learning processing unit 102 can train the learning model using the ground truth image data. Moreover, by performing inference processing that applies the learning model, the position of the object region in the image data to be inferred can be determined.

[0061] (modified version) In the example above, we described a method for generating a likelihood map of an object using the radius of the three axes orthogonal to the object's center coordinates as the first piece of information. However, any other information that includes at least position and size may be used. For example, it could be a VOI surrounding the object or mask image data of an ellipse for a certain cross-section. If the first piece of information is a VOI surrounding the object, for example, the initial likelihood map of the object can be generated in the same way as in the example above by setting the centroid of the VOI as the center coordinate of the object's ellipsoid and the length of each side of the VOI as the diameter of the ellipsoid. Alternatively, if the first piece of information is mask image data of an ellipse for a certain cross-section, a solid of revolution (ellipsoid) of the ellipse can be generated, and then mask image data of the ellipsoid can be generated. In this way, a likelihood map of the object can be generated in the same way as in the example above.

[0062] In the example described above, the generation unit 120 generates ellipsoidal mask image data based on the first information, which is the center coordinates of the object and the radii of the three axes orthogonal to them, and generates an initial likelihood map showing the general area of ​​the object based on this, such that it has an ellipsoidal spread (the likelihood changes concentrically). However, it is not limited to this. For example, the generation unit 120 may use a probability function such as a multivariate Gaussian function instead of a mask image showing the general area of ​​the liver tumor, or it may generate an initial likelihood map from the first information using the Euclidean distance from a certain reference position. These are examples of how the generation unit 120 generates an initial likelihood map in which the likelihood changes concentrically. When using a multivariate Gaussian function, for example, the generation unit 120 uses the center coordinates of the object as the mean value of the multivariate Gaussian function and generates an initial likelihood map showing the general area of ​​the object by determining the degree of spread in each axis direction based on the radius of the object. A more specific method will be illustrated in the second embodiment. When using Euclidean distance from a reference position, for example, the generation unit 120 calculates the Euclidean distance for each voxel using the contour or center coordinates of the ellipsoid region as the reference position, and normalizes the distance values ​​so that the Euclidean distance of each voxel falls between 0 and 1. At this time, the generation unit 120 normalizes the distance values ​​so that the likelihood of an object near the contour of the object is 0.5, and uses the normalized distance values ​​as a likelihood map that indicates the likelihood of an object. The above description by the generation unit 120 is an example of generating an initial likelihood map that represents a rough region by generating mask image data representing a rough distance based on the first information and then performing a distance transformation on the mask image data.

[0063] In the example described above, the generation unit 120 approximates the object as an ellipsoid to generate a likelihood map of the object. However, any known method may be used to realize the object's likelihood map, as long as the shape can be approximated from at least the first information representing its position and size. For example, if the object is close to a cylinder (such as a spine), the generation unit 120 can generate a likelihood map of the object in the same manner as in the example above by generating mask image data of the cylinder using information about the radius and height of the cylinder as the first information. Similarly, if the object is close to a rectangular prism, the generation unit 120 uses the lengths of each side of the rectangular prism as the first information. In addition, the generation unit 120 may approximate the region of the object using a parametric shape model. In this case, the generation unit 120 is given dependent parameters for representing the parametric shape model as the first information. A parametric shape model is, for example, a function of a closed surface or a statistical shape model. That is, the first information includes the parameters of the parametric shape model that represent the general shape of the object. The generation unit 120 then generates a likelihood map that indicates the object's resemblance to the object, based on the first information and the second information, which include a parametric shape model and information on the object's position and size.

[0064] In the example described above, the generation unit 120 used mask image data of the region of an object encompassing the target object as the second piece of information, but is not limited to this. For example, it may also be mask image data showing the region of an object adjacent to (or circumscribing) the target object. If the target object is a liver tumor, since the liver tumor is close to, for example, the kidneys and ribs, the generation unit 120 can achieve the effects of the present invention by setting the likelihood of voxels within the region represented by these mask image data to 0. Alternatively, the generation unit 120 may use mask image data obtained by thresholding the possible voxel values ​​in order to determine the region on the target's training image data in which the target object is likely to exist. In this way, any mask image data generated outside the information processing device according to the present invention may be used as long as it can serve as a constraint on the region of the target object. Furthermore, the generation unit 120 may use a continuous value map that represents the prior probability of the target object. In this case, the generation unit 120 corrects the initial likelihood map of the target object, which is generated based on the first piece of information, by multiplying the value of the prior probability map for each voxel.

[0065] In the example above, the information processing device 100 used a deep learning-based method such as 3D U-Net as its learning model, but it is not limited to this. For example, other machine learning techniques such as Support Vector Machine (SVM) or classification trees may be used as learning models. In this case, the learning method should be an appropriate one depending on the learning model.

[0066] <Second Embodiment> In the first embodiment, the generation unit 120 generates an initial likelihood map showing a rough area of ​​the object based on the first information. Then, the generation unit 120 corrects the initial likelihood map to satisfy constraints based on the second information, thereby generating a likelihood map that indicates the object's resemblance. This step-by-step method was described. In this embodiment, the generation unit 120 determines a sub-region for generating a likelihood map that indicates the object's resemblance to the training image data, based on the second information. The generation unit 120 then calculates the likelihood of the object for the sub-region based on the first information, thereby generating a likelihood map that indicates the object's resemblance. That is, the method by which the information processing device 100 determines the likelihood in each voxel of the likelihood map based on the first information so as to satisfy constraints based on the second information will be described. Here, the sub-region is a region that shows an area in which there is a high probability that the object exists.

[0067] In this embodiment, the information processing device 100 uses the lower and upper limits of the CT values ​​that the target area of ​​the liver tumor can take as second information, and generates a likelihood map that indicates the likelihood of a liver tumor area while imposing constraints such that the likelihood of a liver tumor becomes 0 for voxels outside the range represented by the lower and upper limits.

[0068] More specifically, the information processing device 100 first uses the lower and upper limits of the voxel values ​​that the liver tumor region can take to apply a threshold-based discrimination process to the abdominal CT image data, thereby generating mask image data that indicates the region where the liver tumor is most likely to be located.

[0069] The information processing device 100 then sets the likelihood of a liver tumor for each voxel in the subregion where a liver tumor is likely to exist, based on the first piece of information, which is the radius of the three axes orthogonal to the central coordinates of the liver tumor, and generates a liver tumor likelihood map for the remaining voxels such that the likelihood of a liver tumor becomes 0.

[0070] In this embodiment, the generation unit 120 calculates the likelihood of a liver tumor in each voxel using a multivariate Gaussian function. That is, the information processing device 100 according to this embodiment uses the radii of the three axes orthogonal to the central coordinates of the liver tumor, which are the first piece of information, as parameters of the multivariate Gaussian function to generate a likelihood map that indicates the likelihood of a liver tumor region.

[0071] The configuration of the information processing device according to this embodiment is the same as that of the information processing device 100 according to the first embodiment. Hereinafter, with reference to Figure 1, the functional configuration of the information processing device 100 according to this embodiment will be described, with appropriate omissions of parts that overlap with the information processing device 100 according to the first embodiment.

[0072] Similar to the first embodiment, the storage device 70 stores abdominal CT image data, which is learning image data depicting the object, and the radii of the three axes orthogonal to the central coordinates of the liver tumor as first information. It also stores the lower limit of the CT values ​​that the liver tumor region can take as second information, which represents constraints on the object's region. lower and upper limit t upper It holds.

[0073] The acquisition unit 110 acquires the abdominal CT image data, which is the target learning image data, the radius of the three axes orthogonal to the central coordinates of the liver tumor (the first piece of information), and the lower and upper limits of the CT values ​​that the liver tumor region can take (the second piece of information), and transmits them to the generation unit 120.

[0074] First, the generation unit 120 receives from the acquisition unit 110 the target training image data, which is abdominal CT image data, the first piece of information, which is the radius of the three axes orthogonal to the central coordinates of the liver tumor, and the second piece of information, which is the lower and upper limits of the CT values ​​that the liver tumor region can take. Next, the generation unit 120 uses the second piece of information, the lower and upper limits of the CT values ​​that the liver tumor region can take, to threshold the abdominal CT image data and generate mask image data of the subregion where the liver tumor is likely to exist. Subsequently, the generation unit 120 generates a likelihood map indicating the likelihood of a liver tumor region while imposing a constraint that the likelihood of a liver tumor in voxels outside the subregion where the liver tumor is likely to exist becomes 0. Finally, the generation unit 120 stores the generated likelihood map indicating the likelihood of a liver tumor region in the storage device 70 as ground truth image data for the training image data.

[0075] The training data acquisition unit 130 is the same as in the first embodiment, so its description will be omitted.

[0076] The learning unit 140 is the same as in the first embodiment, so its description will be omitted.

[0077] The hardware configuration of the information processing device 100 according to this embodiment is the same as that of the first embodiment, so a description will be omitted.

[0078] Next, the processing procedure of the ground truth image data generation processing unit 101 in the information processing device 100 in this embodiment will be described using Figure 3. In the following description, parts that overlap with the description of the information processing device 100 according to the first embodiment will be omitted.

[0079] (Step S3100) In step S3100, the acquisition unit 110 acquires the abdominal CT image data, which is the target learning image data, the radius of the three axes orthogonal to the central coordinates of the liver tumor (first information), and the lower and upper limits of the CT values ​​that the liver tumor region can take (second information). Then, it transmits these to the generation unit 120.

[0080] (Step S3200) In step S3200, the generation unit 120 receives abdominal CT image data, which is the target learning image data, the radius of the three axes orthogonal to the central coordinates of the liver tumor as first information, and the lower and upper limits of the CT values ​​that the liver tumor region can take as second information. Next, based on the lower and upper limits of the CT values ​​that the liver tumor region can take, which is the second information, a mask image of the partial region where the liver tumor is likely to be located is generated. Then, based on the mask image data of the partial region where the liver tumor is likely to be located and the radius of the three axes orthogonal to the central coordinates of the liver tumor as first information, a likelihood map indicating the likelihood of it being a liver tumor region is generated. Step S3200 in this embodiment will be described in detail below with reference to Figure 7.

[0081] (Step S7210) In step S7210, the generation unit 120 uses the lower and upper limits of the CT values ​​that the liver tumor region can take as threshold values, which are the second piece of information, and processes the abdominal CT image data using threshold processing to generate mask image data of the region where the liver tumor is likely to be located. Specifically, the voxel value M of the mask image data of the region where the liver tumor is likely to be located is calculated using the following formula. i To decide.

[0082]

number

[0083] Here, I i is the voxel value of abdominal CT image data, t lower ,t upper These are the lower and upper limits of the CT values ​​that a liver tumor region can take, respectively. For example, in the mask image data of a region where a liver tumor is likely to be present, the values ​​of voxels included in the region where a liver tumor is likely to be present are represented as 1, and the values ​​of other voxels are represented as 0.

[0084] (Step S7220) In step S7220, the generation unit 120 generates a likelihood map indicating the likelihood of a liver tumor region based on mask image data of a subregion where a liver tumor is likely to exist and the radii of the triaxial axis orthogonal to the central coordinates of the liver tumor, which is the first information. In this embodiment, the first information further includes parameters of a probability distribution function, and the generation unit 120 generates a likelihood map indicating the likelihood of an object based on the probability distribution function and the first and second information, which include information on the position and size of the object. Here, for a subregion where a liver tumor is likely to exist, a multivariate Gaussian function, which is the probability distribution function, is used to generate a likelihood map of the liver tumor. Specifically, the value of each voxel in the likelihood map of the liver tumor (likelihood of the liver tumor) is set by the following formula.

[0085]

number

[0086] Here,

[0087]

number

[0088] This is a vector representing the coordinates of voxel i in the image coordinate system,

[0089]

number

[0090] is a vector representing the central coordinates of the liver tumor in the image coordinate system. Furthermore, Σ is the covariance matrix, which functions as a parameter determining the degree of spread of the likelihood distribution in the likelihood map of the liver tumor. In particular, the diagonal elements of the covariance matrix represent the variances in the X, Y, and Z axes.

[0091]

number

[0092] These determine the degree of spread in each axial direction. Therefore, the generating unit 120 determines the degree of spread in each axial direction based on the radii of the three orthogonal axes of the liver tumor.

[0093]

number

[0094] Determine the distribution in the X-axis direction.

[0095]

number

[0096] The variances in the Y and Z axes are determined using the same formula. In this way, a likelihood map of liver tumors can be generated such that the likelihood of liver tumors in voxels near the contour of the liver tumor is 0.5.

[0097] After completing the above process, step S3200 is terminated. Next, we return to Figure 3 and explain the subsequent processes.

[0098] (Step S3300) This step is identical to step S3300 in the first embodiment, so its explanation is omitted.

[0099] (Step S3400) This step is identical to step S3400 in the first embodiment, so its explanation is omitted.

[0100] Following the above procedure, the information processing device 100 according to this embodiment generates a likelihood map of liver tumors using the ground truth image data generation processing unit 101. Then, following the same processing procedure as the learning processing unit 102 in the first embodiment, the information processing device 100 uses the likelihood map of liver tumors as a ground truth image and learns a learning model for inferring the likelihood map of liver tumors using a machine learning-based method.

[0101] As described above, the information processing device 100 according to this embodiment generates a likelihood map as ground truth image data for training image data, based on first information regarding the position and size of the object, so as to satisfy constraints based on second information regarding the region of the object. This makes it possible to efficiently and accurately generate ground truth image data that can be used for training a machine learning-based learning model. [Explanation of symbols]

[0102] 70 Storage device 100 Information Processing Devices 101 Processing unit for generating correct image data 102 Learning Processing Unit 110 Acquisition Department 120 Generation part 130 Training Data Acquisition Unit 140 Learning Department

Claims

1. An acquisition unit that acquires first information including information on the position and size of an object in training image data, and second information including information on constraints regarding the region of the object, A generation unit generates a likelihood map indicating the object's resemblance to the training image data, based on the first information and the second information, as the correct image data for the training image data. A training data acquisition unit acquires training data consisting of the ground truth image data, which is a likelihood map generated by the generation unit based on the first information and the second information, and the training image data. A learning unit that performs learning on the learning model based on the loss value obtained based on the estimation result obtained by processing the training image data by the learning model and the ground truth image data, An information processing device characterized by having the following features.

2. The generation unit generates an initial likelihood map relating to the region of the object based on the first information, The information processing apparatus according to claim 1, characterized in that it generates a likelihood map indicating the object's resemblance by correcting the initial likelihood map based on the second information.

3. The information processing apparatus according to claim 2, characterized in that the generation unit generates mask image data indicating the region of the object based on the first information, and generates the initial likelihood map by smoothing the mask image data.

4. The information processing apparatus according to claim 2, characterized in that the generation unit generates mask image data indicating the region of the object based on the first information, and generates the initial likelihood map by performing a distance transformation on the mask image data.

5. The information processing apparatus according to claim 1, characterized in that the generation unit determines a subregion for generating a likelihood map indicating the object's resemblance to the learning image data based on the second information, and generates a likelihood map indicating the object's resemblance to the object by calculating the likelihood of the object for the subregion based on the first information.

6. The information processing apparatus according to claim 5, characterized in that the aforementioned partial region is a region in which the object is highly likely to exist.

7. The information processing apparatus according to any one of claims 1 to 6, wherein the first information further includes parameters of a probability distribution function, and the generation unit generates a likelihood map indicating the likelihood of the object based on the probability distribution function, the first information which includes information on the position and size of the object, and the second information.

8. The information processing apparatus according to any one of claims 1 to 6, wherein the first information further includes parameters of a parametric shape model that represents the general shape of the object, and the generation unit generates a likelihood map indicating the object's resemblance based on the first information, which includes the parametric shape model and information on the position and size of the object, and the second information.

9. The information processing apparatus according to any one of claims 1 to 8, characterized in that the second information is image data.

10. The information processing apparatus according to claim 9, characterized in that the second information is image data indicating the region of an object that encloses the object.

11. An acquisition unit that acquires first information including information on the position and size of an object in training image data, and second information including information on constraints relating to the region of the object, A generation unit generates a likelihood map indicating the object's resemblance to the training image data, based on the first information and the second information, as the correct image data for the training image data. It has, The information processing apparatus is characterized in that the second information is image data indicating the region of an object adjacent to the object.

12. The information processing apparatus according to claim 5, characterized in that the second information is information relating to a threshold for determining an area in which the object is likely to exist.

13. The information processing apparatus according to claim 1, further comprising a control unit that performs inference processing using the learning model learned by the learning unit.

14. An acquisition step of acquiring training image data, first information including information on the position and size of an object in the training image data, and second information including information on constraints regarding the region of the object, A generation step of generating a likelihood map that indicates the object's resemblance to the object, based on the first information and the second information, as the ground truth image data for the training image data, A training data acquisition step for acquiring training data consisting of the ground truth image data, which is a likelihood map generated by the generation step based on the first information and the second information, and the training image data. A learning step in which the learning model is trained based on the loss value obtained based on the estimation result obtained by processing the training image data by the learning model and the ground truth image data, An information processing method characterized by having the following features.

15. An acquisition step of acquiring first information including information on the position and size of an object in training image data, and second information including information on constraints relating to the region of the object, A generation step of generating a likelihood map that indicates the object's resemblance to the object, based on the first information and the second information, as the ground truth image data for the training image data, It has, The information processing method is characterized in that the second information is image data indicating the region of an object adjacent to the object.

16. The information processing method according to claim 14, further comprising an inference step of performing inference processing using the learning model generated by the learning step.

17. A program for a computer to execute the information processing method described in any one of claims 14 to 16.

Images