Information processing apparatus and information processing method

Abstract

An information processing apparatus according to an embodiment includes processing circuitry. The processing circuitry is configured to generate at least a part of a first inference model that, upon receipt of an input of at least one two-dimensional image included in a three-dimensional image, outputs a first inference result corresponding to the two-dimensional image and a second inference model that, upon receipt of an input of two or more of the first inference results, outputs a second inference result corresponding to the three-dimensional image.

Description

[0001] INFORMATION PROCESSING APPARATUS AND INFORMATION PROCESSING

[0002] METHOD

[0003] FIELD

[0004] Embodiments described herein relate generally to an information processing apparatus and an information processing method .

[0005] BACKGROUND

[0006] A technique is known by which a three-dimensional image is input so that an inferring process is performed on the three-dimensional image . This technique is used for deriving a diagnosis name or an image observation by performing the inferring process on a three-dimensional image .

[0007] However, when this technique is applied to three- dimensional images , performance of the inferring process may be degraded if there are a large number of parameters .

[0008] BRIEF DESCRI PTION OF THE DRAWINGS

[0009] FIG . 1 is a block diagram illustrating an exemplary configuration of an information processing system according to a first embodiment ;

[0010] FIG . 2 is a drawing illustrating exemplary data configurations of training data according to the first embodiment ;

[0011] FIG . 3 is a drawing illustrating an example of a f irst training process according to the first embodiment ;

[0012] FIG . 4 is a drawing illustrating an example of a second training process according to the first embodiment ;

[0013] FIG . 5 is a drawing illustrating an example of an inference according to the first embodiment ;

[0014] FIG . 6 is a flowchart illustrating an example of an inferring process performed by an information processing apparatus according to the first embodiment ;

[0015] FIG . 7 is a block diagram illustrating an exemplary configuration of an information processing system according to a second embodiment ;

[0016] FIG . 8 is a flowchart illustrating an example of an inferring process performed by an information processing apparatus according to the second embodiment ;

[0017] FIG . 9 is a block diagram illustrating an exemplary configuration of an information processing system according to a third embodiment ; and

[0018] FIG . 10 is a flowchart illustrating an example of an inferring process performed by an information processing apparatus according to the third embodiment .

[0019] DETAILED DESCRIPTION

[0020] An information processing apparatus including process ing circuitry configured to generate at least a part of a first inference model that , upon receipt of an input of at least one two-dimensional image included in a three-dimensional image , outputs a first inference result corresponding to the two-dimensional image and a second inference model that , upon receipt of an input of two or more of the first inference results , outputs a second inference result corresponding to the three-dimensional image .

[0021] Exemplary embodiments of an information processing apparatus and an information processing method will be explained below, with reference to the accompanying drawings . In the following embodiments , some of the elements referred to by using the same reference characters are assumed to perform the same or similar operations , and duplicate explanations thereof will be omitted, as appropriate .

[0022] FIG . 1 is a block diagram illustrating an exemplary configuration of an information processing system 1 according to a first embodiment . The information processing system 1 includes an image diagnosis apparatus 10 , a first image storage apparatus 20 , a second image storage apparatus 30 , and an information processing apparatus 40 . Further , systems and apparatuses included in the information processing system 1 are connected so as to be able to communicate with one another via a network . The information processing system 1 illustrated in FIG . 1 is merely an example . It is possible to arbitrarily change the quantity of the systems and the apparatuses . Further, one or more other apparatuses that are not illustrated in FIG . 1 may be connected to the network .

[0023] The image diagnosis apparatus 10 is an apparatus configured to image an examined subj ect (hereinafter , "patient" ) . For example , the image diagnosis apparatus 10 may be an apparatus such as an X-ray Computed Tomography (CT) apparatus , a Magnetic Resonance Imaging (MRI ) apparatus , an ultrasound diagnosis apparatus , or a Positron Emission Tomography ( PET) apparatus . However , the image diagnosis apparatus 10 does not necessarily need to be one of the apparatuses described above and may be another type of apparatus .

[0024] The first image storage apparatus 20 and the second image storage apparatus 30 are configured to store therein three-dimensional images such as a three-dimensional medical image 50 . For example , the first image storage apparatus 20 and the second image storage apparatus 30 may each be a server apparatus of a Picture Archiving and Communication System ( PACS ) . However , the first image storage apparatus 20 and the second image storage apparatus 30 do not necessarily need to be in the PACS and may be in other systems or may be other server apparatuses . Further, the first image storage apparatus 20 and the second image storage apparatus 30 may be realized by a single apparatus .

[0025] The first image storage apparatus 20 is configured to store therein training data . The training data is data used for training an inference model . The inference model is a model configured to perform an inferring process on a three- dimensional image . For example , the training data includes the three-dimensional medical image 50 ( see FIG . 2 ) , first correct answer data 520 ( see FIG . 2 ) , and second correct answer data 530 ( see FIG . 2 ) .

[0026] The second image storage apparatus 30 is configured to store therein data subj ect to the inferring process (hereinafter, " inference target data" ) . The inference target data is a three-dimensional medical image 50 subj ect to the inferring process performed by the inference model .

[0027] The information processing apparatus 40 is configured to receive an input of the three-dimensional medical image 50 . Further, the information processing apparatus 40 is configured to generate the inference model including a first inference model 600 and a second inference model 700 .

[0028] More specifically, by using all or a part of two- dimensional medical images 510 included in the three- dimensional medical image 50 , the information processing apparatus 40 is configured to generate the first inference model 600 that outputs first inference results 610 ( see FIG . 4 ) corresponding to the two-dimensional medical images 510 . Further , the information processing apparatus 40 obtains the first inference result 610 with respect to each of the two- dimensional medical images 510 , by having an inferring process performed by the first inference model 600 that has been trained, on each of all or a part of the two-dimensional medical images 510 included in the three-dimensional medical image 50 .

[0029] Further, by inputting the plurality of first inference results 610 to the second inference model 700 , the information processing apparatus 40 is configured to generate the second inference model 700 that outputs a second inference result 710 corresponding to the three-dimensional medical image 50 . Further , the information processing apparatus 40 is configured to perform the inferring proces s on the three-dimensional medical image 50 by using the inference model including the first inference model 600 and the second inference model 700 that were generated . The information processing apparatus 40 will be explained in detail .

[0030] The information processing apparatus 40 includes a network (NW) interface 41 , an input interface 42 , a display 43 , a memory 44 , and processing circuitry 45 .

[0031] The NW interface 41 is connected to the processing circuitry 45 and is configured to control trans fer of various types of data and communication performed with apparatuses connected via the network . For example , the NW interface 41 is reali zed by using a network card, a network adaptor, a Network Interface Controller (NIC) , or the like .

[0032] The input interface 42 is connected to the processing circuitry 45 and is configured to convert input operations received from an operator ( a medical provider) into electric signals , and to output the electric signals to the processing circuitry 45 . More specifically, the input interface 42 is configured to convert the input operations received from the operator into the electric signals and to output the electric signals to the processing circuitry 45 . For example , the input interface 42 is realized by using a trackball , a switch button, a mouse , a keyboard, a touchpad on which input operations can be performed by touching an operation surface thereof , a touch screen in which a display screen and a touchpad are integrally formed, contactless input circuitry using an optical sensor, audio input circuitry, and / or the like . Further, in the present disclosure , the input interface 42 does not necessarily need to include physical operation component parts such as the mouse , the keyboard, and / or the like . Possible examples of the input interface 42 include , for instance , electric signal processing circuitry configured to receive an electric signal corresponding to an input operation from an external input mechanism provided separately from the apparatus and to output the electric signal to controlling circuitry .

[0033] The display 43 is connected to the processing circuitry 45 and is configured to display various types of information and various types of image data output from the processing circuitry 45 . For example , the display 43 is realized by using a liquid crystal display, a Cathode Ray Tube (CRT) display, an organic Electroluminescence (EL) display, a plasma display, a touch panel , or the like .

[0034] The memory 44 is connected to the processing circuitry 45 and is configured to store therein various types of data . Also , the memory 44 is configured to store therein various types of programs that are used for realizing various types of functions as being read and executed by the processing circuitry 45 . For example, the memory 44 is reali zed by using a semiconductor memory element such as a Random Acces s Memory (RAM) or a flash memory, or a hard disk, an optical disk, or the like .

[0035] The processing circuitry 45 is configured to control operations of the entirety of the information processing apparatus 40 . For example , the processing circuitry 45 includes a training data obtaining function 451 , a first training function 452 , a second training function 453 , an inference data obtaining function 454 , and an inferring function 455 . In an embodiment , the processing functions performed by the constituent elements , namely, the training data obtaining function 451 , the first training function 452 , the second training function 453 , the inference data obtaining function 454 , and the inferring function 455 , are stored in the memory 44 in the form of computer-executable programs . The processing circuitry 45 is a processor configured to realize the functions corresponding to the programs by reading and executing the programs from the memory 44 . In other words , the processing circuitry 45 that has read the programs has the functions illustrated within the processing circuitry 45 in FIG . 1 .

[0036] Further, although the example was explained with reference to FIG . 1 in which the single processor realizes the training data obtaining function 451 , the first training function 452 , the second training function 453 , the inference data obtaining function 454 , and the inferring function 455 , it is also acceptable to structure the processing circuitry 45 by combining together a plurality of independent processors , so that the functions are realized as a result of the processors executing the programs . Further, although the example was explained with reference to FIG . 1 in which the single memory such as the memory 44 has stored therein the programs corresponding to the processing functions , it is also acceptable to provide a plurality of memories in a distributed manner, so that the processing circuitry 45 is configured to read the corresponding programs from the individual memories .

[0037] The term "processor" used in the above explanations denotes , for example , a Central Processing Unit (CPU) , a Graphical Processing Unit (GPU) , or circuitry such as an Application Specific Integrated Circuit (ASIC) or a programmable logic device (e . g . , a Simple Programmable Logic Device ( SPLD) , a Complex Programmable Logic Device (CPLD) , or a Field Programmable Gate Array ( FPGA) ) . The one or more processors are configured to reali ze the functions by reading and executing the programs saved in the memory 44 . In this situation, instead of having the programs saved in the memory 44 , it is also acceptable to directly incorporate the programs into the circuitry of the one or more processors . In that situation, the one or more processors realize the functions by reading and executing the programs incorporated in the circuitry thereof .

[0038] The training data obtaining function 451 is configured to obtain at least one piece of training data from the first image storage apparatus 20 . In other words , from the first image storage apparatus 20 , the training data obtaining function 451 is configured to obtain at least one set including a three-dimensional medical image 50 , first correct answer data 520 assigned to at least one two-dimensional medical image 510 included in the three-dimensional medical image 50 , and second correct answer data 530 corresponding to the three-dimensional medical image 50 ,

[0039] FIG . 2 is a drawing illustrating exemplary data configurations of the training data according to the first embodiment . The three-dimensional medical image 50 is a medical image that is three-dimensional . For example , the three-dimensional medical image 50 is a medical image that is three-dimensional and was taken by the image diagnosis apparatus 10 . In the present embodiment , an example will be explained in which the three-dimensional medical image 50 is a chest X-ray CT image . However, the three-dimensional medical image 50 may be a CT image of a site other than the chest , but does not necessarily need to be a CT image and may be an MRI image or a medical image of other types . Further , the three-dimensional medical image 50 includes the one or more two-dimensional medical images 510 . The two-dimensional medical images 510 are two-dimensional images such as slice images .

[0040] The first correct answer data 520 is a degree of certainty that an abnormal shadow is present in a corresponding one of the two-dimensional medical images 510 included in the three-dimensional medical image 50 . For example , the degree of certainty is assigned by a medical doctor or the like . For example, a numerical value presented by the first correct answer data 520 may be 1 . 0 .

[0041] Further, the first correct answer data 520 has all or a part of the two-dimensional medical images 510 included in the three-dimensional medical image 50 , In other words , the first correct answer data 520 does not necessarily need to have all the two-dimensional medical images 510 .

[0042] In the present embodiment , an example is explained in which a piece of first correct answer data 520 is assigned to a single two-dimensional medical image 510 ; however, it is acceptable to assign a piece of first correct answer data 520 to two or more of the two-dimensional medical images 510 . Further , it is also acceptable to assign a piece of first correct answer data 520 to two or more consecutive two- dimensional medical images 510 .

[0043] In an example , it is also acceptable to assign a piece of first correct answer data 520 to three consecutive two- dimensional medical images 510 . In that situation, the first training function 452 (explained later) is configured to train the first inference model 600 configured to infer a correct answer from three consecutive two-dimensional medical images 510 .

[0044] The second correct answer data 530 is a degree of certainty that a speci fic lung disease is present in the three-dimensional medical image 50 . The second correct answer data 530 is assigned to all the three-dimensional medical images 50 used in a training process . For example , the degree of certainty is assigned by a user such as a medical doctor . For example , a numerical value presented by the second correct answer data 530 may be 1 . 0 .

[0045] The first training function 452 is configured to generate at least a part of the first inference model 600 that , upon receipt of an input of at least one of the two- dimensional medical images 510 included in the three- dimensional medical image 50 , outputs the first inference result 610 corresponding to the two-dimensional medical image 510 . More specifically, the first training function 452 is configured to train the first inference model 600 , by using the first correct answer data 520 and at least one of the two-dimensional medical images 510 which is included in the three-dimensional medical image 50 and to which the first correct answer data 520 is assigned .

[0046] FIG . 3 is a drawing illustrating an example of a f irst training process according to the first embodiment . As illustrated in FIG . 3 , the first training function 452 is configured to acquire , from within the training data , set information 540 including sets each made up of a two- dimensional medical image 510 and first correct answer data 520 assigned to the two-dimensional medical image 510 . In other words , the first training function 452 does not acquire certain two-dimensional medical images 510 to which no first correct answer data 520 is assigned .

[0047] The first training function 452 is configured to generate the first inference model 600 by using the set information 540 including the sets each made up of a two- dimensional medical image 510 and corresponding first correct answer data 520 . Accordingly, the first training function 452 is configured to generate the first inference model 600 that , upon receipt of an input of a two-dimensional medical image 510 , for example , infers a likelihood of an abnormal shadow related to a lung disease being present .

[0048] As explained above , the first training function 452 is able to use , in the training process , the set information 540 having a high degree of certainty assigned by the medical doctor or the like with regard to the presence of the abnormal shadow . It is therefore possible to enhance reliability of the training process of the first inference model 600 .

[0049] For example , the first inference model 600 may use a Convolution Neural Network (CNN) ; however , the first inference model 600 does not necessarily need to use a convolution neural network and may use other publicly-known inference models such as a Support Vector Machine ( SVM) that uses pixel values as input vectors , a random forest scheme , or a vision transformer . In the present example , the process of training (generating) the inference model denotes a process of determining a parameter such as a weight of the adopted inference model .

[0050] The second training function 453 is configured to generate at least a part of the second inference model 700 that , upon receipt of an input of two or more of the first inference results 610 output from the first inference model 600 , outputs the second inference result 710 corresponding to the three-dimensional medical image 50 , In other words , after the first inference model 600 was generated, the second training function 453 is configured to generate the second inference model 700 by using the first inference results 610 from the first inference model 600 and the second correct answer data 530 . More specifically, the second training function 453 is configured to train the second inference model 700 , by using set information 620 including the likelihood values being the first inference results 610 inferred by the first inference model 600 and the second correct answer data 530 .

[0051] FIG . 4 is a drawing illustrating an example of a second training process according to the first embodiment . As illustrated in FIG . 4 , a first inferring function 456 is configured to input all or a part of the two-dimensional medical images 510 included in the three-dimensional medical image 50 to the trained first inference model 600 . The first inference model 600 is configured to output the likelihood values being the first inference results 610 respectively corresponding to the two-dimensional medical images 510 included in the three-dimensional medical image 50 . The second training function 453 is configured to input , to the second inference model 700 , the set information 620 including the sets made up of the likelihood values being the first inference results 610 and the second correct answer data 530 assigned to the three-dimensional medical image 50 . Accordingly, the second training function 453 is configured to generate the second inference model 700 that, upon receipt of the input of the likelihood values being the first inference results 610 corresponding to the two-dimensional medical images 510 in the three-dimensional medical image 50 , infers a likelihood of having the specific lung disease . For example , the second inference model 700 may use a SVM; however, the second inference model 700 does not necessarily need to use the SVM and may use , similarly to the first inference model 600 , other publicly-known inference models such as a CNN, a random forest scheme , or a vision transformer . Further, inputs to an SVM usually require that the quantities of elements ( the numbers of dimensions ) be uniform . Although this procedure is omitted from FIG . 4 for the sake of keeping the explanations simple , it is also acceptable to perform a process (e . g . , an interpolation, a truncation, or filling in gaps ) of uniformizing the numbers of dimensions on the likelihood values , before inputting the likelihood values serving as the first inference results 610 to the second inference model 700 .

[0052] Further, although the first training function 452 and the second training function 453 are two separate functions in the present embodiment , it is also acceptable to integrate the two functions into one .

[0053] Further, while the generating process of the first training function 452 involves inputting the three- dimensional medical image 50 to the first inference model 600 , the generating process of the second training function 453 does not involve inputting the three-dimensional medical image 50 to the second inference model 700 .

[0054] The inference data obtaining function 454 is configured to obtain the three-dimensional medical image 50 serving as the inference target data, from the second image storage apparatus 30 .

[0055] The inferring function 455 is configured to perform the inferring process on the three-dimensional medical image 50 , by using the first inference model 600 and the second inference model 700 . Further, the inferring function 455 is configured to perform the inferring process on a three- dimensional image , by inputting the three-dimensional medical image 50 to the first inference model 600 , without inputting the three-dimensional medical image 50 to the second inference model 700 . For example , the inferring function 455 is configured to infer a likelihood of having a lung disease . Alternatively, the inferring function 455 may perform the inferring process on the three-dimensional medical image 50 , by using a single inference model including the first inference model 600 and the second inference model 700 . In other words , the inferring function 455 may perform the inferring process on the three-dimensional image by using the inference model including the first inference model 600 and the second inference model 700 .

[0056] The inferring function 455 includes the first inferring function 456 and a second inferring function 457 . The first inferring function 456 is configured to perform a first inferring process by using the first inference model 600 . The second inferring function 457 is configured to perform a second inferring process by using the second inference model 700 .

[0057] More specif ically, the first inferring function 456 is configured to perform the first inferring process by inputting all or a part of the two-dimensional medical images 510 in the three-dimensional medical image 50 to the first inference model 600 . Accordingly, the first inferring function 456 obtains the first inference results 610 being the likelihood values respectively corresponding to the two- dimensional medical images 510 .

[0058] The second inferring function 457 is configured to perform the second inferring process by inputting, to the second inference model 700 , the first inference results 610 being the likelihood values respectively corresponding to the two-dimensional medical images 510 . Accordingly, the second inferring function 457 obtains the second inference result 710 being the likelihood of having the specific lung disease .

[0059] FIG . 5 is a drawing illustrating an example of an inference according to the first embodiment . The first inferring function 456 is configured to input, to the first inference model 600 , all or a part of the two-dimensional medical images 510 in the three-dimensional medical image 50 serving as the inference target data . Accordingly, the first inferring function 456 obtains the first inference results 610 being the likelihood values respectively corresponding to the two-dimensional medical images 510 .

[0060] Further, the second inferring function 457 is configured to input the first inference results 610 respectively corresponding to the two-dimensional medical images 510 to the second inference model 700 . Accordingly, the second inferring function 457 is configured to obtain the second inference result 710 from the second inference model 700 . For example , the second inferring function 457 is configured to obtain the second inference result 710 being the likelihood of having the specific lung disease .

[0061] Usually, at the time of performing an inferring process on the three-dimensional medical image 50 , the first correct answer data 520 is not assigned to the two-dimensional medical images 510 included in the three-dimensional medical image 50 . However, by applying the first inference model 600 , the first inferring function 456 is able to assign the likelihood values to the two-dimensional medical images 510 , respectively, and to further use the likelihood values as the input to the second inference model 700 . Consequently, the second inferring function 457 is able to perform appropriate inferring processes , regardless of whether the first correct answer data 520 is present or absent .

[0062] Next , an inferring process performed by the information processing apparatus 40 will be explained .

[0063] FIG . 6 is a flowchart illustrating an example of the inferring process performed by the information processing apparatus 40 according to the first embodiment .

[0064] From the first image storage apparatus 20 , the training data obtaining function 451 obtains the training data including the three-dimensional medical image 50 , the first correct answer data 520 , and the second correct answer data 530 ( step S I ) .

[0065] The first training function 452 trains the first inference model 600 by using the first correct answer data 520 and the two-dimensional medical images 510 corresponding to the first correct answer data 520 included in the training data obtained at step SI ( step S2 ) .

[0066] The first inferring function 456 obtains the first inference results 610 respectively corresponding to the two- dimensional medical images 510 , by inputting all or a part of the two-dimensional medical images 510 in the three- dimensional medical image 50 included in the training data, to the first inference model 600 trained at step S2 ( step S3 ) .

[0067] The second training function 453 trains the second inference model 700 by using the likelihood values being the first inference results 610 obtained at step S3 and the second correct answer data 530 ( step S4 ) .

[0068] The inference data obtaining function 454 obtains the three-dimensional medical image 50 serving as the inference target data from the second image storage apparatus 30 ( step S5 ) .

[0069] The inferring function 455 infers the likelihood of having the specific lung disease , by inputting the three- dimensional medical image 50 obtained at step S5 to the first inference model 600 and inputting the likelihood values being the first inference results 610 from the first inference model 600 to the second inference model 700 ( step S6 ) .

[0070] As explained above , the information processing apparatus 40 according to the first embodiment is configured to generate the first inference model 600 that, upon receipt of the input of the two-dimensional medical images 510 included in the three-dimensional medical image 50 , outputs the first inference result 610 with respect to each of the two- dimensional medical images 510 . Further , the information processing apparatus 40 is configured to generate the second inference model 700 that , upon receipt of the input of the first inference results 610 , outputs the second inference result 710 corresponding to the three-dimensional medical image 50 .

[0071] With this configuration , because the first inference model 600 uses the information having a high degree of certainty, it is possible to enhance reliability of the training . Further, even when the second inference model 700 has no information about certainty of each of the two- dimensional medical images 510 , because the highly reliable information is assigned to each of the two-dimensional medical images 510 , it is possible to enhance reliability of the training .

[0072] With the configurations described above , the second inference model 700 has a reduced amount of parameters used in the training, as compared to the situation where the training is performed with sets made up of the three- dimensional medical image 50 and correct answer data .

[0073] For instance , let us discuss an example of a three convolution layers in which the first layer has 64 filters of which each side corresponds to three pixels , and the number of filters is doubled in the next filter and again in the following layer , while the filter size is maintained . Further, let us assume this example uses a simple convolution neural network as an inference model where , after a smoothing process is performed in a smoothing layer, a likelihood is calculated by a fully-connected layer having 128 nodes and an output layer having one node . When performing an inferring process on a three-dimensional image on the basis of inference results from two-dimensional images included in the three-dimensional image , the inference model is configured to receive an input of the outputs of likelihood values of the two-dimensional images inferred via the aforementioned inference model and to further output the final likelihood via a fully-connected layer having 32 nodes and an output layer having one node .

[0074] When the three-dimensional medical image to be input is assumed to be a one-channel image of 64 by 64 by 64 , the number of parameters in the situation using the three- dimensional medical image and the correct answer data is approximately 19 million . When an inferring process is performed on the three-dimensional image on the basis of the inference results from the two-dimensional images included in the three-dimensional image , the number of parameters of the inference model at the former stage ( the two-dimensional images ) is approximately 2 . 83 million, whereas the number of parameters of the inference model at the latter stage (the three-dimensional image ) is 2113 , which total approximately 2 . 83 million . This figure is approximately 15% of the figure in the situation using only the three-dimensional medical image and the correct answer data . This explanations was based on the inference models having simple structures , but if more complicated inference models are used, the number of parameters can further be reduced . By reducing the number of parameters in this manner, the information processing apparatus 40 is able to enhance convergence of the training and to thus enhance reliability of the training as compared to the method described above .

[0075] By using the first inference model 600 and the second inference model 700 trained in this manner, it is possible to perform the appropriate inferring process , even when there is no information about each of the two-dimensional medical images 510 included in the three-dimensional medical image 50 . Consequently, the information processing apparatus 40 is able to enhance performance of the inferring process on the three- dimensional medical image 50 . First Modification Example

[0076] The example was explained in which the first correct answer data 520 and the second correct answer data 530 are the numerical values assigned by the user such as the medical doctor to each of the two-dimensional medical images 510 included in the three-dimensional medical image 50 .

[0077] However, the first correct answer data 520 and the second correct answer data 530 may be numerical values calculated through image processing .

[0078] For example , when having obtained the three-dimensional medical image 50 as the training data, the training data obtaining function 451 may extract feature values from the two-dimensional medical images 510 included in the three- dimensional medical image 50 . The training data obtaining function 451 may be configured to extract the feature values on the basis of pixel values of the two-dimensional medical images 510 or the like .

[0079] Further, the training data obtaining function 451 may be configured to calculate a numerical value indicating reliability, on the basis of a distribution of the feature values , such as a variance value , a center-of-gravity value , the most frequent value , or the like of the feature values . Further, the training data obtaining function 451 may be configured to calculate a degree of certainty that an abnormal shadow’ is present on the basis of the feature values . Further, the training data obtaining function 451 may be configured to use the calculated degree of certainty as the first correct answer data 520 . Preferably, the training data obtaining function 451 may be configured to use the degree of certainty as the first correct answer data 520 when the degree of certainty exceeds a threshold value and to not adopt the degree of certainty as the correct answer data when the degree of certainty is equal to or lower than the threshold value .

[0080] With this configuration , the training data obtaining function 451 is able to reduce burdens that may be imposed by a label assigning process , while maintaining reliability of labels assigned to the two-dimensional medical images 510 .

[0081] 8 Similarly, when there is no second correct answer data 530 corresponding to the three-dimensional medical image 50 , the training data obtaining function 451 may be configured to perform a similar process on the three-dimensional medical image 50 and to use a degree of certainty as the second correct answer data 530 when the degree of certainty is high .

[0082] Further, the example was explained in which the first correct answer data 520 and the second correct answer data 530 are the numerical values indicating the degrees of certainty; however, the first correct answer data 520 and the second correct answer data 530 may be information ( category data) indicating categories such as presence / absence , A / B / C, or the like . In another example, the first correct answer data 520 may be numerical values indicating degrees of certainty or the like , while the second correct answer data 530 may be category data indicating classes or the like . Conversely, the first correct answer data 520 may be category data indicating classes or the like , while the second correct answer data 530 may be numerical values indicating degrees of certainty or the like .

[0083] In yet another example , the category data may be numerical values . For example , when the category data indicates degrees of positivity in having a disease , such as normal / suspected of having the disease / having the disease , the category data may have values such as 0 . 0 / 0 . 5 / 1 . 0 assigned, in accordance with the degrees of positivity . In yet another example , when the categories are classified as having the disease and something other than having the disease , the category data may have 1 . 0 assigned for having the disease and have 0 . 0 assigned for something other than having the disease . Further, when the first correct answer data 520 and the second correct answer data 530 have mutually the same categories , the numerical values may be assigned in mutually-dif ferent manners between the first correct answer data 520 and the second correct answer data 530 . As for the assigning, it is desirable to perform appropriate assigning in accordance with settings of problems .

[0084] For example , let us discuss a situation in which shadows rendered in individual two-dimensional images have small differences , and it is required to provide a rough inference (e . g . , benign / malignant ) as a final result . In that situation , it is desirable to assign detailed numerical values to the first correct answer data 520 and to assign numerical values that are less detailed than the first correct answer data 520 to the second correct answer data 530 . In the example using the three values described above , in the first correct answer data 520 , 0 . 0 / 0 . 5 / 1 . 0 are assigned to the three values of normal / suspected of having the disease / having the disease . Further , in the second correct answer data 530 , 0 . 0 is assigned to " something other than having the disease" which collectively corresponds to being normal and being suspected of having the disease , while 1 . 0 is assigned to having the disease . With this configuration , the first inference model 600 is able to perform the inferring process on the detailed differences in each of the two-dimensional images , whereas the second inference model 700 is able to perform a rough inferring process on the entire three-dimensional image on the basis of the individual di f ferences .

[0085] Second Modification Example

[0086] In the above example , the first training function 452 is configured to train the first inference model 600 . The second training function 453 is configured to train the second inference model 700 , after the first inference model 600 has been generated .

[0087] However, the second training function 453 may be configured to train the second inference model 700 before the generation of the first inference model 600 is completed .

[0088] For example, let us discuss an example in which one of the first inference model 600 and the second inference model 700 is repeatedly trained up to a prescribed maximum number of times . The first training function 452 may be configured to train the first inference model 600 up to a number of times that is set to be smaller than the maximum number of times .

[0089] By using the first inference results 610 from the first inference model 600 , the second training function 453 may be configured to train the second inference model 700 a certain number of times that is smaller than the maximum number of times . After that , the first training function 452 and the second training function 453 may be configured to alternately repeat the generation of the first inference model 600 and the generation of the second inference model 700 that uses the first inference results 610 from the first inference model 600 .

[0090] In other words , the processing circuitry 45 may be configured to perform step S2 a certain number of times being set and to perform steps S3 and S4 a certain number of times being set . The processing circuitry 45 may be configured to repeatedly perform this process . Further , the number of times the first inference model 600 is trained may be the same as or dif ferent from the number of times the second inference model 700 is trained .

[0091] As explained above , because the first training function 452 and the second training function 453 are configured to alternately perform the training on the first inference model 600 a certain number of times and the training on the second inference model 700 a certain number of times , it is possible to enhance relevance between the first inference model 600 and the second inference model 700 . Third Modification Example

[0092] At the time of training the second inference model 700 , the second training function 453 may be configured to input the first correct answer data 520 to the second inference model 700 . More specifically, with respect to certain two- dimensional medical images 510 to which the first correct answer data 520 is assigned in the three-dimensional medical image 50 serving as the training data , the second training function 453 may be configured to input the first correct answer data 520 to the second inference model 700 .

[0093] In contrast , with respect to the other two-dimensional medical images 510 to which the first correct answer data 520 is not assigned, the second training function 453 may be configured to obtain the first inference results 610 obtained by entering inputs to the first inference model 600 . Further, the second training function 453 may be configured to generate the second inference model 700 by using the obtained first inference results 610 , a plurality of pieces of first correct answer data 520 , and the second correct answer data 530 .

[0094] As explained above , with respect to the two-dimensional medical images 510 to which the first correct answer data 520 is assigned, the second training function 453 may be configured to replace the first inference results 610 with the first correct answer data 520 . With this configuration, when the first inference model 600 and the second inference model 700 are generated, in particular, so as to be relevant to each other, as described in Second Modi fication Example , the second training function 453 is able to efficiently proceed with the training process even at an initial stage when capabilities of the first inference model 600 have not developed yet . Fourth Modification Example

[0095] The example was explained above in which the second training function 453 is configured to train the second inference model 700 , by using the first inference results 610 of the three-dimensional medical image 50 used for training the first inference model 600 .

[0096] However , the second training function 453 may be configured to train the second inference model 700 by using first inference results 610 from a three-dimensional medical image 50 different from the three-dimensional medical image 50 used for training the first inference model 600 ,

[0097] As explained above , by using the three-dimensional medical image 50 different from the three-dimensional medical image 50 used for training the first inference model 600 , the second training function 453 is able to enhance independency between the first inference model 600 and the second inference model 700 .

[0098] Fifth Modification Example

[0099] The example was explained in which the processing circuitry 45 of the information processing apparatus 40 includes the training data obtaining function 451 , the first training function 452 , the second training function 453 , the inference data obtaining function 454 , and the inferring function 455 .

[0100] However, one or more of those functions may be included in one or more other apparatuses . For example , the first training function 452 and the second training function 453 may be configured to transmit the first inference model 600 and the second inference model 700 that have been generated to another apparatus . Further, the inference data obtaining function 454 and the inferring function 455 may be included in the other apparatus .

[0101] In other words , the inferring process using the first inference model 600 and the second inference model 700 may be performed by the other apparatus . Further, as for the first inference model 600 and the second inference model 700 , the first inference model 600 and the second inference model 700 may be obtained after being generated by the other apparatus . Furthermore , the inference data obtaining function 454 and the inferring function 455 may be configured to perform the inferring process by using the first inference model 600 and the second inference model 700 having been obtained . Alternatively, the processing circuitry 45 may be configured to obtain one of the first inference model 600 and the second inference model 700 from the other apparatus . By using the one or more inference models that have been trained in this manner , the information processing apparatus 40 is able to shorten time periods for the training .

[0102] Second Embodiment

[0103] FIG . 7 is a block diagram illustrating an exemplary configuration of an information processing system la according to a second embodiment . As for an information processing apparatus 40a according to the second embodiment , the functional configuration, the hardware configuration, and the processing procedures are the same as those in the first embodiment . In the following sections , di fferences from the first embodiment will be explained .

[0104] The information processing apparatus 40a according to the second embodiment is configured to input , to an inference model , the three-dimensional medical image 50 set with a three-dimensional Volume Of Interest (VOI ) .

[0105] For example , when the three-dimensional medical image 50 set with a VOI enclosing a nodule has been input , the information processing apparatus 40a is configured to perform an inferring process related to an image observation on the nodule . Further, the information processing apparatus 40a may perform an inferring process not only on nodules , but also about other image observations .

[0106] From the first image storage apparatus 20 , a training data obtaining function 451a is configured to obtain the three-dimensional medical image 50 , the first correct answer data 520 , and the second correct answer data 530 , as training data . The three-dimensional medical image 50 is a medical image that is three-dimensional and is set with the VOI . In the first correct answer data 520 , numerical values corresponding to an image observation indicated by the second correct answer data 530 are assigned to two-dimensional medical images 510 inside the VOI . The second correct answer data 530 is information indicating the image observation provided by a medical provider such as a medical doctor .

[0107] For example , when the second correct answer data 530 indicates border irregularities , the first correct answer data 520 indicates an irregularity degree which is a degree of border irregularities on cross-sections of the nodule in the two-dimensional medical images 510 with respect to the first correct answer data 520 . Further, the first correct answer data 520 may be assigned only to certain two- dimensional medical images 510 on which a medical provider such as a medical doctor is able to perform the assigning process with certainty . Further, the two-dimensional medical images 510 are cross-sectional images on one of the three types of orthogonal cross-sections ( i . e . , axial images , coronal images , or sagittal images ) . Alternatively, the two- dimensional medical images 510 may be cross-sectional images at an arbitrary angle other than the three types of orthogonal cross-sections .

[0108] A first training function 452a is configured to generate the first inference model 600 by using the three-dimensional medical image 50 set with the VOI and the first correct answer data 520 . More speci fically, the first training function 452a is configured to generate a first inference model 600 that , upon receipt of an input of at least one two- dimensional medical image 510 included in the VOI set in the three-dimensional medical image 50 , outputs a first inference result 610 corresponding to the input two-dimensional medical image 510 . For example , the first training function 452a is configured to generate the first inference model 600 that outputs a numerical value corresponding to the image observation .

[0109] A second training function 453a is configured to generate a second inference model 700 by using the numerical value corresponding to the image observation and serving as the first inference result 610 as well as the second correct answer data 530 indicating the image observation . The second training function 453a is configured to generate the second inference model 700 that , upon receipt of an input of two or more of the first inference results 610 , outputs a second inference result 710 corresponding to the VOI set in the three-dimensional medical image 50 . With this configuration, the second training function 453a is configured to generate the second inference model 700 that , upon receipt of an input of a numerical value corresponding to an image observation and serving as the first inference result 610 , outputs the second inference result 710 indicating an image observation related to the nodule in the VOI .

[0110] From the second image storage apparatus 30 , an inference data obtaining function 454a is configured to obtain a three- dimensional medical image 50 serving as inference target data . When having obtained the three-dimensional medical image 50 , the inference data obtaining function 454a is configured to extract one or more nodules by using a publicly-known extraction technique . After that , on the basis of a result of the extraction, the inference data obtaining function 454a is configured to set the VOI enclosing the one or more extracted nodules . In an example , the inference data obtaining function 454a may set a plurality of VOIs .

[0111] An inferring function 455a includes a first inferring function 456a and a second inferring function 457a .

[0112] The first inferring function 456a is configured to input the three-dimensional medical image 50 set with the VOI and serving as the inference target data, to the first inference model 600 . Accordingly, the first inferring function 456a is configured to obtain the numerical value corresponding to the image observation .

[0113] The second inferring function 457a is configured to input the numerical value corresponding to the image observation to the second inference model 700 . Accordingly, the second inferring function 457a is configured to obtain a second inference result 710 indicating an image observation related to the nodule inside the VOI .

[0114] Next , an inferring process performed by the information processing apparatus 40a will be explained .

[0115] FIG . 8 is flowchart illustrating an example of the inferring process performed by the information processing apparatus 40a according to the second embodiment .

[0116] From the first image storage apparatus 20 , the training data obtaining function 451a obtains training data including the three-dimensional medical image 50 set with at least one VOI , the first correct answer data 520 , and the second correct answer data 530 ( step Si l ) .

[0117] At steps S12 through S15 , the processing circuitry 45a performs the same processes as those at steps S2 through S5 in FIG . 6 .

[0118] The inference data obtaining function 454a extracts the nodule from the three-dimensional medical image 50 serving as the inference target data ( step S16 ) .

[0119] The inference data obtaining function 454a sets at least one VOI enclosing the extracted nodule in the three- dimensional medical image 50 ( step S17 ) . When no nodule is extracted, the inference data obtaining function 454a sets no VOI .

[0120] The inferring function 455a j udges whether or not the three-dimensional medical image 50 is set with at least one VOI ( step S18 ) . When no VOI has been set ( step S18 : No ) , the information processing apparatus 40a ends the inferring process .

[0121] When VOI has been set ( step S 18 : Yes ) , the inferring function 455a infers an image observation related to the nodule , by inputting the three-dimensional medical image 50 set with the VOI to the first inference model 600 and inputting the first inference results 610 from the first inference model 600 to the second inference model 700 ( step S19 ) .

[0122] The inferring function 455a j udges whether or not there is at least one VOI on which the inferring process has not yet been performed ( step S20 ) . When the inferring process has not yet been performed on all the VOIs ( step S20 : No) , the inferring function 455a infers an image observation related to the nodule at step S19 , by using the VOI on which the inferring process has not yet been performed .

[0123] When the inferring process has been performed on all the VOI s ( step S20 : Yes ) , the information processing apparatus 40a ends the inferring process .

[0124] As explained above , the information processing apparatus 40a according to the second embodiment is configured to set the three-dimensional medical image 50 with the three- dimensional volume of interest , by using the publicly-known technique . Further , the information processing apparatus 40a is configured to generate the first inference model 600 that , upon receipt of the input of at least one two-dimensional medical image 510 inside the three-dimensional volume of interest set in the three-dimensional medical image 50 , outputs the first inference result 610 with respect to each of the two-dimensional medical images 510 . Further, the information processing apparatus 40a is configured to generate the second inference model 700 that , upon receipt of the input of the first inference results 610 , outputs the second inference result 710 corresponding to the three- dimensional volume of interest .

[0125] With this configuration, the information processing apparatus 40a is able to perform the training process and the inferring process by using the two-dimensional medical images 510 in the three-dimensional volume of interest within the three-dimensional medical image 50 . Consequently, the information processing apparatus 40a is able to enhance performance of the inferring process on the three-dimensional medical image 50 . Third Embodiment

[0126] FIG . 9 is a block diagram illustrating an exemplary configuration of an information processing system lb according to a third embodiment . As for an information processing apparatus 40b according to the third embodiment , the functional configuration , the hardware configuration , and the processing procedures are the same as those in the first embodiment . In the following sections , differences from the first embodiment will be explained .

[0127] The information processing apparatus 40b according to the third embodiment is configured to input , not the three- dimensional medical image 50 , but a three-dimensional general image to an inference model . For example , the three- dimensional general image may be a three-dimensional ultrasound image . Further, the inference model is configured to perform an inferring process related to a failure of an obj ect . Further, the information processing apparatus 40b may perform not only the inferring process on failures of obj ects , but also other types of inferring processes .

[0128] From the first image storage apparatus 20 , a training data obtaining function 451b is configured to obtain the three-dimensional general image , the first correct answer data 520 , and the second correct answer data 530 as training data . The three-dimensional general image is an ultrasound image that is three-dimensional and is not for medical purposes . In the first correct answer data 520 , numerical values each indicating a degree of certainty that an abnormality such as a scar is present are assigned to the two-dimensional general images in the three-dimensional ultrasound image . Further, the first correct answer data 520 is assigned to the two-dimensional general images in which an abnormality such as a scar is present . The second correct answer data 530 is information about a time period until the obj ect fails and becomes unusable .

[0129] A first training function 452b is configured to generate a first inference model 600 by using the three-dimensional general image and the first correct answer data 520 . Accordingly, the first training function 452b is configured to generate the first inference model 600 that , upon receipt of an input of a three-dimensional general image , outputs the numerical value indicating a degree of certainty that an abnormality such as a scar is present with respect to each of the two-dimensional general images .

[0130] A second training function 453b is configured to generate a second inference model 700 by using the numerical values indicating the degrees of certainty that an abnormality such as a scar is present and serving as the first inference results 610 and the second correct answer data 530 indicating the time period until the obj ect fails and becomes unusable . Accordingly, the second training function 453b is configured to generate the second inference model 700 that , upon receipt of an input of a numerical value indicating a degree of certainty that an abnormality such as a scar is present and serving as the first inference result 610 , outputs a second inference result 710 indicating a time period until the obj ect fails and becomes unusable .

[0131] From the second image storage apparatus 30 , an inference data obtaining function 454b is configured to obtain a three- dimensional general image serving as inference target data .

[0132] An inferring function 455b includes a first inferring function 456b and a second inferring function 457b .

[0133] The first inferring function 456b is configured to input the three-dimensional general image serving as the inference target data to the first inference model 600 . Accordingly, the first inferring function 456b obtains the first inference results 610 representing numerical values each indicating a degree of certainty that an abnormality such as a scar is present .

[0134] The second inferring function 457b is configured to input the numerical values each indicating the degree of certainty that an abnormality such as a scar is present , to the second inference model 700 . Accordingly, the second inferring function 457b obtains a second inference result 710 indicating the time period until the obj ect fails and becomes unusable .

[0135] Next , an inferring process performed by the information processing apparatus 40b will be explained .

[0136] FIG . 10 is a flowchart illustrating an example of the inferring process performed by the information processing apparatus 40b according to the third embodiment .

[0137] From the first image storage apparatus 20 , the training data obtaining function 451b obtains training data including the three-dimensional general image , the first correct answer data 520 , and the second correct answer data 530 ( step S21 ) .

[0138] At steps S22 through S25 , the processing circuitry 45b performs the same processes as those at steps S2 through S5 in FIG . 6 .

[0139] Thus , the information processing apparatus 40b ends the inferring process .

[0140] The information processing apparatus 40b according to the third embodiment is configured to generate the first inference model 600 that , upon receipt of the input of at least one two-dimensional general image included in the three-dimensional ultrasound image that is not for medical purposes , outputs the first inference result 610 corresponding to the two-dimensional general image . Further, the information processing apparatus 40b is configured to generate the second inference model 700 that , upon receipt of the input of the first inference result 610 , outputs the second inference result 710 corresponding to the three- dimensional ultrasound image .

[0141] As explained above , the information processing apparatus 40b is also able to perform the training process and the inferring process on the three-dimensional image that is not for medical purposes . With this configuration , the information processing apparatus 40b is able to enhance performance of the inferring process on the three-dimensional image .

[0142] Other Embodiments

[0143] It is possible to reali ze the present disclosure by supplying a program configured to realize one or more of the functions in any of the above embodiments to a system or an apparatus via a network or a storage medium, so that one or more processors in a computer of the system or the apparatus perform a process of reading and executing the program . Further , it is also possible to realize the present disclosure by using circuitry (e . g . , an ASIC) configured to realize one or more of the functions .

[0144] According to at least one aspect of the embodiments and the like described above , it is possible to enhance performance of the inferring process on the three-dimensional images .

[0145] While certain embodiments have been described, these embodiments have been presented by way of example only, and are not intended to limit the scope of the inventions . Indeed, the novel embodiments described herein may be embodied in a variety of other forms ; furthermore , various omissions , substitutions and changes in the form of the embodiments described herein may be made without departing from the spirit of the inventions . The accompanying claims and their equivalents are intended to cover such forms or modifications as would fall within the scope and spirit of the inventions .

Claims

What is claimed is :1 . An information processing apparatus comprising processing circuitry configured : to generate at least a part of a first inference model that , upon receipt of an input of at least one two- dimensional image included in a three-dimensional image , outputs a first inference result corresponding to the two- dimensional image and a second inference model that , upon receipt of an input of two or more of the first inference results , outputs a second inference result corresponding to the three-dimensional image .2 . The information processing apparatus according to claim 1 , wherein the processing circuitry is configured to generate the first inference model by using said at least one two- dimensional image included in the three-dimensional image .3 . The information processing apparatus according to claim1 , wherein the processing circuitry is configured to obtain second correct answer data corresponding to the three-dimensional image , and to generate the second inference model by using the two or more first inference results and the second correct answer data .4 . The information processing apparatus according to claim2 , wherein the processing circuitry is configured to obtain first correct answer data assigned to said at least one two- dimensional image included in the three-dimensional image, and the processing circuitry is configured to generate the first inference model by using the first correct answer data and the two-dimensional image to which the first correct answer data is assigned .5 . The information processing apparatus according to claim2 , wherein the processing circuitry is configured, after generating the first inference model , to generate the second inference model by using the first inference result from the first inference model .6 . The information processing apparatus according to claim 5 , wherein the processing circuitry is configured to alternately repeat generating the first inference model and generating the second inference model by using the first inference result from the first inference model .7 . The information processing apparatus according to claim3 , wherein , with respect to another two-dimensional image to which first correct answer data assigned to the two- dimensional image is not assigned, the processing circuitry is configured to obtain the first inference result corresponding to said another two-dimensional image .8 . The information processing apparatus according to claim 7 , wherein the processing circuitry is configured to generate the second inference model by using the first inference result , the first correct answer data, and the second correct answer data .9 . The information processing apparatus according to claim3 , wherein the processing circuitry is configured to obtain the second correct answer data being a numerical value or category data with respect to the three-dimensional image .10 . The information processing apparatus according to claim4 , wherein the processing circuitry is configured to obtain the first correct answer data being a numerical value or category data with respect to the two-dimensional image .11 . The information processing apparatus according to claim 1 , wherein the processing circuitry is configured to perform an inferring process on the three-dimensional image , by us ing the first inference model and the second inference model .12 . The information processing apparatus according to claim 1 , wherein the processing circuitry is configured to perform an inferring process on the three-dimensional image by inputting the three-dimensional image to the first inference model , without inputting the three-dimensional image to the second inference model .13 . The information processing apparatus according to claim 1 , wherein the processing circuitry is configured to generate at least a part of the first inference model that , upon receipt of an input of at least one two-dimensional image included in a three-dimensional volume of interest set in the three-dimensional image , outputs the first inference result corresponding to the two-dimensional image and the second inference model that , upon receipt of the input of two or more of the first inference results , outputs the second inference result corresponding to the three-dimensional volume of interest set in the three-dimensional image .14 . An information processing apparatus comprising processing circuitry configured : to perform an inferring process on a three-dimensional image by using an inference model including a first inference model that , upon receipt of an input of at least one two- dimensional image included in a three-dimensional image , outputs a first inference result corresponding to the two- dimensional image and a second inference model that , upon receipt of an input of two or more of the first inference results , outputs a second inference result corresponding tothe three-dimensional image , wherein the second inference model is an inference model trained by using two or more of the first inference results output from the first inference model .15 . An information processing method comprising : generating at least a part of a first inference model that , upon receipt of an input of at least one two- dimensional image included in a three-dimensional image , outputs a first inference result corresponding to the two- dimensional image and a second inference model that , upon receipt of an input of two or more of the first inference results , outputs a second inference result corresponding to the three-dimensional image .

Images