Information processing system, information processing device, information processing method, and program

Abstract

To provide an information processing apparatus configured to select a plurality of learned models having a hierarchical relationship.SOLUTION: An information processing apparatus includes: a storage unit which stores a plurality of learned models comprising a first learned model which classifies medical image data into a class belonging to a first class layer and a second learned model which classifies the medical image data into a class belonging to a second class layer which is lower than the first class layer; an acquisition unit which acquires information on inference; a selection unit which selects a plurality of learned models having a hierarchical relationships on the basis of the information on inference acquired by the acquisition unit; and a notification unit which notifies of the learned models selected by the selection unit.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information processing system, an information processing device, an information processing method, and a program that perform inference using a plurality of trained models. [Background technology]

[0002] There is known a Computer Aided Diagnosis (CAD) system that analyzes medical data acquired by a medical imaging device and presents diagnostic support information to a doctor. The CAD system applies machine learning technology to medical data, such as medical image data, and outputs diagnostic support information.

[0003] Patent Document 1 discloses that in order to improve the reliability of analysis based on machine learning, multiple processed medical signals related to a subject are obtained by implementing at least one of different imaging methods and different signal processing, and inference is performed for each of the multiple processed medical signals using multiple trained models. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-42810 A Summary of the Invention [Problem to be solved by the invention]

[0005] Patent Document 1 does not disclose the selection of multiple trained models in a hierarchical relationship.

[0006] Therefore, an object of the present invention is to provide an information processing system, an information processing device, an information processing method, and a program capable of selecting multiple trained models in a hierarchical relationship. [Means for solving the problem]

[0007] In order to achieve the object of the present invention, the information processing device of the present invention comprises a memory unit that stores multiple trained models including a first trained model that classifies medical image data into classes belonging to a first class hierarchy and a second trained model that classifies the medical image data into classes belonging to a second class hierarchy that is a class hierarchy lower than the first class hierarchy, an acquisition unit that acquires information regarding inference by a user, a selection unit that selects multiple trained models in a hierarchical relationship based on the information regarding inference acquired by the acquisition unit, and a notification unit that notifies of the multiple trained models selected by the selection unit. Effect of the Invention

[0008] According to the present invention, it is possible to select multiple trained models in a hierarchical relationship. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing a configuration of an information processing system according to the present invention. [Diagram 2] FIG. 1 is a diagram showing a configuration of an information processing device according to the present invention. [Diagram 3] FIG. 2 is a diagram showing the arrangement of a storage unit in the information processing device according to the first embodiment of the present invention. [Figure 4] FIG. 1 is a diagram for explaining a flow of selecting a trained model according to the first embodiment of the present invention. [Diagram 5] FIG. 1 is a diagram illustrating an inference flow of a trained model according to the first embodiment of the present invention. [Figure 6] FIG. 11 is a diagram showing the arrangement of a storage unit in an information processing device according to a second embodiment of the present invention. [Figure 7] FIG. 11 is a diagram for explaining a flow of selecting a trained model according to the second embodiment of the present invention. [Figure 8] FIG. 11 is a diagram showing the configuration of a storage unit in an information processing device according to a fourth embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings.

[0011] [First embodiment] FIG. 1 shows the configuration of an information processing system according to the present invention.

[0012] The information processing system of the present invention comprises a medical imaging device 100 that acquires medical data on a subject, an information terminal 110, a network 120, and an information processing device 130. The information terminal 110 may be a single information terminal or multiple information terminals.

[0013] An operation unit 112 and a display unit 114 are connected to the information terminal 110. The operation unit 112 accepts various instructions from an operator and transmits the various instructions to the information terminal 110 and the medical imaging apparatus 100. The operation unit 112 is composed of, for example, a mouse, a keyboard, a button, a panel switch, a foot switch, a trackball, a joystick, etc. The display unit 114 displays a GUI for inputting various instructions in the operation unit 112, and displays image data based on medical data acquired in the medical imaging apparatus 100.

[0014] In FIG. 1, operation unit 112 and display unit 114 are provided separately from information terminal 110, but information terminal 110 may have the functions of operation unit 112 and display unit 114 internally.

[0015] The medical imaging apparatus 100 is an apparatus that acquires medical data of a subject, such as an X-ray CT (Computed Tomography) apparatus, an MRI (Magnetic Resonance Imaging) apparatus, or an ultrasonic diagnostic apparatus.

[0016] X-ray CT equipment is equipped with an X-ray source and an X-ray detector. While rotating the X-ray source and the X-ray detector around the subject, the subject is irradiated with X-rays from the X-ray source and the data detected by the X-ray detector is projected to generate CT data.

[0017] The MRI apparatus generates a predetermined magnetic field for a subject placed within a static magnetic field, and generates MRI data by performing a Fourier transform on the acquired data.

[0018] An ultrasound diagnostic device transmits ultrasound waves to a subject, receives ultrasound waves reflected from the subject, and generates ultrasound data.

[0019] The medical data (CT data, MRI data, ultrasound data, etc.) generated by the medical imaging device 100 is three-dimensional data (volume data) or two-dimensional data. The medical data is, for example, image data relating to a subject. The image data also includes raw data. The medical image data may be moving image data consisting of multiple frame data. The medical data also includes measurement data obtained by performing various measurements using the medical image data.

[0020] The medical imaging apparatus 100 is connected to an information terminal 110. The information terminal 110 is a PC terminal, a mobile phone such as a smartphone, a notebook terminal, a tablet terminal, or the like. The information terminal 110 can also set subject information and can associate medical data acquired from the medical imaging apparatus 100 with the subject information. The information terminal 110 can also display various types of data, such as medical data and measurement data acquired from the medical imaging apparatus 100.

[0021] The information terminal 110 and the information processing device 130 are connected to a network 120. The network 120 includes a communication network outside the hospital, such as wireless communication (Wi-Fi), the Internet, a wireless base station, a provider, a communication line, and the like. The network 120 may also include an intranet, which is a communication network within the hospital. The information terminal 110 can communicate with the information processing device 130 through the network 120. The information terminal 110 can transmit medical data (including medical image data), information related to inference, model selection information, and the like to the information processing device 130. The information processing device 130 can transmit an inference result obtained by performing inference using medical data (including medical image data) to the information terminal 110.

[0022] 2 shows the configuration of the information processing device 130 of the present invention. The information processing device 130 has a teacher data generation unit 200 that generates teacher data using medical image data, and a learning unit 202 that performs learning on the medical image data using the teacher data generated in the teacher data generation unit 200. The information processing device 130 also has a storage unit 204 that stores the trained model generated in the learning unit 202 in association with a class hierarchy, and an inference unit 206 that performs inference using the trained model. The information processing device 130 further has an acquisition unit 208 that acquires information related to inference, a selection unit 210 that selects multiple trained models in a hierarchical relationship based on the information related to inference acquired by the acquisition unit 208, and a notification unit 212 that notifies the multiple trained models selected by the selection unit 210.

[0023] The components (functions) of the information processing device 130 are realized by a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) executing a program (software) stored in a memory.

[0024] The information processing device 130 has a processor and a memory inside. The processor can execute each process of the information processing device 130 according to a program stored in the memory, and can function as a teacher data generating unit 200, a learning unit 202, a storage unit 204, an inference unit 206, an acquisition unit 208, a selection unit 210, a notification unit 212, etc.

[0025] The information on inference acquired by the acquisition unit 208 is, for example, at least one of information indicating medical image data that is the inference target, the inference purpose, and a class.

[0026] The teacher data generating unit 200 is connected to the network 120 and can acquire medical data including medical image data, measurement data, etc. The teacher data generating unit 200 generates teacher data using the medical image data. The teacher data generated by the teacher data generating unit 200 is determined according to the inference task performed by the neural network and the target of classification.

[0027] Examples of inference tasks performed by neural networks include classification tasks to classify medical image data into classes, detection tasks to detect what is captured in what position in the medical image data, and segmentation tasks to extract target areas from medical image data.

[0028] When training a neural network to perform a classification task, the training data generation unit 200 generates training data that pairs medical image data with a correct answer label, which is a label indicating an object that appears in the medical image data.

[0029] On the other hand, when performing a detection task using a neural network, the teacher data generation unit 200 generates teacher data that pairs medical image data with a correct image to which a ROI (Region Of Interest) indicating the position of an object captured in the medical image data and a correct label that is a label indicating the object have been assigned.

[0030] When the task performed by the neural network is segmentation, the teacher data generation unit 200 generates teacher data that pairs medical image data with a correct image to which position information of pixels of an object reflected in the medical image data and a correct label that indicates the object have been assigned.

[0031] For example, when training a neural network to perform the task of segmenting the presence or absence, type, and area of ​​a lesion from medical image data obtained from the information terminal 110, the teacher data generation unit 200 generates teacher data that pairs medical image data having a lesion area with a correct answer image to which information on a correct answer label indicating the type of lesion for the medical image data and position information of the pixels of the lesion have been assigned.

[0032] The teacher data generating unit 200 may also perform preprocessing of medical data depending on the neural network that is learning in the learning unit 202. For example, when the target of inference by the neural network is medical image data, the acquired medical image data is subjected to processes such as noise removal, filtering, image cropping, and resolution change. When the target of inference is natural language such as text, the teacher data generating unit 200 performs preprocessing of the data to be processed, such as performing morphological analysis and applying vector conversion technology, depending on the target of inference of the neural network and the task.

[0033] 2 shows a form in which the teacher data generating unit 200 is included inside the information processing device 130, the information terminal 110 may include the teacher data generating unit 200 therein. That is, the teacher data generating unit 200 may be included as a component of the information terminal 110. For example, after the teacher data as described above is generated by the information terminal 110, the inference device may be trained by the learning unit 202 of the information processing device 130 via the network 120.

[0034] The learning unit 202 is connected to the teacher data generating unit 200. The learning unit 202 generates a trained model by using a neural network to learn medical image data in association with teacher data. Here, the trained model indicates parameters determined by performing a learning process up to a predetermined standard and information on a model corresponding to the parameters. Note that the trained model may be used to train another model as transfer learning, or further learning processing may be performed on the trained model.

[0035] A neural network includes multiple layers. Among neural networks, CNN (Convolutional Neural Network), which is a type of deep learning technology, has multiple intermediate layers between the input layer and the output layer, although not shown. The multiple intermediate layers include a convolution layer, a pooling layer, an upsampling layer, and a synthesis layer. The convolution layer is a layer that performs convolution processing on a group of input values. In the convolution layer, the input medical image data is convolved to extract features of the medical image data.

[0036] The pooling layer is a layer that performs processing to make the number of output value groups smaller than the number of input value groups by thinning out or synthesizing the input value groups. The upsampling layer is a layer that performs processing to make the number of output value groups larger than the number of input value groups by duplicating the input value groups or adding values ​​interpolated from the input value groups. The synthesis layer is a layer that inputs value groups such as the output value groups of a certain layer or pixel values ​​constituting medical image data from multiple sources and synthesizes them by concatenating or adding them. The number of intermediate layers can be changed at any time depending on the learning content.

[0037] The storage unit 204 is connected to the learning unit 202. The storage unit 204 stores a plurality of class hierarchies and a trained model corresponding to each of the plurality of class hierarchies. The class hierarchies correspond to classes classified by the trained models, for example, the first class hierarchies are classes for classifying organ regions, and the second class hierarchies are classes for detecting abnormalities from a specific region. The trained models for classifying classes belonging to the class hierarchies are stored in the storage unit 204 in association with the class hierarchies. In addition, among the trained models corresponding to the first class hierarchies, for example, a trained model for identifying a lung field region and among the trained models corresponding to the second class hierarchies, a trained model for detecting an abnormal site from a lung field region are in a relationship defined as a hierarchical relationship. The hierarchical relationship indicates a flow of inference by the trained models, and, for example, when a CT image is input, a first trained model for inferring a lung field region from the CT image is applied, and a second trained model for inferring an abnormal region from the inferred lung field region is stored in the storage unit 204 as a hierarchical relationship.

[0038] The trained model stored in the storage unit 204 is a trained model trained to detect and extract, for example, organ regions and lesion regions in medical image data. Here, the trained model is generated using, for example, a neural network, but in addition to CNN and RNN (Recurrent Neural Network), which are deep learning technologies among neural network technologies, and models derived from CNN and RNN, other machine learning technologies such as support vector machines, logistic regression, and random forests may be used, or a rule-based method may be used.

[0039] The inference unit 206 can acquire medical data including medical image data, measurement data, etc. from the acquisition unit 208. The inference unit 206 is connected to the storage unit 204, and can perform inference using a trained model selected by the selection unit 210, which will be described later, from among the trained models stored in the storage unit 204. The inference unit 206 performs inference on newly generated medical image data, for example, using a trained model trained to extract the type and area of ​​a lesion in medical image data. If the newly generated medical image data contains a lesion, the inference unit 206 can output the type and area of ​​the lesion. In addition, when multiple trained models in a hierarchical relationship are selected by the selection unit 210, the inference unit 206 performs inference processing using a higher trained model among the multiple trained models in the hierarchical relationship, and performs inference processing using a lower trained model using an input based on the inference result. That is, the inference unit 206 performs inference using the multiple trained models in the selected hierarchical relationship.

[0040] FIG. 3 shows the configuration of the storage unit 204 of the present invention. The storage unit 204 stores a plurality of trained models and auxiliary information trained in the learning unit 202 in association with a class hierarchy. The class hierarchy corresponds to a class to be inferred by the trained model, and a trained model that classifies into a class belonging to the class hierarchy is stored in association with the class hierarchy. The higher the class, the earlier the class hierarchy should be inferred for medical image data acquired by the medical imaging device 100, and the lower the class, the later the class hierarchy should be inferred for the medical image data. The inference unit 206 executes an inference process using a trained model corresponding to a higher class hierarchy, thereby improving the inference performance of the trained model corresponding to a lower class hierarchy. In addition, as a function of supporting the trained model corresponding to the lower class hierarchy, for example, extraction of an organ region, reduction of noise, etc., are executed by the trained model corresponding to the higher class hierarchy. Here, the storage unit 204 is composed of, for example, a first class hierarchy 301 and a second class hierarchy 302, and for example, the first class hierarchy 301 corresponds to a class for inferring an organ region, and the second class hierarchy 302 corresponds to a class for inferring an abnormality from a specific region. Note that the class hierarchy may be composed of three or more class hierarchies. In each class hierarchy constituting the multiple class hierarchies in the storage unit 204, multiple trained models for inferring a class corresponding to each class hierarchy are stored.

[0041] A first trained model 310 and first incidental information 330 assigned to the first trained model 310 are stored in the first class hierarchy 301 in the storage unit 204. Also, a second trained model 312 and second incidental information 332 assigned to the second trained model are stored in the first class hierarchy 301 in the storage unit 204.

[0042] In the second class hierarchy 302 in the memory unit 204, 2-1 additional information 334 is assigned to the 2-1 trained model 314, 2-2 additional information 336 is assigned to the 2-2 trained model 316, 2-3 additional information 338 is assigned to the 2-3 trained model 318, and 2-4 additional information 324 is assigned to the 2-4 trained model 320 and stored.

[0043] Here, the first trained model 310 in the first class hierarchy 301 and the 2-1 trained model 314 and the 2-2 trained model 316 in the second class hierarchy 302 are in a hierarchical relationship. Here, the hierarchical relationship is shown diagrammatically connected by lines. Also, the second trained model 312 in the first class hierarchy 301 is in a hierarchical relationship with the 2-3 trained model 318 and the 2-4 trained model 320 in the second class hierarchy. The hierarchical relationship corresponds to, for example, a plurality of inference flows executed on medical image data. For example, for a first inference and a flow in which a second inference is performed in response to the first inference to obtain an output, the storage unit 204 stores the trained model that performs the first inference and the trained model that performs the second inference as a hierarchical relationship. Note that here, the hierarchical relationship is shown by lines for simplicity, but for example, the accompanying information accompanying each of the trained models stores information indicating the trained model that performs the second inference performed after the first inference as a hierarchical relationship. In addition, in addition to the case where the incidental information attached to the trained model corresponding to a higher class hierarchy stores information indicating the trained model corresponding to a lower class hierarchy, the incidental information attached to the trained model corresponding to a lower class hierarchy may store information indicating the trained model corresponding to the higher class hierarchy. Furthermore, the incidental information attached to the trained model corresponding to the higher class hierarchy and the incidental information attached to the trained model corresponding to the lower class hierarchy may store information indicating each other's models in a bidirectional manner.

[0044] The storage unit 204 stores a plurality of trained models in association with a plurality of class hierarchies and stores the hierarchical relationship between the plurality of trained models, thereby allowing the selection unit 210 described later to select a trained model in a hierarchical relationship, thereby reducing the time and effort required for the user to select a model and input the output and input for the selected plurality of models. Note that, although FIG. 3 shows a form in which six trained models are stored in the storage unit 204 together with accompanying information, the storage unit 204 can also store seven or more trained models together with accompanying information. In addition, each of the plurality of trained models is different in one of the task to be inferred, the class indicating the classification target, the model structure, and the teacher data. In addition, each trained model can be specified or the trained model can be identified based on the accompanying information assigned to each trained model. Furthermore, the trained models in the hierarchical relationship can be further specified or identified based on the accompanying information.

[0045] The inference unit 206 selects an appropriate trained model based on the attached information using the selection unit 210 in response to input from the information terminal 110, and uses the selected trained model to perform inference processing on the data to be inferred obtained from the information terminal 110.

[0046] The inference unit 206 performs inference on the medical image data to be inferred, for example, using a trained model that has been trained to extract a lesion area or an organ area in the medical image data. When multiple trained models in a hierarchical relationship are selected by the selection unit 210, the inference unit 206 performs inference using a trained model corresponding to a higher class hierarchy, and performs inference on an input based on an inference result of the inference using a trained model corresponding to a lower class hierarchy. The inference unit 206 performs inference processing using a trained model corresponding to a higher class hierarchy, and executes inference processing using a trained model corresponding to a lower class hierarchy in a hierarchical relationship with the trained model, thereby improving the accuracy of the inference processing to be output.

[0047] The inference unit 206 may notify, via the notification unit 212, the inference results of each of the multiple trained models in a hierarchical relationship selected by the selection unit 210, or may notify, via the notification unit 212, the inference results of the trained model corresponding to the lowest level.

[0048] The acquisition unit 208 acquires information about inference from the information terminal 110. The information about inference includes at least one of, for example, medical image data of an inference target, and information indicating an inference purpose and a class. When a user operates, for example, the information terminal 110 to transmit medical image data of an inference target, the acquisition unit 208 acquires the transmitted information about inference and transmits the acquired information to the selection unit 210. When the acquired information about inference is insufficient, the acquisition unit 208 executes a notification to prompt the user to transmit additional information via the notification unit 212. The acquisition unit 208 may also acquire selection information of a trained model. In response to the acquired selection information, the selection unit 210 selects a trained model, and the inference unit 206 performs inference using the selected trained model. The selection information is input by the user via, for example, the operation unit 112 of the information terminal 110.

[0049] The selection unit 210 selects multiple hierarchically related trained models from the storage unit 204 as trained models for use in inference by the inference unit 206, based on information regarding inference acquired by the acquisition unit 208.

[0050] For example, when the information on the inference acquired by the acquisition unit 208 is medical image data, the selection unit 210 calculates the degree of conformance with the trained models corresponding to each of the multiple class hierarchies, and selects multiple trained models in a hierarchical relationship according to the degree of conformance. The calculation of the degree of conformance may be performed, for example, by acquiring the probability that an object corresponding to a class corresponding to a class hierarchy exists in the target medical image data using a trained model. The degree of conformance may also be detected by other detection methods. For example, the degree of conformance may be calculated based on the similarity between the trained model corresponding to each class hierarchy and the teacher data. The selection unit 210 calculates the degree of conformance, and selects a trained model from trained models corresponding to class hierarchies in which the calculated degree of conformance is greater than a predetermined value. The selection unit 210 selects a trained model corresponding to the lowest hierarchy among trained models in which the degree of conformance is greater than a predetermined value, and also selects a trained model corresponding to a higher hierarchy in the hierarchical relationship from the supplementary information of the selected trained model. In addition, when there are three or more class hierarchies, three or more trained models are selected according to the hierarchical relationship. Note that the selection by the selection unit 210 is not limited to this, and multiple trained models in a hierarchical relationship may be selected so as to include the model with the highest degree of fitness.

[0051] The selection unit 210 may also select a trained model corresponding to the highest class hierarchy from among trained models whose fitness is greater than a predetermined value. The selection unit 210 selects trained models corresponding to lower classes from the supplementary information of the trained model corresponding to the selected highest class hierarchy. When there are three or more class hierarchies, three or more trained models are selected according to the relationship of the hierarchies.

[0052] Furthermore, when the information on inference acquired by the acquisition unit 208 is information on an inference purpose or a class, the selection unit 210 calculates the compatibility between the inference purpose or the class information and the class hierarchy in the storage unit 204. For example, the compatibility is determined as the similarity between a vector of the inference purpose or the class information and a vector of a class constituting the class hierarchy. Alternatively, the compatibility may be determined as the likelihood that the trained model has performed an inference process using a vector of the inference purpose or the class information as input.

[0053] Furthermore, when the information acquired by the acquisition unit 208 is information on an inference purpose or a class, and medical image data, the selection unit 210 first calculates the degree of conformance from the information on the inference purpose or the class by the above-mentioned method. Next, the selection unit 210 calculates the degree of conformance for the medical image data by the above-mentioned method. Finally, the selection unit 210 selects multiple trained models in a hierarchical relationship by the above-mentioned method based on the degree of conformance based on both degrees of conformance. The selection unit 210 may select a trained model by comparing each of the two degrees of conformance with a predetermined value, or may select a trained model by comparing the degree of conformance calculated by combining the two degrees of conformance by summing or multiplying them with a predetermined value. When multiple trained models in a hierarchical relationship are selected by the selection unit 210, information on the selected trained model is transmitted to the notification unit 212.

[0054] If the acquisition unit 208 acquires model selection information by a user, for example, from the information terminal 110, after or before the notification by the notification unit 212, the selection unit 210 selects multiple trained models in a hierarchical relationship based on the selection information.

[0055] The notification unit 212 notifies information on the trained model selected by the selection unit 210. The notification by the notification unit 212 is sent to the information terminal 110 connectable to the information processing device 130 via the network 120. The notification unit 212 also notifies the information terminal of the inference result by the inference unit 206.

[0056] That is, the information processing device 130 in this embodiment includes a storage unit 204 that stores multiple trained models including a first trained model that classifies medical image data into a class belonging to a first class hierarchy and a second trained model that classifies the medical image data into a class belonging to a second class hierarchy that is a class hierarchy lower than the first class hierarchy, an acquisition unit 208 that acquires information regarding inference, a selection unit 210 that selects multiple trained models in a hierarchical relationship based on the information regarding inference acquired by the acquisition unit 208, and a notification unit that notifies the multiple trained models selected by the selection unit 210. Thus, multiple trained models in a hierarchical relationship can be selected from multiple trained models, and the user's effort in selecting a trained model from multiple trained models and inputting outputs and inputs for the selected multiple trained models can be reduced.

[0057] Hereinafter, a flow from the information related to inference to the selection unit 210 selecting a trained model to be used for inference will be described with reference to FIG.

[0058] In step S400, the acquisition unit 208 acquires information related to the inference. When the acquisition unit 208 acquires information related to the inference, the process proceeds to step S401, and when the acquisition unit 208 has not acquired information related to the inference, the process ends.

[0059] In step S401, the selection unit 210 calculates the compatibility with each class hierarchy by any of the above-mentioned methods, and determines whether the calculated compatibility is greater than a predetermined value. If the compatibility calculated by the selection unit 210 is greater than the predetermined value, the process proceeds to step S402. If the compatibility calculated by the selection unit 210 is equal to or less than the predetermined value, the process ends.

[0060] In step S402, the selection unit 210 determines whether there are two or more trained models determined to be greater than the predetermined value. If there are two or more trained models determined to be greater than the predetermined value, the selection unit 210 proceeds to step S403, and if there are less than two trained models, the selection unit 210 proceeds to step S406.

[0061] In step S403, the selection unit 210 determines whether multiple trained models are in a hierarchical relationship. The selection unit 210 makes the determination by referring to accompanying information attached to each of the two or more trained models. If the selection unit 210 determines that there is no trained model in a hierarchical relationship, the process proceeds to step S404, and if the selection unit 210 determines that there is a trained model in a hierarchical relationship, the process proceeds to step S405.

[0062] In step S404, the selection unit 210 selects the trained model with the highest degree of conformance from among the multiple trained models as the trained model to be used for inference by the inference unit 206.

[0063] In step S405, a plurality of trained models in a hierarchical relationship are selected as trained models to be used for inference by the inference unit 206.

[0064] In step S406, the notification unit 212 notifies the trained model selected by the selection unit 210.

[0065] Next, a flow of inference performed by the inference unit 206 using the trained model selected by the selection unit 210 will be described with reference to FIG.

[0066] When a trained model is selected by the selection unit 210 in step S500, the process proceeds to step S501, and when no model is selected, the process ends.

[0067] In step S501, the inference unit 206 determines whether or not the number of trained models selected by the selection unit 210 is two or more. If the number of trained models is less than two, the process proceeds to step S502. If the number of trained models is two or more, the process proceeds to step S503.

[0068] In step S502, the inference unit 206 executes inference processing on the medical image data using the trained model selected by the selection unit 210. When the inference processing using the trained model is executed, the inference result is transmitted to the notification unit 212, and the process proceeds to step S506.

[0069] In step S503, the inference unit 206 determines whether or not two or more trained models are in a hierarchical relationship. If they are not in a hierarchical relationship, the process proceeds to step S504. If they are in a hierarchical relationship, the process proceeds to step S505.

[0070] In step S504, the inference unit 206 executes an inference process using each of the two or more trained models that are not in a hierarchical relationship. When the inference process using the trained models is executed, the inference result is transmitted to the notification unit 212, and the process proceeds to step S506.

[0071] In step S505, the inference unit 206 performs inference processing in order from the trained model corresponding to the higher class hierarchy among the multiple trained models in a hierarchical relationship. A case will be described in which the selected trained models are the first trained model 310 corresponding to the first class hierarchy 301 and the 2-1 trained model 314 corresponding to the second class hierarchy 302. The inference unit 206 applies the first trained model 310 to the medical image data, performs inference processing, and obtains a first inference result. Next, the inference unit 206 inputs the 2-1 trained model 314 based on the first inference result by the first trained model 310, performs inference processing, and obtains a second inference result. When the inference processing by the trained model is performed, the inference result is transmitted to the notification unit 212, and the process proceeds to step S506.

[0072] In step S506, the notification unit 212 notifies the inference result based on the trained model selected by the selection unit 210. The notification by the notification unit 212 is, for example, the information terminal 110 capable of communicating with the information processing device 130 via the network 120. Note that the notification by the notification unit 212 may be confirmed, for example, on a display unit (not shown) provided in the information processing device 130 or by the user accessing the information processing device 130, as long as it can be notified to the user who desires the inference.

[0073] The information processing device 130 in this embodiment can select multiple trained models in a hierarchical relationship from multiple trained models, and can perform inference processing using the selected multiple trained models in a hierarchical relationship.

[0074] (Modification) In the above-described embodiment, the selection unit 210 calculates the degree of compatibility between the information regarding inference and the class hierarchy, and selects a trained model to be subjected to inference processing by the inference unit 206 from trained models whose degree of compatibility is greater than a predetermined value.

[0075] Here, when a trained model corresponding to the lowest or highest class hierarchy of multiple trained models in a hierarchical relationship selected by the selection unit 210 is in a hierarchical relationship with a trained model corresponding to an even higher or lower class hierarchy, the selection unit 210 selects the trained model as a recommended trained model, and the notification unit 212 may issue a notification recommending the selection of the trained model.

[0076] [Second embodiment] In the above embodiment, the selection unit 210 calculates the compatibility between the information on inference and the class corresponding to the class hierarchy, and selects a trained model corresponding to the lowest or highest class hierarchy among trained models whose compatibility is greater than a predetermined value. In this embodiment, the selection unit 210 selects a trained model corresponding to the highest class hierarchy among trained models whose compatibility is greater than a predetermined value, a trained model corresponding to the lowest class hierarchy, and, if another class hierarchy is stored in the storage unit 204 between the highest class hierarchy and the lowest class hierarchy, further selects a trained model from another class hierarchy between them. Hereinafter, this embodiment will be described with reference to FIG. 6.

[0077] The storage unit 204 further has an Nth class hierarchy 603. N is a natural number equal to or greater than 3, and is set according to the number of class hierarchies.

[0078] Here, for simplicity, the explanation will be given assuming that N=3. In this embodiment, the selection unit 210 selects a trained model corresponding to the highest class hierarchy among trained models whose fitness is greater than a predetermined value, and further selects a trained model corresponding to the lowest class hierarchy. When both trained models selected by the selection unit 210 are in a hierarchical relationship (when searchable from the auxiliary information of at least one of the trained models) and there is a different class hierarchy between the class hierarchies to which the two models correspond, the trained model in the hierarchical relationship among the trained models corresponding to the different class hierarchies in between is selected as a trained model for further inference by the inference unit 206. For example, a case will be explained in which the selection unit 210 selects the first trained model 310 as the trained model corresponding to the highest class hierarchy among trained models whose fitness is greater than a predetermined value, and the N-1th trained model 614 as the trained model corresponding to the lowest class hierarchy. The selection unit 210 determines whether the first trained model 310 and the N-1th trained model 614 are in a hierarchical relationship. The selection unit 210 determines whether there is another class hierarchy between the first class hierarchy 301, which is determined to be in a hierarchical relationship by the selection unit 210 and corresponds to the first trained model 310, and the Nth class hierarchy, which corresponds to the (N-1)th trained model 614. If there is another class hierarchy, the selection unit 210 further selects trained models that are in a hierarchical relationship with the first trained model 310 and the (N-1)th trained model 614 from among trained models corresponding to different class hierarchies in between.

[0079] The flow of this embodiment will be described with reference to Fig. 7. In step S700, the selection unit 210 determines whether or not there is another hierarchy between the class hierarchy of the highest class and the lowest class corresponding to the trained model in which the calculated fitness is greater than a predetermined value and which has a hierarchical relationship. If the selection unit 210 determines that there is no other hierarchy, the process proceeds to step S405 described above. If the selection unit 210 determines that there is another hierarchy, the process proceeds to step S702.

[0080] In step S702, the selection unit 210 further selects, from among the trained models corresponding to other hierarchies, trained models that are in a hierarchical relationship with the trained model corresponding to the highest class and the trained model corresponding to the lowest class (can be searched from the accompanying information). Note that the hierarchical relationship is defined as being in a hierarchical relationship between the first trained model 310 and the 2-1st trained model 314, as well as between the first trained model 314 and the N-1st trained model 614, if both trained models can be searched from any of the accompanying information, for example.

[0081] The information processing device 103 in this embodiment enables the selection unit 210 to specify the range when selecting an appropriate trained model from multiple trained models, thereby reducing the effort required by the user to select a trained model from multiple trained models and to input the output and input for the selected multiple trained models.

[0082] [Third embodiment] In the above-described embodiments, a configuration has been described in which multiple trained models in a hierarchical relationship are selected by the selection unit 210. In this embodiment, however, a case will be described in which multiple trained models in a hierarchical relationship are selected by the selection unit 210.

[0083] A case will be described in which the selection unit 210 selects, for example, the first trained model 310 in FIG. 3 and the 2-1 trained model 314 in a hierarchical relationship with the first trained model 310 as first candidates, and the second trained model 312 and the 2-3 trained model 318 in a hierarchical relationship with the second trained model 312 as second candidates. When multiple candidates are selected by the selection unit 210, the selection unit 210 further compares the fitness of the multiple candidates. The comparison of fitness by the selection unit 210 is, for example, to select a candidate having a trained model that classifies a class with the highest fitness as a trained model to be inferred by the inference unit 206. Alternatively, a candidate having a large value obtained by adding up the fitness of each of the trained models in the candidate may be selected. When the difference in fitness between the multiple candidates selected by the selection unit 210 is small, the notification unit 212 issues a notification recommending the selection of one of the two candidates.

[0084] According to this embodiment, even when multiple trained models in multiple hierarchical relationships for performing inference on medical image data are selected as candidates, multiple trained models in an appropriate hierarchical relationship can be selected.

[0085] [Fourth embodiment] This embodiment further describes the selection of a trained model by the selection unit 210 when the trained model corresponding to the lowest hierarchy selected from among trained models larger than a predetermined value is a trained model that is in a hierarchical relationship with multiple trained models.

[0086] 8, the storage unit 204 according to the fourth embodiment stores a first class hierarchy 801 and a second class hierarchy 802. The first class hierarchy 801 stores a first trained model 810, first incidental information 830 associated with the first trained model 810, a second trained model 812, and second incidental information 832 associated with the second trained model 812. The second class hierarchy 802 further stores a 2-2 trained model 816 in a hierarchical relationship with the first trained model 810 and the second trained model 812, and 2-2 incidental information 836 associated with the 2-2 trained model.

[0087] Assume that, through the selection process by the selection unit 210, the first trained model 810 and the first candidate trained model 2-2 816, and the second trained model 812 and the second candidate trained model 2-2 are selected as trained models whose calculated fitness is greater than a predetermined value and which are in a hierarchical relationship.

[0088] When multiple candidates are selected, the appropriate candidate is selected by comparing the total fitness values ​​of the multiple candidates. In addition, when the storage unit 204 stores three or more class hierarchies such as the class hierarchy described in Fig. 6, and the candidates include trained models that are in a hierarchical relationship with multiple trained models, the selection unit 210 compares the fitness values ​​of multiple candidates including the trained models and selects multiple trained models that are in a hierarchical relationship.

[0089] According to this embodiment, even when multiple trained models in multiple hierarchical relationships are selected as candidates, multiple trained models in an appropriate hierarchical relationship can be selected.

[0090] (Other variations) A computer program for realizing the functions of the above-described embodiments can be supplied to a computer via a network or a storage medium (not shown), and the computer program can be executed. This is a computer program for causing a computer to execute the above-described information processing method. In other words, the computer program is a program for causing a computer to realize the functions of an information processing device. The storage medium stores the computer program. [Explanation of symbols]

[0091] 130 Information processing equipment 204 Storage section 206 Reasoning section 208 Acquisition Department 210 Selection section 212 Notification Department

Claims

1. A storage unit that stores a plurality of trained models including a first trained model that classifies image data into a class belonging to a first class hierarchy, and a second trained model that classifies image data into a class belonging to a second class hierarchy that is a class hierarchy lower than the first class hierarchy; An acquisition unit that acquires information related to inference; A selection unit that selects a plurality of trained models in a hierarchical relationship based on information about the inference acquired by the acquisition unit; A notification unit that notifies the plurality of trained models selected by the selection unit; 13. An information processing device comprising:

2. 2. An information processing apparatus according to claim 1, wherein the information relating to the inference is at least one of information indicating an inference target, an inference purpose, and a class.

3. The information processing device according to claim 1 or 2, further comprising an inference unit that performs inference using a plurality of trained models in the hierarchical relationship selected by the selection unit.

4. The information processing device according to claim 3 , characterized in that the inference unit performs inference in order from a trained model that classifies into a class belonging to a higher hierarchy among the multiple trained models in the hierarchical relationship selected by the selection unit.

5. The information processing device according to any one of claims 1 to 4, characterized in that the selection unit calculates a degree of conformance with a trained model corresponding to each class hierarchy from information regarding the inference, and selects the multiple trained models according to the calculated degree of conformance.

6. The information processing device according to claim 5 , wherein the selection unit selects a plurality of trained models from trained models whose degree of conformance with the trained model corresponding to each of the class hierarchies is greater than a predetermined value.

7. The information processing device according to claim 6, characterized in that the selection unit selects, from among the trained models whose degree of conformance with the trained models corresponding to each class hierarchy is greater than a predetermined value, a trained model corresponding to the lowest class hierarchy and a trained model in a higher class hierarchy that is in the hierarchical relationship with the trained model corresponding to the lowest class hierarchy.

8. The information processing device according to claim 6 or 7, characterized in that the selection unit selects a trained model corresponding to the highest class hierarchy and a trained model corresponding to a lower class hierarchy that has a hierarchical relationship with the trained model corresponding to the highest class hierarchy, from among the trained models whose degree of conformance with the trained models corresponding to each class hierarchy is greater than a predetermined value.

9. The information processing device according to any one of claims 6 to 8, characterized in that the selection unit selects a trained model corresponding to the highest hierarchy and a trained model corresponding to the lowest hierarchy among the trained models whose degree of conformance with the trained models corresponding to each hierarchy is greater than a predetermined value, and if there is a hierarchy between the highest hierarchy and the lowest hierarchy, further selects the trained model corresponding to the hierarchy.

10. The acquisition unit further acquires selection information of the trained model, The information processing device according to claim 1 , wherein a plurality of trained models in a hierarchical relationship are selected based on the selection information.

11. The information processing device according to claim 5 , characterized in that the selection unit calculates the goodness of fit using a trained model having classes corresponding to the first class hierarchy and classes corresponding to the second class hierarchy as classes for classification.

12. A storage unit that stores a plurality of trained models including a first trained model that classifies data to be inferred into classes belonging to a first class hierarchy, and a second trained model that classifies data to be inferred into classes belonging to a second class hierarchy that is a class hierarchy lower than the first class hierarchy; An acquisition unit that acquires information related to inference; A selection unit that selects a plurality of trained models in a hierarchical relationship based on information about the inference acquired by the acquisition unit; A notification unit that notifies the plurality of trained models selected by the selection unit; 13. An information processing device comprising:

13. An acquisition step of acquiring information regarding inference; a selection step of selecting a plurality of trained models in a hierarchical relationship from a storage unit that stores a plurality of trained models including a first trained model that classifies image data into a class belonging to a first class hierarchy and a second trained model that classifies image data into a class belonging to a second class hierarchy that is a class hierarchy lower than the first class hierarchy, based on information regarding the inference acquired by the acquisition step; A notification step of notifying the plurality of trained models selected by the selection step; 13. An information processing method comprising:

14. An acquisition step of acquiring information regarding inference; a selection step of selecting a plurality of trained models in a hierarchical relationship from a storage unit that stores a plurality of trained models including a first trained model that classifies data to be inferred into a class belonging to a first class hierarchy based on information regarding inference acquired by the acquisition step, and a second trained model that classifies data to be inferred into a class belonging to a second class hierarchy that is a class hierarchy lower than the first class hierarchy; A notification step of notifying the plurality of trained models selected by the selection step; 13. An information processing method comprising:

15. The information processing method according to claim 13 or 14, further comprising an inference step of performing inference in order from a trained model that classifies into a class belonging to a higher hierarchy among the multiple trained models in the hierarchical relationship selected by the selection step.

16. A program for causing a computer to execute the information processing method according to any one of claims 13 to 15.

17. An information processing system including an information processing device having a storage unit that stores a plurality of trained models including a first trained model that classifies image data into a class belonging to a first class hierarchy and a second trained model that classifies image data into a class belonging to a second class hierarchy that is a class hierarchy lower than the first class hierarchy, and an information terminal that is capable of communicating with the information processing device via a network, The information processing device includes: An acquisition unit that acquires information related to inference; A selection unit that selects a plurality of trained models in a hierarchical relationship based on information about the inference acquired by the acquisition unit; A notification unit that notifies the information terminal of the plurality of trained models selected by the selection unit; An information processing system comprising:

18. An information processing system including an information processing device having a storage unit that stores a plurality of trained models including a first trained model that classifies data to be inferred into a class belonging to a first class hierarchy, and a second trained model that classifies data to be inferred into a class belonging to a second class hierarchy that is a class hierarchy lower than the first class hierarchy, and an information terminal that is capable of communicating with the information processing device via a network, The information processing device includes: An acquisition unit that acquires information related to inference; A selection unit that selects a plurality of trained models in a hierarchical relationship based on information about the inference acquired by the acquisition unit; A notification unit that notifies the information terminal of the plurality of trained models selected by the selection unit; An information processing system comprising:

19. The information terminal includes a display unit that displays a plurality of trained models notified by the notification unit; The information processing system according to claim 17 or 18, further comprising an operation unit for performing an operation to select a trained model for inference based on the notified plurality of trained models.

20. The information processing system according to claim 19, characterized in that the acquisition unit in the information processing device acquires selection information of a trained model selected by the operation unit, and selects a plurality of trained models in a hierarchical relationship based on the selection information.

21. The information processing device further includes an inference unit that performs inference using a plurality of trained models in the selected hierarchical relationship, The information processing system according to any one of claims 17 to 20, characterized in that the inference unit performs inference in order from a trained model that classifies a class belonging to a higher hierarchy among the multiple trained models in the selected hierarchical relationship.

Images