Information processing system, information processing device, information processing method and program

Abstract

To provide an information processing system which can secure confidentiality of an inference model which executes inference on data to be inferred, while protecting privacy of the data to be inferred.SOLUTION: In an information processing system, a first information processing device is configured of: a first inference unit which is configured of an input layer, and at least some of intermediate layers, and performs first inference processing with respect to medical data to be inferred, by using a first partial model corresponding to a plurality of second partial models; and a first output unit which outputs a result of the first inference processing and selection information to a second information processing device. The second information processing device is configured of a second inference unit which performs second inference processing by inputting the first inference result with respect to the second partial model selected from the plurality of second partial models on the basis of the selection information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an information processing device, an information processing method, and a program that perform inference processing using an inference model. [Background technology]

[0002] Systems are known that apply machine learning techniques to medical data, such as medical images acquired by medical imaging devices (modalities) and medical information acquired from medical information systems, to make inferences about specific diseases (disease detection, benign / malignant distinction, prognosis prediction, risk prediction, etc.).

[0003] Non-patent document 1 discloses a technology in which, when an image is input, image compression is performed by a first information processing device located locally, and the compressed results are transferred to a second information processing device located in a remote cloud service to perform disease segmentation.

[0004] Non-Patent Document 2 discloses a technology in which a model consisting of an encoder and decoder is trained using a medical image as input, and feature extraction is performed using an inference model (encoder section). [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Zihao Liu et al., “Orchestrating Medical Image Compression and Remote Segmentation Networks”, International Conference on Medical Image Computing and Computer-Assisted Intervention-MICCAI2020. [Non-patent document 2] Behzad Bozorgtabar et al., “SALAD: Self-Supervised Aggregation Learning for Anomaly Detection on X-Rays”, International Conference on Medical Image Computing and Computer-Assisted Intervention-MICCAI2020. Summary of the Invention [Problem to be solved by the invention]

[0006] In the technology described in Non-Patent Document 1, when performing segmentation of a medical image, the medical image, which is the data to be inferred, is transmitted to a cloud service. In this case, the data to be inferred, such as the medical image, is transmitted from within the hospital to an external information processing device, making it difficult to protect the privacy of the data to be inferred.

[0007] On the other hand, as in the technology described in Non-Patent Document 2, there are cases where an inference model is built on an information processing device managed by the user who performs the inference. In such cases, it is difficult to prevent the user from copying the inference model and distributing it to a third party, or from unauthorized use such as modifying the inference model, so the confidentiality of the inference model cannot be guaranteed.

[0008] The present invention aims to provide an information processing system, information processing method, and program that can protect the privacy of the data to be inferred while ensuring the confidentiality of the inference model that performs inference on the data to be inferred. [Means for solving the problem]

[0009] In order to solve the above problem, an information processing system according to one aspect of the present invention includes a first information processing device and a second information processing device capable of communicating with the first information processing device via a network, An information processing system that performs inference processing on data to be inferred acquired by a first information processing device using a trained inference model based on a multi-layer neural network, wherein the inference model is configured by combining a first partial model, a second partial model selected from a plurality of second partial models, and a third partial model corresponding to the second partial model, the first partial model being configured with an input layer of the inference model and a part of an intermediate layer, each of the plurality of second partial models being configured with a layer different from the intermediate layer included in the first partial model among the intermediate layers of the inference model, and the third partial model being configured with an output layer of the inference model;The first information processing device includes a first acquisition unit that acquires medical data to be inferred and selection information of a partial model to be applied to the medical data to be inferred; , recommended The medical data processing device includes a first inference unit that performs a first inference process on the medical data to be analyzed, a third inference unit, and a first output unit that outputs the result of the first inference process and selected information to a second information processing device, , th One inference Processing a second acquisition unit for acquiring the result and selection information; , th The second inference part and The first inference unit performs a first inference process on medical data to be inferred using a first partial model corresponding to the plurality of second partial models, the second inference unit performs a second inference process by inputting a result of the first inference process to a second partial model selected from the plurality of second partial models based on the selection information, and the third inference unit performs a third inference process by inputting a result of the second inference process to a third partial model corresponding to the second partial model. . [Effects of the Invention]

[0010] According to the present invention, it is possible to protect the privacy of the data to be inferred while ensuring the confidentiality of the inference model that performs inference on the data to be inferred. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram showing the configuration of an information processing system according to a first embodiment. [Figure 2] Schematic diagram of an inference model according to the first embodiment. [Figure 3] FIG. 2 is a diagram showing the hardware configuration of a first information processing apparatus according to the first embodiment. [Figure 4] FIG. 3 is a flowchart showing an inference process of the information processing system according to the first embodiment. [Figure 5] FIG. 10 is a flowchart showing a modified example of the inference process of the information processing system according to the second embodiment. [Figure 6] FIG. 10 is a diagram showing the configuration of an information processing system according to a third embodiment. [Figure 7] Schematic diagram of an inference model according to the third embodiment. [Figure 8] FIG. 11 is a flowchart showing an inference process of an information processing system according to a third embodiment. [Figure 9] FIG. 10 is a diagram showing the configuration of an information processing system according to a fourth embodiment. [Figure 10] Schematic diagram of an inference model according to the fourth embodiment. [Figure 11] FIG. 10 is a flowchart showing a learning process of an information processing system according to a fourth embodiment. [Figure 12] FIG. 10 is a schematic diagram of an information processing system according to a fifth embodiment. [Figure 13] Schematic diagram of an inference model according to the fifth embodiment. [Figure 14] FIG. 13 is a flowchart showing a learning process of the information processing system according to the fifth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] The present invention is preferably applicable to medical data such as raw data (signal data) acquired by a modality and diagnostic medical data generated from raw data by image reconstruction. Modalities include, for example, X-ray CT scanners, MRI scanners, SPECT scanners, PET scanners, and electrocardiographs. Note that the data to be inferred and the training data may include not only medical data but also information related to patient privacy, such as age, gender, and disease information.

[0013] Hereinafter, the inference process using an inference model in the information processing system of the present invention will be described in the first, second, and third embodiments. Furthermore, the learning process of the inference model in the information processing system of the present invention will be described in the fourth and fifth embodiments. As will be described later, the inference model used in the inference process is not limited to the inference model generated through the learning process of the fourth and fifth embodiments of the present invention. The inference model used in the inference process is a trained inference model trained based on machine learning or deep learning using a known method or the learning process described in the present invention. Here, the trained inference model is only required to have undergone a learning process to satisfy predetermined conditions, and the trained inference model may be used as the target of additional learning, transfer learning, fine tuning, etc. Therefore, the learning process using the learning process described below may be performed as additional learning of a trained inference model trained using a known method, or the learning process may be performed in the reverse order.

[0014] Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings.

[0015] [First embodiment] An information processing system 1 of the present invention will now be described with reference to Fig. 1. The information processing system 1 of the present invention is composed of a first information processing device 2, a second information processing device 3, and a network 4 that connects both information processing devices so that they can communicate with each other.

[0016] Here, with reference to FIG. 2, the configuration of an information processing system 1 according to the present invention will be described. The information processing system 1 is composed of a first information processing device 2 and a second information processing device 3. The first information processing device 2 and the second information processing device 3 each have a partial model, which is part of an inference model that performs inference processing on medical data to be inferred and outputs the execution result. The second information processing device 3 also has multiple partial models. Each of the multiple partial models has different layers and parameters subsequent to the layer constituting the partial model in the first information processing device 2, and is a partial model that has been trained to classify different tasks and classes. Furthermore, the partial model in the first information processing device 1 corresponds to two or more of the multiple partial models in the subsequent stages. The inference model here is a trained model based on a neural network composed of an input layer, a middle layer, and an output layer. Parameters for outputting inference results are determined through training processing, and a model that pairs the parameters with a network model is defined as an inference model.

[0017] The first information processing device 2 executes a first inference process using a first partial model composed of an input layer and at least a part of the intermediate layers among the trained inference models described above. The second information processing device 3 executes a second inference process using a second partial model that matches the selected information among a plurality of second partial models composed of layers different from the first partial models among the trained inference models described above. The configuration of each information processing device will be described below.

[0018] Here, the first information processing device 2 is an information processing device that can be operated by a user of the inference model who has the authority to manage the medical data to be inferred, for example, a medical professional. On the other hand, the second information processing device 3 is an information processing device owned by a model provider who has the authority to manage the inference model used for inference. The second information processing device 3 exists on a server external to the first information processing device 2 and is configured to be able to communicate via a network 4.

[0019] The first information processing device 2 includes a first acquisition unit 11 that acquires medical data to be inferred and selection information for selecting a plurality of second partial models to be applied to the medical data to be inferred. The first information processing device 2 further includes a first inference unit 12 that performs a first inference process on the medical data to be inferred using a first partial model corresponding to a plurality of second partial models composed of layers different from those constituting the first partial model, among trained inference models based on a neural network composed of an input layer, an intermediate layer, and an output layer, which performs inference processing on the medical data. Here, the plurality of second partial models are trained partial models trained with parameters of a specific trained inference model fixed during training. Here, the specific trained inference model is, for example, an inference model trained to classify classes correlated with a plurality of second partial models, or a trained inference model generated by training one of a plurality of second partial models. The selection information for the second partial model is selected by user input based on information on the classes classified by each of the partial models. The selection information may be set by simply selecting or inputting a class that the user wants to infer, or the order in which inference processes are applied may be set among multiple partial models, and selection information for selecting multiple partial models may be set in response to the user's input or selection of some of the classes to be inferred or inference tasks.

[0020] The first information processing device 2 further includes an output unit 13 that outputs the first inference result from the first inference process and the selection information acquired by the first acquisition unit 11 to the second information processing device 3. The second information processing device 3 has a plurality of second partial models composed of layers different from the layers that make up the first partial models. The first information processing device 2 also has a memory unit 10 that stores the first partial models that are part of the trained inference model and medical data to be inferred. It also has an inference result acquisition unit 14 that acquires the result of the second inference by the second information processing device 3, which is another information processing device, and a display control unit 15 that displays the acquired inference result on a display device.

[0021] (Storage unit 10) The storage unit 10 stores a first partial model including an input layer of a trained inference model and medical data to be inferred. The storage unit 10 stores, as the first partial model, a network corresponding to the partial model and trained parameters corresponding to the network, in association with each other. The medical data to be inferred may be medical data automatically transferred from a modality or an external image server. Note that a portion of a trained inference model refers to a continuous portion from one layer to another, but is not limited to this, and may also be a continuous portion from one neuron to another neuron or an isolated neuron. The partial model may also be multiple non-adjacent portions of the trained inference model.

[0022] (First Acquisition Section 11) The first acquisition unit 11 acquires medical data to be inferred from the storage unit 10, and further acquires selection information of a plurality of second partial models to be applied to the medical data to be inferred. The first acquisition unit 11 then transmits the acquired medical data to be inferred and the selection information to the first inference unit 12.

[0023] (First Inference Part 12) The first inference unit 12 acquires a first partial model from the storage unit 10 and performs a first inference using the first partial model on the medical data to be inferred. The first inference unit 12 also transmits a first inference result using the first partial model to the output unit 13. In this embodiment, the first partial model is a first partial model that is composed of an input layer and at least a part of the intermediate layers among the trained inference models and corresponds to a plurality of second partial models in the second information processing device 3, and transmits an output from the intermediate layer to the output unit 13. Here, the output from the intermediate layer is tensor information, and if the inference model is a model based on CNN, the output is a feature map.

[0024] (Output section 13) The output unit 13 transmits the first inference result and the selection information to another information processing device, that is, the second information processing device 3. Here, the selection information is information indicating the multiple second partial models to be applied and information including the application order of the multiple second partial models.

[0025] (Inference result acquisition unit 14) The inference result acquisition unit 14 acquires the inference result of the second inference process for the medical data to be inferred from the second information processing device 3. When the inference result acquisition unit 14 acquires the inference result, it transmits the inference result to the display control unit 15.

[0026] (Display control unit 15) The display control unit 15 controls the display of the inference results acquired by the inference result acquisition unit 14 on a display device. The display device may be a display attached to the information processing device or a mobile terminal of a hospital staff member via an external server. Here, the first information processing device 2 may be configured by a computer including a processor, memory, storage, etc. In this case, a program stored in the storage is loaded into the memory and executed by the processor, thereby realizing the functions and processing of the memory unit 10, first acquisition unit 11, first inference unit 12, output unit 13, inference result acquisition unit 14, display control unit 15, etc. However, this configuration is not limited thereto. For example, all or part of the configuration of the first information processing device 2 may be realized by a dedicated processor (such as an ASIC) or FPGA. Alternatively, part of the arithmetic processing may be executed by a processor such as a GPU or DSP. Furthermore, the first information processing device 2 may be configured by a single piece of hardware or multiple pieces of hardware. For example, the functions and processing of the first information processing device 2 may be realized by multiple computers working together using cloud computing or distributed computing.

[0027] 3 shows an example of a specific configuration of the first information processing device 2. In this example, the first information processing device 2 has a CPU 20, a GPU 21, a RAM 22, a ROM 23, and a storage device 24, which are connected via a system bus 25. In addition, the first information processing device 2 is connected to a display device 26 and an input device 27 such as a mouse and a keyboard.

[0028] By configuring the first information processing device 2 as described above, the user of the inference model, who is the administrator of the medical data, can obtain multiple inference results as needed while protecting the privacy of the medical data, without having to transmit the medical data to be inferred to an external information processing device. Furthermore, even when multiple inference results are desired, since the first partial model is a partial model corresponding to multiple second partial models, it is possible to reduce the number of communications between information processing devices and the data volume, and further reduce the demand for machine resources by performing some of the inference processing on another information processing device.

[0029] Furthermore, the provider of the inference model can ensure the confidentiality of the inference model by installing only a portion of the inference model in the first information processing device 2. Furthermore, since the provider of the inference model provides the user with the corresponding partial model for multiple second partial models, confidentiality can be ensured regarding the network structure and parameters related to the second partial models.

[0030] Here, the second information processing device 3 resides on a server external to the first information processing device 2 and includes a second acquisition unit 70 that acquires the results of the first inference processing and selection information for multiple second partial models to be applied from the first information processing device 2. Furthermore, the second information processing device 3 includes a second inference unit 72 that performs a second inference processing using, as input, a result of the first inference processing of the trained inference model for the medical data to be inferred, the first partial model being composed of an input layer and at least some of the intermediate layers, among trained inference models based on a neural network composed of an input layer, an intermediate layer, and an output layer, for performing an inference processing on medical data, and using multiple second partial models composed of layers different from those constituting the first partial model in the inference processing. The second information processing device 3 also includes a storage unit 71 that stores the second partial models.

[0031] In this embodiment, each of the multiple second partial models is composed of an output layer and an intermediate layer that is different from the intermediate layer that constitutes the first inference model among the trained inference models. Furthermore, each of the multiple second partial models is a partial model that differs in at least one of the inference task that it performs and the class that it infers. Furthermore, the second partial models may have different network configurations in consideration of the properties of the classes, etc.

[0032] (Storage unit 71) The storage unit 71 stores a network corresponding to each of the plurality of second partial models and the learned parameters corresponding to the network in association with each other. Note that a partial model refers to a continuous portion from one layer to another, but is not limited to this and may also be a continuous portion from one neuron to another neuron or an isolated neuron. Furthermore, a partial model may be multiple non-adjacent portions of an inference model.

[0033] (Second inference part 72) The second inference unit 72 acquires a second partial model that matches the selection information from the storage unit 71 and performs a second inference using the second partial model on the medical data to be inferred. The inference result from the second inference is then transmitted to the first information processing device 2. Here, if the selection information includes multiple second partial models, the second inference is performed on the multiple second partial models using the first inference result as input. If the selection information also includes the application order of the second partial models, the inference process is performed in accordance with the application order. Furthermore, based on the inference result from the second partial model whose application order is i, it may be determined whether or not an inference is performed using the (i+1)th second partial model. The multiple second partial models selected by the selection information are partial models that classify correlated class groups. Each second partial model has a different inference task or class to infer. For example, there may be a second partial model that classifies the presence or absence of nodules, a second partial model that detects nodules, and a second partial model that extracts nodule regions. Furthermore, if the inference difficulty differs between class classification, detection, and extraction, the user may set the selection information to determine the application order according to the inference difficulty. Furthermore, each second partial model may be configured differently in the number of layers constituting the network, the depth, etc. Here, in this embodiment, since the multiple second partial models are models having an output layer, multiple inference results are output in the second information processing device. On the other hand, the multiple second partial models may be configured only with an intermediate layer, and the output from the intermediate layer may be transmitted to the first information processing device 2.

[0034] By configuring the second information processing device 3 as described above, the provider of the inference model can install only a portion of the trained inference model in the first information processing device 2 and keep the portion of the trained inference model within the second information processing device 3 that the provider owns and manages, thereby ensuring the confidentiality of the inference model. Furthermore, if the output layer is within the second information processing device 3, it is also possible to detect unauthorized use of an inference model including a second partial model based on the output from the output layer. In addition, by providing multiple second partial models for a first partial model, the user is able to reduce the effort required to create input data for each inference model. The above-described information processing system can obtain multiple inference results as needed while protecting the privacy of the medical data to be inferred and ensuring the confidentiality of the inference model that performs inference on the medical data to be inferred.

[0035] Here, the inference process performed by the information processing system 1 in this embodiment will be described with reference to FIG.

[0036] In step S40, the first acquisition unit 11 in the first information processing device 2 acquires the medical data of the inference target and selection information of the plurality of second partial models. Upon acquiring the medical data of the inference target and the selection information of the plurality of second partial models, the first acquisition unit 11 transmits the acquired medical data of the inference target to the first inference unit 12, and proceeds to the next step.

[0037] In step S41, the first inference unit 12 in the first information processing device 2 executes a first inference process on the data to be inferred, using a first partial model that is composed of an input layer and at least some of the intermediate layers and that corresponds to a plurality of second partial models that are composed of layers different from those that constitute the first partial model. After executing the first inference process, the first inference unit 12 transmits the inference result of the first inference process and selection information of the plurality of second partial models to the output unit 13, and proceeds to the next step.

[0038] In step S42, the output unit 13 in the first information processing device 2 outputs the inference result of the first inference and the selection information of the plurality of second partial models to the second information processing device 2, and the process proceeds to the next step.

[0039] In step S43, the second inference unit 72 of the second information processing device 3 performs a second inference process using multiple second partial models composed of layers different from those constituting the first partial model. In this embodiment, the second partial model is composed of an inference model including an intermediate layer different from the intermediate layer constituting the first partial model. The multiple second partial models also have an output layer, and each outputs an inference result for the data to be inferred. Each of the multiple second partial models is a partial model with a different inference class or inference task. Here, the multiple second partial models are trained partial models trained with the parameters of the first partial model fixed during training for a specific trained inference model. Here, the specific trained inference model is, for example, an inference model trained to classify classes correlated with multiple second partial models, or a trained inference model generated by training one of the multiple second partial models. After transmitting the inference result for the medical data to be inferred to the first information processing device 2, the second inference unit 72 proceeds to the next step. If inference is performed using multiple second partial models based on the selection information, the second inference unit 72 transmits the multiple inference results to the first information processing device 2. The second inference unit 72 may store the inference result of the second inference process in the storage unit 71 and present the inference result in response to external access. The output destination is not limited to the first information processing device 2, and the result may be sent to a specified information terminal or a contact.

[0040] In step S44, the inference result acquisition unit 14 in the first information processing device 2 acquires a second inference result, which is an inference result based on a plurality of second partial models, from the second inference unit 72 of the second information processing device 3. Upon acquiring the second inference result, the inference result acquisition unit 14 transmits the second inference result to the display control unit 15 and proceeds to the next step.

[0041] In step S45, the display control unit 15 causes the display device 25 to display the inference results for the medical data to be inferred. The inference results displayed on the display device 25 are the inference results obtained by performing a series of inference processes (first inference process and second inference process) on the medical data to be inferred by the first information processing device 2 and the second information processing device 3. Furthermore, if there are multiple inference results based on the second partial model, the display control unit 15 displays the multiple inference results in a comparable manner. For example, the display control unit 15 may display the multiple inference results side by side, switch between them, or display a composite image in which each inference result is superimposed on the same image. Furthermore, the display control unit 15 may display the inference results in association with the medical data to be inferred, or may display information about the inference model used for the inference.

[0042] Therefore, the information processing system 1 described in the present invention can perform multiple inferences as needed with reduced computational costs while protecting the privacy of the medical data to be inferred and ensuring the confidentiality of the inference model that performs inference on the medical data to be inferred. Furthermore, since the first partial model is a partial model that corresponds to multiple second partial models, it is possible to reduce the effort required to select multiple inference models to obtain multiple inference results and the effort required to process the inference data to match the inference models.

[0043] (Modification of the first embodiment) In the first embodiment, it was described that the second inference unit 72 performs inference using multiple second partial models that match the selection information based on selection information for multiple second partial models. It was also described that the second inference unit 72 performs inference using the second partial models in accordance with the application order of the second partial models included in the selection information. In this modified example, the inference result using the i-th second partial model is input to the i+1-th second partial model based on the application order. For example, a case will be described in which a brain tumor region is roughly extracted in the i-th second partial model, and then the brain tumor region is more finely extracted in the i+1-th second partial model. By inputting the brain tumor region extracted by the i-th second partial model as reference information into the i+1-th second partial model, the extraction accuracy of the brain tumor region in the i+1-th second partial model is improved.

[0044] In addition to the effects of the above-described embodiment, the configuration of this modified example makes it possible to perform inference processing with even higher accuracy by combining inference results of multiple second partial models.

[0045] [Second embodiment] In the first embodiment, the second inference unit 72 performs inference using multiple second partial models that match the selection information based on selection information for multiple second partial models. The second inference unit 72 also performs inference using the second partial models in accordance with the application order of the second partial models included in the selection information. In this embodiment, the second inference unit 72 further determines whether to perform inference processing using the (i+1)th second partial model based on the inference result using the i-th second partial model, which has the highest application order. If the second inference unit 72 determines not to perform inference processing using the (i+1)th second partial model as a result of this determination, it does not perform inference processing using the corresponding second partial model. This configuration reduces computational costs and the effort required for the user to compare multiple inference results. Below, the processing flow of the information processing system in this embodiment will be described using Figure 5. Note that descriptions of flows similar to those in the above-mentioned embodiments will be omitted as appropriate.

[0046] In step S53, the second acquisition unit 70 acquires the first inference result based on the first partial model and selection information of multiple second partial models from the first information processing device 2. The second acquisition unit 70 transmits both pieces of acquired information to the second inference unit 72. The second inference unit 72 determines the inference order of the second partial models in accordance with the application order of the models included in the acquired selection information, and performs inference processing using the i-th second partial model.

[0047] In step S54, the second inference unit 72 determines whether to perform inference processing using the (i+1)th or later partial models based on the result of the inference processing using the i-th second partial model. If the inference result using the i-th second partial model is related to the inference processing related to the (i+1)th or later partial models, the second inference unit 72 determines not to perform inference processing using other partial models based on the result of the i-th partial model. For example, a case will be described in which multiple second partial models included in the selection information are each partial models that classify, detect, or extract specific classes. In this case, if it is determined that there is no specific class using the second partial model that performs classification, the second inference unit 72 determines not to perform inference processing using the partial model that performs detection or extraction. If it is determined that there is no second partial model to perform inference on, the process proceeds to step S45 and ends this step.

[0048] With the configuration of this embodiment, even when a plurality of second partial models are selected by the selection information, in addition to the effects of the above-described embodiment, it is possible to perform inference processing using an appropriate partial model.

[0049] [Third embodiment] In the first and second embodiments, an information processing system was described that includes a first information processing device 2 having a first partial model corresponding to a plurality of second partial models composed of layers different from those constituting the first partial model, among trained inference models based on a neural network composed of an input layer, an intermediate layer, and an output layer, which performs inference processing on medical data, and a second information processing device 3 having a plurality of second partial models composed of layers different from those constituting the first partial model, among the trained inference models. In this embodiment, the first information processing device 2 further includes a third inference unit that performs inference using a plurality of third partial models including an output layer, and the configuration of an information processing system that can output to the first information processing device will be described using FIG. 6. The inference model in this embodiment will also be described using FIG. 7.

[0050] Here, differences from the first and second embodiments will be particularly described, and overlapping portions will be omitted as appropriate.

[0051] Similar to the first embodiment, the information processing system 1 in this embodiment is configured from a first information processing device 2, a second information processing device 3, and a network 4 that communicatively connects the information processing devices. The information processing system 1 is also configured from three partial models.

[0052] A first information processing device 2 on the user side has a trained inference model based on a neural network composed of an input layer, an intermediate layer, and an output layer, which performs inference processing on medical data, and the first partial model is composed of an input layer and some of the intermediate layers and corresponds to a plurality of second partial models composed of layers different from those constituting the first partial model, and further has a third partial model composed of some of the intermediate layers and an output layer. The third partial model is stored corresponding to each of the plurality of second partial models, and performs the third inference processing using as input the output from the intermediate layer which is the result of the inference processing from the second partial model.

[0053] On the other hand, the second information processing device 3 on the administrator side of the inference model has multiple second partial models consisting of at least a portion of the intermediate layers of the inference models between the intermediate layer of the first partial model and the intermediate layer of the third partial model. The number of partial models is variable, and it is sufficient that a partial model having an input layer and a partial model having an output layer are provided in the first information processing device 2 on the user side who has the authority to manage the medical data to be inferred. The number of first information processing devices 2 is also not important as long as they are information processing devices managed by the user side. Furthermore, the determination of whether to perform the (i+1)th inference based on the inference result of the i-th partial model, as described in the second embodiment, may be performed by the second inference unit 72 that receives output from the third inference unit 51. Alternatively, the third inference unit 51 may determine whether or not to perform inference processing for the (i+1)th partial model based on the output from the i-th third partial model corresponding to the i-th second partial model. Here, the determination of whether or not inference processing is necessary by the second inference unit 72 or the third inference unit 51 is a determination of the second partial model that matches the selected information and the third partial model that corresponds to the second partial model.

[0054] The information processing system 1 described in this embodiment not only ensures the confidentiality of the inference model for the medical data while protecting the privacy of the medical data to be inferred, but also ensures privacy regarding the inference results, since multiple inference results are output to the user. Furthermore, based on the inference results of the i-th partial model, the second inference unit 72 or the third inference unit 51 determines whether or not to perform inference processing using the (i+1)-th partial model, thereby reducing the computational cost of unnecessary partial models. Furthermore, even when updating parameters of partial models using techniques such as additional learning (described below), the correct answer data required to calculate the loss function does not need to be sent to an external device. For example, if the inference model performs image quality enhancement, if the output layer is located on an information processing device other than the user's, a high-quality image of the medical data to be inferred is generated by the second information processing device 3, making it difficult to protect the privacy of the medical data. Therefore, by having the first information processing device 2 have a partial model with an output layer and performing inference using the partial model, privacy regarding the output can be protected.

[0055] The information processing device 1 of this embodiment has at least a memory unit 10, a first acquisition unit 11, a first inference unit 12, an output unit 13, an inference result acquisition unit 14, a display control unit 15, and further a third inference unit 51 that performs a third inference using third partial models corresponding to each of the above-mentioned multiple second partial models.

[0056] The information processing device 2 according to this embodiment is configured to include a storage unit 71 and a second inference unit 72. Each of the multiple second partial models used in the second inference unit 72 is configured from a network made up of an intermediate layer between the intermediate layer of the first partial model and the intermediate layer of the third partial model among the trained inference models.

[0057] The inference process in this embodiment will be described below with reference to FIG.

[0058] Up to step S42, the flow is the same as in the first embodiment, so a description thereof will be omitted.

[0059] In step S73, the second information processing device 3 performs a second inference using a plurality of second partial models, using the inference result by the first inference unit 12 as input. Each of the plurality of second partial models is formed from a network made up of an intermediate layer located between the intermediate layer of the first partial model and the intermediate layer of the third partial model among the trained inference models. The second inference unit 72 transmits the output from the intermediate layer of the second partial model to the first information processing device as the result of the second inference, and then proceeds to the next step.

[0060] In step S74, the third inference unit 51 in the first information processing device 2 performs a third inference using a plurality of third partial models with the second inference result as input. The third inference unit 51 transmits the result of the third inference to the inference result acquisition unit 14 and proceeds to the next step.

[0061] In step S75, the inference result acquisition unit 14 acquires the third inference result as the inference result for the medical data that is the inference target, and transmits the acquired inference result to the display control unit 15.

[0062] Although explanation of step S45 will be omitted, the inference results displayed on the display device 25 are the inference results obtained by performing a series of inference processes (first inference process, second inference process, and third inference process) on the medical data to be inferred by the first information processing device 2 and the second information processing device 3.

[0063] This embodiment not only protects the privacy of the medical data to be inferred while ensuring the confidentiality of the inference model for the medical data, but also enables inference processing to be performed while protecting the privacy of multiple inference results.

[0064] Below, in the fourth and fifth embodiments, the learning process of the inference model in the present invention will be explained. Note that the inference model used in the inference process described above is not limited to the inference model generated through the learning process of the fourth and fifth embodiments. Furthermore, the inference model in the following embodiments may be an inference model that has not undergone learning processing, or a trained inference model that has undergone learning processing.

[0065] [Fourth embodiment] In the fourth embodiment, the configuration for learning an information processing system 800 of the present invention will be described with reference to Figure 9. The information processing system 800 is composed of a first information processing device 900 which is an information processing device on the user side of the inference model, a second information processing device 1000 which is an information processing device on the provider side of the inference model, and a network 1100 connecting the information processing devices. Also, with reference to Figure 9, the network configuration of the inference model corresponding to each information processing device will be described.

[0066] The first information processing device 900 performs inference processing on medical data. The first partial model is composed of an input layer and at least a portion of the intermediate layers of an inference model based on a neural network, the inference model being composed of the input layer, an intermediate layer, and an output layer. The second information processing device 1000 also has a plurality of second partial models composed of layers different from those constituting the first partial model. In this embodiment, the second partial model is a partial model composed of a portion of the intermediate layer and an output layer. By providing the first partial model including the input layer in the first information processing device 900 on the user side and providing a plurality of second partial models, which are part of the inference model, in the second information processing device 1000 on the inference model provider side, it is possible to perform learning processing of the inference model while protecting the privacy of the medical data and ensuring the confidentiality of the inference model. The first partial model may be configured as a public network, and the second partial model may be configured as a private network. The model provider can further enhance the confidentiality of the inference model by using the second partial model as a private network.

[0067] The first information processing device 900 has a memory unit 901 that stores teacher data and information about the inference model. It further includes a teacher data acquisition unit 902 that acquires teacher data from the memory unit 900, and a first learning unit 903 that learns a first partial model based on the acquired teacher data. The memory unit 900 may be configured as a storage device managed by a user of the inference model. When the learning process for the first partial model is completed, the first learning unit 903 stores information about the learned first partial model in the memory unit 901. If the first partial model corresponds to multiple second partial models, the parameters of the first partial model for which the learning process has been completed are not updated, and the parameters of the second partial model are learned and updated to generate multiple second partial models corresponding to multiple tasks or classes.

[0068] The second information processing device 1000 has a memory unit 1001 that stores information about an inference model, and a second learning unit 1002 that learns a second partial model.

[0069] Here, the learning process performed by the learning unit refers to forward propagating training data to a partial model and updating the parameters of the partial model using error information obtained by the backpropagation method. The training data is composed of training data and correct labels, where the training data is, for example, medical data, and the correct labels are information indicating the objects captured in the medical data. The correct labels may also be set as correct data indicating the object captured for each pixel. The first information processing device 900 can select an appropriate model even when multiple first and second partial models exist by transmitting model selection information to the second information processing device 1000.

[0070] An example of the learning process of the information processing system 800 in this embodiment will be described below with reference to the flowchart in FIG.

[0071] In step S50, the teacher data acquisition unit 902 acquires teacher data in which learning data and correct labels are paired from the storage unit 901. The teacher data acquisition unit 902 transmits information about the learning data to the first learning unit 903, transmits the correct labels to the second information processing device 1000, and then proceeds to the next step.

[0072] In step S51, the first learning unit 903 acquires the learning data transmitted from the teacher data acquisition unit 902 and information on the first partial model from the storage unit 901. Here, the first learning unit 903 may transmit information indicating the acquired first partial model to the second information processing device 1000.

[0073] In step S52 , the second learning unit 1002 acquires information on the second partial model from the storage unit 1001 and information on the correct label from the teacher data acquisition unit 902 .

[0074] In step S53, the first learning unit 903 inputs the learning data to the first partial model and performs forward propagation, thereby performing a first learning process, which is part of the learning process. When the first learning process is completed, the first learning unit 903 transmits data generated by the first learning process, such as tensors, to the second learning unit 1002.

[0075] In step S54, the second learning unit 1002 inputs the parameters transmitted from the first learning unit 903 into the second partial model and forward propagates them, thereby performing a second learning process that is part of the learning process.

[0076] In step S55, the second learning unit 1002 compares the output of the second partial model obtained by forward propagation using the second partial model including the output layer in its network configuration with the correct label and obtains error information using a loss function. The second learning unit 1002 also determines whether learning is complete. The second learning unit 1002 determines the end of the learning process based on whether the calculated error information is less than a predetermined value or whether the learning process has been performed a predetermined number of times. If the second learning unit 1002 determines that the learning process is complete, the flow ends. On the other hand, if it determines that the learning process should be continued, the process proceeds to step S56. Note that step S55 may be determined by the first learning unit 903 before the start of the first learning process.

[0077] In step S56, the second learning unit 1002 updates the parameters of the second partial model based on the error information calculated in step S55. Here, parameters refer to, for example, weights and biases. The second learning unit 1002 then transmits the error information from the intermediate layer close to the output layer to the input layer by backpropagation (backpropagation error method). The second learning unit 1002 transmits the error information from the intermediate layer close to the input layer that constitutes the second partial model to the first learning unit 903, and then proceeds to the next step.

[0078] In step S57, the first learning unit 903 updates the parameters of the first partial model based on the error information transmitted from the second learning unit 1002. After updating the parameters of the first partial model, the process proceeds to step S53. Note that updating the parameters is not essential; as described above, the parameters of the first partial model may be fixed during learning, and only the parameters of the second partial model may be learned. Here, the first partial model is, for example, an inference model trained to classify classes correlated with multiple second partial models, or a trained inference model generated by training one of the multiple second partial models. Note that, as described in step S55, the first learning unit 903 may determine the end of the learning process at this timing.

[0079] By configuring the information processing system 800 of the present invention in this manner, it is possible to train multiple inference models while ensuring the privacy of medical data and the confidentiality of the inference models. The number of partial models is not limited to two, and the learning process described in this embodiment is applicable as long as a partial model including an input layer is present in an information processing device on the user side of the inference model. Furthermore, if there are multiple inference models, information for selecting a partial model corresponding to each of the multiple inference models may be transmitted from the first learning unit 903 to the second learning unit 1002. The partial model may be selected by the user or by the information processing device in accordance with the input data.

[0080] [Fifth embodiment] In the fourth embodiment, a learning process in an information processing system has been described in which a first information processing device has a first partial model consisting of an input layer and at least a part of the intermediate layers, and a second information processing device has a second partial model consisting of a layer different from the first partial model.

[0081] In this embodiment, learning of an information processing system in which a first information processing device has a third partial model including at least an intermediate layer different from the first and second partial models, and an output layer will be described with reference to Fig. 12. The network configuration of the inference model in this embodiment will also be described with reference to Fig. 13. Note that explanations of parts that overlap with the third embodiment will be omitted as appropriate.

[0082] The information processing system 1200 of this embodiment is composed of a first information processing device 1300 which is an information processing device on the user side of the inference model, a second information processing device 1400 which is an information processing device on the provider side of the inference model, and a network 1100 connecting the information processing devices.

[0083] The first information processing device 1300 has a first partial model consisting of the input layer and at least a part of the intermediate layer of an inference model based on a neural network, which is composed of an input layer, an intermediate layer, and an output layer, and further has a third partial model including at least the output layer of the inference model.

[0084] The second information processing device 1400 has a second partial model consisting of at least a portion of the intermediate layers of the inference model.

[0085] According to the configuration of this embodiment, the first information processing device 1300 further has a third partial model including an output layer, so that the learning process of the inference model can be performed without transmitting the training data and correct answer labels that constitute the teacher data to the second information processing device 1400. Furthermore, the provider of the inference model can ensure the confidentiality of the inference model by having a second partial model that includes at least a part of the intermediate layers that constitute the inference model. Note that, in this embodiment, as long as the first information processing device 1300 has a partial model including an input layer and a partial model including an output layer, the number of partial models, the number of information processing devices, etc. can be designed as appropriate.

[0086] The first information processing device 1300 has a memory unit 1301 that stores teacher data and information on inference models. It also includes a teacher data acquisition unit 902 that acquires teacher data from the memory unit 1301, a first learning unit 1303 that learns a first partial model based on the acquired teacher data, and a third learning unit that learns a third partial model.

[0087] The second information processing device 1400 includes a memory unit 1001 that stores information about an inference model, and a second learning unit that learns a second partial model.

[0088] Here, the learning process refers to a series of processes that update the parameters of the partial models by forward propagating the learning data that constitutes the teacher data to the partial models and backpropagating (backpropagation) the error information between the correct label and the output value from the output layer. The teacher data is composed of the learning data and the correct label.

[0089] An example of the learning process of the information processing system 800 in this embodiment will be described below with reference to the flowchart in Fig. 14. Note that differences from Fig. 11 will be particularly described here.

[0090] In step S130, the third learning unit 1304 acquires information on the third partial model from the storage unit 1301 and information on the correct label from the teacher data acquisition unit 1302.

[0091] In step S132, the second learning unit 1402 inputs the parameters transmitted from the first learning unit 903 into the second partial model and performs forward propagation, thereby performing a second learning process which is part of the learning process.

[0092] In step S133, the third learning unit 1304 inputs the parameters transmitted from the second learning unit 1402 into the third partial model and forward propagates them, thereby performing a third learning process that is part of the learning process.

[0093] In step S134, the third learning unit 1304 compares the output of the third partial model obtained by forward propagation using the third partial model including the output layer in its network configuration with the correct label and acquires error information using a loss function. The third learning unit 1304 also determines whether learning is complete. The third learning unit 1304 determines the completion of learning by determining whether the calculated error information is less than a predetermined value or whether the learning process has been performed a predetermined number of times. If the third learning unit 1304 determines that the learning process has ended, the flow ends. On the other hand, if the third learning unit 1304 determines that the learning process should continue, the process proceeds to step S53. The determination of the end of the learning process in step S134 may be made by the first learning unit 1303 before the start of the first learning process.

[0094] In step S135, the third learning unit 1304 updates the parameters of the third partial model based on the error information calculated in step S134. Here, parameters refer to, for example, weights and biases. The third learning unit 1304 transmits the error information from the intermediate layer close to the output layer to the input layer by backpropagation. The third learning unit 1304 transmits the error information from the intermediate layer close to the input layer that constitutes the third partial model to the second learning unit 1402, and then proceeds to the next step.

[0095] In step S136, the second learning unit 1402 updates the parameters of the second partial model based on the error information transmitted from the third learning unit 1304. The second learning unit 1402 transmits the error information from the intermediate layer closer to the output layer side to the input layer side by backpropagation, and transmits the output from the intermediate layer closer to the input layer side to the first learning unit 1303, and then proceeds to the next step.

[0096] In step S137, the first learning unit 1303 updates the parameters of the first partial model based on the error information transmitted from the second learning unit 1402. As described above, since the parameters of the first partial model are trained to correspond to multiple second partial models, parameters generated by a specific learning process may be fixed as the parameters of the first partial model. The specific learning process refers to an inference model trained to classify classes correlated with multiple third partial models, or a trained inference model generated by training one of the multiple third partial models. Once the backpropagation of the error information using the updated or fixed parameters of the first partial model is completed, the process proceeds to step S53. As described in step S134, the first learning unit 1303 may determine the end of the learning process at this timing.

[0097] By configuring the information processing system 1200 of the present invention in this way, it is possible to perform training processes for multiple inference models while ensuring the privacy of medical data and the confidentiality of inference models. Furthermore, there is no need to transmit the training data and correct labels that make up the training data from the information processing device on the user side of the inference model, which further ensures the confidentiality of medical data.

[0098] Note that the inference model learned by the fourth and fifth embodiments may be used as the inference model that performs the inference processing in the first and second embodiments. The fourth and fifth learning processes are also effective as a method for additional learning of the inference model that performs the inference processing.

[0099] [Variation 1] In the above-described learning process, the partial model including the output layer is in the first information processing device and in the second information processing device.

[0100] In this modified example, we will explain a configuration in which, depending on the output of the inference model, it is set whether a partial model including an output layer is placed on an information processing device on the user side of the inference model or on the provider side of the inference model.

[0101] For example, if the output of an inference model is equivalent to the input training data, such as by increasing the resolution of the input data, the inference model including the output layer is placed on an information processing device on the user side of the inference model. On the other hand, if the output of an inference model is for classifying or detecting medical data, the inference model including the output layer is placed on an information processing device on the provider side of the inference model. By configuring partial models according to the output of the inference model, it is possible to perform the learning process while balancing machine resources and the confidentiality of the medical data. Note that configuring partial models according to the output of the inference model is also effective in the inference process. For example, a first partial model that performs a first inference may be selected from multiple first models according to the medical data to be inferred, or a second classification model that performs a second inference may be selected from multiple second partial models according to the medical data to be inferred.

[0102] [Variation 2] In the above-described learning process, the case where the learning process of the inference model is performed using the backpropagation method has been described.

[0103] In this modified example, we will explain a case where the learning process for an inference model is performed using a learning method other than the backpropagation method.

[0104] For example, methods such as Synthetic Gradient, which trains a model to estimate the gradient that will be obtained for each layer, or Feedback Alignment, which uses a fixed random matrix when backpropagating errors, or Target Prop, which propagates targets instead of errors, or any other method would be acceptable.

[0105] (Other Examples) The present invention can also be realized by executing the following process: software (programs) that realize the functions of the above-described embodiments are supplied to a system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device reads and executes the programs.

Claims

1. An information processing system comprising a first information processing device and a second information processing device capable of communicating with the first information processing device via a network, the information processing system performing inference processing on inference target data acquired by the first information processing device using a trained inference model based on a multi-layer neural network, the inference model is configured by combining a first partial model, a second partial model selected from a plurality of second partial models, and a third partial model corresponding to the second partial model, the first partial model is composed of an input layer and a part of an intermediate layer of the inference model, each of the plurality of second partial models is composed of an intermediate layer of the inference model that is different from an intermediate layer included in the first partial model; the third partial model is configured to include an output layer of the inference model, The first information processing device a first acquisition unit that acquires medical data to be inferred and selection information of a partial model to be applied to the medical data to be inferred; a first inference unit that performs a first inference process; a third inference unit that performs a third inference process; a first output unit that outputs a result of the first inference process and the selection information to a second information processing device; The second information processing device a second acquisition unit that acquires a result of the first inference process and the selection information; a second inference unit that performs a second inference process; the first inference unit performs the first inference processing on the medical data to be inferred using a first partial model corresponding to the plurality of second partial models; the second inference unit performs the second inference process by inputting a result of the first inference process to a second partial model selected from the plurality of second partial models based on the selection information; An information processing system characterized in that the third inference unit performs the third inference process by inputting the results of the second inference process to a third partial model corresponding to the second partial model.

2. a first information processing device and a second information processing device capable of communicating with the first information processing device via a network; The first information processing device includes: a first acquisition unit that acquires medical data to be inferred and selection information of a partial model to be applied to the medical data to be inferred; a first inference unit that performs a first inference process on the medical data to be inferred, using a first partial model that is composed of the input layer and at least a part of the intermediate layers of a neural network composed of an input layer, an intermediate layer, and an output layer, and that corresponds to a plurality of second partial models that are composed of layers different from those that compose the first partial model; a first output unit that outputs a result of the first inference processing and the selection information to a second information processing device; The second information processing device a second acquisition unit that acquires the first inference result and the selection information from the first information processing device; a second inference unit that performs a second inference process by inputting the first inference result into a second partial model selected from the plurality of second partial models based on the selection information, The information processing system is characterized in that the selection information is information including an application order of the plurality of second partial models.

3. The first information processing device further includes an inference result acquisition unit, 3. The information processing system according to claim 2, wherein the inference result acquisition unit acquires the inference result by the second inference unit.

4. The information processing system described in claim 2 or 3, characterized in that the multiple second partial models in the second information processing device include a second partial model including the output layer, and the second inference unit performs second inference using the second partial model.

5. 2. The information processing system according to claim 1, wherein the selection information includes an application order of the plurality of second partial models.

6. 6. The information processing system according to claim 1, wherein each of the plurality of second partial models is a partial model generated by additional learning with parameters of the first partial model fixed.

7. 7. The information processing system according to claim 1, wherein each of the plurality of second partial models is a partial model that differs in at least one of an inference task and a class to be inferred.

8. The second inference unit selects, from the plurality of second partial models, An information processing system as described in any one of claims 2 to 5, characterized in that an inference process is performed using a predetermined second partial model, and depending on the results of the inference process, it is determined whether or not to perform inference process using another partial model.

9. 9. The information processing system according to claim 8, wherein the second inference unit does not perform inference processing using other partial models depending on the result of the inference processing.

10. 10. The information processing system according to claim 1, wherein the output unit outputs an output from the intermediate layer that constitutes the first partial model to a second information processing device.

11. 9. The information processing system according to claim 8, wherein the second inference unit inputs the result of the inference process to another second partial model.

12. An information processing device as the second information processing device in an information processing system comprising a first information processing device and a second information processing device capable of communicating with the first information processing device via a network, the information processing device performing inference processing of inference target data acquired by the first information processing device using a trained inference model based on a multi-layer neural network, the inference model is configured by combining a first partial model, a second partial model selected from a plurality of second partial models, and a third partial model corresponding to the second partial model, the first partial model is composed of an input layer and a part of an intermediate layer of the inference model, each of the plurality of second partial models is composed of an intermediate layer of the inference model that is different from an intermediate layer included in the first partial model; the third partial model is configured to include an output layer of the inference model, The information processing device includes an acquisition unit that acquires, from the first information processing device, a result of a first inference process on the medical data of the inference target using the first partial model corresponding to the plurality of second partial models, and selection information of a partial model to be applied to the medical data of the inference target; a second inference unit that performs a second inference process by inputting a result of the first inference process to a second partial model selected from the plurality of second partial models based on the selection information; an output unit that outputs the result of the second inference process to the first information processing device that has a third partial model corresponding to the second partial model.

13. an acquisition unit that acquires, from another information processing device, a result of a first inference process on medical data to be inferred, using a first partial model that is composed of the input layer and at least some of the intermediate layers of a neural network composed of an input layer, an intermediate layer, and an output layer, and that corresponds to a plurality of second partial models composed of layers different from those that constitute the first partial model, and selection information of a partial model to be applied to the medical data to be inferred; a second inference unit that performs a second inference process by inputting the first inference result into a second partial model selected based on the selection information; and 2. An information processing apparatus according to claim 1, wherein the selection information is information including an application order of the plurality of second partial models.

14. An information processing method executed by a processor in an information processing system including a first information processing device and a second information processing device capable of communicating with the first information processing device via a network, the information processing system performing inference processing on inference target data acquired by the first information processing device using a trained inference model based on a multi-layer neural network, the information processing method being used in the second information processing device, the information processing method comprising: the inference model is configured by combining a first partial model, a second partial model selected from a plurality of second partial models, and a third partial model corresponding to the second partial model, the first partial model is composed of an input layer and a part of an intermediate layer of the inference model, each of the plurality of second partial models is composed of an intermediate layer of the inference model that is different from an intermediate layer included in the first partial model; the third partial model is configured to include an output layer of the inference model, an acquisition step in which the information processing device acquires, from the first information processing device, a result of a first inference process on medical data to be inferred, using the first partial models corresponding to the plurality of second partial models, and selection information of partial models to be applied to the medical data to be inferred; a second inference step of performing a second inference process by inputting a result of the first inference process to a second partial model selected from the plurality of second partial models based on the selection information; an output step of outputting a result of the second inference process to the first information processing device having a third partial model corresponding to the second partial model; An information processing method comprising:

15. An information processing method executed by a processor, comprising: an acquisition step of acquiring, from another information processing device, a result of a first inference process on medical data to be inferred, using a first partial model that is composed of the input layer and at least some of the intermediate layers of a neural network composed of an input layer, an intermediate layer, and an output layer, and that corresponds to a plurality of second partial models composed of layers different from those that constitute the first partial model, and selection information of the partial model to be applied to the medical data to be inferred; a second inference step of performing a second inference process by inputting the first inference result into a second partial model selected based on the selection information, The information processing method, wherein the selection information is information including an application order of the plurality of second partial models.

Images