Medical information processing device and medical information processing method

Abstract

To protect highly confidential patient medical data, as well as the intellectual property rights of models. [Solution] The medical information processing device comprises a first acquisition unit, an encryption unit, and a second acquisition unit. The first acquisition unit receives imaging data and applies a first part of an inference model to the imaging data to obtain an intermediate analysis result. The encryption unit encrypts the intermediate analysis result based on a homomorphic encryption scheme. The second acquisition unit transmits the encrypted intermediate analysis result to a remote server and processes it using a second part of the inference model to obtain an encrypted analysis result.

Description

Technical Field

[0001] The embodiments disclosed in this specification and the drawings relate to a medical information processing apparatus and a medical information processing method.

Background Art

[0002] Imaging data can be processed using models that perform estimation tasks such as image classification and image segmentation. Image classification is a process of classifying an image based on its content. Image segmentation is a process of dividing an image into a plurality of image segments based on the characteristics of each image segment such as color, intensity, and texture. These models are artificial intelligence methods using an architecture that includes one or more of neural networks, transformers, and encoders and decoders.

[0003] The encoder / decoder structure is an architecture commonly used in deep learning. The encoder takes in input data (e.g., a medical image) and creates a low-dimensional, efficient latent representation by extracting useful features. The size of the latent representation is constant, but the size of the input data varies. The decoder receives the new representation and generates the required output data (e.g., a segmentation mask). Also, the size of the output varies.

[0004] Image analysis models may be provided as a service by a third-party company. Since image analysis models are exclusive custom solutions, third parties may generally restrict access to these models by storing them on one or more dedicated servers.

[0005] Image analysis models are routinely used in medical contexts for diagnosing and detecting pathological features. These models are applicable to patient medical imaging data generated as part of clinical analysis procedures. Protecting patient medical data and safeguarding patient privacy are critical issues for healthcare providers. Therefore, healthcare providers are often reluctant to transfer patient medical data from secure servers.

[0006] Given the need to protect the privacy of patients' medical data and the intellectual property of third-party models, it is extremely difficult to safely analyze patients' medical data using models owned by third parties. [Prior art documents] [Patent Documents]

[0007] [Patent Document 1] Chinese Patent Application Publication No. 114093465 Specification [Overview of the Initiative] [Problems that the invention aims to solve]

[0008] One of the problems that the embodiments disclosed herein and in the drawings aim to solve is the protection of highly confidential patient medical data, as well as the protection of intellectual property rights of the model. However, the problems that the embodiments disclosed herein and in the drawings aim to solve are not limited to the above problem. Problems corresponding to the effects of each configuration shown in the embodiments described later can also be positioned as other problems. [Means for solving the problem]

[0009] The medical information processing device according to the embodiment comprises a first acquisition unit, an encryption unit, and a second acquisition unit. The first acquisition unit receives imaging data and applies a first portion of an inference model to the imaging data to obtain intermediate analysis results. The encryption unit encrypts the intermediate analysis results based on a homomorphic encryption scheme. The second acquisition unit transmits the encrypted intermediate analysis results to a remote server and processes them using a second portion of the inference model to obtain encrypted analysis results.

[0010] The configuration of the present invention will be better understood from the following detailed description, which is presented as an example with reference to the drawings. [Brief explanation of the drawing]

[0011] [Figure 1A] Figure 1A shows two schematic block diagrams illustrating known systems that securely transfer and analyze confidential imaging data between a trusted local site and an untrusted remote server, based on an encryption scheme. [Figure 1B] Figure 1B is a block diagram of a known system that securely transfers and analyzes confidential imaging data between a trusted local site and an untrusted remote server, based on a homomorphic encryption scheme and a two-part estimation model. [Figure 2] Figure 2 is a schematic diagram showing an example of the apparatus according to the embodiment. [Figure 3A] Figure 3A is a schematic block diagram showing an example of a system for performing estimation related to imaging data according to the embodiment. [Figure 3B] Figure 3B shows an example of segmentation results, including encryption analysis results and confidential analysis results, according to the embodiment. [Figure 4] Figure 4 is a schematic block diagram showing an example of a system that performs estimation on imaging data based on the UNET model according to the embodiment. [Figure 5] Figure 5 is a flowchart showing an example of a method for performing estimation on imaging data according to the embodiment. [Modes for carrying out the invention]

[0012] Based on the embodiment, a medical information processing device is provided comprising a first acquisition unit, an encryption unit, and a second acquisition unit. The first acquisition unit receives imaging data and applies a first part of an inference model to the imaging data to obtain intermediate analysis results. The encryption unit encrypts the intermediate analysis results based on a homomorphic encryption scheme. The second acquisition unit transmits the encrypted intermediate analysis results to a remote server and processes them using a second part of the inference model to obtain encrypted analysis results.

[0013] A medical information processing method is provided based on an embodiment. The method includes receiving imaging data, applying a first part of an inference model to the imaging data to obtain an intermediate analysis result, encrypting the intermediate analysis result based on a homomorphic encryption scheme, and transmitting the encrypted intermediate analysis result to a remote server and processing it using a second part of the inference model to obtain an encrypted analysis result.

[0014] Figure 1A shows two schematic block diagrams illustrating known systems for securely transferring and analyzing confidential imaging data between a trusted local site and an untrusted remote server, based on cryptographic schemes. The confidential imaging data is analyzed using estimation models. The upper part of Figure 1A shows a system with a non-homomorphic encryption scheme, and the lower part of Figure 1A shows a system with a homomorphic encryption scheme.

[0015] In this specification, imaging data refers to confidential imaging data relating to one or more individuals and is not intended to be made public. In one embodiment, confidential imaging data refers to medical imaging data relating to one or more patients. Medical imaging data is acquired as appropriate by imaging a patient as part of a clinical diagnostic procedure using techniques such as MRI, ultrasound, computed tomography (CT scan), X-ray, or whole-slide imaging. Imaging data may be 2D, 3D, or a video sequence. Furthermore, imaging data refers to raw imaging data obtained directly from the imaging device or raw imaging data that has undergone preprocessing.

[0016] An encryption scheme is a method of encoding the original data representation, in plaintext, into another form, ciphertext. Ciphertext is effectively a scrambled version of the plaintext. An encryption key is generated by an algorithm and used to convert the plaintext into ciphertext. A decryption key is also generated by an algorithm and used to decrypt the ciphertext. Ideally, the private key is only made public to authorized parties. Symmetric encryption is an encryption scheme that uses the same key for both encryption and decryption. Public-key encryption is an encryption scheme where the encryption key is different from the decryption key. The encryption key is made public, but the decryption key is kept secret.

[0017] Generally, ciphertext generated by conventional encryption schemes does not maintain the same structure as the plaintext. Therefore, decrypting the ciphertext after applying a function to it does not yield equivalent results compared to applying the same function to the plaintext. In contrast, homomorphic encryption encrypts the plaintext such that the resulting ciphertext structure corresponds to that of the plaintext. As a result, a limited number of functions can transform the homomorphically encoded ciphertext, which corresponds to the original plaintext. Decrypting the transformed ciphertext back into plaintext yields the same result as applying the same set of functions to the original plaintext. The limited number of functions that can be applied to homomorphically encrypted data include polynomial functions consisting only of addition and / or multiplication.

[0018] In this specification, an estimation model means a model or algorithm for deriving meaning from an image based on a set of input values. In particular, in the case of medical imaging data, an estimation algorithm means a model for identifying, classifying, segmenting, and / or quantifying patterns or pixels within an image for use in clinical analysis and / or diagnosis. For example, an estimation algorithm is an image segmentation algorithm that assigns a label to each pixel in an image, where each label corresponds to a type of cell or tissue. In another example, an estimation model is a classification model that determines whether an image is relevant to a clinical field, for example, whether an image corresponds to the presence or absence of cancer in a sample.

[0019] Refer to the upper part of FIG. 1A to which a non-homomorphic encryption method is applied. The first trusted local site 101 receives the first non-public image data 110. The first trusted local site 101 encrypts the first non-public image data 110 according to the public-key non-homomorphic encryption method to obtain the first encrypted data 111. The first trusted local site 101 further transmits the first encrypted data 111 to the first untrusted remote server 141. Also, the first trusted local site 101 transmits the decryption key to the untrusted remote server 141. The first untrusted remote server 141 decrypts the first encrypted data 111 to obtain the first non-public image data 110. Further, the first untrusted remote server 141 applies an estimation model to the first non-public image data 110 to generate the first non-public analysis result 112. The estimation model is an image classification model or an image segmentation model, etc. The first untrusted remote server 141 encrypts the first non-public analysis result 112 according to the public-key non-homomorphic encryption method to obtain the first encrypted analysis result 113. Also, the first untrusted remote server 141 transmits the first encrypted analysis result 113 to the first trusted local site 101. When a different encryption method from that of the local site 101 is used in the remote server 141, the remote server 141 separately transmits the decryption key to the local site 101. The first trusted local site 101 decrypts the first encrypted non-public analysis result 113 to obtain the first non-public analysis result 112.

[0020] Refer to the lower part of FIG. 1A to which the homomorphic encryption method is applied. The second trusted local site 102 receives the second non-public image data 120. The second trusted local site 102 encrypts the second non-public image data 120 according to the public key homomorphic encryption method to obtain the second encrypted data 121. Further, the second trusted local site 102 transmits the second encrypted data 121 to the second untrusted remote server 142. Also, the second untrusted remote server 142 applies the estimation model to the second encrypted data 121 to generate the second encrypted analysis result 123. The estimation model is, for example, an image classification model or an image segmentation model suitable for execution on homomorphically encrypted data. For example, the non-linear operations in the estimation model can be replaced with a polynomial approximation formula including only addition and multiplication. Also, the second untrusted remote server 142 transmits the second encrypted analysis result 123 to the second trusted local site 102. The second trusted local site 102 decrypts the second encrypted non-public analysis result 123 to obtain the second non-public analysis result 124.

[0021] It can be seen from the lower part of FIG. 1A that when encrypting non-public image data based on the homomorphic encryption method, the non-public analysis result can be generated in fewer steps than when using the non-homomorphic encryption method shown in the upper part of FIG. 1A. Further, when using the homomorphic encryption method, there is no need to exchange encryption keys between the trusted local site and the untrusted remote site.

[0022] An estimation model can be represented as a function that receives an image as input and outputs a predicted value for that image. The architecture of an estimation model is a modular structure, etc. For example, an estimation model can also be represented as two functions, or two parts. The estimation model can be divided into a first part and a second part. The first part of the estimation model receives input data, performs a first process on the input data to generate an intermediate analysis result, and the second part of the estimation model receives the intermediate analysis result, performs a second process on the intermediate analysis result to output a predicted value for the image as the final analysis result. Note that the first and second parts of the estimation model can each be represented as a function, and each function may contain two or more subfunctions. For example, an estimation model such as a neural network with many layers or functions can be divided into a first part and a second part, each containing multiple layers, or functions.

[0023] Figure 1B is a block diagram of a known system for securely transferring and analyzing confidential imaging data between a trusted local site and an untrusted remote server, based on a homomorphic encryption scheme and a two-part estimation model. The system components shown in Figure 1B are equivalent to those of the system shown in the lower part of Figure 1A, except that the untrusted remote server in Figure 1B is specifically configured to apply the estimation model as an autoencoder. The autoencoder has two functions: an encoder 150 and a decoder 160. The encoder 150 can be considered the first part of the estimation model, and the decoder 160 the second part of the estimation model. The encoder 150 converts the second encrypted data 121 into an intermediate analysis result 122. The intermediate analysis result 122 is a latent representation of the second encrypted data. The latent representation is typically lower dimensional than the second encrypted data 121. The decoder 160 converts the intermediate analysis result 122 into a reconstruction of the second encrypted data 121. The encoder 150 and decoder 160 are neural networks, respectively. The decoder 160 is trained to perform image segmentation so that its output includes homomorphically encrypted image segmentation information relating to the second non-public image data 120. The decoder's output is the second cryptographic analysis result 123.

[0024] As shown in Figure 1B, running the entire estimation model on homomorphic encrypted data presents a challenge due to its exorbitant cost. Both the encryption process itself and the development of the estimation model for homomorphic encrypted data are time-consuming and computationally expensive. Furthermore, only polynomial operations, such as addition and multiplication, can be performed on homomorphic encrypted data. This is problematic because operations such as activation functions, which are common characteristics of artificial intelligence methods, must be approximated by polynomial functions, potentially leading to a decrease in the accuracy of the estimation model.

[0025] Next, with reference to Figure 2, the apparatus 205 according to the embodiment will be described. Figure 2 is a schematic diagram showing an example of the apparatus according to the embodiment. The apparatus 205 may also be called a medical information processing device. The apparatus 205 processes imaging data such as non-public medical imaging data. The apparatus 205 applies a first part of the estimation model to the unencrypted imaging data to obtain an intermediate analysis result. Furthermore, the apparatus 205 encrypts the intermediate analysis result according to a homomorphic encryption scheme and transmits the encrypted intermediate analysis result to a remote server. At the remote server, a second part of the estimation model is applied to the encrypted intermediate analysis result. In other embodiments, the apparatus 205 may process any data as appropriate.

[0026] The device 205 includes an arithmetic unit 212. The arithmetic unit 212 is a computer, server, or the like. The arithmetic unit 212 can be connected to a display screen 216 or other display device. The arithmetic unit is also connected to one or more input devices 218, such as a computer keyboard or mouse. The arithmetic unit 212 receives data from a memory 240, which may also be called a data storage unit or storage. The memory 240 stores imaging data and a first part of the estimation model. The memory 240 also stores software for performing public-key homomorphic encryption.

[0027] In another embodiment, the computing unit 212 receives imaging data from one or more data storage units (not shown) instead of or in addition to the memory 240. For example, the computing unit 212 receives imaging data from one or more remote data storage units (not shown), including cloud-based storage.

[0028] The computing unit 212 includes a data processing circuit 222. The processing circuit 222 comprises a central processing unit (CPU) and a graphical processing unit (GPU). The processing circuit 222 provides processing resources for automatically or semi-automatically processing imaging data.

[0029] The processing circuit 222 includes an estimation circuit 224 for applying a first part of the estimation model to the imaging data, an encryption circuit 226 for encrypting the partially analyzed imaging data based on a homomorphic encryption scheme and transmitting the partially analyzed encrypted imaging data to a remote server for further analysis, and a decryption circuit 228 for receiving and decrypting the encrypted analysis results from the remote server. The estimation circuit 224 is an example of the first and second acquisition units. The encryption circuit 226 is an example of the encryption unit. The decryption circuit 228 is an example of the decryption unit.

[0030] In this embodiment, each circuit 224, 226, 228 is implemented in the CPU and / or GPU by a computer program that includes computer-readable instructions for performing a method according to the embodiment. In other embodiments, the circuits may be implemented as one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs).

[0031] Furthermore, the arithmetic unit 212 includes the computer's hard drive and other components, namely RAM, ROM, a data bus, an operating system including various device drivers, and hardware devices including a graphics card. For clarity, Figure 1 does not show these components.

[0032] Here, Figure 3A is a schematic block diagram showing an example of a system for performing estimations related to imaging data according to the embodiment. The apparatus 205 in Figure 2 can be implemented as part of a system 300 for secure transfer and analysis of non-public imaging data based on homomorphic encryption, as shown in Figure 3A. The system 300 comprises a trusted local site 301 and an untrusted remote server 341. The trusted local site 301 comprises an apparatus 305, which corresponds to apparatus 205. Apparatus 305 stores a first portion 350 of the estimation model. In one embodiment, apparatus 305 is implemented by two or more computing units, which are connected to each other via a secure network. In one embodiment, the trusted local site 301 is a computer network of a hospital, medical institution, or research institution.

[0033] The untrusted remote server 341 stores the second part 360 of the estimation model. In one embodiment, the untrusted remote server 341 is implemented by two or more computing units. In one embodiment, the untrusted remote server 341 is cloud-based. The trusted local site 301 does not have access rights to the untrusted remote server 341, and the untrusted remote server 341 does not have access rights to the trusted local site 301. In one embodiment, data transfer between the trusted local site 301 and the untrusted remote server 340 is performed using a file sharing service such as Dropbox, a secure file sharing service such as Box or FFP, or a network protocol such as SSH.

[0034] The combination of the first and second parts 350, 360 of the estimation model constitutes a function that takes imaging data as input and outputs predicted values. The first part 350 of the estimation model processes the imaging data before encryption. That is, the first part 350 of the estimation model includes non-polynomial functions such as activation functions. The second part 360 of the estimation model processes homomorphically encrypted data. Therefore, the second part 360 of the estimation model includes only addition and multiplication. In one embodiment, the second part 360 of the estimation model has a polynomial approximation formula for a function that includes non-polynomial operations. The polynomial approximation formula consists only of addition and / or multiplication.

[0035] In addition to or alternatively, in one embodiment, the estimation model is divided into a first part 350 and a second part 360 so that the proprietary model is applied only to encrypted data. For example, the first part 350 of the estimation model, hosted on a trusted local site 301, is a public model such as an open-source model. In one embodiment, the first part 350 of the estimation model may be a general-purpose feature extractor such as ResNet.

[0036] The second part 360 of the estimation model is a proprietary model hosted on the remote server 341 and is not publicly accessible. The second part 360 of the estimation model is a custom-made model that provides predictions for a specific type of medical imaging data. For example, the second part 360 of the estimation model is customized to perform a specific type of estimation task for a specific type of imaging data. Alternatively, for example, the second part 360 of the estimation model is customized to perform a segmentation task for MRI brain data.

[0037] In addition to or alternatively, in one embodiment, the estimation model is divided into a first part 350 and a second part 360 to perform computationally intensive tasks on unencrypted data. For example, the first part 350 of the estimation model, hosted on a trusted local site 301, has computationally intensive tasks that can be performed more quickly on raw image data. The second part 360 of the estimation model, hosted on a remote server 341, includes less computationally intensive operations to reduce the computational load required for homomorphic encrypted data analysis. The first part of the estimation model comprises a first set of neural network layers, and the second part of the estimation model comprises a second set of neural network layers. The first set of neural network layers is greater than the second set of neural network layers. In addition to or alternatively, the first part of the estimation model includes operations on a first number, and the second part of the estimation model includes operations on a second number. The first number is greater than the second number.

[0038] Figure 3A is examined in more detail. Device 305 receives imaging data 310, such as confidential medical imaging data. The imaging data 310 is generated by an imaging device connected to device 305. Alternatively, the imaging data 310 is generated by another imaging device that is part of a trusted local site 301. Alternatively, the imaging data 310 is generated at a location away from the trusted local site 301 and securely transmitted to a server at the trusted local site 301.

[0039] The device 305 applies the first part 350 of the estimation model to the imaging data 310 and obtains an intermediate analysis result 312. The device 305 also encrypts the intermediate analysis result 312 based on a homomorphic encryption scheme and obtains an encrypted intermediate analysis result 313. The homomorphic encryption scheme is a leveled homomorphic encryption scheme or a fully homomorphic encryption scheme, etc. The intermediate analysis result 312 is encrypted based on a public key generated by a public-key homomorphic encryption scheme. The private key that pairs with the public key is also generated based on the same public-key homomorphic encryption scheme used to generate the public key.

[0040] Homomorphic encryption schemes are based on the linear algebra problem LWE (Learning with Errors) and its extension, the Ring-LWE (RLWE) problem. This scheme is based on the solution to the system of linear equations As + e = b, where A and s are the secret key, s is a small error term, and b is the public key. In LWE, the matrix input is an integer modulo a prime number p. In RLWE, the input is a polynomial of degree (p-1). As the prime number increases, the scheme becomes more complex and secure, but also more complex and computationally expensive. Both the secret and public keys are obtained from valid vectors in the choice space by sampling uniformly or based on a selection distribution.

[0041] Device 305 stores the secret key. Furthermore, device 305 sends the encrypted intermediate analysis result 313 to an untrusted remote server 341.

[0042] In one embodiment, the public-key homomorphic encryption scheme is the CKKS (Cheon-Kim-Kim-Song) scheme. The CKKS encryption scheme enables addition and multiplication of encrypted data. The CKKS encryption scheme may also be implemented using libraries such as HEAAN (Cryptolab), SEAL (Microsoft), or HeLib (IBM).

[0043] In other embodiments, the public-key homomorphic encryption scheme is the BFV (Brakerski-Fan-Vaikuntanathan) scheme or the BGV (Brakerski-Gentry-Vaikuntanathan) scheme, which enables modulo operations on encrypted integers. In these embodiments, it is necessary to convert the values ​​of the input data into integer values. The BFV scheme can be implemented using libraries such as SEAL (Microsoft), PALISADE (New Jersey Institute of Technology), or Lattigo (Laboratory for Data Security). The BGV scheme can be implemented using libraries such as PALISADE (New Jersey Institute of Technology), HELib (IBM), or Lol (Crockett and Peikert).

[0044] The untrusted remote server 341 receives the encrypted intermediate analysis result 313 and applies the second part 360 of the estimation model to the encrypted intermediate analysis result 313 to obtain the encrypted analysis result 314. The untrusted remote server 341 transmits the encrypted analysis result 314 to the device 305. The device 305 decrypts the encrypted analysis result 314 using the secret key and obtains the confidential analysis result 315. Here, Figure 3B shows an example of a segmentation result including the encrypted analysis result 314 and the confidential analysis result 315 according to the embodiment.

[0045] In one embodiment, different servers are present in the medical setting, each performing different processing steps on the imaging data. For example, the first device 305 is an imaging server that applies a first part of an estimation model to the imaging data. Using the same or a different server 305 as the server that performed the image analysis, a public-private key pair is generated and the intermediate analysis results are encrypted. A different server 305 may be used to receive, decrypt, and store the encrypted analysis results 314. In this embodiment, a key exchange takes place between the server 305 that generated the public-private key pair and the server 305 that decrypts the encrypted analysis results 314. The server 305 that generated the public-private key pair sends the private key to the server 305 that decrypts the encrypted analysis results 314.

[0046] In an embodiment in which the imaging data 310 includes multiple different images 310, the apparatus 305 applies a first portion 350 of the estimation model to each of the images individually to obtain multiple intermediate analysis results 312. Furthermore, the apparatus 305 individually encrypts the multiple intermediate analysis results 312 and sends each encrypted intermediate analysis result 313 to an untrusted remote server 341. The untrusted remote server 341 applies a second portion 360 of the estimation model to each of the multiple encrypted intermediate analysis results 313 to obtain multiple encrypted analysis results 314. The apparatus 305 individually decrypts the multiple encrypted analysis results 314 to obtain multiple analysis results 315.

[0047] In this embodiment, the second portion 360 of the estimation model is applied to homomorphic encrypted data, but the second portion 360 of the estimation model can also be effectively applied to unencrypted data. For example, the second portion 360 of the estimation model can be applied to unencrypted intermediate analysis results 312, and results equivalent to analysis results 315 can be obtained.

[0048] The first and second parts of the estimation model are each trained using training data that includes medical images and corresponding ground truth data. During training, the parameters of each of the first and second parts of the estimation model are modified to minimize the loss function. For segmentation task training, the ground truth data includes contours or regions of the corresponding medical images that relate to specific tissue types or pathological features. For classification task training, the ground truth data includes classifications of the corresponding medical images. The ground truth data is defined manually by a clinical expert or generated by software. The first and second parts of the estimation model are trained separately. The first and second parts of the estimation model are trained using the same or different training datasets. The first part of the estimation model can be trained at a trusted local site or anywhere else. The second part of the estimation model is trained on an untrusted remote server. In one embodiment, the first part of the estimation model is trained at a trusted local site using private medical data.

[0049] In one embodiment, the estimation model is an autoencoder, where the first part 350 of the estimation model is an encoder and the second part 360 of the estimation model is a decoder. In this embodiment, the intermediate analysis result 312 is a latent spatial representation of the imaging data 310. The encrypted analysis result 314 is the encrypted segmented imaging data 310. The analysis result 315 is the segmented imaging data 310. The analysis result 315 also includes segmentation of the imaging data 310, which is MRI imaging data of the brain. The analysis result 315 includes an image region 320 corresponding to a predicted brain tumor. The estimation architecture disclosed herein can be modified to perform estimation on other types of medical imaging data. Multiple different characteristics are estimated from the data. For example, different types of abnormal tissue are estimated.

[0050] In another embodiment, the estimation model is a vision transformer (not shown). The vision transformer is described in "Dosovitskiy, Alexey, et al. "An image is worth 16x16 words: Transformers for image recognition at scale." arXiv preprint arXiv:2010.11929(2020)". Such a vision transformer receives an image as input and generates an image classification as output.

[0051] The vision transformer includes a first module that receives an image as input and generates image patches. The vision transformer also includes a second module that generates patch embeddings and positional embeddings corresponding to the patches in the input image. Furthermore, the vision transformer includes a transformer encoder that receives the patch embeddings and positional embeddings of the input image and outputs a latent spatial representation of the patch embeddings and positional embeddings. Finally, the vision transformer includes a multilayer perceptron (MLP) head that receives the latent spatial representation and outputs a classification of the image data.

[0052] A vision transformer according to one embodiment is divided into a first part 350 and a second part 360 as follows. The first part 350 includes a first module that receives imaging data 310 and generates image patches, a second module that receives image patches and generates patch embeddings and positional embeddings, and a transformer encoder that receives the patch embeddings and positional embeddings and generates latent spatial representations of the patch embeddings and positional embeddings. The second part 360 includes an MLP head that receives the latent spatial representations of the patch embeddings and positional embeddings and outputs a classification. In this embodiment, the intermediate analysis result 312 is the latent spatial representation of the patch embeddings and positional embeddings of the imaging data 310. The encrypted analysis result 314 includes the encrypted classification of the imaging data 310. The analysis result 315 includes the classification of the imaging data 310.

[0053] In one embodiment, a first part 350 of the estimation model comprises a general-purpose transformer, and a second part 360 of the estimation model comprises a custom MLP head specialized for the encrypted latent space of medical imaging data. Since the transformer is generally trained on a large amount of publicly available datasets, the general-purpose transformer can be effectively applied to imaging data. In contrast, the MLP head is fitted by a third party to a specific classification problem. Confidentiality can be maintained by hosting the MLP head on an untrusted remote server 341.

[0054] The transformer encoder is stored on a trusted local site and operates on unencrypted raw data. In contrast, a more custom, project-specific MLP head is stored on a remote server. The transformer encoder consists of L pairs of identical layers. Each pair includes a self-aware layer that considers adjacent patches and extracts features from each patch, and a feedforward layer that non-linearly transforms the output of the self-aware layer. Similar to the encoder / decoder architecture described above, incorporating a deep encoder into the transformer makes it easier to solve more complex problems, but immediately leads to a significant increase in computational cost. The MLP head used for classification is typically a simple network consisting of a small number of layers. The types of layers included in an MLP are fully connected layers and activation functions.

[0055] In one embodiment, the vision transformer is applied to segmentation rather than classification. In this embodiment, the MLP head is replaced with a decoder or upsampling network, and the analysis result 315 includes segmentation of the imaging data 310.

[0056] In one embodiment, the transformer is trained on a larger, more general dataset and then fine-tuned for a specific task application using high-quality data highly relevant to the estimation task being performed. The transformer may be fine-tuned at a local site using private data, including relevant medical imaging data and corresponding ground truth data. However, for security reasons, it is preferable that models fine-tuned with private hospital data not be moved from the local site. The MLP head (or decoder and upsampling network) is first trained on a larger, more general dataset and then fine-tuned for a specific task application using high-quality data. Since the classification MLP head (or segmentation decoder and upsampling network) resides on a remote server, it cannot be fine-tuned using private data from a secure site. Instead, the MLP head (or decoder and upsampling network) is fine-tuned using the model owner's data, including medical imaging data and corresponding ground truth data.

[0057] Here, Figure 4 is a schematic block diagram showing an example of a system that performs estimation on imaging data based on a UNET model according to an embodiment. In other embodiments, the estimation model is an image segmentation model such as UNET. UNET is described below: "Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. "U-net: Convolutional networks for biomedical image segmentation.", Medical Image Computing and Computer-Assisted Intervention - MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18. Springer International Publishing, 2015." In these embodiments, the architecture of the estimation model comprises a contraction path including an encoder layer and an expansion path including a decoder layer. In such embodiments, the first part 350 of the estimation model comprises encoder layers 450a-450n. The second part 360 of the estimation model comprises decoder layers 460a-460n. The first encoder layer 450a receives the imaging data 310 and outputs a latent representation of the imaging data 310. This latent representation of the imaging data 310 is called the intermediate analysis result 512a. Each of the second to nth encoder layers 450b-450n receives an input based on the output of the preceding encoder layer and outputs a latent representation of the input. These latent representations of the input are called intermediate analysis results 412b-412n. The device 305 individually encrypts the intermediate analysis results 412b-412n to obtain a set of encrypted intermediate analysis results 413a-413n.

[0058] The remote server 341 receives multiple encrypted intermediate analysis results 413a-413n, respectively. Each decoder layer 460a-460n receives the corresponding encrypted intermediate analysis result 413a-413n from the multiple encrypted intermediate analysis results 413a-413n obtained by an architecture known as skip connection. Each decoder layer 460a-460n also receives the output from the subsequent decoder layers, with the exception of the last decoder layer 460n. Each decoder layer 460a-460n generates an output (not shown) based on its own input. The last decoder layer 460n does not transform its input, but decoder layers 460a through 460n-1 transform their respective inputs. For example, decoder layer 460n receives only 413n and generates an output. Decoder layer 460b receives the outputs of 413b and decoder layer 460c as input and generates an output by transforming those inputs. The decoder layer 460a receives the outputs of 413b and decoder layer 460b as input and generates an output by transforming the input. The output of the first decoder layer 460a includes an encrypted analysis result 314 obtained by encrypting the segmented imaging data 310. The analysis result 315 is the segmented imaging data 310.

[0059] In one embodiment, each encoder layer 450a-450n comprises two consecutive 3x3 convolutions followed by a normalized linear unit (ReLU) function. The second through nth encoder layers 450b-450n each receive the output of a preceding encoder layer, to which a 2x2 maximum pooling operation with a stride of 2 is applied, with twice the number of feature channels. Each skip layer has a copy / crop function. The first through (n-1)th decoder layers 460a-460n-1 each receive the output from a subsequent decoder layer to which a 2x2 ascending convolution that halves the number of feature channels is applied. The first through (n-1)th decoder layers each comprise two consecutive 3x3 convolutions followed by a normalized linear unit (ReLU) function. The first decoder layer 460a further comprises a 1x1 convolution layer.

[0060] In other embodiments, the encrypted intermediate analysis results 313 are hosted on a data custodian server. The encrypted intermediate analysis results 313 are made public for research or commercial purposes. The encrypted intermediate analysis results 313 may be provided to customers or users as a service. In one embodiment, a customer's device queries this data using an API. Because the encrypted intermediate analysis results 313 are based on homomorphic encryption, the encrypted intermediate analysis results 313 can be analyzed while protecting patient confidentiality.

[0061] In another embodiment, to improve the computational efficiency of device 305 during homomorphic encryption, device 305 utilizes a cryptographic accelerator. A cryptographic accelerator is a coprocessor specifically designed to perform cryptographic operations intensively. In yet another embodiment, to improve computational efficiency on encrypted data from an untrusted server, the remote server utilizes an accelerator. An accelerator is a coprocessor specifically designed to perform computationally intensive tasks on homomorphic encrypted data. In addition to or alternatively, a graphical processing unit (GPU) may be implemented at the trusted local site 301 and / or the remote server 341 to reduce execution time.

[0062] Here, Figure 5 is a flowchart showing an example of a method for performing estimation on imaging data according to the embodiment. The method for securely transferring and analyzing imaging data will be described below with reference to Figure 5. In step 510, the device 305 receives the imaging data 310. In step 520, the device 305 applies the first part 350 of the estimation model to the imaging data 310 and obtains an intermediate analysis result 312. In step 530, the device 305 generates a pair of public and private keys based on a homomorphic encryption scheme. The device 305 encrypts the intermediate analysis result 312 using the public key and obtains an encrypted intermediate analysis result 313. In step 540, the device 305 sends the encrypted intermediate analysis result 313 to an untrusted remote server 341.

[0063] In step 550, the untrusted remote server 341 receives the encrypted intermediate analysis result 313. In step 560, the untrusted remote server 341 applies the second part 360 of the estimation model to the encrypted intermediate analysis result 313 and obtains the encrypted analysis result 314. In step 570, the untrusted remote server 341 transmits the encrypted analysis result 314 to the device 305.

[0064] In step 580, device 305 receives the encryption analysis result 314. In step 590, device 305 decrypts the encryption analysis result 314 using the private key and obtains the non-public analysis result 315. Alternatively, if the device 305 performing decryption is different from the device 305 performing encryption, the private key is first sent from the device 305 that generated the private key-public key pair to the device 305 performing decryption.

[0065] In step 600, the device 305 outputs the confidential analysis results 315. The confidential analysis results 315 may optionally be displayed on the monitor for examination by clinicians or researchers.

[0066] The system and method described herein are advantageous because they provide a method for analyzing patient data encrypted based on homomorphic encryption using a proprietary model. This method reduces both execution time and computational resource utilization compared to methods that apply estimation methods only to homomorphic encrypted data. The system and method described herein enable the protection of intellectual property rights of proprietary models and patient privacy. This is achieved by splitting the estimation model between a trusted local site and an untrusted remote server so that only the calculations necessary for encrypted data are performed. For example, a general-purpose model, such as a general-purpose encoder, is applied to raw patient data at a trusted local site. Calculations on unencrypted data can be performed faster and consume fewer resources than calculations on encrypted data. Furthermore, it becomes possible to include nonlinearity in the calculations performed on unencrypted data. This avoids the loss of accuracy that can occur due to the approximation of nonlinear functions by addition and subtraction, which is necessary in the case of homomorphic encrypted data. The output of a general-purpose model, such as a general-purpose encoder, is encrypted and sent to a remote server where a proprietary model, such as a custom decoder, is stored. Therefore, sensitive patient information does not leave the trusted site. Performing local analysis on unencrypted data does not infringe on the intellectual property rights of the owner of a proprietary model. The solutions provided herein do not limit healthcare providers to only a subset of locally available models. Furthermore, the solutions provided herein enable owners of proprietary models to commercialize their models, for example, by implementing a pay-per-use model.

[0067] A medical image processing method is provided by embodiment. In this method, a first part of an estimation model is used to perform calculations on unencrypted data on a secure and reliable server, the output is encrypted using a homomorphic encryption scheme, and a second part of the model stored on a remote server is executed on the homomorphically encrypted output.

[0068] This model can be used for medical image segmentation. The model has an encoder / decoder architecture in which the encoder is kept locally and the decoder is kept on a remote server. Images are encrypted using off-the-shelf general-purpose feature detectors such as ResNet. The data is encrypted using level homomorphic encryption. The data is also encrypted using fully homomorphic encryption. By partitioning the model, the number of calculations performed on the remote server can be minimized. Alternatively, the model may be partitioned so that only the final layer of the network is placed on the remote server. Alternatively, the model may be partitioned so that more complex and costly calculations are performed on a trusted local server. The first part of the model stored on the local server may include nonlinear functions. The introduction of a GPU can reduce the execution time of this method.

[0069] While specific circuits have been described herein, in other embodiments, one or more functions of these circuits can be implemented by a single processing resource or other component. Alternatively, functions implemented by a single circuit can be implemented by combining two or more processing resources or other components. A single circuit includes multiple components, whether spaced apart or not, that implement the functions of that circuit. Multiple circuits include single components that implement the functions of those circuits.

[0070] Although one embodiment has been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. The novel methods, apparatus, and systems can be implemented in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents.

[0071] A medical information processing device is provided comprising a first acquisition unit, an encryption unit, and a second acquisition unit, based on various embodiments. The first acquisition unit receives imaging data and applies a first part of an inference model to the imaging data to obtain intermediate analysis results. The encryption unit encrypts the intermediate analysis results based on a homomorphic encryption scheme. The second acquisition unit transmits the encrypted intermediate analysis results to a remote server and processes them using a second part of the inference model to obtain encrypted analysis results.

[0072] The first acquisition unit may further receive the encrypted analysis results from the remote server, and may further include a decryption unit that decrypts the encrypted analysis results and acquires the analysis results.

[0073] Furthermore, the inference model comprises a first part and a second part. The inference model may receive image data as input and provide analysis results based on the image data as output. The analysis results may include classification of the image data and / or segmentation of the image data.

[0074] Furthermore, the first part of the inference model may be a public model.

[0075] Furthermore, the first portion of the inference model may include an encoder.

[0076] Furthermore, the first part of the inference model may include nonlinear operations.

[0077] Furthermore, the second part of the inference model may be a proprietary model.

[0078] Furthermore, the second part of the inference model may include a decoder.

[0079] Furthermore, the second part of the inference model does not necessarily have to include nonlinear operations.

[0080] Furthermore, the second part of the inference model may include only addition and / or multiplication.

[0081] Furthermore, the first and second portions of the inference model may each comprise multiple neural network layers. The second portion of the inference model may have fewer neural network layers than the first portion of the inference model.

[0082] Furthermore, the first part of the inference model may include a greater number of operations than the second part of the inference model.

[0083] Furthermore, the encryption unit may generate a pair of public and private keys according to a homomorphic encryption scheme and encrypt the intermediate analysis result based on the public key, thereby encrypting the intermediate analysis result.

[0084] The decryption unit may also decrypt the encrypted analysis result by decrypting the encrypted analysis result based on the secret key.

[0085] The medical information processing device may also include one or more first servers and one or more second servers. The generation of the public key-private key pair is performed in one of the one or more first servers. The decryption of the analysis result is performed in one of the one or more second servers. The decryption unit may further transmit the private key from one of the one or more first servers to one of the one or more second servers.

[0086] Furthermore, the imaging data may be confidential medical imaging data.

[0087] Furthermore, the medical information processing device may be connected to a secure network of a hospital or medical research institution.

[0088] Furthermore, the homomorphic encryption scheme may be a leveled homomorphic encryption scheme or a fully homomorphic encryption scheme. Furthermore, the medical information processing device may utilize an implemented GPU.

[0089] Furthermore, the medical information processing device may utilize a cryptographic accelerator.

[0090] A medical information processing method is provided based on various embodiments. The method comprises receiving imaging data, applying a first part of an inference model to the imaging data to obtain an intermediate analysis result, encrypting the intermediate analysis result based on a homomorphic encryption scheme, and transmitting the encrypted intermediate analysis result to a remote server and processing it using a second part of the inference model to obtain an encrypted analysis result.

[0091] The method may further include receiving the encrypted analysis results from a remote server and decrypting the encrypted analysis results to obtain the analysis results.

[0092] While certain embodiments are described, these embodiments are presented for illustrative purposes only and are not intended to limit the scope of the invention. In practice, the novel methods and systems described herein can be embodied in a variety of other forms. Furthermore, various omissions, substitutions, and modifications in the forms of methods and systems described herein may be made without departing from the spirit of the invention. The claims of the appended claims and equivalents thereof are intended to cover forms and modifications that fall within the scope of the invention. [Explanation of Symbols]

[0093] 205 Equipment 212 Arithmetic equipment 216 Display screen 218 Input device 222 Processing Circuit 224 Estimation circuit 226 Encryption circuit 228 Decoding Circuit

Claims

1. A first acquisition unit receives imaging data, applies a first part of an inference model to the imaging data, and obtains an intermediate analysis result. An encryption unit that encrypts the aforementioned intermediate analysis results based on a homomorphic encryption scheme, A second acquisition unit obtains encrypted analysis results by sending encrypted intermediate analysis results to a remote server and processing them using the second part of the inference model, A medical information processing device equipped with [a specific feature].

2. The first acquisition unit receives the encrypted analysis results from the remote server, The system further includes a decryption unit that decrypts the encrypted analysis results and obtains the analysis results. The medical information processing device according to claim 1.

3. The inference model comprises a first part and a second part, The inference model receives image data as input and provides analysis results based on the image data as output. The analysis results include the classification and / or segmentation of the image data. The medical information processing device according to claim 1.

4. The first part of the aforementioned inference model is a public model. The medical information processing device according to claim 1.

5. The first part of the inference model comprises an encoder. The medical information processing device according to claim 1.

6. The first part of the inference model includes nonlinear operations, The medical information processing device according to claim 1.

7. The second part of the aforementioned inference model is a proprietary model. The medical information processing device according to claim 1.

8. The second part of the inference model comprises a decoder. The medical information processing device according to claim 1.

9. The second part of the inference model does not include nonlinear operations. The medical information processing device according to claim 1.

10. The second part of the aforementioned inference model includes only addition and / or multiplication operations. The medical information processing device according to claim 1.

11. The first part of the inference model and the second part of the inference model each comprise a plurality of neural network layers, The second part of the inference model has fewer neural network layers than the first part of the inference model. The medical information processing device according to claim 1.

12. The first part of the inference model includes a greater number of operations than the second part of the inference model. The medical information processing device according to claim 1.

13. The encryption unit generates a pair of public and private keys according to a homomorphic encryption scheme, and encrypts the intermediate analysis result based on the public key, thereby encrypting the intermediate analysis result. The medical information processing device according to claim 2.

14. The decryption unit decrypts the encrypted analysis result by decrypting the encrypted analysis result based on the secret key. The medical information processing device according to claim 13.

15. The system further comprises one or more first servers and one or more second servers, The generation of the public key and private key pair is performed on one of the one or more first servers. The decoding of the analysis results is performed on one of the one or more second servers. The decryption unit transmits the secret key from one of the one or more first servers to one of the one or more second servers. The medical information processing device according to claim 14.

16. The aforementioned imaging data is confidential medical imaging data. The medical information processing device according to claim 1.

17. The aforementioned medical information processing device is connected to a secure network of a hospital or medical research institution. The medical information processing device according to claim 1.

18. The aforementioned homomorphic encryption scheme is a leveled homomorphic encryption scheme or a fully homomorphic encryption scheme. The medical information processing device according to claim 1.

19. The aforementioned medical information processing device utilizes the implemented GPU. The medical information processing device according to claim 1.

20. The aforementioned medical information processing device utilizes a cryptographic accelerator. The medical information processing device according to claim 1.

21. Receiving imaging data, Applying the first part of the inference model to the imaging data to obtain the intermediate analysis results, The intermediate analysis results are encrypted based on a homomorphic encryption scheme, and The encrypted intermediate analysis results are sent to a remote server and processed using the second part of the inference model to obtain the encrypted analysis results. A medical information processing method including [the specified term].

22. Receiving the encrypted analysis results from the remote server, and Decrypting the encrypted analysis results to obtain the analysis results, Further including, The medical information processing method according to claim 21.

Images