Method for processing homomorphic encryption data and apparatus therefor

Abstract

A method for processing homomorphic encryption data by cooperation of a client device and a server is proposed. The method may comprise the steps of: determining a cut layer for splitting a privacy preserving machine learning (PPML) model into at least two models; obtaining intermediate inference data from input data by using a first partial PPML model corresponding to a front end of the determined cut layer; performing homomorphic encryption for the intermediate inference data; and obtaining homomorphically encrypted final inference data from the homomorphically encrypted intermediate inference data by using a second partial PPML model corresponding to a rear end of the determined cut layer.

Description

Method for processing homomorphic encrypted data and apparatus for the same

[0001] The present invention relates to a method for processing homomorphic encrypted data and an apparatus for the same. More specifically, it relates to a method for processing homomorphic encrypted data and an apparatus for the same for optimizing load balancing or computational load between a client and a server.

[0002] While generating added value through data is a new growth engine for the Fourth Industrial Revolution, the use of personal information or personal credit information poses a potential threat that could infringe upon individual privacy. Therefore, methods to technically protect individual privacy are being researched.

[0003] Data analysis based on existing encryption technology ensures the confidentiality of data through encryption during data transmission, reception, and storage, but since encrypted data must be decrypted for analysis, there is a possibility of data leakage and leakage of decryption key information. Additionally, it lacks quantum resistance, which can pose a threat in the era of quantum computing.

[0004] Privacy-Preserving Machine Learning (PPML) refers to technologies and methodologies that enable the training of machine learning models or the performance of predictions while protecting sensitive data. This is a field gaining attention due to increasing interest in personal information protection, and various technologies are utilized.

[0005] Among these, homomorphic encryption is an encryption technique that allows computations to be performed even while data remains encrypted. It has the advantage of significantly reducing the risk of data exposure because it enables model training and inference while keeping sensitive data encrypted.

[0006] Furthermore, because homomorphic encryption possesses quantum resistance, it is being considered as a suitable solution in situations where existing cryptographic systems are threatened by the emergence of quantum computers. In particular, as it can be used in the currently active field of AI / ML, it is emerging as an essential element for PPML.

[0007] However, applying homomorphic encryption to current systems is difficult due to the increase in plaintext computation and data size being tens of times greater. In particular, in PPML, which applies homomorphic encryption to AI / ML, the amount of computation and size act as a significant burden.

[0008] Although data size can be reduced using Split ML, if the characteristics and environment of the device and cloud are not taken into account, load balancing issues may occur.

[0009] In order to maximize the efficiency of homomorphic encryption operations and memory usage, there is a need for measures to efficiently process homomorphically encrypted personal data and adaptively optimize model separation according to device and cloud status and usage environments.

[0010] The present invention proposes a method for processing homomorphic encrypted data and an apparatus for the same.

[0011] The problems to be solved by the present invention are not limited to the problems to be solved above, and other problems not mentioned will be clearly understood by those skilled in the art to which the present invention belongs from the description below.

[0012] A method for processing homomorphically encrypted data through cooperation between a client device and a server is proposed, and the method may include the steps of: determining a cut layer for splitting a privacy preserving machine learning (PPML) model into at least two; obtaining intermediate inference data from input data using a first partial PPML model corresponding to the front end of the determined cut layer at the client device; performing homomorphic encryption on the intermediate inference data at the client device; obtaining homomorphically encrypted final inference data from the homomorphically encrypted intermediate inference data using a second partial PPML model corresponding to the rear end of the determined cut layer at the server; and obtaining final inference data by decrypting the homomorphically encrypted final inference data at the client device.

[0013] Additionally or alternatively, the step of determining the cut layer in the above method may include a step of estimating the inference time of the client device or the inference time of the server for each candidate cut layer using client device-related information or server-related information.

[0014] Additionally or alternatively, the method may include the step of obtaining a total inference time by summing the inference time of the client device and the inference time of the server for each candidate cut layer, and determining the candidate cut layer corresponding to the shortest inference time among the obtained total inference times as the cut layer.

[0015] Additionally or alternatively, the above method may include the step of determining a lower time and an upper time for the total inference time for each of the above candidate cut layers.

[0016] Additionally or alternatively, the lower limit time may be determined based on preset time limit information, and the upper limit time may be determined based on memory information of the client device.

[0017] Additionally or alternatively, the method may include a step of determining the cut layer by considering the total inference time for each candidate cut layer and the lower limit time or the upper limit time.

[0018] Additionally or alternatively, the step of determining the cut layer may include a step of estimating the power consumption or memory usage of the client device for each candidate cut layer using information related to the client device.

[0019] Additionally or alternatively, the method may include the step of obtaining an estimate of the power consumption or memory usage of the client device for each candidate cut layer, and determining the candidate cut layer corresponding to the smallest value among the obtained estimates as the cut layer.

[0020] Additionally or alternatively, the step of obtaining the intermediate inference data may include separating the input data into sensitive data and general data, and obtaining sensitive data-based intermediate inference data from the sensitive data using the first partial PPML model.

[0021] Additionally or alternatively, the step of obtaining the homomorphically encrypted final inference data may include obtaining the homomorphically encrypted first partial inference data from the homomorphically encrypted sensitive data-based intermediate inference data using the second partial PPML model, obtaining the second partial inference data from the general data using the ML model for the general data, and performing an ensemble operation by applying weights to the first partial inference data and the second partial inference data, respectively, to obtain the final inference data.

[0022] Additionally or alternatively, the sensitive data is separated into multiple parts, and the method may include the step of obtaining sensitive data-based intermediate inference data from each sensitive data using the first part PPML model.

[0023] Additionally or alternatively, the method may include the step of periodically adjusting the cut layer based on the load balance of the client device and the server.

[0024] In addition, a computer-readable medium is proposed that stores a computer program for performing at least some of the methods described above.

[0025] Additionally, a client device is proposed, wherein the client device includes a processor and a transceiver, and the processor may be configured to obtain intermediate inference data from input data using a first partial PPML model corresponding to the front end of a cut layer for splitting a privacy preserving machine learning (PPML) model into at least two, perform homomorphic encryption on the obtained intermediate inference data, transmit the homomorphically encrypted intermediate inference data to a server through the transceiver, receive homomorphically encrypted final inference data obtained from the homomorphically encrypted intermediate inference data from the server using a second partial PPML model, and decrypt the received homomorphically encrypted final inference data to obtain final inference data.

[0026] Additionally or alternatively, the processor may be configured to estimate the inference time of the client device or the inference time of the server for each candidate cut layer by utilizing information related to the client device or information related to the server.

[0027] Additionally or alternatively, the processor may be configured to obtain a total inference time by summing the inference time of the client device and the inference time of the server for each candidate cut layer, and to determine the candidate cut layer corresponding to the shortest inference time among the obtained total inference times as the cut layer.

[0028] Additionally or alternatively, the processor may be configured to estimate the power consumption or memory usage of the client device for each candidate cut layer using information related to the client device.

[0029] Additionally or alternatively, the processor may be configured to separate the input data into sensitive data and general data, and to obtain sensitive data-based intermediate inference data from the sensitive data using the first partial PPML model.

[0030] Additionally or alternatively, the processor may be configured to periodically adjust the cut layer based on the load balance of the client device and the server.

[0031] Additionally, a server device is proposed, said server device including a processor and a transceiver, said processor may be configured to determine a cut layer for splitting a privacy preserving machine learning (PPML) model into at least two parts, transmit information about said determined cut layer to a client device through said transceiver, obtain homomorphically encrypted final inference data from said homomorphically encrypted intermediate inference data using a second part PPML model corresponding to the rear end of said determined cut layer, and transmit said obtained homomorphically encrypted final inference data to the client device through said transceiver.

[0032] The above-mentioned problem-solving methods are merely some of the embodiments of the present invention, and various embodiments reflecting the technical features of the present invention can be derived and understood by those skilled in the art based on the detailed description of the present invention to be described below.

[0033] The present invention has the following technical effects.

[0034] It is possible to efficiently process homomorphically encrypted personal data and adaptively optimize model separation based on device and cloud status and usage environment, thereby protecting sensitive personal data while maximizing the efficiency of homomorphic encryption operations and memory usage.

[0035] The effects according to the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art from the following detailed description of the invention.

[0036] The accompanying drawings, which are included as part of the detailed description to aid in understanding the present invention, provide embodiments of the present invention and explain the technical concept of the present invention together with the detailed description.

[0037] Figure 1 illustrates a system diagram for processing or analyzing existing encrypted data.

[0038] Figure 2 illustrates a conventional homomorphic encryption-based PPML system.

[0039] Figure 3 shows the structure of an existing improved machine learning model.

[0040] FIG. 4 illustrates a system for homomorphic encryption according to the present invention.

[0041] Figure 5 shows a flowchart of an artificial intelligence inference method through sensitive data separation according to the present invention.

[0042] FIG. 6 illustrates a system for homomorphic encryption with sensitive data separation applied according to the present invention.

[0043] Figure 7 illustrates a split model based on VGC16.

[0044] Figure 8 illustrates a conceptual diagram of how a split ML model is applied to homomorphic encryption.

[0045] Figure 9 shows the relationship between the cut layer and the inference time in a split ML model with homomorphic encryption applied.

[0046] FIG. 10 shows a flowchart of an inference scheme for a split ML model with homomorphic encryption applied according to the present invention.

[0047] Figure 11 illustrates the relationship between the total inference time and the number of cut layers related to determining the optimal cut layer according to the present invention.

[0048] FIG. 12 illustrates a flowchart for a method to determine a cut layer for a split ML model with homomorphic encryption applied according to the present invention.

[0049] Figure 13 shows the ML model information of VGG16.

[0050] Figure 14 shows the total inference time and the lower and upper limits of the cut layer according to the present invention.

[0051] FIG. 15 illustrates a system in which sensitive data separation and a split ML model according to the present invention are applied.

[0052] FIG. 16 illustrates a system in which sensitive data separation and a split ML model according to the present invention are applied.

[0053] FIG. 17 illustrates a block diagram of a device for processing homomorphic encrypted data according to the present invention.

[0054] FIG. 18 illustrates a block diagram of a device for processing homomorphic encrypted data according to the present invention.

[0055] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Identical or similar components regardless of drawing symbols will be assigned the same reference number, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably solely for the ease of drafting the specification and do not inherently possess distinct meanings or roles. Furthermore, in describing embodiments disclosed in this specification, if it is determined that a detailed description of related prior art could obscure the essence of the embodiments disclosed in this specification, such detailed description will be omitted. Additionally, the attached drawings are intended only to facilitate understanding of the embodiments disclosed in this specification; the technical concept disclosed in this specification is not limited by the attached drawings, and it should be understood that they include all modifications, equivalents, and substitutions that fall within the spirit and technical scope of the present invention.

[0056] Terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. These terms are used solely for the purpose of distinguishing one component from another.

[0057] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0058] A singular expression includes a plural expression unless the context clearly indicates otherwise.

[0059] In this application, terms such as “comprising” or “having” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0060]

[0061] Figure 1 illustrates a system for processing or analyzing existing encrypted data.

[0062] A basic explanation of data analysis based on existing encryption technology will be provided. A client (individual) can encrypt data using an encryption key and transmit it to a server. The server can perform data analysis by decrypting the data using a decryption key and then performing operations. The server encrypts the data analysis results again and transmits them to the client, and the client can obtain the data analysis results by decrypting them.

[0063] This method requires the premise that the server's trust is guaranteed, as it exposes the decryption key and plaintext to the server for data analysis. Furthermore, the cryptographic algorithm systems currently in use may be vulnerable to the threat of quantum computing.

[0064]

[0065] Figure 2 illustrates a conventional homomorphic encryption-based PPML system.

[0066] Homomorphic encryption is a technology that enables computation without decryption and can securely guarantee data ownership and usage due to its quantum-resistant Learning With Error (LWE) characteristics. However, it results in increased computation speed and data size compared to plaintext, posing a significant burden, particularly when providing AI / ML services. Specifically, while simple data analysis is possible, it is currently difficult to provide real-time customer services.

[0067] As the size of customer data for service provision increases, the size of homomorphic encryption also increases, and the size of ML used also increases, requiring more computation and memory.

[0068]

[0069] Figure 3 shows the structure of an existing improved machine learning model.

[0070] By reducing the computational portion processed by homomorphic encryption, the computation time or the burden on the server can be reduced. One such method is Split ML, which in this specification refers to machine learning performed by dividing the entire ML model into multiple parts, as illustrated in FIG. 3. Since the client-side artificial intelligence model processes data in plaintext, it requires relatively less memory and has a relatively fast computation speed. In this case, the client side homomorphically encrypts the extracted Feature Map (Cut layer) and sends it to the server side, and since the server side computes the homomorphically encrypted data through the remaining artificial intelligence model, it has advantages in terms of speed and memory usage compared to processing the entire artificial intelligence model over homomorphic encryption.

[0071] Client devices intending to provide personal data can range from MCU (microcontroller unit)-based IoT devices to devices equipped with high-spec GPUs.

[0072] However, while this Split ML can yield overall speed and memory benefits, it may also lead to an increased load on client devices. Furthermore, since client devices intended to provide personal data can range from MCU-based IoT devices to those with high-end GPUs, it is necessary to adaptively determine the Cut layer of Split ML by considering device specifications, server load, and the latency tolerance of the provided service.

[0073] Therefore, to maximize the efficiency of homomorphic encryption operations and memory usage, we propose a method to efficiently process homomorphically encrypted personal data and adaptively optimize model separation according to device and cloud status and usage environment.

[0074]

[0075] FIG. 4 illustrates a system for homomorphic encryption according to the present invention.

[0076] A system can be designed to split personal data into sensitive data and general data, apply homomorphic encryption only to the sensitive data and apply it to private ML in the cloud, and pass the general data to non-private ML to perform inference on the plaintext.

[0077] The final value can be determined by using ensemble techniques for the homomorphic encryption inference part (sensitive data) and the plaintext inference part (general data). Ensemble techniques are a method in machine learning that combines multiple models to create a single, powerful model. Because individual models compensate for each other's weaknesses, they generally demonstrate better predictive performance than a single model.

[0078] The result value based on the ensemble can be expressed as follows.

[0079]

[0080] H(x) is the final decision-maker (result),

[0081] w m is the weight (w) for the m-th partial model inference m The smaller this is, the less influence the m-th submodel has),

[0082] h m is the m-th partial model,

[0083] F represents the function used in the final decision unit.

[0084] Since isomorphic functions place a heavy burden on computation, the server may delegate F to the client and deliver the ensemble result just before that.

[0085] Since the extent to which the sensitive data portion and the general data portion each influence the actual decision varies depending on the service (model), the impact of each is w p (Weights for private data part model inference) and w n The weights (for model inference on the general data portion) are weights that are learned together during training, and these values ​​are used to determine the final ensemble value. Various methods such as voting or averaging can be used for the ensemble method, and private ML or non-private ML can be separated into multiple parts and applied with varying degrees of influence.

[0086] Since plaintext operations are much faster and consume less memory compared to homomorphic encryption operations, this method allows for memory and speed improvements on both the client and server sides by homomorphically encrypting only sensitive data and processing homomorphic encryption operations on it, while also ensuring secure processing because the sensitive data is homomorphically encrypted.

[0087] Sensitive data can be represented as an image or a Feature Map with processed images as shown in Fig. 4, or strings or numerical values ​​can be used as shown in the table below.

[0088] Name Age Gender Address Income Daniel 23 Male Jung-gu, Daegu 1.5 million won Sean 25 Male Seo-gu, Daegu 2 million won Phillip 27 Male Buk-gu, Daegu 2 million won Kate 32 Female Dongnae-gu, Busan 1.8 million won Helen 38 Female Songpa-gu, Seoul 1.8 million won Jamie 46 Female Mapo-gu, Seoul 2.5 million won

[0089] Name, age, income, etc., may be considered sensitive data, while gender, address, etc., may be considered general data.

[0090]

[0091] In this specification, sensitive data may be defined as data including both unique identification information and sensitive information.

[0092] Sensitive information includes the following items defined in the Personal Information Protection Act.

[0093] ① Thoughts, beliefs,

[0094] ② Joining or withdrawing from a labor union or political party

[0095] ③ Political views,

[0096] ④ Information regarding health, sexual life, etc.,

[0097] ⑤ Other personal information that is likely to significantly infringe upon the privacy of the data subject

[0098] Unique identification information refers to resident registration numbers, passport numbers, driver's license numbers, alien registration numbers, etc.

[0099] To distinguish sensitive data among personal data, one can use a server providing relevant services or download information about the list to perform the distinction on the client side. Alternatively, this process can also be entrusted to a server that separates sensitive data using homomorphic encryption.

[0100] The following is a scenario in which homomorphic encryption operations can be efficiently processed by distinguishing between sensitive personal data and non-sensitive data (general data).

[0101] 1. Dangerous Person Detection

[0102] A photo of a visitor captured by the front door camera is sent to the server and compared with the dangerous person database to determine if the person is dangerous. In this case, only the face portion of the photo is homomorphically encrypted.

[0103] 2. Detection of hazardous environments

[0104] Photos taken by the robot vacuum cleaner are encrypted to detect risk factors (falls by the elderly, burglary, fire, operation of unused appliances, etc.) and send alarms. In this case, only the face or exposed body parts within the photo are homomorphically encrypted.

[0105] 3. Vehicle Image Analysis

[0106] Photos of a vehicle's interior or exterior are analyzed and utilized for driver state analysis, detection of external hazards, etc. In this case, only the parts of the photo where the face or body is exposed are homomorphically encrypted.

[0107] 4. Home training

[0108] Analysis and feedback provided on body posture during exercise. In this case, only exposed parts of the face or body in the photo are homomorphically encrypted.

[0109] Meanwhile, although general data is depicted in Fig. 4 as being transmitted to the server (cloud service) without encryption, general data can be transmitted with encryption using a general symmetric key applied. The server (cloud service) can decrypt the received encrypted general data using the symmetric key and then perform inference using a non-private ML model.

[0110]

[0111] FIG. 5 shows a flowchart of an artificial intelligence inference method through sensitive data separation according to the present invention. This method can be broadly divided into four steps.

[0112] First step (Symmetric key, shared separation policy)

[0113] The client (100) and the server (200) share a symmetric key with each other in advance (S1). The method of sharing the symmetric key may be a form of public key encryption used in existing HTTPS, etc., and quantum-resistant encryption such as PQC (Post-Quantum Cryptography) may also be used to enhance overall security performance.

[0114] Additionally, the client (100) and the server (200) share a policy for separating sensitive data and general data (S2). The server (200) receives the security level desired by the client (100) and can determine a policy for separating sensitive data. The separation policy may include a separation model or information related to a separation algorithm. Furthermore, the separation policy allows for separation into N and M (integers such that N and M >= 1) rather than just two types of sensitive data and non-sensitive data (general data).

[0115] Second step (processing on the client)

[0116] Client (100) each has personal data sensitive data (D P ) and non-sensitive data (D N It can be separated into ) (S3). Sensitive data may include specific body parts such as a face in the case of an image, and may include string or numeric data such as age, name, annual salary, etc. The client (100) can encrypt each of the separated data (S4).

[0117] For sensitive data, the server (200) does not decrypt it, but applies homomorphic encryption to perform analysis / prediction (HE(D P )), symmetric encryption is applied to non-sensitive data (E(D N )). Sensitive data may be resized or padded to fit the input of the ML model before homomorphic encryption is applied. The client (100) sensitive data (DP ) and non-sensitive data((D N Personal data HE(D) encrypted with homomorphic encryption and symmetric keys, respectively P ), E(D N ) can be transmitted to the server (200) (S5).

[0118] Third step (processing on the server)

[0119] The server (200) can decrypt (S6) the non-sensitive data among the received personal data using a symmetric key and transmit it to a non-private ML model, and transmit the sensitive data to a private ML model as is, while homomorphically encrypted. In each ML model, inference is performed on the sensitive data and the non-sensitive data (S7).

[0120] The server (200) determines that for the results of performing inference in each ML model, the respective influence (degree of influence on the decision) is w p Wow w n The final value is determined using the applied ensemble technique (see Equation 1) (S8). Since the ensemble HE(∑) is in a homomorphic encrypted state, the server cannot know the result. The server (200) can transmit the ensemble HE(∑) to the client (100) (S9).

[0121] Fourth step (Processing in client 2)

[0122] The client (100) receives the homomorphic encrypted inference result HE(∑) from the server (200) and decrypts it using a homomorphic encryption key that only the client (100) has (or knows) (S10). A final inference result can be obtained by performing processing that is difficult or time-consuming to compute in the homomorphic encrypted state (e.g., Softmax, etc.) after decrypting the homomorphic encryption (S11).

[0123] Figure 6 illustrates a system in a scenario where sensitive data is divided into two.

[0124] In the case of PPML with homomorphic encryption applied, it can be divided into multiple parts because the amount of computation is large. Also, if there is too much general data and the non-private ML model takes a long time to compute, the general data can also be divided into multiple parts for processing, and this information should be included in the separation policy and shared with the client (100).

[0125] Meanwhile, although general data is shown in FIG. 6 as being transmitted to the server (200) without encryption, general data may be transmitted with encryption using a general symmetric key applied. The server (200) can decrypt the received encrypted general data using a symmetric key and then perform inference using a non-private ML model.

[0126]

[0127] Figure 7 illustrates a split model based on VGC16.

[0128] VGC (Visual Geometry Group) 16 is a widely known deep neural network model in the field of computer vision.

[0129] To increase the overall speed of homomorphic encryption, methods can be used to reduce the data being homomorphically encrypted and, at the same time, reduce the size of the ML model that is computed in the actual encrypted state.

[0130] A method to reduce the size of an ML model is the Split ML model, and the closer the Cut Layer is to the end of the ML model, the faster the speed is guaranteed.

[0131]

[0132] Figure 8 illustrates a conceptual diagram of how a split ML model is applied to homomorphic encryption.

[0133] The client (100) executes the ML model prior to the cut layer in the plaintext area to obtain intermediate data (Smashed data), homomorphically encrypts it, and transmits it to the server (200). The server (200) applies the homomorphically encrypted intermediate data to the model after the cut layer to obtain an inference result, but the value of the inference result is homomorphically encrypted so the server (200) cannot know the result. The server (200) transmits the inference result to the client (100), and the client (100) can decrypt the received inference result to extract the result value.

[0134]

[0135] Figure 9 shows the relationship between the cut layer and the inference time in a split ML model with homomorphic encryption applied.

[0136] As the cut layer is positioned at the end of the ML model, the overall inference time decreases, but the load on the client (100) side increases. Conversely, as the cut layer is positioned at the beginning of the ML model, the load on the client (100) side decreases, but the ML model part that computes homomorphic encryption increases, so not only does the overall inference time increase, but the load on the server (200) also increases.

[0137] Therefore, the optimal cut layer position, i.e., CL Optimal When determining the lower bound, the decision must be made between the upper bound and the lower bound, the lower bound is influenced by the minimum inference time requirement (i.e., the entire inference must be completed within the minimum inference time), and the upper bound may be related to the capacity / capability of the client (100).

[0138]

[0139] The following describes the method for determining the cut layer.

[0140] FIG. 10 shows a flowchart of an inference scheme for a split ML model with homomorphic encryption applied according to the present invention.

[0141] First step (sharing between client and server)

[0142] The client (100) shares the capacity or capability to perform ML inference computation with the server (200) (S101). The server (200) can determine a cut layer between a lower limit and an upper limit (S102).

[0143] FIG. 11 illustrates the relationship between the total inference time and the number of cut layers related to determining the optimal cut layer according to the present invention. In order to determine the optimal cut layer as shown in FIG. 11, information regarding the total inference time for each candidate cut layer is required. The information regarding the total inference time is determined by the sum of the inference time at the client (100) and the inference time at the server (200). The inference time of the server (200) is the time inferred by applying homomorphic encryption after the cut layer, and the inference time of the client (device) represents the time inferred from the plaintext up to the cut layer. As shown in FIG. 11, the server (200) can determine the optimal cut layer by considering the upper and lower limits shown in FIG. 9 based on the predicted total inference time.

[0144] The server (200) can transmit information about the cut layer (e.g., location) or information about the partial ML model that the client needs to execute to the client (100) (S103).

[0145] Second step (processing on the client)

[0146] The client (100) infers information D from the plaintext using a partial ML model prior to the cut layer, and as a result, D smashed = ML pre (D) can be obtained (S104). After that, the client (100) homomorphically encrypts the inference result to obtain homomorphically encrypted information, HE(D smashed Can obtain ) (S105). HE(D smashed ) can be referred to as homomorphically encrypted intermediate inference data.

[0147] The client (100) has homomorphically encrypted intermediate inference data, HE(D smashed ) can be transmitted to the server (200) (S106).

[0148] Third step (processing on the server)

[0149] The server (200) infers the homomorphically encrypted inference result, i.e., the homomorphically encrypted intermediate inference data received by inferring the partial ML model prior to the cut layer, by inferring the remaining partial ML model after the cut layer to obtain the homomorphically encrypted final inference result, ML post (HE(D smashed Can obtain )) (S107).

[0150] As explained earlier, the homomorphically encrypted final inference result can be represented as HE(Result) because it is in a homomorphically encrypted state, and the server (200) cannot know the result.

[0151] The server (200) can transmit the homomorphically encrypted final inference result to the client (100) (S108).

[0152] Fourth step (Processing on the client side 2)

[0153] The client (100) receives the final inference result HE (Result) that is homomorphically encrypted from the server (200) and can decrypt it using a homomorphic encryption key that only the client (100) has or knows (S109). By performing processing that is difficult or time-consuming to compute in the homomorphically encrypted state (e.g., Softmax) after decryption, the client (100) can obtain the final inference result (S110).

[0154]

[0155] FIG. 12 illustrates a flowchart for a method to determine a cut layer for a split ML model with homomorphic encryption applied according to the present invention.

[0156] FIG. 12 describes a method for determining a cut layer based on “inference time,” but the present invention is not limited thereto. That is, in addition to “inference time,” the present invention may determine a cut layer based on the power consumption, memory usage, computational amount, or heat generation of the client device (or server).

[0157] The procedure for determining the cut layer is divided into three main steps. It consists of a step of predicting the inference time of the client device (100) and the server (200) respectively and summing them to predict the total inference time, a step of setting a lower limit and an upper limit for the total inference time, and a step of determining the cut layer that minimizes the total inference time and periodically adjusting the cut layer based on load balancing.

[0158] Although the determination of the cut layer is described as being performed by the server (200) in FIG. 11, the present invention is not limited thereto. That is, the determination of the cut layer may be performed by the client (100), or the determination of the cut layer may be performed by a third device or other device.

[0159] First step

[0160] The first step includes a step (S1010) of predicting the inference time of the client (100) and server (200) according to the cut layer of the PPML model and summing them to obtain an estimated total inference time.

[0161] The information used to estimate the total inference time may include client device capacity / capability (310), server capacity / capability (320), and / or ML model information. The client capacity / capability (310) information may include device name or CPU / GPU information.

[0162] To determine the capacity / capability of a client device or server, the determination can be made based on CPU / GPU benchmark scores of each device or device, or by delivering a sample ML model to the client device or server and using actual performance figures.

[0163] Figure 13 shows the ML model information of VGG16. Referring to Figure 13, it can be seen that the ML model's ToolBox (Conv, fc, maxpool, etc.) exists, and that the time required for other ToolBoxes, excluding conv (Convolution) and fc (Fully Connected Layer), is negligible. Additionally, since the rightmost column represents the number of parameters in the ML model, the amount of computation is proportional to the parameter values ​​and the size of the input image. Therefore, it is possible to predict the inference time according to the cut layer by applying a ratio to the parameter values ​​and the size of the input image with a previously obtained standard value related to the capacity / capability of the client device or server (i.e., the benchmarking score mentioned earlier or the value measured using a sample ML model) (S1011, S1012).

[0164] The estimated inference time for the client device and the estimated inference time for the server are summed (S1021), and an example of this is illustrated in FIG. 11.

[0165] Second step

[0166] The second step includes determining the lower and upper limits of the total inference time (S1030). In determining the lower and upper limits of the total inference time, memory information (350) among the capacity / capability of the device and time limit information (340) provided by the service provider are used. If the total inference time graph is defined as y=f(x), the lower limit of the cut layer can be defined as follows.

[0167]

[0168] Trequire represents time limit information (340).

[0169] The lower limit is related to the size (capacity) of the device's available memory, and based on the size of the available memory, the maximum cut layer at which the device can operate can be determined, and this value corresponds to the upper limit.

[0170] The function g can be estimated using ML model information or can be held in the form of an LUT on the server (200) through actual measurement.

[0171]

[0172] UB represents the upper bound, g is a function representing the relationship between available memory size and the upper bound, M available represents the available memory size or capacity.

[0173] Figure 14 shows the total inference time and the lower and upper limits of the cut layer according to the present invention.

[0174] third step

[0175] The third step may include a step (S1040) of determining an optimal cut layer within a lower and upper limit range. At this time, the optimal cut layer may be periodically adjusted based on the load balance (360) (memory, CPU / GPU usage, etc.) of the client (100) and the server (200).

[0176] From the perspective of inference time, determining the cut layer with the shortest time corresponds to the optimal value.

[0177] The client (100) may have various characteristics, and in particular, for ultra-lower power (ULP) devices such as drones or low-power sensors, the power consumption of the device may be an important factor rather than the inference time. In such cases, the optimal cut layer can be determined by using power information, such as power limit information, instead of time limit information (340).

[0178] In particular, since the power consumption of a wireless device may be higher when transmitting wireless data than when operating an ML model, it is necessary to go through steps such as prediction, setting lower / upper limits, and setting cut layers as described above. In this case, a prediction related to the power consumption of the device is required, and the server (200) must know the power threshold of the device in advance.

[0179] As such, various indicators such as power consumption, memory usage, computation amount, and heat generation can be applied to the y-axis of the graph exemplified in Fig. 14 instead of inference time.

[0180]

[0181] FIG. 15 illustrates a system in which sensitive data separation and a split ML model according to the present invention are applied.

[0182] Referring to FIG. 15, homomorphic encryption is performed on the inferred result by applying separation of sensitive data and ML model splitting on the client device side, and accordingly, inference on general data and inference through the remaining ML model of the homomorphically encrypted intermediate inference result are performed on the server side.

[0183] The separation of sensitive data and general data of personal data and the application of a split ML model can be used in combination. In this case, the split ML model may be applied only to the sensitive data (as shown in Fig. 15), or the split ML model may also be applied to the general data.

[0184] In addition, sensitive data can be separated into multiple parts, and a split ML model can be applied. Figure 16 illustrates the application of a split ML model to multiple sensitive data.

[0185] The client device (100) can separate personal data into sensitive data and general data, perform partial inference on the sensitive data to obtain an intermediate inference result, and transmit it to the server (200) by homomorphically encrypting it. Additionally, the client device (100) can transmit general data to the server (200).

[0186] The server (200) can perform inference on homomorphically encrypted intermediate inference results using the remainder of a private ML model. Additionally, the server (200) can perform inference on general data using a non-private ML model.

[0187] After that, the server (200) [influences] each ML model (i.e., weight w p , w n The ensemble can be computed by applying ) and the result can be transmitted to the client device (100).

[0188] Meanwhile, although general data is depicted in FIG. 15 as being transmitted to the server (cloud service) without encryption, general data can be transmitted with encryption using a general symmetric key applied. The server (cloud service) can decrypt the received encrypted general data using the symmetric key and then perform inference using a non-private ML model.

[0189]

[0190] FIG. 17 illustrates a block diagram of a device for processing homomorphic encrypted data according to the present invention.

[0191] FIG. 17 illustrates a client device (100). The client device (100) may include a transceiver (110), a processor (120), and a memory (130).

[0192] The processor (120) may be configured to obtain intermediate inference data from input data using a first part PPML model corresponding to the front of a cut layer for splitting a privacy preserving machine learning (PPML) model into at least two.

[0193] Additionally, the processor (120) may be configured to perform homomorphic encryption on the acquired intermediate inference data.

[0194] The processor (120) may be configured to transmit homomorphically encrypted intermediate inference data to the server (200) via the transceiver (110). Additionally, the processor (120) may be configured to receive homomorphically encrypted final inference data obtained from the homomorphically encrypted intermediate inference data from the server (200) using a second part PPML model.

[0195] The processor (120) may be configured to obtain the final inference data by decrypting the received homomorphic encrypted final inference data.

[0196] Additionally, the processor (120) may be configured to separate input data into sensitive data and general data, and to obtain intermediate inference data based on sensitive data from the sensitive data using a first partial PPML model. Additionally, the sensitive data may be separated into multiple parts, and accordingly, the processor (120) may be configured to obtain intermediate inference data based on sensitive data from each of the sensitive data using a first partial PPML model.

[0197] The memory (130) is configured to provide a buffer or storage space necessary for obtaining intermediate inference data, homomorphic encryption, decryption, or transmitting and receiving data.

[0198] Even if not described with reference to FIG. 17, the apparatus (100) for processing homomorphic encrypted data of the present invention may perform the operation according to the present invention as described above in FIG. 3 to FIG. 16.

[0199]

[0200] FIG. 18 illustrates a block diagram of a device for processing homomorphic encrypted data according to the present invention.

[0201] FIG. 18 illustrates a server (200). The server (200) may include a transceiver (210), a processor (220), and memory (230).

[0202] The processor (220) can be configured to receive homomorphically encrypted intermediate inference data from the client device (100) through the transceiver (210).

[0203] The processor (220) may be configured to obtain homomorphically encrypted final inference data from homomorphically encrypted intermediate inference data using a second part PPML model corresponding to the rear end of a determined cut layer. Then, the processor (220) may be configured to transmit the obtained homomorphically encrypted final inference data to a client device (100) through a transceiver (210).

[0204] Additionally, the processor (220) may be configured to obtain homomorphically encrypted first part inference data from intermediate inference data based on homomorphically encrypted sensitive data using a second part PPML model. The processor (220) may be configured to obtain second part inference data from general data using an ML model for general data. The processor (220) may be configured to perform ensemble operations by applying weights to the first part inference data and the second part inference data, respectively, to obtain final inference data.

[0205] The memory (230) is configured to provide a buffer or storage space necessary for obtaining inference data, homomorphic encryption, decryption, or transmitting and receiving data.

[0206] Even if not described with reference to FIG. 18, the apparatus (200) for processing homomorphic encrypted data of the present invention may perform the operation according to the present invention as described above in FIG. 3 to FIG. 16.

[0207]

[0208] Additionally, the processor (120) or the processor (220) may be configured to determine a cut layer for splitting a privacy preserving machine learning (PPML) model into at least two. The processor that determined the cut layer may share or transmit information about the cut layer to an interacting counterpart processor.

[0209] A processor (120) or a processor (220) may be configured to estimate the inference time of a client device (100) or a server (200) for each candidate cut layer by using information related to a client device or information related to a server.

[0210] A processor (120) or a processor (220) may be configured to obtain a total inference time by summing the inference time of the client device (100) and the inference time of the server (200) for each candidate cut layer. Additionally, the processor (120) or a processor (220) may be configured to determine the candidate cut layer corresponding to the shortest inference time among the total inference times obtained as the cut layer.

[0211] The processor (120) or the processor (220) may be configured to determine a lower time limit and an upper time limit for the total inference time for each candidate cut layer. In this case, the lower time limit may be determined based on preset time limit information, and the upper time limit may be determined based on memory information of the client device.

[0212] The processor (120) or processor (220) may be configured to determine the cut layer by considering the total inference time for each candidate cut layer and the lower time or upper time.

[0213] As another embodiment for determining a cut layer, the processor (120) or the processor (220) may be configured to estimate the power consumption or memory usage of the client device (100) for each candidate cut layer using information related to the client device.

[0214] A processor (120) or a processor (220) may be configured to obtain an estimate of the power consumption or memory usage of a client device (100) for each candidate cut layer, and to determine the candidate cut layer corresponding to the smallest value among the obtained estimates as the cut layer.

[0215] Additionally, the processor (120) or the processor (220) may be configured to periodically adjust the cut layer based on the load balance of the client device (100) and the server (200).

[0216]

[0217] In addition, as another aspect of the present invention, the operation of the aforementioned proposal or invention may be provided as code that can be implemented, carried out, or executed by a "computer" (a comprehensive concept including a system on chip (SoC) or a (micro)processor, etc.), or as a computer-readable storage medium or computer program product that stores or contains said code, and the scope of the present invention may be extended to said code or as a computer-readable storage medium or computer program product that stores or contains said code.

[0218]

[0219] The detailed description of the preferred embodiments of the present invention disclosed above is provided to enable those skilled in the art to implement and practice the present invention. Although the present invention has been described with reference to preferred embodiments, those skilled in the art will understand that various modifications and changes can be made to the present invention as described in the following claims. Accordingly, the present invention is not intended to be limited to the embodiments shown herein, but to be given the broadest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for processing homomorphic encrypted data through cooperation between a client device and a server, A step of determining a cut layer for splitting a privacy preserving machine learning (PPML) model into at least two; In the above client device, a step of obtaining intermediate inference data from input data using a first partial PPML model corresponding to the front end of the determined cut layer; A step of performing homomorphic encryption on the intermediate inference data at the above client device; A step of obtaining homomorphically encrypted final inference data from the homomorphically encrypted intermediate inference data using a second partial PPML model corresponding to the rear end of the determined cut layer in the above server; and The above client device includes the step of decrypting the homomorphically encrypted final inference data to obtain the final inference data. , method.

2. In claim 1, the step of determining the cut layer is, The method includes the step of estimating the inference time of the client device or the inference time of the server for each candidate cut layer using client device-related information or server-related information. , method.

3. In Paragraph 2, The method includes the step of obtaining a total inference time by summing the inference time of the client device and the inference time of the server for each candidate cut layer, and determining the candidate cut layer corresponding to the shortest inference time among the obtained total inference times as the cut layer. , method.

4. In Paragraph 3, A step comprising determining a lower time and an upper time for the total inference time for each of the above candidate cut layers , method.

5. In Paragraph 4, The above lower limit time is determined based on preset time limit information, and The above upper limit time is determined based on the memory information of the client device. , method.

6. In Paragraph 4, A step of determining the cut layer by considering the total inference time for each of the above candidate cut layers and the lower time or the upper time. , method.

7. In paragraph 1, the step of determining the cut layer is, A step comprising estimating the power consumption or memory usage of the said client device for each candidate cut layer using information related to the client device, method.

8. In Paragraph 7, The method includes the step of obtaining an estimate of the power consumption or memory usage of the client device for each candidate cut layer, and determining the candidate cut layer corresponding to the smallest value among the obtained estimates as the cut layer. , method.

9. In paragraph 1, the step of acquiring the intermediate inference data is, The method includes the step of separating the above input data into sensitive data and general data, and obtaining sensitive data-based intermediate inference data from the sensitive data using the first partial PPML model. , method.

10. In claim 9, the step of obtaining the homomorphically encrypted final inference data is, Using the above second part PPML model, homomorphically encrypted first part inference data is obtained from the homomorphically encrypted sensitive data-based intermediate inference data, and Using an ML model for the above general data, second partial inference data is obtained from the above general data, and A step comprising performing an ensemble operation by applying weights to the first partial inference data and the second partial inference data, respectively, to obtain final inference data. , method.

11. In Paragraph 9, The above sensitive data is separated into multiple parts, and The step of obtaining sensitive data-based intermediate inference data from each sensitive data using the above-mentioned first partial PPML model , method.

12. In Paragraph 1, A step of periodically adjusting the cut layer based on the load balance of the client device and the server. , method.

13. A computer-readable medium storing a computer program for performing a method according to any one of paragraphs 1 through 12.

14. As a client device, It includes a processor and a transceiver, The above processor is, Intermediate inference data is obtained from input data using a first partial PPML model corresponding to the front of a cut layer for splitting a privacy preserving machine learning (PPML) model into at least two parts, and Homomorphic encryption is performed on the above-mentioned intermediate inference data, and Through the above transceiver, the homomorphically encrypted intermediate inference data is transmitted to a server, and the homomorphically encrypted final inference data obtained from the homomorphically encrypted intermediate inference data is received from the server using a second part PPML model, and Configured to obtain final inference data by decrypting the received homomorphically encrypted final inference data. , client device.

15. In paragraph 14, the above processor is: Configured to estimate the inference time of the client device or the inference time of the server for each candidate cut layer using client device-related information or server-related information. , client device.

16. In paragraph 15, the above processor is: The system is configured to obtain a total inference time by summing the inference time of the client device and the inference time of the server for each candidate cut layer, and to determine the candidate cut layer corresponding to the shortest inference time among the obtained total inference times as the cut layer. , client device.

17. In paragraph 14, the above processor is: Configured to estimate the power consumption or memory usage of the said client device for each candidate cut layer using information related to the client device. , client device.

18. In paragraph 14, the above processor is: The above input data is separated into sensitive data and general data, and is configured to obtain sensitive data-based intermediate inference data from the sensitive data using the above first partial PPML model. , client device.

19. In paragraph 14, the above processor is: Configured to periodically adjust the cut layer based on the load balance of the client device and the server. , client device.

20. As a server device, It includes a processor and a transceiver, The above processor is, Determine a cut layer to split a privacy preserving machine learning (PPML) model into at least two, and Information regarding the determined cut layer is transmitted to a client device through the above transceiver, and Using a second partial PPML model corresponding to the rear end of the above-determined cut layer, homomorphically encrypted final inference data is obtained from homomorphically encrypted intermediate inference data, and The above-mentioned homomorphically encrypted final inference data is configured to be transmitted to the client device through the above-mentioned transceiver. , server device.

Images