A method and device for reasoning calculation based on LLM model

Abstract

An inference calculation method based on an LLM model is applied to a service device on which a target LLM model is deployed; the model parameters of the target LLM model include a first part of parameters and a second part of parameters, wherein the first part of parameters are parameters that require privacy protection; the service device has a TEE and a GPU, including: calculating inference data and the first part of parameters in the TEE to obtain a first calculation result; encrypting the inference data in the TEE and transmitting it to the GPU, and the GPU calculating based on the second part of parameters and the encrypted inference data to obtain a second calculation result; decrypting the second calculation result in the TEE, and generating an inference result corresponding to the inference data based on the first calculation result and the decrypted second calculation result.

Description

Technical Field

[0001] The embodiments of this specification belong to the field of artificial intelligence technology, and more particularly to an inference calculation method and device based on an LLM model. Background Art

[0002] The data held by different data holders may usually contain the user's private information, so sharing data between different data holders may infringe the user's privacy. Based on this, in some solutions, in order to open up the data flow between multiple data holders and avoid data silos, secure multi-party computing technology can usually be used to support joint training between multiple parties and tap the value of data. At the same time, it can ensure that the plaintext content of user data will not be leaked when multiple parties interact with each other during model training.

[0003] With the continuous development of the LLM (Large Language Model) model, the use of secure multi-party computing technology to jointly train an LLM model by combining data held by multiple data holders has gradually matured, allowing the LLM model to achieve very good results in different downstream tasks in many fields. Summary of the invention

[0004] This specification proposes an inference calculation method based on an LLM model, which is applied to a service device that deploys a target LLM model; the model parameters of the target LLM model include a first part of parameters and a second part of parameters, wherein the first part of parameters are parameters that require privacy protection; the service device has a TEE and a GPU, and the method includes:

[0005] Calculating the inference data and the first part of parameters in the TEE to obtain a first calculation result;

[0006] The inference data is encrypted in the TEE and transmitted to the GPU, and the GPU performs calculation based on the second part of parameters and the encrypted inference data to obtain a second calculation result;

[0007] The second calculation result is decrypted in the TEE, and an inference result corresponding to the inference data is generated based on the first calculation result and the decrypted second calculation result.

[0008] Optionally, the target LLM model is an LLM model obtained by fine-tuning a pre-trained initial LLM model using a LoRA method; the first part of parameters are low-rank LoRA parameters.

[0009] Optionally, generating an inference result corresponding to the inference data based on the first calculation result and the decrypted second calculation result includes:

[0010] A weighted sum calculation is performed on the first calculation result and the second calculation result to obtain an inference result corresponding to the inference data.

[0011] Optionally, encrypting the inference data in the TEE includes:

[0012] encrypting the inference data in the TEE based on a one-time key generated for the inference data;

[0013] Decrypting the second calculation result in the TEE includes:

[0014] The second calculation result is decrypted in the TEE based on the one-time key generated for the inference data.

[0015] Optionally, the one-time key is a one-time random key generated based on an OTP algorithm and having the same length as the inference data;

[0016] Encrypting the inference data in the TEE based on a one-time key generated for the inference data includes:

[0017] Performing an addition calculation on the inference data and the one-time random key in the TEE to encrypt the inference data;

[0018] Decrypting the second calculation result in the TEE based on the one-time key generated for the inference data includes:

[0019] A subtraction calculation is performed on the second calculation result and the one-time random key in the TEE to decrypt the second calculation result.

[0020] Optionally, the second calculation result is a matrix operation result obtained by performing a matrix multiplication operation on the second part of parameters and the inference data after encryption;

[0021] Before performing a subtraction calculation on the second calculation result and the one-time random key in the TEE to decrypt the second calculation result, the method further includes:

[0022] Performing a matrix multiplication operation on the second part of parameters and the one-time random key in the TEE to obtain a target matrix operation result;

[0023] Performing a subtraction calculation on the second calculation result and the one-time random key in the TEE to decrypt the second calculation result includes:

[0024] In the TEE, a matrix subtraction operation is performed on the second calculation result and the target matrix operation result to decrypt the second calculation result.

[0025] Optionally, before calculating the inference data and the first part of parameters in the TEE to obtain a first calculation result, the method further includes:

[0026] Obtain the inference data uploaded by the user to the TEE through an encrypted data channel;

[0027] This specification also proposes an inference computing device based on an LLM model, which is applied to a service device that deploys a target LLM model; the model parameters of the target LLM model include a first part of parameters and a second part of parameters, wherein the first part of parameters are parameters that require privacy protection; the service device has a TEE and a GPU, and the device includes:

[0028] A first calculation module, which calculates the inference data and the first part of parameters in the TEE to obtain a first calculation result;

[0029] A transmission module, which encrypts the reasoning data in the TEE and transmits it to the GPU, and the GPU performs calculation based on the second part of parameters and the encrypted reasoning data to obtain a second calculation result;

[0030] The second computing module decrypts the second computing result in the TEE, and generates an inference result corresponding to the inference data based on the first computing result and the decrypted second computing result.

[0031] In the above technical solution, on the one hand, by executing the inference calculation of the LLM model on the inference data provided by the user in a secure computing environment built on the service device based on TEE and GPU, the inference data can be prevented from being leaked to the outside during the inference calculation process on the inference data provided by the user;

[0032] On the other hand, by executing the calculation of the first part of the inference data and the LLM model parameters that need to be privacy protected in TEE, and encrypting the inference data in TEE, and then transmitting it to the GPU, the GPU executes the calculation of the second part of the inference data and the LLM model parameters that do not need to be privacy protected. This can ensure that the inference data provided by the user and the first part of the parameters that need to be privacy protected are not leaked to the outside, while making the most of the performance advantages of the GPU in inference calculations, and further improving the computing performance of inference calculations based on confidential inference data. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0034] Figure 1 is a flow chart of an inference calculation method based on an LLM model shown in an embodiment of this specification;

[0035] Figure 2 It is a flowchart of performing reasoning calculation based on an LLM model using a Transformer architecture shown in an embodiment of this specification;

[0036] Figure 3 It is a schematic diagram of performing reasoning calculation based on an LLM model fine-tuned and trained by the LoRA method shown in an embodiment of this specification;

[0037] Figure 4 is a schematic structural diagram of an electronic device shown in an embodiment of this specification;

[0038] Figure 5 It is a block diagram of an inference calculation based on the LLM model shown in an embodiment of this specification. DETAILED DESCRIPTION

[0039] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be described clearly and completely below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0040] In the scenario where data held by multiple data holders are combined to jointly train an LLM model, the data held by each party may involve user privacy; therefore, in practical applications, secure multi-party computing technology can usually be introduced during the training process to ensure that the plaintext content of the training data is not leaked.

[0041] However, after the LLM model is trained, when the user uses the LLM model for reasoning, the reasoning data input by the user into the LLM model may also involve the user's privacy; therefore, if the reasoning data input by the user into the LLM model is not protected, the user's privacy may still be leaked.

[0042] Based on this, this specification proposes a secure reasoning framework for the LLM model based on TEE and GPU.

[0043] Under this framework, the service device on which the target LLM model is deployed can be equipped with TEE and GPU respectively. The model parameters of the target LLM model may include a first part of parameters and a second part of parameters, wherein the first part of parameters are parameters that require privacy protection.

[0044] When the user needs to use the trained target LLM model for inference, the inference data and the first part of the parameters can be calculated in the TEE to obtain a first calculation result, and the inference data can be encrypted in the TEE and transmitted to the GPU. The GPU calculates the second calculation result based on the second part of the parameters and the encrypted inference data, and returns the second calculation result to the TEE.

[0045] After receiving the second calculation result returned by the GPU, the TEE can decrypt the second calculation result in the TEE, and generate an inference result corresponding to the inference data based on the first calculation result and the decrypted second calculation result.

[0046] In the above technical solution, on the one hand, by executing the inference calculation of the LLM model on the inference data provided by the user in a secure computing environment built on the service device based on TEE and GPU, the inference data can be prevented from being leaked to the outside during the inference calculation process on the inference data provided by the user;

[0047] On the other hand, by executing the calculation of the first part of the inference data and the LLM model parameters that need to be privacy protected in TEE, and encrypting the inference data in TEE, and then transmitting it to the GPU, the GPU executes the calculation of the second part of the inference data and the LLM model parameters that do not need to be privacy protected. This can ensure that the inference data provided by the user and the first part of the parameters that need to be privacy protected are not leaked to the outside, while making the most of the performance advantages of the GPU in inference calculations, and further improving the computing performance of inference calculations based on confidential inference data.

[0048] See also Figure 1 , Figure 1 This is a flowchart of an inference calculation method based on an LLM model shown in this specification. The solution can be applied to a service device deployed with a target LLM model; the method includes the following execution process:

[0049] Step 102: Calculate the inference data and the first part of parameters in the TEE to obtain a first calculation result;

[0050] The above-mentioned target LLM model can be a general domain large model or a vertical domain large model, which is not particularly limited in this specification.

[0051] It should be noted that the so-called general domain big model refers to a general big model that is trained by widely collecting data samples from various fields and can handle reasoning tasks in various fields. The so-called vertical domain big model refers to a big model that can handle reasoning tasks in a specific target field by further fine-tuning the general domain big model based on the data samples of the general domain big model.

[0052] For example, in one example, the target LLM model may be a LLM service model corresponding to the target service scenario obtained by fine-tuning the initial LLM model using the service data generated in the preset target service scenario as training samples. The initial LLM model may be an open source LLM model.

[0053] The above-mentioned inference data specifically refers to the data that needs to be input into the LLM model for inference calculation. The specific type of the inference data usually depends on the specific type of input data supported by the above-mentioned target LLM model and is not particularly limited in this specification.

[0054] For example, in a traditional LLM model for text reasoning in natural language tasks, the reasoning data may specifically be text data, while for a multimodal LLM model, the reasoning data may include data types such as text, audio, image, and video.

[0055] In this specification, the model parameters of the target LLM model may specifically include a first part of parameters and a second part of parameters. Correspondingly, the inference calculation based on the target LLM model on the inference data provided by the user may also generally include a first part of calculation of the inference data and the first part of parameters, and a second part of calculation of the inference data and the second part of parameters.

[0056] It should be noted that the first part of the parameters may be parameters that require privacy protection; and the second part of the parameters may be parameters that do not require privacy protection.

[0057] For example, in practical applications, the first part of the parameters can be a parameter matrix generated by fine-tuning the initial LLM model using the service data generated in a specific service scenario as training samples. Since these service data are usually not public data and may involve user data privacy, this part of the parameters usually requires additional privacy protection. The second part of the parameters can be a parameter matrix generated by fine-tuning the initial LLM model using public data as training samples. Since public data usually does not involve user data privacy, this part of the parameters usually does not require additional privacy protection.

[0058] In one embodiment shown, the target LLM model may also be a LLM model obtained by fine-tuning the pre-trained initial LLM model using the LoRA (Low-rank adaptation) method; in this case, the first part of the parameters may be a low-rank LoRA parameter matrix. The second part of the parameters may be the original parameter matrix of the initial LLM model.

[0059] It should be noted that LoRA is a method for fine-tuning the LLM model. This method can limit the modification of the LLM model parameters to a lower dimension, thereby reducing the number of parameters that actually need to be learned. This method enables efficient updates on larger models while maintaining the knowledge learned during pre-training. In a standard neural network, the original model parameters are usually a high-dimensional parameter matrix. The basic idea of ​​LoRA is to replace the original parameter matrix of the LLM model with the product of two low-rank parameter matrices. For example, suppose the original parameter matrix is , containing p×q parameters. The basic idea of ​​LoRA is to replace the original weight matrix W with the product of two low-rank parameter matrices A and B, where the low-rank parameter matrix A contains p×r parameters and the low-rank parameter matrix B contains r×q parameters, where the value of r is much smaller than p and q, so the combination of low-rank parameter matrices AA and B will have a parameter amount much smaller than p×q.

[0060] In this specification, in order to provide privacy protection for the above-mentioned first part parameters and the inference data provided by the user, a TEE (Trusted execution environment) can be installed on the service device as a secure execution environment for the above-mentioned first part calculation.

[0061] In this case, when the user needs to use the trained target LLM model for reasoning, the service device can obtain the reasoning data provided by the user, and calculate the reasoning data provided by the user and the first part of the parameters loaded in the TEE in the TEE to obtain a first calculation result.

[0062] Among them, in one embodiment shown, in order to protect the privacy of the reasoning data provided by the user, the user can upload the reasoning data to the TEE carried by the service device through an encrypted data channel.

[0063] For example, in one example, an encrypted data channel can be pre-created between the user client and the TEE carried by the service device. When the user needs to use the trained target LLM model for inference, the data upload operation can be performed on the user client, and the inference data can be uploaded to the TEE carried by the service device using the encrypted data channel.

[0064] The specific type of the encrypted data channel is not particularly limited in this specification.

[0065] For example, in actual applications, the encrypted data channel between the user client and the service device can be an encrypted connection for secure data transmission created between the user client and the service device by running a specific encryption protocol (such as TLS / SSL, IPSec and other encryption protocols).

[0066] It should be noted that, in practical applications, the LLM model is usually based on a neural network architecture, and a neural network is usually a multi-layer network architecture, and each layer in the neural network usually involves different calculations; therefore, the reasoning calculation based on the target LLM model can usually include multiple sub-calculations with a preset execution order.

[0067] For example, see Figure 2 Taking the target LLM model as an LLM model using the Transformer architecture as an example, the multiple sub-calculations with a preset execution order included in the reasoning calculation based on the target LLM model may specifically include:

[0068] Sub-computation 1: Embedding calculation, which is a sub-computation involved in the embedding layer in the Transformer network architecture. The function of Embedding calculation is to encode the inference data input by the user into a vector.

[0069] Sub-computation 2: Normalization calculation (Layernorm) of the vector output by the embedding layer. The function of the normalization calculation is to normalize the values ​​corresponding to the elements contained in the vector output by the embedding layer.

[0070] It should be noted that the sub-computations 1 and 2 listed above are also referred to as sub-computations for performing data preprocessing on the input inference data.

[0071] Sub-computation 3: Matrix operation for calculating the QKV vector. This calculation is a sub-computation included in the multi-head attention layer in the Transformer architecture. The function of this calculation is to calculate the QKV vector based on the normalized vector. It should be explained that the QKV vector contains the Query vector, Key vector, and Value vector.

[0072] Sub-computation 4: BMM (Batch Matrix Multiplication) calculation for the Query vector and Key vector in the QKV vector. This calculation is also a sub-computation included in the multi-head attention layer in the Transformer architecture. The function of this calculation is to calculate the dot product between the Query vector and the Key vector to obtain an attention score matrix.

[0073] Sub-calculation 5: Softmax calculation for the above attention score matrix. This calculation is also a sub-calculation included in the multi-head attention layer in the Transformer architecture. The function of this calculation is to use the Softmax function to convert the score of each position in the attention score matrix into a probability distribution to obtain an attention weight matrix.

[0074] Sub-calculation 6: BMM calculation for the attention weight matrix and the Value vector in the QKV vector. This calculation is also a sub-calculation included in the multi-head attention layer in the Transformer architecture. The function of this calculation is to perform a weighted summation of the attention weight matrix and the Value vector to obtain a weighted Value vector.

[0075] Sub-computation 7: Performs matrix operations on the weighted Value vector. This calculation is also a sub-computation included in the multi-head attention layer in the Transformer architecture. The function of this calculation is to project the output of the multi-head attention layer (that is, the weighted Value vector) to the specified vector space to convert the output of the multi-head attention layer to the required dimension and obtain the projected Value vector.

[0076] Sub-computation 8: Normalization calculation (Layernorm) of the vector input to the feedforward network. This calculation is a sub-computation involved in the feedforward network in the Transformer network architecture. The function of this calculation is to normalize the values ​​corresponding to the elements contained in the vector input to the feedforward network. For example, Figure 2 The vector of the input feedforward network shown is a vector generated by weighted summing the projected Value vector output by the multi-head attention layer and the vector obtained by Embedding calculation of the input inference data.

[0077] Sub-computation 9: The matrix operation for linear transformation of vectors contained in FC (fully connected layers) 1 in the feedforward network. The function of this calculation is to linearly change the vector of input FC1.

[0078] Sub-calculation 10: Perform activation calculation on the linearly changed vector output by FC1. This calculation is also a sub-calculation involved in the feedforward network in the Transformer network architecture. The function of this calculation is to use an activation function (such as the ReLU activation function) to activate the linearly changed vector output by FC1.

[0079] Sub-calculation 11: The matrix operation for linear transformation of the vector contained in FC2 in the feedforward network. The function of this calculation is to linearly change the vector input to FC2.

[0080] It should be noted that Figure 2 The multiple sub-computations with a preset execution order included in the LLM model using the Transformer architecture are shown, which is only an illustrative embodiment provided in this specification. In actual applications, the multiple sub-computations with a preset execution order included in the LLM model using the Transformer architecture can be based on specific needs. Figure 2 The above various sub-calculations are flexibly adjusted based on the above, and will not be listed one by one in this manual.

[0081] In this case, when the service device calculates the inference data provided by the user and the first part of the parameters loaded in the TEE in the TEE, the multiple sub-calculations can be performed in sequence on the inference data based on the first part of the parameters in accordance with the above-mentioned preset execution order to obtain a first calculation result corresponding to the inference data. The first calculation result represents the inference result obtained by performing the above-mentioned inference calculation on the inference data based on the first part of the parameters of the above-mentioned target LLM model.

[0082] For example, see Figure 2 , still taking the above-mentioned target LLM model as an LLM model using the Transformer architecture as an example, after the service device obtains the inference data uploaded to the TEE by the user through the encrypted data channel, it can first execute sub-computation 1 in the TEE, perform embedding calculation on the inference data, and encode the inference data into a vector.

[0083] When sub-computation 1 is completed, the encoded vector output by sub-computation 1 can be used as input data for sub-computation 2, and sub-computation 2 is continued to be executed in the TEE to normalize the values ​​corresponding to the elements contained in the vector.

[0084] After sub-computation 2 is completed, the normalized vector output by sub-computation 2 can be used as input data for sub-computation 3, and sub-computation 3 is continued to be executed in TEE to calculate the QKV vector based on the vector.

[0085] After sub-computation 3 is completed, the QKV vector output by sub-computation 3 can be used as the input data of sub-computation 4, and sub-computation 4 can continue to be executed in TEE. The BMM calculation is performed on the Query vector and Key vector in the QKV vector to obtain an attention score matrix.

[0086] After sub-computation 4 is completed, the attention score matrix output by sub-computation 4 can be used as the input data of sub-computation 5, and sub-computation 5 can be continued to be executed in TEE to perform Softmax calculation on the above attention score matrix to obtain an attention weight matrix.

[0087] After sub-computation 5 is completed, the attention weight matrix output by sub-computation 5 can be used as the input data of sub-computation 6, and sub-computation 6 can continue to be executed in TEE. The BMM calculation is performed on the attention weight matrix and the Value vector in the QKV vector to obtain a weighted Value vector.

[0088] After sub-computation 6 is completed, the weighted Value vector output by sub-computation 6 can be used as input data for sub-computation 7. Sub-computation 7 is continued to be executed in the TEE, and the projection matrix operation is performed on the weighted Value vector.

[0089] When sub-computation 7 is completed, the projected Value vector output by sub-computation 7 can be used as the input data of sub-computation 8 and input into the feedforward network included in the Transformer architecture. Sub-computation 8 is continued to be executed in TEE to perform normalization calculation on the vector input into the feedforward neural network. For example, Figure 2 The vector of the input feedforward network shown is a vector generated by performing a weighted sum operation on the projected Value vector output by sub-computation 7 and the encoded vector output by sub-computation 1.

[0090] After sub-calculation 8 is completed, the normalized vector output by sub-calculation 8 can be used as input data of sub-calculation 9 and input into FC1 in the feedforward network. Sub-calculation 9 is continued to be executed in TEE, and a linear change is performed on the vector input into FC1.

[0091] After sub-calculation 9 is completed, the linearly changed vector output by sub-calculation 9 can be used as input data of sub-calculation 10, and sub-calculation 10 is continued to be executed in TEE, and activation calculation is performed on the linearly changed vector to obtain an activation vector.

[0092] After sub-computation 10 is completed, the activation vector output by sub-computation 10 can be used as input data of sub-computation 11 and input into FC2 in the feedforward network. Sub-computation 11 is continued to be executed in TEE, and a linear change is performed on the activation vector input into FC2.

[0093] Please continue to see Figure 2 When sub-calculation 11 is completed, the vector after linear change output by calculation 11 and the vector input to the feedforward network after sub-calculation 7 are completed can be weighted summed to obtain the first calculation result corresponding to the above-mentioned inference data.

[0094] It should be noted that, in practical applications, if the model framework adopted by the target LLM model is a framework with relatively strong scalability, the reasoning calculation based on the target LLM model may also include multiple reasoning calculations executed in parallel; in this case, each reasoning calculation in the multiple reasoning calculations may respectively include multiple sub-calculations with the above-mentioned preset execution order.

[0095] For example, taking the LLM model using the Transformer architecture as an example, the Transformer architecture usually contains several basic Transformer Blocks, and supports the expansion of the number of Transformer Blocks. Each Transformer Block has a complete Transformer network structure, which means that these Transformer Blocks can perform inference calculations in parallel based on the inference data input by the user.

[0096] Step 104: encrypt the reasoning data in the TEE and transmit it to the GPU, and the GPU performs calculation based on the second part of parameters and the encrypted reasoning data to obtain a second calculation result;

[0097] In this specification, a graphics processing unit (GPU) may also be mounted on the server device as an execution environment for the calculation of the second part.

[0098] In this case, after the service device obtains the reasoning data provided by the user, in addition to calculating the first calculation result by combining the reasoning data provided by the user with the first part of the parameters loaded in the TEE, the reasoning data can also be encrypted in the TEE to obtain the ciphertext data corresponding to the reasoning data. Then, the ciphertext data is transmitted to the GPU through the IO channel between the TEE and the GPU, and the GPU continues to calculate the reasoning data with the second part of the parameters loaded in the GPU to obtain the second calculation result.

[0099] Wherein, as mentioned above, the inference calculation based on the target LLM model can generally include multiple sub-calculations with a preset execution order. When the service device calculates the encrypted inference data with the second part of parameters loaded in the GPU in the GPU, it can still perform the multiple sub-calculations in sequence for the encrypted inference data based on the second part of parameters in accordance with the above-mentioned preset execution order to obtain a second calculation result corresponding to the inference data. Wherein, the second calculation result represents the inference result obtained by performing the above-mentioned inference calculation on the inference data based on the second part of parameters of the above-mentioned target LLM model.

[0100] For example, see Figure 2 , you can execute the same as Figure 2 The same calculation process is to execute the multiple sub-calculations in sequence for the encrypted reasoning data based on the second part of parameters to obtain a second calculation result corresponding to the encrypted reasoning data, and the specific process will not be repeated here.

[0101] It should be noted that the encryption method used to encrypt the inference data in TEE is not specifically limited in this specification; for example, in practical applications, encryption methods that will not affect subsequent inference calculations can be given priority.

[0102] In one embodiment shown, the encryption method used when encrypting the inference data in the TEE may specifically be an encryption method based on a one-time key.

[0103] In this case, a one-time key can be generated for the reasoning data in the TEE; wherein, the one-time key can specifically be a symmetric key; when it is necessary to encrypt the reasoning data provided by the user in the TEE, the reasoning data can be encrypted in the TEE based on the one-time key generated for the reasoning data to obtain ciphertext data corresponding to the reasoning data, and then the ciphertext data can be transmitted to the GPU through the IO channel between the TEE and the GPU, and the GPU performs the above-mentioned second part of the calculation based on the ciphertext data.

[0104] Correspondingly, when the above-mentioned second part of the calculation is completed, the GPU can transmit the second calculation result of the above-mentioned second part of the calculation to the TEE through the IO channel between the TEE and the GPU, and can continue to decrypt the second calculation result in the TEE based on the one-time key to obtain the second calculation result in plain text.

[0105] It should be noted that the encryption algorithm used to generate the one-time key is not particularly limited in this specification. In practical applications, it can be flexibly selected based on specific security requirements.

[0106] For example, in one embodiment shown, the encryption algorithm used to generate the above-mentioned one-time key can specifically be an OTP (One-time padding) algorithm. The OTP algorithm is a very secure encryption method. The algorithm will generate a random key for the data to be encrypted, and each key is a one-time key and is only used once, which makes it very difficult to crack. In the OTP algorithm, when encrypting plaintext, a randomly generated one-time key is used for encryption. This random one-time key usually has the same length as the plaintext and will only be used once. Moreover, when encrypting the plaintext with this random one-time key, it is usually only necessary to add the plaintext and the one-time key to obtain the ciphertext data. When decrypting, it is only necessary to subtract the ciphertext data from the same one-time key to restore the original plaintext data.

[0107] In this case, when the OTP algorithm is used as the encryption algorithm, the one-time key can be a one-time random key with the same length as the inference data generated based on the OTP algorithm. When the inference data is encrypted in the TEE based on the one-time key generated for the inference data, an addition operation can be performed on the inference data and the one-time random key in the TEE to obtain the ciphertext data corresponding to the inference data.

[0108] When the second calculation result is decrypted in TEE based on the one-time key generated for the inference data, a subtraction operation can be performed on the second calculation result and the one-time random key in TEE to restore the plaintext content of the second calculation result.

[0109] Step 106: decrypt the second calculation result in the TEE, and generate an inference result corresponding to the inference data based on the first calculation result and the decrypted second calculation result.

[0110] In this specification, after the GPU calculates the encrypted inference data and the second part of the parameters loaded in the GPU to obtain the second calculation result, the GPU can return the second calculation result to the TEE through the IO channel between the TEE and the GPU. After receiving the second calculation result returned by the GPU, the TEE can also decrypt the second calculation result in the TEE.

[0111] Among them, the decryption method used for decrypting the second calculation result usually corresponds to the encryption method used when encrypting the above-mentioned inference data in TEE.

[0112] In one embodiment shown, the encryption method used when encrypting the inference data in the TEE is still taken as an example, where the encryption method based on a one-time key is used. In this case, the second calculation result can continue to be decrypted in the TEE based on the one-time key generated for the above-mentioned inference data to obtain the second calculation result in plain text.

[0113] For example, in one embodiment shown, the encryption algorithm used to generate the one-time key is still taken as the OTP algorithm. In this case, when the OTP algorithm is used as the encryption algorithm, the one-time key can be a one-time random key with the same length as the inference data generated based on the OTP algorithm. When the second calculation result is decrypted in the TEE based on the one-time key generated for the inference data, a subtraction operation can be performed on the second calculation result and the one-time random key in the TEE to restore the plain text content of the second calculation result.

[0114] It should be noted that since the above-mentioned second calculation result is usually a matrix operation result obtained by performing a matrix multiplication operation on the second part of the parameters of the above-mentioned target LLM model and the encrypted inference data; therefore, when performing a subtraction operation on the second calculation result and the one-time random key in the TEE to decrypt the second calculation result, it is necessary to consider the influence of the second part of the parameters of the target LLM model on the decryption result.

[0115] In one embodiment shown, the second part of the parameters of the target LLM model can also be pre-loaded in the TEE. Before the second calculation result is decrypted in the TEE, a matrix multiplication operation can be performed in the TEE on the second part of the parameters of the target LLM model loaded in the TEE and the generated one-time random key to obtain the target matrix operation result.

[0116] Among them, it should be noted that by performing matrix multiplication operations on the model parameter matrix of the target LLM model and the generated one-time random key in advance in the TEE, the time consumed by this part of the calculation can be excluded from the overall time consumed by the inference calculation on the inference data, thereby reducing the inference time for the inference calculation on the inference data.

[0117] When the second calculation result is decrypted in TEE based on the one-time key generated for the inference data, a matrix subtraction operation can be performed in TEE on the second calculation result and the pre-calculated target matrix operation result to restore the plaintext content of the second calculation result.

[0118] For example, suppose the second part of the target LLM model is parameterized by Indicates that the one-time random key generated for the inference data based on the OTP algorithm is used Indicates that the inference data is represented by X, then in TEE, the inference data X and the one-time random key The ciphertext data of the input data X obtained by adding can be expressed as , based on the second part of the parameters on the GPU The second calculation result of the input data X obtained after performing matrix multiplication on the ciphertext data can be expressed as .

[0119] The second calculation result of the input data X on the TEE To decrypt is actually to solve Through simple mathematical operations, we can know Therefore, the second part of the parameters can be pre-set on the TEE Perform matrix multiplication with the one-time random key R to obtain the target matrix operation result , in the right When decrypting, the calculation result of the input data X is , and the pre-computed target matrix operation result By performing matrix subtraction, the calculation results in plain text can be restored .

[0120] After the second calculation result is decrypted in the TEE, the first calculation result and the decrypted second calculation result may be merged in the TEE, and based on the first calculation result and the second calculation result, an inference result corresponding to the inference data may be jointly generated.

[0121] The specific method for combining the first reasoning result and the second reasoning result is not particularly limited in this specification.

[0122] For example, in one embodiment shown, the first reasoning result and the second reasoning result may be weighted and summed to obtain a reasoning result corresponding to the reasoning data.

[0123] See also Figure 3 , Figure 3 This is a schematic diagram of performing inference calculation on inference data based on the target LLM model shown in this specification.

[0124] like Figure 3 As shown, taking the above-mentioned target LLM model as an example, which is an LLM model obtained by fine-tuning the pre-trained initial LLM model using the LoRA method, in addition to performing inference calculations on the inference data based on the first part of the parameters of the target LLM model (i.e., the low-rank LoRA parameters) in the TEE to obtain a first calculation result corresponding to the inference data, the same calculation process can also be executed in the GPU, and inference calculations are performed on the ciphertext data corresponding to the inference data based on the second part of the parameters of the target LLM model to obtain a second calculation result corresponding to the ciphertext data, and the second calculation result is decrypted in the TEE.

[0125] After the second calculation result is decrypted in the TEE, a weighted sum calculation may be performed on the first calculation result and the second calculation result in the TEE to obtain an inference result corresponding to the inference data.

[0126] Since the LoRA parameters of the target LLM model are usually generated by fine-tuning the initial LLM model with the service data generated in a specific service scenario as training samples, and these service data are usually not public data and may involve user data privacy, the LoRA parameter matrix of the target LLM model is loaded into TEE, and the LoRA parameter matrix is ​​used for inference calculation in a closed and secure computing environment such as TEE, the model parameters in the LoRA parameter matrix can be effectively protected, thereby avoiding the leakage of user data privacy involved in the parameters contained in the LoRA parameter matrix.

[0127] On the contrary, since the initial model parameters of the target LLM model (i.e., the second part of the parameters) are usually a parameter matrix generated by fine-tuning the initial LLM model using public data as training samples, the model parameter matrix usually does not involve user data privacy. Therefore, by loading the model parameter matrix of the target LLM model into the GPU and using the model parameter matrix for inference calculations in a public computing environment such as the GPU, there is no risk of leaking user data privacy.

[0128] Corresponding to the embodiments of the aforementioned method, this specification also provides embodiments of an apparatus, an electronic device, and a storage medium.

[0129] Figure 4 is a schematic structural diagram of an electronic device provided by an exemplary embodiment. Figure 4 At the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, a memory 408, and a non-volatile memory 410, and may also include other required hardware. One or more embodiments of this specification may be implemented based on software, such as the processor 402 reading the corresponding computer program from the non-volatile memory 410 into the memory 408 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0130] like Figure 5 As shown, Figure 5 This is a block diagram of an inference calculation based on an LLM model according to an exemplary embodiment of the present specification. The device can be applied to Figure 4 In the electronic device shown, the technical solution of this specification is implemented. The model parameters of the target LLM model include a first part of parameters and a second part of parameters, wherein the first part of parameters are parameters that require privacy protection; the service device has a TEE and a GPU; the device 500 includes:

[0131] A first calculation module 501 calculates the inference data and the first part of parameters in the TEE to obtain a first calculation result;

[0132] A transmission module 502 encrypts the reasoning data in the TEE and transmits it to the GPU, and the GPU performs calculation based on the second part of parameters and the encrypted reasoning data to obtain a second calculation result;

[0133] The second calculation module 503 decrypts the second calculation result in the TEE, and generates an inference result corresponding to the inference data based on the first calculation result and the decrypted second calculation result.

[0134] Accordingly, the present specification also provides an electronic device, which includes a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to implement the steps in all the method flows described above.

[0135] Accordingly, the present specification also provides a computer-readable storage medium on which executable computer program instructions are stored; wherein, when the instructions are executed by a processor, the steps in all the method flows described above are implemented.

[0136] Accordingly, the present specification also provides a computer program product having executable computer program instructions stored thereon; wherein, when the computer program instructions are executed by a processor, the steps in all the method flows described above are implemented.

[0137] In the 1990s, it was very clear whether the improvement of a technology was hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with hardware entity modules. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0138] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, and the memory controller may also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that, in addition to implementing the controller in a purely computer-readable program code manner, the controller may be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the devices for implementing various functions included therein may also be considered as structures within the hardware component. Or even, the devices for implementing various functions may be considered as both software modules for implementing the method and structures within the hardware component.

[0139] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a server system. Of course, the present application does not exclude that with the development of computer technology in the future, the computer that implements the functions of the above embodiments may be, for example, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0140] Although one or more embodiments of the present specification provide method operation steps as described in the embodiments or flow charts, more or fewer operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiments is only one way of executing the order of many steps, and does not represent the only execution order. When the device or terminal product in practice is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment). The terms "include", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, product or device including a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements inherent to such a process, method, product or device. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or device including the elements. For example, if the words first, second, etc. are used to represent the name, they do not represent any specific order.

[0141] For the convenience of description, the above devices are described in various modules according to their functions. Of course, when implementing one or more of the present specification, the functions of each module can be implemented in the same or more software and / or hardware, or the module implementing the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0142] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0143] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0144] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0145] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0146] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0147] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage, graphene storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0148] It should be understood by those skilled in the art that one or more embodiments of the present specification may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of the present specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0149] One or more embodiments of this specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of this specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0150] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. In the description of this specification, the description of the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of this specification. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.

[0151] The above description is only an example of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. For those skilled in the art, one or more embodiments of this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification shall be included in the scope of the claims.

Claims

1. An inference calculation method based on an LLM model, applied to a service device that deploys a target LLM model; the model parameters of the target LLM model include a first part of parameters and a second part of parameters, wherein the first part of parameters are parameters that require privacy protection; The service device has a TEE and a GPU, and the method includes: In the TEE, the inference data and the first part of parameters are inferred to obtain a first calculation result; The inference data is encrypted in the TEE and transmitted to the GPU, and the GPU performs inference calculation based on the second part of parameters and the encrypted inference data to obtain a second calculation result; The second calculation result is decrypted in the TEE, and an inference result corresponding to the inference data is generated based on the first calculation result and the decrypted second calculation result.

2. According to the method as claimed in claim 1, the target LLM model is an LLM model obtained by fine-tuning the pre-trained initial LLM model using the LoRA method; the first part of parameters is a low-rank LoRA parameter.

3. The method according to claim 1, generating an inference result corresponding to the inference data based on the first calculation result and the decrypted second calculation result, comprising: A weighted sum calculation is performed on the first calculation result and the second calculation result to obtain an inference result corresponding to the inference data.

4. The method according to claim 1, encrypting the inference data in the TEE, comprising: encrypting the inference data in the TEE based on a one-time key generated for the inference data; Decrypting the second calculation result in the TEE includes: The second calculation result is decrypted in the TEE based on the one-time key generated for the inference data.

5. The method according to claim 4, wherein the one-time key is a one-time random key generated based on an OTP algorithm and having the same length as the inference data; Encrypting the inference data in the TEE based on a one-time key generated for the inference data includes: Performing an addition calculation on the inference data and the one-time random key in the TEE to encrypt the inference data; Decrypting the second calculation result in the TEE based on the one-time key generated for the inference data includes: A subtraction calculation is performed on the second calculation result and the one-time random key in the TEE to decrypt the second calculation result.

6. The method according to claim 5, wherein the second calculation result is a matrix operation result obtained by performing a matrix multiplication operation on the second part of parameters and the encrypted reasoning data; Before performing a subtraction calculation on the second calculation result and the one-time random key in the TEE to decrypt the second calculation result, the method further includes: Performing a matrix multiplication operation on the second part of parameters and the one-time random key in the TEE to obtain a target matrix operation result; Performing a subtraction calculation on the second calculation result and the one-time random key in the TEE to decrypt the second calculation result includes: In the TEE, a matrix subtraction operation is performed on the second calculation result and the target matrix operation result to decrypt the second calculation result.

7. The method according to claim 1, before performing inference calculation on the inference data and the first part of parameters in the TEE to obtain the first calculation result, further comprising: Obtain the inference data uploaded by the user to the TEE through an encrypted data channel.

8. An inference computing device based on an LLM model, applied to a service device that deploys a target LLM model; the model parameters of the target LLM model include a first part of parameters and a second part of parameters, wherein the first part of parameters are parameters that require privacy protection; The service device has a TEE and a GPU, and the device includes: A first calculation module performs inference calculation on the inference data and the first part of parameters in the TEE to obtain a first calculation result; A transmission module, which encrypts the inference data in the TEE and transmits it to the GPU, and the GPU performs inference calculation based on the second part of parameters and the encrypted inference data to obtain a second calculation result; The second computing module decrypts the second computing result in the TEE, and generates an inference result corresponding to the inference data based on the first computing result and the decrypted second computing result.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images