A method and device for reasoning calculation based on LLM model

Abstract

A method for reasoning and computing based on an LLM model is applied to a service device that deploys a target LLM model; wherein the reasoning and computing based on the target LLM model includes multiple sub-computations with a preset execution order; the method includes: obtaining reasoning data uploaded by a user to a TEE carried by the service device through an encrypted data channel; executing the multiple sub-computations in sequence on the reasoning data based on a model parameter matrix of the target LLM model according to the preset execution order to obtain a first reasoning result corresponding to the reasoning data; wherein the multiple sub-computations include at least one nonlinear sub-computation executed in the TEE; and at least one linear sub-computation executed in a GPU carried by the service device; the input data of the at least one linear sub-computation is encrypted in the TEE and then transmitted from the TEE to the GPU.

Description

Technical Field

[0001] The embodiments of this specification belong to the field of artificial intelligence technology, and more particularly to an inference calculation method and device based on an LLM model. Background Art

[0002] The data held by different data holders may usually contain the user's private information, so sharing data between different data holders may infringe on the user's privacy. Based on this, in some solutions, in order to open up the data flow between multiple data holders and avoid the data island phenomenon, secure multi-party computing technology can usually be used to achieve data sharing between different data holders; for example, in actual applications, in the application scenario of jointly training artificial intelligence models based on data held by multiple data holders, secure multi-party computing technology can be used to participate in training based on ciphertext data, thereby ensuring that the plaintext content of the data will not be leaked when multiple parties interact with each other during the model training process.

[0003] With the continuous development of the LLM (Large Language Model) model, the use of secure multi-party computing technology to jointly train an LLM model by combining data held by multiple data holders has gradually matured, allowing the LLM model to achieve very good results in different downstream tasks in many fields. Summary of the invention

[0004] This specification proposes an inference calculation method based on an LLM model, which is applied to a service device that deploys a target LLM model; wherein the inference calculation based on the target LLM model includes multiple sub-calculations with a preset execution order; the method includes:

[0005] Obtaining the inference data uploaded by the user to the TEE carried by the service device through an encrypted data channel;

[0006] According to the preset execution order, the multiple sub-calculations are sequentially performed on the reasoning data based on the model parameter matrix of the target LLM model to obtain a first reasoning result corresponding to the reasoning data;

[0007] Among them, the multiple sub-calculations include at least one nonlinear sub-calculation performed in the TEE; and at least one linear sub-calculation performed in the GPU carried by the service device; the at least one linear sub-calculation is a linear matrix operation based on the model parameter matrix of the target LLM model; the input data of the at least one linear sub-calculation is encrypted in the TEE and then transmitted from the TEE to the GPU; the calculation result of the at least one linear sub-calculation is transmitted from the GPU to the TEE and decrypted in the TEE.

[0008] Optionally, the inference calculation performed based on the target LLM model includes multiple inference calculations executed in parallel; each of the multiple inference calculations respectively includes the multiple sub-calculations having the preset execution order.

[0009] Optionally, the target LLM model is an LLM model obtained by fine-tuning a pre-trained initial LLM model using a LoRA method; the model parameters of the target LLM model also include a low-rank LoRA parameter matrix;

[0010] The method further comprises:

[0011] In the TEE, in accordance with the row order, the polynomial calculations are sequentially performed on the reasoning data based on the LoRA parameter matrix to obtain a second reasoning result corresponding to the reasoning data;

[0012] In the TEE, an inference result corresponding to the inference data is generated based on the first inference result and the second inference result.

[0013] Optionally, generating an inference result corresponding to the inference data based on the first inference result and the second inference result includes:

[0014] A weighted sum calculation is performed on the first reasoning result and the second reasoning result to obtain an reasoning result corresponding to the reasoning data.

[0015] Optionally, encrypting the input data of the linear sub-computation in the TEE includes:

[0016] Encrypting the input data of the linear sub-computation in the TEE based on a one-time key generated for the input data of the linear sub-computation;

[0017] The decryption process is performed in the TEE on the calculation result of the linear sub-calculation, including:

[0018] In the TEE, a calculation result of the linear sub-computation is decrypted based on the one-time key generated for the input data of the linear sub-computation.

[0019] Optionally, the one-time key is a one-time random key generated based on an OTP algorithm and having the same length as the input data of the linear sub-calculation;

[0020] The input data of the linear sub-computation is encrypted in the TEE based on a one-time key generated for the input data of the linear sub-computation, including:

[0021] Performing an addition operation on the input data of the linear sub-computation and the one-time random key in the TEE to encrypt the input data of the linear sub-computation;

[0022] Optionally, the calculation result of the linear sub-calculation is a matrix operation result obtained by performing a matrix multiplication operation on the model parameter matrix and the input data of the linear sub-calculation after encryption processing;

[0023] Before decrypting the calculation result of the linear sub-calculation in the TEE based on the one-time key generated for the input data of the linear sub-calculation, the method further includes:

[0024] Performing a matrix multiplication operation on the model parameter matrix and the one-time random key in the TEE to obtain a target matrix operation result;

[0025] Decrypting the calculation result of the linear sub-calculation in the TEE based on the one-time key generated for the input data of the linear sub-calculation includes:

[0026] In the TEE, a matrix subtraction operation is performed on the calculation result of the linear sub-calculation and the target matrix operation result to decrypt the calculation result of the linear sub-calculation.

[0027] Optionally, the input data of the at least one linear sub-calculation is input data that has been subjected to sparse processing in the TEE; the input data after the sparse processing is encrypted in the TEE and then transmitted from the TEE to the GPU.

[0028] Optionally, the input data of at least part of the linear sub-calculations in the at least one linear sub-calculation is input data that has been sparsely processed in the TEE; wherein, the at least part of the linear sub-calculations is a linear sub-calculation in the at least one linear sub-calculation that uses the calculation result of the nonlinear sub-calculation as input data.

[0029] Optionally, the at least part of the linear sub-calculation is a linear sub-calculation in which a calculation result of a nonlinear sub-calculation in which an activation function is introduced into the at least one nonlinear sub-calculation is used as input data.

[0030] Optionally, performing sparse processing on the input data of the linear sub-calculation includes:

[0031] Invalid elements included in the input data of the linear sub-computation are deleted.

[0032] Optionally, deleting the invalid element contained in the input data of the linear sub-calculation includes:

[0033] Obtaining pre-estimated position information of invalid elements contained in input data of the linear sub-calculation;

[0034] The data element corresponding to the position information and included in the input data of the linear sub-calculation is deleted.

[0035] Optionally, estimating position information of invalid elements contained in input data of the linear sub-computation includes:

[0036] Acquire a plurality of input data samples corresponding to the linear sub-computation; wherein the plurality of input data samples are input data samples in plain text generated in the process of respectively performing the inference calculation on a plurality of inference data samples in plain text;

[0037] Respectively counting the position information of the invalid elements contained in the multiple input data samples, and performing an intersection operation on the counted position information of the invalid elements contained in the multiple input data samples to obtain the position information of the invalid elements contained in the same position in the multiple input data samples;

[0038] The position information of the invalid elements at the same position included in the plurality of input data samples is stored as an estimation result of the position information of the invalid elements included in the linear sub-calculation.

[0039] Optionally, the at least part of the linear sub-calculation is a linear sub-calculation in which a calculation result of a nonlinear sub-calculation in which a ReLU activation function is introduced in the at least one nonlinear sub-calculation is used as input data; accordingly, the invalid element is a 0 element;

[0040] Deleting invalid elements contained in the input data of the linear sub-calculation, including:

[0041] Delete the zero elements contained in the input data of the linear sub-computation.

[0042] Optionally, the GPU is loaded with the original parameter matrix of the target LLM model after low-bit-width processing; the at least one linear sub-calculation is a linear matrix operation based on the original parameter matrix after the low-bit-width processing; the input data of the at least one linear sub-calculation is the input data subjected to the low-bit-width quantization processing in the TEE; the input data after the low-bit-width quantization processing is encrypted in the TEE and then transmitted from the TEE to the GPU; the calculation result of the at least one linear sub-calculation is transmitted from the GPU to the TEE and decrypted after being restored to the calculation result of the standard bit-width in the TEE.

[0043] Optionally, the data supported by the GPU for processing is a floating point number of a first bit width; the low bit width quantization processing includes converting the value of the data from the floating point number of the first bit width into a fixed point number of a second bit width; wherein the second bit width is lower than the first bit width;

[0044] The original parameter matrix is ​​subjected to low bit width processing, including:

[0045] Converting the parameter value of each model parameter contained in the original parameter matrix from a floating point number of the first bit width to a fixed point number of the second bit width;

[0046] Performing low bit width quantization processing on input data of the at least one linear sub-calculation in the TEE includes:

[0047] In the TEE, a value of input data of the at least one linear sub-calculation is converted from a floating point number of the first bit width to a fixed point number of the second bit width.

[0048] Optionally, restoring the calculation result of the at least one linear sub-calculation returned by the GPU to a calculation result of a standard bit width in the TEE includes:

[0049] In the TEE, the calculation result of the at least one linear sub-calculation returned by the GPU is converted from a fixed-point number of the second bit width to a floating-point number of the first bit width.

[0050] Optionally, the low bit width quantization is SmoothQuant quantization; the first bit width is 32 bits; the second bit width is 8 bits; the low bit width quantization processing includes converting the data value from a 32-bit floating point number to an 8-bit integer.

[0051] Optionally, the target LLM model is an LLM model using a Transformer architecture; the linear sub-calculation includes one or more combinations shown below:

[0052] The matrix operations used to calculate the QKV vectors included in the multi-head attention layer in the Transformer architecture;

[0053] The multi-head attention layer in the Transformer architecture includes a matrix operation for Projection of the output results of the multi-head attention layer;

[0054] The fully connected layer in the feedforward network in the Transformer architecture contains matrix operations that perform linear transformations on vectors.

[0055] This specification also proposes an inference computing device based on an LLM model, which is applied to a service device that deploys a target LLM model; wherein the inference computing based on the target LLM model includes multiple sub-computations with a preset execution order; the device includes:

[0056] An acquisition module, which acquires the inference data uploaded by the user to the TEE carried by the service device through an encrypted data channel;

[0057] A computing module, in accordance with the preset execution order, sequentially performs the multiple sub-calculations on the inference data based on the model parameter matrix of the target LLM model to obtain a first inference result corresponding to the inference data; wherein the multiple sub-calculations include at least one nonlinear sub-calculation performed in the TEE; and at least one linear sub-calculation performed in the GPU carried by the service device; the at least one linear sub-calculation is a linear matrix operation based on the model parameter matrix of the target LLM model; the input data of the at least one linear sub-calculation is encrypted in the TEE and then transmitted from the TEE to the GPU; the calculation result of the at least one linear sub-calculation is transmitted from the GPU to the TEE and decrypted in the TEE.

[0058] In the above embodiment, on the one hand, by performing the inference calculation of the LLM model on the inference data provided by the user in a secure computing environment built on the service device based on TEE and GPU, the inference data can be prevented from being leaked to the outside during the inference calculation process on the inference data provided by the user;

[0059] On the other hand, by executing the nonlinear operations included in the reasoning calculation in the TEE, and encrypting the input data of the linear operations included in the reasoning calculation in the TEE, and then transmitting it to the GPU for execution in the GPU, it is possible to maximize the use of the performance advantages of the GPU in executing linear operations while ensuring that the reasoning data provided by the user and the intermediate calculation results obtained by reasoning calculations on the reasoning data are not leaked to the outside, thereby further improving the computing performance of reasoning calculations based on confidential reasoning data. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0061] Figure 1 is a flow chart of an inference calculation method based on an LLM model shown in an embodiment of this specification;

[0062] Figure 2 It is a flowchart of performing reasoning calculation based on an LLM model using a Transformer architecture shown in an embodiment of this specification;

[0063] Figure 3 It is a schematic diagram of performing reasoning calculation based on an LLM model fine-tuned and trained by the LoRA method shown in an embodiment of this specification;

[0064] Figure 4 is a schematic structural diagram of an electronic device shown in an embodiment of this specification;

[0065] Figure 5 It is a block diagram of an inference calculation based on the LLM model shown in an embodiment of this specification. DETAILED DESCRIPTION

[0066] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0067] In the scenario where data held by multiple data holders are combined to jointly train an LLM model, the data held by each party may involve user privacy; therefore, in practical applications, secure multi-party computing technology can usually be introduced during the training process to ensure that the plaintext content of the training data is not leaked.

[0068] However, after the LLM model is trained, when the user uses the LLM model for reasoning, the reasoning data input by the user into the LLM model may also involve the user's privacy; therefore, if the reasoning data input by the user into the LLM model is not protected, the user's privacy may still be leaked.

[0069] Based on this, this specification proposes a secure reasoning framework for the LLM model based on TEE and GPU.

[0070] Under this framework, the service device on which the target LLM model is deployed can be equipped with TEE and GPU respectively. The reasoning calculation based on the target LLM model can include multiple sub-calculations with a preset execution order; wherein the multiple sub-calculations include at least one nonlinear sub-calculation executed in the TEE; and at least one linear sub-calculation executed in the GPU. The at least one linear sub-calculation is a linear matrix operation based on the model parameter matrix of the target LLM model.

[0071] When the user needs to use the trained target LLM model for reasoning, the reasoning data can be uploaded to the TEE carried by the service device through an encrypted data channel. After obtaining the reasoning data uploaded by the user to the TEE carried by the service device through the encrypted data channel, the service device can perform the multiple sub-calculations on the reasoning data in sequence according to the preset execution order and based on the model parameter matrix of the target LLM model to obtain a first reasoning result corresponding to the reasoning data.

[0072] In the process of executing the multiple sub-calculations in sequence according to the preset execution order, the nonlinear sub-calculations included in the multiple sub-calculations can be executed in the TEE, and the linear sub-calculations included in the multiple sub-calculations can be executed in the GPU.

[0073] When executing these linear sub-calculations, the input data of these linear sub-calculations can be encrypted in the TEE, and then the TEE transmits the encrypted ciphertext data corresponding to the input data to the GPU, and the GPU executes these linear sub-calculations based on the ciphertext data. After completing these linear sub-calculations, the GPU can transmit the calculation results to the TEE and decrypt them in the TEE.

[0074] In the above technical solution, on the one hand, by executing the inference calculation of the LLM model on the inference data provided by the user in a secure computing environment built on the service device based on TEE and GPU, the inference data can be prevented from being leaked to the outside during the inference calculation process on the inference data provided by the user;

[0075] On the other hand, by executing the nonlinear operations included in the reasoning calculation in the TEE, and encrypting the input data of the linear operations included in the reasoning calculation in the TEE, and then transmitting it to the GPU for execution in the GPU, it is possible to maximize the use of the performance advantages of the GPU in executing linear operations while ensuring that the reasoning data provided by the user and the intermediate calculation results obtained by reasoning calculations on the reasoning data are not leaked to the outside, thereby further improving the computing performance of reasoning calculations based on confidential reasoning data.

[0076] See also Figure 1 , Figure 1 This is a flowchart of an inference calculation method based on an LLM model shown in this specification. The solution can be applied to a service device deployed with a target LLM model; the method includes the following execution process:

[0077] Step 102: Obtain the inference data uploaded by the user to the TEE carried by the service device through the encrypted data channel;

[0078] The above-mentioned target LLM model can be a general domain large model or a vertical domain large model, which is not particularly limited in this specification.

[0079] It should be noted that the so-called general domain big model refers to a general big model that is trained by widely collecting data samples from various fields and can handle reasoning tasks in various fields. The so-called vertical domain big model refers to a big model that can handle reasoning tasks in a specific target field by further fine-tuning the general domain big model based on the data samples of the general domain big model.

[0080] For example, in one example, the target LLM model may be a LLM service model corresponding to the target service scenario obtained by fine-tuning the initial LLM model using the service data generated in the preset target service scenario as training samples. The initial LLM model may be an open source LLM model.

[0081] The above-mentioned inference data specifically refers to the data that needs to be input into the LLM model for inference calculation. The specific type of the inference data usually depends on the specific type of input data supported by the above-mentioned target LLM model and is not particularly limited in this specification.

[0082] For example, in a traditional LLM model for text reasoning in natural language tasks, the reasoning data may specifically be text data, while for a multimodal LLM model, the reasoning data may include data types such as text, audio, image, and video.

[0083] In this specification, a TEE (Trusted execution environment) can be installed on the service device as a secure execution environment for the reasoning calculation of the LLM model. When the user needs to use the trained target LLM model for reasoning, in order to protect the privacy of the reasoning data provided by the user, the user can upload the reasoning data to the TEE installed on the service device through an encrypted data channel.

[0084] For example, in one example, an encrypted data channel can be pre-created between the user client and the TEE carried by the service device. When the user needs to use the trained target LLM model for inference, the data upload operation can be performed on the user client, and the inference data can be uploaded to the TEE carried by the service device using the encrypted data channel.

[0085] The specific type of the encrypted data channel is not particularly limited in this specification.

[0086] For example, in actual applications, the encrypted data channel between the user client and the service device can be an encrypted connection for secure data transmission created between the user client and the service device by running a specific encryption protocol (such as TLS / SSL, IPSec and other encryption protocols).

[0087] Step 104: According to the preset execution order, the multiple sub-calculations are sequentially performed on the inference data based on the model parameter matrix of the target LLM model to obtain a first inference result corresponding to the inference data; wherein the multiple sub-calculations include at least one nonlinear sub-calculation performed in the TEE; and at least one linear sub-calculation performed in the GPU carried by the service device; the at least one linear sub-calculation is a linear matrix operation based on the model parameter matrix of the target LLM model; the input data of the at least one linear sub-calculation is encrypted in the TEE and then transmitted from the TEE to the GPU; the calculation result of the at least one linear sub-calculation is transmitted from the GPU to the TEE and decrypted in the TEE.

[0088] In practical applications, the LLM model is usually based on a neural network architecture, and a neural network is usually a multi-layer network architecture, and each layer in the neural network usually involves different calculations; therefore, the reasoning calculation based on the target LLM model can specifically include multiple sub-calculations with a preset execution order.

[0089] For example, see Figure 2 Taking the target LLM model as an LLM model using the Transformer architecture as an example, the multiple sub-calculations with a preset execution order included in the reasoning calculation based on the target LLM model may specifically include:

[0090] Sub-computation 1: Embedding calculation, which is a sub-computation involved in the embedding layer in the Transformer network architecture. The function of Embedding calculation is to encode the inference data input by the user into a vector.

[0091] Sub-computation 2: Normalization calculation (Layernorm) of the vector output by the embedding layer. The function of the normalization calculation is to normalize the values ​​corresponding to the elements contained in the vector output by the embedding layer.

[0092] It should be noted that the sub-computations 1 and 2 listed above are also referred to as sub-computations for performing data preprocessing on the input inference data.

[0093] Sub-computation 3: Matrix operation for calculating the QKV vector. This calculation is a sub-computation included in the multi-head attention layer in the Transformer architecture. The function of this calculation is to calculate the QKV vector based on the normalized vector. It should be explained that the QKV vector contains the Query vector, Key vector, and Value vector.

[0094] Sub-computation 4: BMM (Batch Matrix Multiplication) calculation for the Query vector and Key vector in the QKV vector. This calculation is also a sub-computation included in the multi-head attention layer in the Transformer architecture. The function of this calculation is to calculate the dot product between the Query vector and the Key vector to obtain an attention score matrix.

[0095] Sub-calculation 5: Softmax calculation for the above attention score matrix. This calculation is also a sub-calculation included in the multi-head attention layer in the Transformer architecture. The function of this calculation is to use the Softmax function to convert the score of each position in the attention score matrix into a probability distribution to obtain an attention weight matrix.

[0096] Sub-calculation 6: BMM calculation for the attention weight matrix and the Value vector in the QKV vector. This calculation is also a sub-calculation included in the multi-head attention layer in the Transformer architecture. The function of this calculation is to perform a weighted summation of the attention weight matrix and the Value vector to obtain a weighted Value vector.

[0097] Sub-computation 7: Performs matrix operations on the weighted Value vector. This calculation is also a sub-computation included in the multi-head attention layer in the Transformer architecture. The function of this calculation is to project the output of the multi-head attention layer (that is, the weighted Value vector) to the specified vector space to convert the output of the multi-head attention layer to the required dimension and obtain the projected Value vector.

[0098] Sub-computation 8: Normalization calculation (Layernorm) of the vector input to the feedforward network. This calculation is a sub-computation involved in the feedforward network in the Transformer network architecture. The function of this calculation is to normalize the values ​​corresponding to the elements contained in the vector input to the feedforward network. For example, Figure 2 The vector of the input feedforward network shown is a vector generated by weighted summing the projected Value vector output by the multi-head attention layer and the vector obtained by Embedding calculation of the input inference data.

[0099] Sub-computation 9: The matrix operation for linear transformation of vectors contained in FC (fully connected layers) 1 in the feedforward network. The function of this calculation is to linearly change the vector of input FC1.

[0100] Sub-calculation 10: Perform activation calculation on the linearly changed vector output by FC1. This calculation is also a sub-calculation involved in the feedforward network in the Transformer network architecture. The function of this calculation is to use an activation function (such as the ReLU activation function) to activate the linearly changed vector output by FC1.

[0101] Sub-calculation 11: The matrix operation for linear transformation of the vector contained in FC2 in the feedforward network. The function of this calculation is to linearly change the vector input to FC2.

[0102] It should be noted that Figure 2The multiple sub-computations with a preset execution order included in the LLM model using the Transformer architecture are shown, which is only an illustrative embodiment provided in this specification. In actual applications, the multiple sub-computations with a preset execution order included in the LLM model using the Transformer architecture can be based on specific needs. Figure 2 The above various sub-calculations are flexibly adjusted based on the above, and will not be listed one by one in this manual.

[0103] In one embodiment shown, if the model framework adopted by the target LLM model is a framework with relatively strong scalability, the reasoning calculation based on the target LLM model may include multiple reasoning calculations executed in parallel; in this case, each reasoning calculation in the multiple reasoning calculations may respectively include multiple sub-calculations with the above-mentioned preset execution order.

[0104] For example, taking the LLM model using the Transformer architecture as an example, the Transformer architecture usually contains several basic Transformer Blocks, and supports the expansion of the number of Transformer Blocks. Each Transformer Block has a complete Transformer network structure, which means that these Transformer Blocks can perform inference calculations in parallel based on the inference data input by the user.

[0105] After the service device obtains the inference data uploaded to the TEE by the user through the encrypted data channel, it can use the model parameter matrix of the target LLM model to sequentially execute the above-mentioned multiple sub-calculations included in the above-mentioned inference calculation for the inference data to obtain the first inference result corresponding to the inference data.

[0106] The reasoning calculation based on the target LLM model includes multiple sub-calculations with a preset execution order, which can usually include at least one nonlinear sub-calculation and at least one linear sub-calculation. The linear sub-calculation can usually be a linear matrix operation based on the parameter matrix of the LLM model.

[0107] For example, see Figure 2, still taking the target LLM model as an example, the LLM model using the Transformer architecture, among the multiple sub-calculations with a preset execution order included in the inference calculation based on the target LLM model, the sub-calculations belonging to linear calculations include sub-calculations 3, 7, 9, and 11. These linear sub-calculations usually need to perform linear matrix operations based on the parameter matrix and input data of the LLM model. Except for sub-calculations 3, 7, 9, and 11, all other sub-calculations are nonlinear sub-calculations.

[0108] In the process of executing the above-mentioned multiple sub-calculations in sequence according to the preset execution order, the service device can, on the one hand, execute the nonlinear sub-calculations included in the multiple sub-calculations in the TEE. On the other hand, the nonlinear sub-calculations included in the multiple sub-calculations can be executed in the GPU.

[0109] Among them, compared with TEE, GPU usually lacks security protection for input data and intermediate calculation results during the calculation process; therefore, in practical applications, the input data of the linear sub-calculations contained in the above multiple sub-calculations can be encrypted in TEE to obtain the ciphertext data corresponding to the input data. Then, the ciphertext data is transmitted to the GPU through the IO channel between TEE and GPU, and the GPU executes these linear sub-calculations based on the ciphertext data.

[0110] It should be noted that the input data of the linear sub-calculation included in the above-mentioned multiple sub-calculations may generally be the output data of the upstream sub-calculation of the linear sub-calculation.

[0111] Of course, if the upstream sub-computation of the linear sub-computation is also a linear sub-computation, since the calculation result output by the upstream linear sub-computation is itself in the form of ciphertext, the calculation result output by the upstream linear sub-computation does not need to be repeatedly encrypted when used as input data for the downstream sub-computation.

[0112] After these linear sub-calculations are executed on the GPU, the GPU can also transmit the calculation results of these linear sub-calculations to the TEE through the IO channel between the TEE and the GPU, and then decrypt the calculation results of these linear sub-calculations in the TEE.

[0113] For example, see Figure 2, still taking the above target LLM model as an LLM model using the Transformer architecture as an example, after the service device obtains the inference data uploaded to the TEE by the user through the encrypted data channel, it can first execute sub-computation 1 in the TEE, perform embedding calculation on the inference data, and encode the inference data into a vector. Among them, sub-computation 1 is a nonlinear sub-computation.

[0114] When sub-computation 1 is completed, the encoded vector output by sub-computation 1 can be used as the input data of sub-computation 2, and sub-computation 2 is continued to be executed in TEE to normalize the values ​​corresponding to the elements contained in the vector. Among them, sub-computation 2 is also a nonlinear sub-computation.

[0115] When sub-calculation 2 is completed, the normalized vector output by sub-calculation 2 can be used as the input data of sub-calculation 3, and sub-calculation 3 can be continued to be executed to calculate the QKV vector based on the vector. Among them, since sub-calculation 3 is a linear sub-calculation, the normalized vector output by sub-calculation 2 can be encrypted in the TEE to obtain the ciphertext vector corresponding to the vector, and then the ciphertext vector is transmitted to the GPU through the IO channel between the TEE and the GPU. After receiving the ciphertext vector, the GPU can continue to execute sub-calculation 3 using the ciphertext vector as input data, and perform linear matrix multiplication on the ciphertext vector based on the model parameter matrix of the target LLM model loaded in the GPU to calculate the QKV vector.

[0116] When sub-computation 3 is completed, the GPU can return the QKV vector output by sub-computation 3 to the TEE through the IO channel between the TEE and the GPU. Since the QKV vector is calculated based on the ciphertext vector, after receiving the QKV vector, the TEE can decrypt the QKV vector in the TEE to obtain the plaintext QKV vector, and then use the plaintext QKV vector as the input data of sub-computation 4, continue to execute sub-computation 4 in the TEE, and perform BMM calculations on the Query vector and Key vector in the QKV vector to obtain an attention score matrix. Among them, sub-computation 4 is a nonlinear sub-computation.

[0117] After sub-computation 4 is completed, the attention score matrix output by sub-computation 4 can be used as the input data of sub-computation 5, and sub-computation 5 can be continued to be executed in TEE to perform Softmax calculation on the above attention score matrix to obtain an attention weight matrix. Among them, sub-computation 5 is also a nonlinear sub-computation.

[0118] After sub-computation 5 is completed, the attention weight matrix output by sub-computation 5 can be used as the input data of sub-computation 6, and sub-computation 6 is continued to be executed in TEE, and the BMM calculation is performed on the attention weight matrix and the Value vector in the QKV vector to obtain the weighted Value vector. Among them, sub-computation 6 is also a nonlinear sub-computation.

[0119] When sub-calculation 6 is completed, the weighted Value vector output by sub-calculation 6 can be used as the input data of sub-calculation 7, and sub-calculation 7 can be continued to be executed, and the matrix operation of Projection projection can be performed on the weighted Value vector. Among them, since sub-calculation 7 is a linear sub-calculation, the weighted Value vector output by sub-calculation 6 can be encrypted in TEE to obtain the ciphertext vector corresponding to the weighted Value vector, and then the ciphertext vector is transmitted to the GPU through the IO channel between TEE and GPU. After receiving the ciphertext vector, the GPU can continue to execute sub-calculation 7 with the ciphertext vector as input data, perform linear matrix multiplication operation on the ciphertext vector based on the model parameter matrix of the target LLM model loaded in the GPU, and perform Projection projection on the ciphertext vector to obtain the projected Value vector.

[0120] When sub-computation 7 is completed, the GPU can return the projected Value vector output by sub-computation 7 to the TEE through the IO channel between the TEE and the GPU. After receiving the projected Value vector, the TEE can decrypt the projected Value vector in the TEE to obtain the projected Value vector in plain text, and then use the projected Value vector in plain text as the input data of sub-computation 8, input it into the feedforward network included in the Transformer architecture, continue to execute sub-computation 8 in the TEE, and perform normalization calculations on the vectors input into the feedforward neural network. For example, Figure 2 The vector of the input feedforward network shown is a vector generated by performing a weighted sum operation on the projected Value vector output by sub-computation 7 and the encoded vector output by sub-computation 1. Among them, sub-computation 8 is a nonlinear sub-computation.

[0121] When sub-calculation 8 is completed, the normalized vector output by sub-calculation 8 can be used as input data of sub-calculation 9, input into FC1 in the feedforward network, and sub-calculation 9 is continued to be executed, and the vector input into FC1 is linearly changed. Among them, since sub-calculation 9 is a linear sub-calculation, the normalized vector output by sub-calculation 8 can be encrypted in TEE to obtain the ciphertext vector corresponding to the normalized vector, and then the ciphertext vector is transmitted to the GPU through the IO channel between TEE and GPU. After receiving the ciphertext vector, the GPU can continue to execute sub-calculation 9 with the ciphertext vector as input data, and perform a linear matrix multiplication operation on the ciphertext vector based on the model parameter matrix of the target LLM model loaded in the GPU, so as to linearly change the ciphertext vector and obtain the vector after the linear change.

[0122] When sub-computation 9 is completed, the GPU can return the linearly changed vector output by sub-computation 9 to the TEE through the IO channel between the TEE and the GPU. After receiving the linearly changed vector, the TEE can decrypt the linearly changed vector in the TEE to obtain the linearly changed vector in plain text, and then use the linearly changed vector in plain text as the input data of sub-computation 10, continue to execute sub-computation 10 in the TEE, and perform activation calculation on the linearly changed vector to obtain an activation vector. Among them, sub-computation 10 is a nonlinear sub-computation.

[0123] When sub-calculation 10 is completed, the activation vector output by sub-calculation 10 can be used as the input data of sub-calculation 11, input into FC2 in the feedforward network, and sub-calculation 11 can be continued to be executed, and the activation vector input into FC2 can be linearly changed. Among them, since sub-calculation 11 is a linear sub-calculation, the activation vector output by sub-calculation 10 can be encrypted in TEE to obtain the ciphertext vector corresponding to the normalized vector, and then the ciphertext vector can be transmitted to the GPU through the IO channel between TEE and GPU. After receiving the ciphertext vector, the GPU can continue to execute sub-calculation 11 with the ciphertext vector as input data, and perform a linear matrix multiplication operation on the ciphertext vector based on the model parameter matrix of the target LLM model loaded in the GPU, so as to linearly change the ciphertext vector and obtain the vector after the linear change.

[0124] Please continue to see Figure 2, when sub-calculation 11 is completed, the vector after linear change output by calculation 11 and the vector input to the feedforward network after sub-calculation 7 are completed can be weighted summed to obtain the first inference result corresponding to the above-mentioned inference data. It should be noted that the first inference result represents the inference result obtained by performing the above-mentioned inference calculation on the inference data based on the model parameter matrix of the above-mentioned LLM model.

[0125] It can be seen from the above embodiments that the nonlinear sub-computation included in the above-mentioned multiple sub-computations can be independently executed by the TEE in the TEE, while the linear sub-computation included in the above-mentioned multiple sub-computations can be executed by the cooperation of the TEE and the GPU by using data interaction in the form of ciphertext.

[0126] In this way, not only can the inference data provided by the user and the intermediate calculation results obtained by inference calculation based on the inference data be ensured not to be leaked to the outside, but also the performance advantage of GPU in performing linear operations can be maximized to further improve the computing performance of inference calculation based on dense inference data.

[0127] It should be noted that, in the process of TEE and GPU cooperating with each other to execute nonlinear sub-calculations, the encryption method used to encrypt the input data of nonlinear sub-calculations is not specifically limited in this specification; for example, in practical applications, encryption methods that will not affect linear calculations can be given priority.

[0128] In one embodiment shown, the encryption method used when encrypting the input data of the linear sub-computation in the TEE may specifically be an encryption method based on a one-time key.

[0129] In this case, a one-time key can be generated in the TEE for the input data of each linear sub-calculation; wherein the one-time key can specifically be a symmetric key; when the input data of any linear sub-calculation needs to be encrypted, the input data of the linear sub-calculation can be encrypted in the TEE based on the one-time key generated for the input data of the linear sub-calculation to obtain ciphertext data corresponding to the input data, and then the ciphertext data can be transmitted to the GPU through the IO channel between the TEE and the GPU, and the GPU executes the linear sub-calculation based on the ciphertext data.

[0130] Correspondingly, when the linear sub-calculation is completed, the GPU can transmit the calculation result of the linear sub-calculation to the TEE through the IO channel between the TEE and the GPU, and can continue to decrypt the calculation result in the TEE based on the one-time key to obtain the calculation result in plain text, and then continue to use the plain text calculation result of the linear sub-calculation as input data for downstream sub-calculations.

[0131] It should be noted that the encryption algorithm used to generate the one-time key is not particularly limited in this specification. In practical applications, it can be flexibly selected based on specific security requirements.

[0132] For example, in one embodiment shown, the encryption algorithm used to generate the above-mentioned one-time key can specifically be an OTP (One-time padding) algorithm. The OTP algorithm is a very secure encryption method. The algorithm will generate a random key for the data to be encrypted, and each key is a one-time key and is only used once, which makes it very difficult to crack. In the OTP algorithm, when encrypting plaintext, a randomly generated one-time key is used for encryption. This random one-time key usually has the same length as the plaintext and will only be used once. Moreover, when encrypting the plaintext with this random one-time key, it is usually only necessary to add the plaintext and the one-time key to obtain the ciphertext data. When decrypting, it is only necessary to subtract the ciphertext data from the same one-time key to restore the original plaintext data.

[0133] In this case, when the OTP algorithm is used as the encryption algorithm, the one-time key can be a one-time random key generated based on the OTP algorithm and having the same length as the input data of the linear sub-calculation. When the input data of the linear sub-calculation is encrypted in the TEE based on the one-time key generated for the input data of the linear sub-calculation, the input data of the linear sub-calculation and the one-time random key can be added in the TEE to obtain the ciphertext data corresponding to the input data.

[0134] When the calculation result of the linear sub-calculation is decrypted in the TEE based on the one-time key generated for the input data of the linear sub-calculation, a subtraction operation can be performed on the calculation result of the linear sub-calculation and the one-time random key in the TEE to restore the plaintext content of the calculation result.

[0135] It should be noted that the calculation result of the linear sub-calculation is usually the result obtained by performing a matrix multiplication operation on the model parameter matrix of the above-mentioned target LLM model and the input data of the linear sub-calculation; therefore, when performing a subtraction operation on the calculation result of the linear sub-calculation and the one-time random key in the TEE to decrypt the calculation result, it is necessary to consider the influence of the model parameter matrix of the target LLM model on the decryption result.

[0136] In one embodiment shown, the model parameter matrix of the target LLM model can also be pre-loaded in the TEE. Before the calculation result of the linear sub-calculation is decrypted in the TEE, a matrix multiplication operation can be performed in the TEE on the model parameter matrix of the target LLM model loaded in the TEE and the generated one-time random key to obtain the target matrix operation result.

[0137] Among them, it should be noted that by performing matrix multiplication operations on the model parameter matrix of the target LLM model and the generated one-time random key in advance in the TEE, the time consumed by this part of the calculation can be excluded from the overall time consumed by the inference calculation on the inference data, thereby reducing the inference time for the inference calculation on the inference data.

[0138] When the calculation result of the linear sub-calculation is decrypted in the TEE based on the one-time key generated for the input data of the linear sub-calculation, a matrix subtraction operation can be performed in the TEE on the calculation result of the linear sub-calculation and the pre-calculated target matrix operation result to restore the plaintext content of the calculation result.

[0139] For example, suppose the model parameter matrix of the target LLM model is Indicates that the one-time random key generated based on the OTP algorithm for the input data of a linear sub-computation is used Indicates that the input data of the linear sub-computation is represented by X, then in TEE, the input data X and the one-time random key The ciphertext data of the input data X obtained by adding can be expressed as , based on the model parameter matrix on the GPU The calculation result of the input data X obtained after matrix multiplication of the ciphertext data can be expressed as .

[0140] The calculation result of the input data X on the TEE To decrypt is actually to solve Through simple mathematical operations, we can know , so the model parameter matrix can be pre-calibrated on TEE Perform matrix multiplication with the one-time random key R to obtain the target matrix operation result , in the right When decrypting, the calculation result of the input data X is , and the pre-computed target matrix operation result By performing matrix subtraction, the calculation results in plain text can be restored .

[0141] It should be noted that, by encrypting the input data of the linear sub-calculations included in the above-mentioned multiple sub-calculations, the ciphertext data obtained by encrypting it in the TEE is transmitted to the GPU through the IO channel between the TEE and the GPU, and the GPU executes these linear sub-calculations in the form of ciphertext data. Although it can ensure that the intermediate calculation results obtained by inference calculation on the inference data provided by the user are not leaked to the outside, in actual applications, there may be multiple linear sub-calculations included in the above-mentioned multiple sub-calculations. Therefore, in the process of executing these linear sub-calculations, the TEE uses the IO channel between the GPU to frequently transmit input data, which will inevitably increase the data transmission load of the IO channel.

[0142] In one embodiment shown, the input data of the linear sub-calculations included in the above-mentioned multiple sub-calculations can also be sparsely processed in the TEE. After the sparsely processed input data of these linear sub-calculations, the sparsely processed input data can be encrypted in the TEE to obtain the ciphertext data corresponding to the sparsely processed input data. Then, the ciphertext data is transmitted to the GPU through the IO channel between the TEE and the GPU, and the GPU executes these linear sub-calculations based on the ciphertext data.

[0143] By increasing the sparsification of the input data for these linear sub-calculations, the amount of data transmitted by the TEE when transmitting input data to the GPU can be further reduced, reducing data transmission overhead, thereby improving the overall computational efficiency of inference calculations based on dense inference data.

[0144] In one embodiment shown, among at least one linear sub-calculation included in the above-mentioned multiple sub-calculations, there may be a linear sub-calculation that uses the calculation result of a nonlinear sub-calculation as input data. Due to the nonlinear sub-calculation, there is a certain probability of introducing some nonlinear elements in the calculation result that have no effect or little effect on the final reasoning result. In this case, when the input data of the above-mentioned at least one linear sub-calculation is sparsely processed in the TEE, the input data of at least part of the linear sub-calculation that uses the calculation result of the nonlinear sub-calculation as input data in the above-mentioned at least one linear sub-calculation may be sparsely processed.

[0145] For example, see Figure 2 , Figure 2 Among the linear sub-calculations shown, the sub-calculations belonging to the linear calculations include sub-calculations 3, sub-calculations 7, sub-calculations 9, and sub-calculations 11. These linear sub-calculations all use the output results of the nonlinear sub-calculations as input data. Therefore, the input data of sub-calculations 3, sub-calculations 7, sub-calculations 9, and sub-calculations 11 can be sparsely processed in the TEE.

[0146] For sub-computation 3, the input data of sub-computation 3 is the normalized vector output by sub-computation 2. After sub-computation 2 is executed, the normalized vector output by sub-computation 2 can be sparsely processed in TEE, and the sparsely processed vector can be encrypted in TEE to obtain the ciphertext vector corresponding to the vector, and then the ciphertext vector is transmitted to the GPU through the IO channel between TEE and GPU, and the GPU uses the ciphertext vector as input data to continue to execute sub-computation 3.

[0147] For sub-computation 7, the input data of sub-computation 7 is the weighted Value vector output by sub-computation 6. When sub-computation 6 is executed, the weighted Value vector output by sub-computation 6 can be sparsely processed in TEE, and the sparsely processed vector can be encrypted in TEE to obtain the ciphertext vector corresponding to the vector, and then the ciphertext vector is transmitted to the GPU through the IO channel between TEE and GPU, and the GPU uses the ciphertext vector as input data to continue to execute sub-computation 7.

[0148] For sub-computation 9, the input data of sub-computation 8 is the normalized vector output by sub-computation 8. After sub-computation 8 is executed, the normalized vector output by sub-computation 8 can be sparsely processed in TEE, and the sparsely processed vector can be encrypted in TEE to obtain the ciphertext vector corresponding to the vector, and then the ciphertext vector is transmitted to the GPU through the IO channel between TEE and GPU, and the GPU uses the ciphertext vector as input data to continue to execute sub-computation 9.

[0149] For sub-computation 11, the input data of sub-computation 11 is the activation vector output by sub-computation 10. When sub-computation 10 is executed, the activation vector output by sub-computation 10 can be sparsely processed in TEE, and the sparsely processed vector can be encrypted in TEE to obtain the ciphertext vector corresponding to the vector, and then the ciphertext vector is transmitted to the GPU through the IO channel between TEE and GPU, and the GPU uses the ciphertext vector as input data to continue to execute sub-computation 11.

[0150] In one embodiment shown, in at least one linear sub-calculation included in the above-mentioned multiple sub-calculations, in addition to the linear sub-calculation that may use the calculation result of the nonlinear sub-calculation as input data, there may be further nonlinear sub-calculations that introduce activation functions in the calculation. The probability of introducing some nonlinear elements that have no effect or little effect on the final inference result in the calculation results of these nonlinear sub-calculations that introduce activation functions in the calculation is usually greater than that of ordinary nonlinear sub-calculations that do not introduce activation functions in the calculation.

[0151] In this case, when the input data of at least one of the above-mentioned linear sub-calculations is sparsely processed in the TEE, only the calculation result of the nonlinear sub-calculation in which the activation function is introduced into the calculation in the at least one of the above-mentioned linear sub-calculations can be sparsely processed as the input data of at least part of the linear sub-calculations.

[0152] For example, see Figure 2 , Figure 2 Among the linear sub-calculations shown, the linear sub-calculations include sub-calculations 3, 7, 9, and 11. These linear sub-calculations all use the output results of the nonlinear sub-calculations as input data. Figure 2 Among the nonlinear sub-calculations shown, only sub-calculations 10 introduces an activation function, so only the input data of sub-calculations 11 can be sparsely processed in TEE, while the input data of sub-calculations 3, 7, and 9 do not need to be sparsely processed.

[0153] It should be noted that the specific type of the activation function introduced in the calculation of at least part of the nonlinear sub-calculations is not particularly limited in this specification.

[0154] In this specification, the sparse processing of the input data of the linear sub-computation may specifically include deleting the part of the input data that has no effect or little effect on the final reasoning result. It should be noted that the specific type of elements that have no effect or little effect on the final reasoning result is not specifically limited in this specification, and in practical applications, it can be flexibly determined from the input data based on specific reasoning requirements.

[0155] In one embodiment shown, since the invalid elements contained in the input data usually have no effect or little effect on the final reasoning result, the sparse processing of the input data of the linear sub-computation may specifically include deleting the invalid elements contained in the input data of the linear sub-computation.

[0156] For example, in one example, taking the activation function introduced in the calculation of the above-mentioned linear sub-calculation as the ReLU activation function, after the ReLU activation function is introduced in the calculation, the activated vector outputted usually contains a large number of 0 elements, and the 0 element usually has no substantial effect on the final reasoning result. Therefore, when the activation function introduced in the calculation of the linear sub-calculation is the ReLU activation function, the invalid elements contained in the input data of the linear sub-calculation may be the 0 elements contained in the input data. In this case, when the TEE performs sparse processing on the input data of the linear sub-calculation, the 0 elements contained in the input data may be deleted.

[0157] Of course, in addition to deleting invalid elements contained in the input data of the linear sub-calculation, in actual applications, when performing sparse processing on the input data of the linear sub-calculation, other forms of data elements that have no effect or little effect on the final reasoning result can also be deleted, and they will not be listed one by one in this manual.

[0158] In one embodiment shown, in order to improve the accuracy of deleting invalid elements included in the input data, the position information of the invalid elements included in the input data of the at least part of the linear sub-calculations may be estimated in advance. In this case, when the input data of any linear sub-calculations in the at least part of the linear sub-calculations are subjected to sparse processing, the pre-estimated position information of the invalid elements included in the input data of the linear sub-calculations may be obtained; then, the data elements included in the input data of the linear sub-calculations corresponding to the position information may be deleted.

[0159] It should be noted that the specific method of estimating the position information of the invalid elements included in the input data of at least part of the linear sub-calculation is not particularly limited in this specification.

[0160] In one embodiment shown, since the input data of at least part of the above-mentioned linear sub-computation is usually the calculation result output by the upstream nonlinear sub-computation running in the TEE, and the TEE is a closed and secure computing environment, the secure calculation also adopts the form of ciphertext; therefore, based on the calculation result in the ciphertext form output by the upstream nonlinear sub-computation, it is usually impossible to intuitively estimate the position of the invalid elements contained therein.

[0161] In this case, in order to accurately estimate the position information of invalid elements contained in the input data of the above-mentioned at least part of the linear sub-calculation, when estimating the position information contained in the input result of any linear sub-calculation in the above-mentioned at least part of the linear sub-calculation, a plurality of inference data samples in plain text form can be prepared in advance, and the above-mentioned inference calculations can be performed on these inference data samples in plain text form respectively, and then a plurality of input data samples in plain text form corresponding to the linear sub-calculation generated in the process of performing the above-mentioned inference calculations on these inference data samples in plain text form respectively are obtained.

[0162] For example, the multiple plaintext input data samples corresponding to the linear sub-calculation, that is, the plaintext calculation results output by the nonlinear sub-calculation upstream of the linear sub-calculation during the process of performing the above-mentioned inference calculations on these plaintext inference data samples respectively.

[0163] After obtaining multiple plaintext input data samples corresponding to the linear sub-calculation, the position information of the invalid elements contained in the multiple input data samples can be counted respectively, and the position information of the invalid elements contained in the multiple input data samples can be further intersected to obtain the position information of the invalid elements in the same position contained in the multiple input data samples. The position information of the invalid elements in the same position contained in the multiple input data samples is the estimated result of the position information of the invalid elements contained in the input data of the linear sub-calculation, and the estimated result can be saved for subsequent use.

[0164] In this way, the problem of not being able to intuitively estimate the positions of invalid elements contained in the encrypted calculation results output by the upstream nonlinear sub-calculation can be solved. In addition, by performing intersection operations on the position information of invalid elements contained in multiple input data samples corresponding to the linear sub-calculation, the accuracy of the estimation results can be further improved.

[0165] In another embodiment shown, the original parameter matrix of the target LLM model and the input data of the linear sub-calculations included in the above-mentioned multiple sub-calculations can also be subjected to low-bit-width quantization processing respectively according to the same quantization method.

[0166] It should be noted that low-bit-width quantization of data usually refers to converting the value of the data from a higher bit-width to a lower bit-width. The reason for using the same quantization method to perform low-bit-width quantization on the original parameter matrix and the input data is to ensure that the parameter values ​​of the model parameters contained in the original parameter matrix and the values ​​of the data elements contained in the input data have the same bit-width, so as not to affect the linear matrix operations performed on the original parameter matrix and the input data in the inference calculation stage.

[0167] In this case, the original parameter matrix of the target LLM model loaded in the GPU can specifically be the original parameter matrix after the low-bit-width quantization processing. Correspondingly, the linear sub-calculation included in the above-mentioned multiple sub-calculations can be linear matrix operations based on the original parameter matrix after the low-bit-width quantization processing. Of course, unlike the above-mentioned linear sub-calculations, the nonlinear sub-calculations included in the above-mentioned multiple sub-calculations can still be executed in the TEE according to the default data bit width supported by the service device.

[0168] When executing the linear sub-calculations included in the above-mentioned multiple sub-calculations, the input data of these linear sub-calculations can be subjected to low-bit-width quantization processing in TEE, and these input data can be quantized into low-bit-width data. After the low-bit-width quantization processing is performed on the input data of these linear sub-calculations, the input data after the low-bit-width quantization processing can be encrypted in TEE to obtain ciphertext data corresponding to the input data after the low-bit-width quantization processing.

[0169] Then, the ciphertext data is transmitted to the GPU through the IO channel between the TEE and the GPU, and the GPU performs these linear sub-calculations based on the ciphertext data. After the GPU completes these linear sub-calculations based on the ciphertext data, the calculation results can be transmitted to the TEE through the IO channel between the TEE and the GPU. After the calculation results of these linear sub-calculations are restored to the calculation results of the standard bit width in the TEE, the calculation results are then decrypted.

[0170] By adding low-bit-width quantization processing to the input data of these linear sub-calculations, these input data can be quantized into low-bit-width data, further reducing the amount of data transmission when the TEE transmits input data to the GPU, reducing data transmission overhead, and thus improving the overall computational efficiency of inference calculations based on dense inference data.

[0171] In one embodiment shown, since the operations supported by the GPU equipped with the service device are generally floating-point operations, and although floating-point operations have higher calculation accuracy, they are generally more complex than fixed-point operations; therefore, in the process of performing the above-mentioned low-bit-width quantization processing, you can choose to quantize the data into fixed-point numbers to participate in the calculation.

[0172] Assuming that the data supported by the GPU for processing is a floating point number with a first bit width, when performing the above-mentioned low bit width quantization processing, the data value can be specifically selected to be converted from a floating point number with a first bit width to a fixed point number with a second bit width; wherein the second bit width is lower than the first bit width. For example, in one example, the first bit width can be 32 bits, and the second bit width can be 8 bits.

[0173] In this case, when the original parameter matrix of the target LLM model is subjected to low-bit width processing, the parameter values ​​of each model parameter contained in the original parameter matrix can be converted from floating-point numbers of the first bit width to fixed-point numbers of the second bit width. Since the original parameter matrix does not involve the user's data privacy, the low-bit width processing of the original parameter matrix can be performed in the GPU.

[0174] Correspondingly, when low-bit-width quantization processing is performed in TEE on the input data of the linear sub-calculation included in the above-mentioned multiple sub-calculations, the value of the input data of the linear sub-calculation can also be converted from a floating-point number of the first bit width to a fixed-point number of the second bit width in TEE.

[0175] When the bit width of the calculation results of the linear sub-calculations included in the above-mentioned multiple sub-calculations is restored in the TEE, the calculation results of these linear sub-calculations can be converted from fixed-point numbers of the second bit width to floating-point numbers of the first bit width to restore the calculation results to the standard bit width.

[0176] It should be noted that, when performing the above-mentioned low-bit-width quantization processing, the quantization method used is not particularly limited in this specification, and in practical applications, it can be flexibly selected based on actual needs.

[0177] For example, in one embodiment shown, the quantization method used in the above-mentioned low-bit-width quantization processing may specifically be SmoothQuant quantization. SmoothQuant quantization is a quantization method for the LLM model, through which the parameter values ​​of the model parameters contained in the parameter matrix of the LLM model and the numerical values ​​of the calculation results of each sub-calculation output in the LLM model can be quantized into int8 integers. Assuming that the data supported by the GPU for processing is 32-bit floating-point numbers, the SmoothQuant quantization method is used to perform low-bit-width quantization processing on the above-mentioned original parameter matrix and the above-mentioned input data, and the values ​​of the model parameters contained in the above-mentioned original parameter matrix and the above-mentioned input data can be converted from 32-bit floating-point numbers into 8-bit integers.

[0178] It should be noted that, although the encryption method used to encrypt the input data of the nonlinear sub-computation in the TEE can be flexibly selected based on the needs, those encryption methods that will not affect the linear calculation, it should be emphasized that since the input data of the nonlinear sub-computation and the original matrix parameters of the target LLM model are both quantized with low bit width, when selecting the encryption method, if the encryption method selected is to use the principle of matrix operation of the generated key and the plaintext data to achieve encryption, it is usually necessary to perform the same low bit width quantization on the generated key.

[0179] In one embodiment shown, still taking the encryption method based on a one-time key as an example, assuming that the encryption algorithm used to generate the above-mentioned one-time key is still the OTP algorithm, then the one-time random key generated based on the OTP algorithm and having the same length as the input data of the above-mentioned linear sub-calculation needs to be pre-processed with the same low-bit width. For example, assuming that the low-bit width quantization processing performed on the original parameter matrix and input data is to convert the value of the data from a floating-point number of the first bit width to a fixed-point number of the second bit width, then the one-time random key needs to be quantized to a fixed-point number of the second bit width in the same quantization method.

[0180] In this case, when encrypting the input data of the above-mentioned linear sub-calculation in the TEE, the one-time random key that has been processed with a low bit width in the same quantization manner may be used, and an addition calculation may be performed on the input data of the linear sub-calculation and the one-time random key in the TEE to encrypt the input data of the linear sub-calculation;

[0181] Correspondingly, in the TEE, the calculation result of the above-mentioned linear sub-calculation is decrypted, and the one-time random key after low-bit width processing in the same quantization method can still be used. In the TEE, subtraction calculation is performed on the calculation result of the linear sub-calculation and the one-time random key to decrypt the input data of the linear sub-calculation.

[0182] For example, in practical applications, the calculation result of the above linear sub-calculation is usually a matrix calculation result obtained by performing matrix multiplication calculation on the original parameter matrix after low bit width processing and the input data after encryption processing. The original parameter matrix after low bit width processing can also be pre-loaded in TEE for use in decrypting the calculation result.

[0183] Before decrypting the calculation result of the above-mentioned linear sub-calculation in TEE, the original parameter matrix after low-bit width processing loaded in TEE can be read, and matrix multiplication calculation can be performed on the original parameter matrix after low-bit width processing and the one-time random key after low-bit width processing in TEE to obtain the target matrix calculation result. When decrypting the calculation result of the above-mentioned linear sub-calculation in TEE, the calculation result of the above-mentioned linear sub-calculation and the calculated target matrix calculation result can be subjected to matrix subtraction calculation to decrypt the calculation result of the linear sub-calculation.

[0184] In one embodiment shown, the target LLM model may specifically be an LLM model obtained by fine-tuning a pre-trained initial LLM model using a LoRA (Low-rank adaptation) method; in this case, the model parameters of the target LLM model may include a low-rank LoRA parameter matrix in addition to the model parameter matrix.

[0185] It should be noted that LoRA is a method for fine-tuning the LLM model. This method can limit the modification of the LLM model parameters to a lower dimension, thereby reducing the number of parameters that actually need to be learned. This method enables efficient updates on larger models while maintaining the knowledge learned during pre-training. In a standard neural network, the original model parameters are usually a high-dimensional parameter matrix. The basic idea of ​​LoRA is to replace the original parameter matrix of the LLM model with the product of two low-rank parameter matrices. For example, suppose the original parameter matrix is , containing p×q parameters. The basic idea of ​​LoRA is to replace the original weight matrix W with the product of two low-rank parameter matrices A and B, where the low-rank parameter matrix A contains p×r parameters and the low-rank parameter matrix B contains r×q parameters, where the value of r is much smaller than p and q, so the combination of low-rank parameter matrices AA and B will have a parameter amount much smaller than p×q.

[0186] Among them, the specific implementation process of fine-tuning the LLM model using the LoRA method will not be described in detail in this manual.

[0187] See also Figure 3 , Figure 3 This is a schematic diagram of performing inference calculation on inference data based on the target LLM model shown in this specification.

[0188] like Figure 3 As shown, if the target LLM model is an LLM model obtained by fine-tuning the pre-trained initial LLM model using the LoRA method, in addition to Figure 1 In the process shown in FIG. 1 , the inference data is inferred based on the model parameter matrix of the target LLM model to obtain a first inference result corresponding to the inference data. In addition, the inference result corresponding to the inference data can also be executed in the TEE. Figure 1 The same calculation process is used to perform inference calculation on the inference data based on the LoRA parameter matrix of the target LLM model to obtain a second inference result corresponding to the inference data.

[0189] For example, the above-mentioned multiple sub-calculations can also be performed in sequence in the TEE according to the above-mentioned preset execution order and based on the LoRA parameter matrix loaded in the TEE for the inference data to obtain a second inference result corresponding to the inference data. The specific implementation process will not be repeated here.

[0190] Among them, it should be emphasized that, unlike the inference calculation based on the model parameter matrix of the target LLM model, when the inference calculation is performed on the inference data based on the LoRA parameter matrix of the target LLM model in the TEE, both the linear sub-calculation and the nonlinear sub-calculation will be performed in the TEE. The second inference result specifically represents the inference result obtained by performing the above-mentioned inference calculation on the inference data based on the LoRA parameter matrix of the above-mentioned LLM model.

[0191] Please continue to see Figure 3After the decryption of the first inference result is completed in the TEE, the first inference result and the second inference result can be merged in the TEE, and based on the first inference result and the second inference result, an inference result corresponding to the inference data is jointly generated. The specific method used to merge the first inference result and the second inference result is not particularly limited in this specification; for example, in one example, the first inference result and the second inference result can be weighted and summed to obtain an inference result corresponding to the inference data.

[0192] Since the LoRA parameter matrix of the target LLM model is usually a parameter matrix generated by fine-tuning the initial LLM model using the service data generated in a specific service scenario as training samples, and these service data are usually not public data and may involve user data privacy, the LoRA parameter matrix of the target LLM model is loaded into TEE, and the LoRA parameter matrix is ​​used for inference calculation in a closed and secure computing environment such as TEE, the model parameters in the LoRA parameter matrix can be effectively protected, thereby avoiding the leakage of user data privacy involved in the parameters contained in the LoRA parameter matrix.

[0193] On the contrary, since the initial model parameter matrix of the target LLM model is usually a parameter matrix generated by fine-tuning the initial LLM model using public data as training samples, the model parameter matrix usually does not involve user data privacy. Therefore, by loading the model parameter matrix of the target LLM model into the GPU and using the model parameter matrix for inference calculations in a public computing environment such as the GPU, there is no risk of leaking user data privacy.

[0194] Corresponding to the embodiments of the aforementioned method, this specification also provides embodiments of an apparatus, an electronic device, and a storage medium.

[0195] Figure 4 is a schematic structural diagram of an electronic device provided by an exemplary embodiment. Figure 3 At the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, a memory 408, and a non-volatile memory 410, and may also include other required hardware. One or more embodiments of this specification may be implemented based on software, such as the processor 402 reading the corresponding computer program from the non-volatile memory 410 into the memory 408 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0196] like Figure 5 As shown, Figure 5 This is a block diagram of an inference calculation based on an LLM model according to an exemplary embodiment of the present specification. The device can be applied to Figure 4 In the electronic device shown, the technical solution of this specification is implemented. Wherein, the reasoning calculation based on the target LLM model includes multiple sub-calculations with a preset execution order; the device 500 includes:

[0197] An acquisition module 501 acquires inference data uploaded by a user to the TEE carried by the service device through an encrypted data channel;

[0198] The computing module 502 executes the multiple sub-calculations in sequence for the inference data based on the model parameter matrix of the target LLM model in accordance with the preset execution order to obtain a first inference result corresponding to the inference data; wherein the multiple sub-calculations include at least one nonlinear sub-calculation executed in the TEE; and at least one linear sub-calculation executed in the GPU carried by the service device; the at least one linear sub-calculation is a linear matrix operation based on the model parameter matrix of the target LLM model; the input data of the at least one linear sub-calculation is encrypted in the TEE and then transmitted from the TEE to the GPU; the calculation result of the at least one linear sub-calculation is transmitted from the GPU to the TEE and decrypted in the TEE.

[0199] Accordingly, the present specification also provides an electronic device, which includes a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to implement the steps in all the method flows described above.

[0200] Accordingly, the present specification also provides a computer-readable storage medium on which executable computer program instructions are stored; wherein, when the instructions are executed by a processor, the steps in all the method flows described above are implemented.

[0201] Accordingly, the present specification also provides a computer program product having executable computer program instructions stored thereon; wherein, when the computer program instructions are executed by a processor, the steps in all the method flows described above are implemented.

[0202] In the 1990s, it was very clear whether the improvement of a technology was hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with hardware entity modules. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0203] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, and the memory controller may also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that, in addition to implementing the controller in a purely computer-readable program code manner, the controller may be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the devices for implementing various functions included therein may also be considered as structures within the hardware component. Or even, the devices for implementing various functions may be considered as both software modules for implementing the method and structures within the hardware component.

[0204] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a server system. Of course, the present application does not exclude that with the development of computer technology in the future, the computer that implements the functions of the above embodiments may be, for example, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0205] Although one or more embodiments of the present specification provide method operation steps as described in the embodiments or flow charts, more or fewer operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiments is only one way of executing the order of many steps, and does not represent the only execution order. When the device or terminal product in practice is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment). The terms "include", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, product or device including a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements inherent to such a process, method, product or device. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or device including the elements. For example, if the words first, second, etc. are used to represent the name, they do not represent any specific order.

[0206] For the convenience of description, the above devices are described in various modules according to their functions. Of course, when implementing one or more of the present specification, the functions of each module can be implemented in the same or more software and / or hardware, or the module implementing the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0207] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0208] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0209] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0210] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0211] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0212] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage, graphene storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0213] It should be understood by those skilled in the art that one or more embodiments of the present specification may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of the present specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0214] One or more embodiments of this specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of this specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0215] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. In the description of this specification, the description of the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of this specification. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.

[0216] The above description is only an example of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. For those skilled in the art, one or more embodiments of this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification shall be included in the scope of the claims.

Claims

1. An inference calculation method based on an LLM model, applied to a service device deployed with a target LLM model; wherein: The target LLM model is an LLM model obtained by fine-tuning the pre-trained initial LLM model using the LoRA method; The inference calculation based on the target LLM model includes multiple sub-calculations with a preset execution order; the method includes: Obtaining the inference data uploaded by the user to the TEE carried by the service device through an encrypted data channel; According to the preset execution order, the multiple sub-calculations are sequentially performed on the inference data based on the model parameter matrix of the target LLM model to obtain a first inference result corresponding to the inference data; wherein the multiple sub-calculations include at least one nonlinear sub-calculation performed in the TEE; and at least one linear sub-calculation performed in the GPU carried by the service device; the at least one linear sub-calculation is a linear matrix operation based on the model parameter matrix of the target LLM model; the input data of the at least one linear sub-calculation is encrypted in the TEE and then transmitted from the TEE to the GPU; the calculation result of the at least one linear sub-calculation is transmitted from the GPU to the TEE and decrypted in the TEE; the model parameters of the target LLM model also include a low-rank LoRA parameter matrix; In the TEE, in accordance with the execution order, the polynomial calculations are sequentially performed on the reasoning data based on the LoRA parameter matrix to obtain a second reasoning result corresponding to the reasoning data; In the TEE, an inference result corresponding to the inference data is generated based on the first inference result and the second inference result.

2. According to the method of claim 1, the inference calculation based on the target LLM model includes multiple inference calculations executed in parallel; each of the multiple inference calculations respectively includes the multiple sub-calculations with the preset execution order.

3. The method according to claim 1, generating an inference result corresponding to the inference data based on the first inference result and the second inference result, comprising: A weighted sum calculation is performed on the first reasoning result and the second reasoning result to obtain an reasoning result corresponding to the reasoning data.

4. The method according to claim 1, wherein the encryption processing is performed on the input data of the linear sub-computation in the TEE, comprising: Encrypting the input data of the linear sub-computation in the TEE based on a one-time key generated for the input data of the linear sub-computation; The decryption process is performed in the TEE on the calculation result of the linear sub-calculation, including: In the TEE, a calculation result of the linear sub-computation is decrypted based on the one-time key generated for the input data of the linear sub-computation.

5. The method according to claim 4, wherein the one-time key is a one-time random key generated based on an OTP algorithm and having the same length as the input data of the linear sub-calculation; The input data of the linear sub-computation is encrypted in the TEE based on a one-time key generated for the input data of the linear sub-computation, including: An addition operation is performed in the TEE on the input data of the linear sub-computation and the one-time random key to encrypt the input data of the linear sub-computation.

6. The method according to claim 5, wherein the calculation result of the linear sub-calculation is a matrix operation result obtained by performing a matrix multiplication operation on the model parameter matrix and the input data of the linear sub-calculation after encryption; Before decrypting the calculation result of the linear sub-calculation in the TEE based on the one-time key generated for the input data of the linear sub-calculation, the method further includes: Performing a matrix multiplication operation on the model parameter matrix and the one-time random key in the TEE to obtain a target matrix operation result; Decrypting the calculation result of the linear sub-calculation in the TEE based on the one-time key generated for the input data of the linear sub-calculation includes: In the TEE, a matrix subtraction operation is performed on the calculation result of the linear sub-calculation and the target matrix operation result to decrypt the calculation result of the linear sub-calculation.

7. In the method as claimed in claim 1, the input data of at least one linear sub-calculation is the input data that has been sparsely processed in the TEE; the input data after the sparse processing is encrypted in the TEE and then transmitted from the TEE to the GPU.

8. The method of claim 7, wherein the input data of at least part of the linear sub-computation in the at least one linear sub-computation is the input data that has been sparsely processed in the TEE; wherein, The at least part of the linear sub-calculation is a linear sub-calculation in which the calculation result of the nonlinear sub-calculation is used as input data in the at least one linear sub-calculation.

9. The method as claimed in claim 8, wherein the at least part of the linear sub-calculation is a linear sub-calculation in which a calculation result of a nonlinear sub-calculation in which an activation function is introduced into the at least one nonlinear sub-calculation is used as input data.

10. The method according to claim 8, The input data of the linear sub-calculation is subjected to sparse processing, including: Invalid elements included in the input data of the linear sub-computation are deleted.

11. The method according to claim 10, wherein deleting the invalid elements contained in the input data of the linear sub-calculation comprises: Obtaining pre-estimated position information of invalid elements contained in input data of the linear sub-calculation; The data element corresponding to the position information and included in the input data of the linear sub-calculation is deleted.

12. The method according to claim 11, Estimating position information of invalid elements contained in input data of the linear sub-computation, including: Acquire a plurality of input data samples corresponding to the linear sub-computation; wherein the plurality of input data samples are input data samples in plain text generated in the process of respectively performing the inference calculation on a plurality of inference data samples in plain text; Respectively counting the position information of the invalid elements contained in the multiple input data samples, and performing an intersection operation on the counted position information of the invalid elements contained in the multiple input data samples to obtain the position information of the invalid elements contained in the same position in the multiple input data samples; The position information of the invalid elements at the same position included in the plurality of input data samples is stored as an estimation result of the position information of the invalid elements included in the linear sub-calculation.

13. The method according to claim 12, wherein the at least part of the linear sub-calculation is a linear sub-calculation in which a calculation result of a nonlinear sub-calculation in which a ReLU activation function is introduced into the at least one nonlinear sub-calculation is used as input data; and correspondingly, the invalid element is a 0 element; Deleting invalid elements contained in the input data of the linear sub-calculation, including: Delete the zero elements contained in the input data of the linear sub-computation.

14. The method as claimed in claim 1, wherein the original parameter matrix of the target LLM model after low-bit-width processing is loaded into the GPU; the at least one linear sub-calculation is a linear matrix operation based on the original parameter matrix after the low-bit-width processing; the input data of the at least one linear sub-calculation is the input data subjected to the low-bit-width quantization processing in the TEE; the input data after the low-bit-width quantization processing is encrypted in the TEE and then transmitted from the TEE to the GPU; the calculation result of the at least one linear sub-calculation is transmitted from the GPU to the TEE, and is decrypted after being restored to the calculation result of the standard bit-width in the TEE.

15. The method according to claim 14, wherein the data supported by the GPU for processing is a floating point number of the first bit width; the low bit width quantization processing comprises converting the value of the data from the floating point number of the first bit width into a fixed point number of the second bit width; wherein, The second bit width is lower than the first bit width; The original parameter matrix is ​​subjected to low bit width processing, including: Converting the parameter value of each model parameter contained in the original parameter matrix from a floating point number of the first bit width to a fixed point number of the second bit width; Performing low bit width quantization processing on input data of the at least one linear sub-calculation in the TEE includes: In the TEE, a value of input data of the at least one linear sub-calculation is converted from a floating point number of the first bit width to a fixed point number of the second bit width.

16. The method according to claim 15, wherein in the TEE, the calculation result of the at least one linear sub-calculation returned by the GPU is restored to a calculation result of a standard bit width, comprising: In the TEE, the calculation result of the at least one linear sub-calculation returned by the GPU is converted from a fixed-point number of the second bit width to a floating-point number of the first bit width.

17. The method as claimed in claim 16, wherein the low bit width quantization is SmoothQuant quantization; the first bit width is 32 bits; the second bit width is 8 bits; and the low bit width quantization processing includes converting the data value from a 32-bit floating point number to an 8-bit integer.

18. The method according to claim 1, wherein the target LLM model is an LLM model using a Transformer architecture; The linear sub-calculation includes one or more combinations of the following: The matrix operations used to calculate the QKV vectors included in the multi-head attention layer in the Transformer architecture; The multi-head attention layer in the Transformer architecture includes a matrix operation for Projection of the output results of the multi-head attention layer; The fully connected layer in the feedforward network in the Transformer architecture contains matrix operations that perform linear transformations on vectors.

19. An inference computing device based on an LLM model, applied to a service device deployed with a target LLM model; wherein: The target LLM model is an LLM model obtained by fine-tuning the pre-trained initial LLM model using the LoRA method; The inference calculation based on the target LLM model includes multiple sub-calculations with a preset execution order; the device includes: An acquisition module, which acquires the inference data uploaded by the user to the TEE carried by the service device through an encrypted data channel; A computing module, in accordance with the preset execution order, sequentially performs the multiple sub-calculations on the inference data based on the model parameter matrix of the target LLM model to obtain a first inference result corresponding to the inference data; wherein the multiple sub-calculations include at least one nonlinear sub-calculation performed in the TEE; and at least one linear sub-calculation performed in the GPU carried by the service device; the at least one linear sub-calculation is a linear matrix operation based on the model parameter matrix of the target LLM model; the input data of the at least one linear sub-calculation is encrypted in the TEE and then transmitted to the GPU by the TEE; the calculation result of the at least one linear sub-calculation is transmitted to the TEE by the GPU and decrypted in the TEE; the model parameters of the target LLM model also include a low-rank LoRA parameter matrix; in the TEE, in accordance with the execution order, sequentially performs the multiple sub-calculations on the inference data based on the LoRA parameter matrix to obtain a second inference result corresponding to the inference data; in the TEE, based on the first inference result and the second inference result, generates an inference result corresponding to the inference data.

20. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 18 are implemented.

21. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 18.

22. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 18 are implemented.

Images