Input protection method and device of large language model, electronic equipment and intelligent system

Abstract

The invention discloses an input protection method and device of a large language model, electronic equipment and an intelligent system, and relates to the technical field of artificial intelligence. The method comprises the steps that input layer data of a large language model are obtained from a cloud server, the input layer data comprise a word list and an input layer vector matrix, and the word list comprises a plurality of segmented words; adding disturbance to each vector of the input layer vector matrix to obtain a disturbance vector matrix; performing fine tuning training on the large language model based on the disturbance vector matrix to obtain a target vector matrix; performing word segmentation processing on the input data to obtain a first corresponding relationship, and obtaining a second corresponding relationship between the first corresponding relationship and the target vector matrix, the first corresponding relationship representing a corresponding relationship between each segmented word in the word list and the index value, and the second corresponding relationship representing a corresponding relationship between each segmented word in the word list and the index value; the second corresponding relation represents the corresponding relation of each segmented word in the word list, the index value and each target vector in the target vector matrix, and the second corresponding relation is stored in local equipment. Therefore, the privacy of data input by the user is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an input protection method for a large language model, an input protection device for a large language model, a computer-readable storage medium, an electronic device, and an artificial intelligence system. Background Art

[0002] With the widespread application of large language models (such as GPT, Llama, etc.), artificial intelligence systems have shown great potential and flexibility in processing various natural language tasks (such as text generation, machine translation, semantic analysis, etc.). However, these large models are usually deployed on cloud servers, and the data input by users needs to be transmitted to the cloud for processing. Although this architecture brings efficient utilization of computing resources and improved model performance, it also raises major challenges in protecting user data privacy.

[0003] In a large language model, the Embedding layer serves as the input layer of the model and is responsible for converting discrete user inputs, such as tokens in text, into continuous vector representations. The working principle of the Embedding layer is similar to a lookup table, mapping each input token into a vector space of fixed dimensions so that the model can understand and process these inputs. However, this also means that if the unprocessed input data is transmitted directly to the cloud, the cloud server will be able to directly obtain the user's input content, which may bring the risk of privacy leakage in the scenario of sensitive information processing. Summary of the invention

[0004] The present application aims to solve at least one of the technical problems in the related art to a certain extent. To this end, the first purpose of the present application is to propose an input protection method for a large language model, which is applied to a local device, the large language model is stored in a cloud server, the large language model includes an input layer, and the method includes: obtaining input layer data of the large language model from the cloud server, the input layer data includes a vocabulary and an input layer vector matrix, and the vocabulary includes multiple segmented words; adding perturbations to each vector of the input layer vector matrix to obtain a perturbation vector matrix; fine-tuning the large language model based on the perturbation vector matrix to obtain a target vector matrix; performing segmentation processing on the input data to obtain a first correspondence, and obtaining a second correspondence between the first correspondence and the target vector matrix, wherein the first correspondence represents the correspondence between each segmented word in the vocabulary and the index value, and the second correspondence represents the correspondence between each segmented word in the vocabulary, the index value and each target vector in the target vector matrix, and the second correspondence is stored in the local device. The protection method of the present application downloads the input layer vector matrix of the large language model stored in the cloud server to the local device, and the local device perturbs and fine-tunes each vector of the input layer vector matrix to obtain the target vector, and directly uploads the target vector to the cloud server for calculation. In this way, while ensuring the accuracy of the model, the original one-to-one correspondence between word segmentation and vector is broken, thereby effectively preventing the cloud model from reversely inferring the user's original input text through the vector, and improving the privacy of the user's input text.

[0005] The second objective of the present application is to propose an input protection device for a large language model.

[0006] The third object of the present application is to provide a computer-readable storage medium.

[0007] The fourth objective of the present application is to provide an electronic device.

[0008] The fifth objective of this application is to propose an artificial intelligence system.

[0009] To achieve the above-mentioned purpose, the first aspect of the present application proposes an input protection method for a large language model, which is applied to a local device. The large language model is stored in a cloud server. The large language model includes an input layer. The method includes: obtaining input layer data of the large language model from the cloud server, the input layer data includes a vocabulary and an input layer vector matrix, and the vocabulary includes multiple word segments; adding perturbations to each vector of the input layer vector matrix to obtain a perturbation vector matrix; fine-tuning the large language model based on the perturbation vector matrix to obtain a target vector matrix; performing word segmentation processing on the input data to obtain a first corresponding relationship, and obtaining a second corresponding relationship between the first corresponding relationship and the target vector matrix, wherein the first corresponding relationship represents the corresponding relationship between each word segment in the vocabulary and the index value, and the second corresponding relationship represents the corresponding relationship between each word segment in the vocabulary, the index value and each target vector in the target vector matrix, and the second corresponding relationship is stored in the local device.

[0010] According to one embodiment of the present application, a disturbance is added to each vector of the input layer vector matrix to obtain a disturbance vector matrix, including: determining each disturbance vector based on the sum of each vector of the input layer vector matrix and a preset disturbance; determining the disturbance vector matrix based on each disturbance vector; wherein the preset disturbances are the same or different, and the preset disturbances include one of additive noise, random projection, and Gaussian noise.

[0011] According to one embodiment of the present application, a large language model is fine-tuned and trained based on a perturbation vector matrix to obtain a target vector matrix, including: forward propagating the perturbation vector matrix and the vocabulary based on a forward propagation algorithm to calculate a loss function; calculating a gradient according to the loss function based on a back propagation algorithm; adjusting the parameters of the perturbation vector matrix based on the gradient and an optimization algorithm to minimize the loss function, and using the parameters corresponding to the minimized loss function as the target vector matrix.

[0012] According to one embodiment of the present application, the optimization algorithm includes an Adam or SGD optimization algorithm.

[0013] According to one embodiment of the present application, the above method also includes: performing shuffle processing on the vocabulary in the first correspondence relationship to obtain a first target correspondence relationship and a second target correspondence relationship, wherein the first target correspondence relationship is the correspondence relationship between the word segmentation and the index value after shuffle processing, and the second target correspondence relationship is the correspondence relationship between the index value after shuffle processing and each target vector in the target vector matrix.

[0014] According to one embodiment of the present application, the first target correspondence relationship is stored in a local device, and the second target correspondence relationship is stored in a cloud server.

[0015] To achieve the above-mentioned purpose, the second aspect of the present application proposes an input protection device for a large language model, which is applied to a local device. The large language model is stored in a cloud server. The large language model includes an input layer. The device includes: a first acquisition module, which is used to obtain input layer data of the large language model from the cloud server, and the input layer data includes a vocabulary and an input layer vector matrix. The vocabulary includes multiple word segments; a processing module, which is used to add perturbations to each vector of the input layer vector matrix to obtain a perturbation vector matrix; a training module, which is used to fine-tune the large language model based on the perturbation vector matrix to obtain a target vector matrix; a second acquisition module, which is used to perform word segmentation processing on the input data to obtain a first corresponding relationship, and to obtain a second corresponding relationship between the first corresponding relationship and the target vector matrix, wherein the first corresponding relationship represents the corresponding relationship between the word segmentation and the index value, and the second corresponding relationship represents the corresponding relationship between each word segmentation in the vocabulary, the index value and each target vector in the target vector matrix, and the second corresponding relationship is stored in the local device.

[0016] To achieve the above-mentioned purpose, the third aspect of the present application proposes a computer-readable storage medium on which a program is stored, and when the program is executed by a processor, the input protection method of the aforementioned large language model is implemented.

[0017] To achieve the above-mentioned purpose, the fourth aspect of the present application proposes an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the input protection method of the aforementioned large language model is implemented.

[0018] To achieve the above objectives, the fifth embodiment of the present application proposes an artificial intelligence system, including the aforementioned electronic device and a cloud server.

[0019] According to the input protection method, device, electronic device and intelligent system of the large language model of the embodiment of the present application, it is applied to the local device, the large language model is stored in the cloud server, the large language model includes an input layer, and the method includes: obtaining the input layer data of the large language model from the cloud server, the input layer data includes a vocabulary and an input layer vector matrix, and the vocabulary includes multiple segmented words; adding perturbations to each vector of the input layer vector matrix to obtain a perturbation vector matrix; fine-tuning the large language model based on the perturbation vector matrix to obtain a target vector matrix; performing segmentation processing on the input data to obtain a first corresponding relationship, and obtaining a second corresponding relationship from the first corresponding relationship and the target vector matrix, wherein the first corresponding relationship represents the corresponding relationship between each segmented word and the index value in the vocabulary, and the second corresponding relationship represents the corresponding relationship between each segmented word, the index value and each target vector in the target vector matrix in the vocabulary, and the second corresponding relationship is stored in the local device. The protection method of the present application is to download the input layer vector matrix of the large language model stored in the cloud server to the local device, and the local device performs perturbation and fine-tuning processing on each vector of the input layer vector matrix to obtain the target vector, and directly upload the target vector to the cloud server for calculation. In this way, while ensuring the accuracy of the model, the original one-to-one correspondence between word segmentation and vector is broken, thereby effectively preventing the cloud model from reversely inferring the user's original input text through the vector, thereby improving the privacy of the user's input text. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a schematic diagram of the implementation principle of the Embedding layer according to some embodiments of the present application;

[0021] Figure 2 is a flowchart of a method for input protection of a large language model according to some embodiments of the present application;

[0022] Figure 3 A schematic diagram of an out-of-order processing process according to some embodiments of the present application;

[0023] Figure 4 A schematic diagram of a local device-cloud server distributed deployment according to some embodiments of the present application;

[0024] Figure 5 is a flow chart of a method for input protection of a large language model according to other embodiments of the present application;

[0025] Figure 6 is a block diagram of an input protection device for a large language model according to some embodiments of the present application;

[0026] Figure 7 is a block diagram of an electronic device according to some embodiments of the present application;

[0027] Figure 8is a block diagram of an artificial intelligence system according to some embodiments of the present application. DETAILED DESCRIPTION

[0028] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0029] The following describes in detail the input protection method, device, electronic device and intelligent system of the large language model according to the embodiments of the present application with reference to the accompanying drawings.

[0030] In some embodiments, reference Figure 1 The calculation process of the Embedding layer (input layer) of the large language model includes text input, word segmentation, token mapping, embedding search, generation of the Embedding matrix, and subsequent processing. The calculation process of the Embedding layer of the large language model is as follows:

[0031] (1) Text input:

[0032] The user enters a natural language text, such as "This is a iiput test.".

[0033] (2) Word segmentation:

[0034] The input text can be split into tokens by using a tokenizer. Tokens can be words, subwords, or characters, depending on the token segmentation strategy used by the model. For example, "This is a iiput test." may be split into tokens including "This", "is", "a", "iiput", "test", ".".

[0035] (3)Token mapping:

[0036] Each token is converted to a corresponding index value by looking up the vocabulary. The vocabulary is built by the model during training, and it contains all possible tokens and their corresponding unique indexes. For example, the index of "This" in the vocabulary may be 2023, the index of "is" may be 2003, the index of "a" may be 1037, the index of "iiput" may be 7952, the index of "test" may be 13209, and the index of "." may be 5841.

[0037] (4) Embedded search:

[0038] These index values ​​are passed to the Embedding layer as indices into the lookup table to obtain the corresponding Embedding vector. An Embedding vector is a continuous value vector of fixed dimension, which is usually learned by the model during training. Each token has a corresponding Embedding vector, which captures the semantic information of the token.

[0039] (5) Generate Embedding Matrix:

[0040] The model combines the embedding vectors corresponding to all tokens in the input sentence into a matrix, where each row represents the embedding vector of a token. For the input sentence "This is a iiput test.", the generated embedding vector matrix may be a matrix of size 6×d, where d is the dimension of the embedding vector.

[0041] (6) Subsequent processing:

[0042] The embedding matrix will be used as input to subsequent neural network layers for further feature extraction, context understanding, and task reasoning.

[0043] Usually the entire Embedding layer is deployed locally, but since the generation of the above-mentioned Embedding vector corresponds one-to-one to the input word segmentation, even if the word segmenter and word embedding matrix of the entire Embedding layer are downloaded to the local deployment, the cloud can also infer the input text content based on the Embedding vector, resulting in privacy leakage.

[0044] Based on this, the present application proposes an input protection method for a large language model, which is applied to a local device. The large language model is stored in a cloud server. The large language model includes an input layer. The input layer vector matrix of the large language model stored in the cloud server is downloaded to the local device. The local device perturbs and fine-tunes each vector of the input layer vector matrix to obtain a target vector, and uploads the target vector to the cloud server for calculation. In this way, while ensuring the accuracy of the model, the original one-to-one correspondence between word segmentation and vector is broken, thereby effectively preventing the cloud model from reversely inferring the user's original input text through the vector, and improving the privacy of the user's input text.

[0045] Figure 2 Flowchart of a method for input protection of a large language model according to some embodiments of the present application. Figure 2 The input protection method of the large language model of the embodiment of the present application may include the following steps:

[0046] S110, obtaining input layer data of the large language model from the cloud server, the input layer data including a vocabulary and an input layer vector matrix, the vocabulary including a plurality of segmented words.

[0047] Specifically, the input layer (Embedding layer) data of the large language model is obtained from the cloud server. The input layer data includes a vocabulary and an input layer vector matrix, wherein the vocabulary includes multiple word segments, and the input layer vector matrix includes multiple vectors, and the word segments correspond to the vectors one by one. For example, the vocabulary includes word segmentation W0, word segmentation W1, ..., word segmentation Wi, and the input layer vector matrix includes vector V0, vector V1, ..., vector Vi, word segmentation W0 corresponds to vector V0, word segmentation W1 corresponds to vector V1, ..., word segmentation Wi corresponds to vector Vi. This process ensures that the local device has an initial input layer representation that is consistent with the cloud server.

[0048] S120, adding perturbation to each vector of the input layer vector matrix to obtain a perturbation vector matrix.

[0049] Specifically, the local device performs a perturbation operation on each vector of the input layer vector matrix to obtain a perturbation vector. For example, perturbation ε is added to vector V0 to obtain perturbation vector V0+ε, perturbation ε is added to vector V1 to obtain perturbation vector V1+ε, ..., perturbation ε is added to vector Vi to obtain perturbation vector Vi+ε, and finally the perturbation vector matrix is ​​determined according to perturbation vector V0+ε, perturbation vector V1+ε, ..., perturbation vector Vi+ε. This breaks the one-to-one correspondence between each word segmentation and each vector of the input layer vector matrix.

[0050] S130, fine-tuning the large language model based on the perturbation vector matrix to obtain a target vector matrix.

[0051] Specifically, while maintaining the accuracy of downstream tasks, in order to further enhance the privacy protection effect, the large language model is fine-tuned based on the perturbation vector matrix to obtain the target vector matrix. For example, the target vector V0' is obtained after fine-tuning based on the perturbation vector V0+ε, the target vector V1' is obtained after fine-tuning based on the perturbation vector V1+ε, ..., the target vector Vi' is obtained after fine-tuning based on the perturbation vector Vi+ε, and finally the target vector matrix is ​​determined based on the target vector V0', target vector V1', ..., target vector Vi'.

[0052] S140, perform word segmentation processing on the input data to obtain a first correspondence, and obtain a second correspondence between the first correspondence and the target vector matrix, wherein the first correspondence represents the correspondence between each word segment in the vocabulary and the index value, and the second correspondence represents the correspondence between each word segment in the vocabulary, the index value and each target vector in the target vector matrix, and the second correspondence is stored in the local device.

[0053] Specifically, after receiving the input data, the local device uses the word segmenter to obtain the correspondence between each word segment and the index value, that is, the first correspondence. For example, after the input data is segmented, the word segment W0 and the word segment W1 are obtained, and then the index value T0 corresponding to the word segment W0 and the index value T1 corresponding to the word segment W1 can be determined by looking up the two-dimensional mapping table between the word segment and the index value, and the two-dimensional mapping table includes multiple word segments and the index value corresponding to each word segment.

[0054] Then, the second corresponding relationship is obtained according to the first corresponding relationship and the target vector matrix. For example, the index value T0 and the index value T1 can be determined according to the first corresponding relationship. Based on the index value T0 and the index value T1, the target vector V0' corresponding to the index value T0 and the target vector V1' corresponding to the index value T1 are determined by searching the two-dimensional mapping table between the index value and the target vector. The two-dimensional mapping table includes multiple index values ​​and the target vector corresponding to each index value. In other words, based on the index value T0, the corresponding word segmentation W0 and the target vector V0' can be determined, and based on the index value T1, the corresponding word segmentation W1 and the target vector V1' can be determined, that is, the second corresponding relationship is determined.

[0055] The second correspondence is stored in the local device, and when needed, the target vector in the second correspondence is directly uploaded to the cloud model for calculation. Since the target vector has been perturbed and fine-tuned, it can effectively prevent the cloud server from reversely inferring the user's original input data through the target vector.

[0056] The protection method of the present application downloads the input layer vector matrix of the large language model stored in the cloud server to the local device, and the local device perturbs and fine-tunes each vector of the input layer vector matrix to obtain the target vector, and directly uploads the target vector to the cloud server for calculation. In this way, while ensuring the accuracy of the model, the original one-to-one correspondence between word segmentation and vector is broken, thereby effectively preventing the cloud model from reversely inferring the user's original input text through the vector, and improving the privacy of the user's input text.

[0057] In some embodiments, a disturbance is added to each vector of the input layer vector matrix to obtain a disturbance vector matrix, including: determining each disturbance vector based on the sum of each vector of the input layer vector matrix and a preset disturbance; determining the disturbance vector matrix based on each disturbance vector; wherein the preset disturbances are the same or different, and the preset disturbances include one of additive noise, random projection, and Gaussian noise.

[0058] Exemplarily, the input layer vector matrix includes vector V0, vector V1,…, vector Vi, the preset disturbance corresponding to each vector includes preset disturbance ε0, preset disturbance ε1,…, preset disturbance εi, the sum of vector V0 and preset disturbance ε0 is disturbance vector V0+ε0, the sum of vector V1 and preset disturbance ε1 is disturbance vector V1+ε1,…, the sum of vector Vi and preset disturbance εi is disturbance vector Vi+εi, finally, the disturbance vector matrix is ​​determined according to the disturbance vector V0+ε0, disturbance vector V1+ε1,…, disturbance vector Vi+εi, wherein the disturbance vector matrix is ​​a matrix of size i×d, where d represents the dimension of the disturbance vector.

[0059] It should be noted that the preset disturbance includes one of additive noise, random projection, and Gaussian noise. Each vector may be the same as or different from the corresponding preset disturbance. When each vector is different from the corresponding preset disturbance, the preset disturbance needs to be set according to a certain rule to facilitate subsequent processing by the cloud server.

[0060] In some embodiments, a large language model is fine-tuned and trained based on a perturbation vector matrix to obtain a target vector matrix, including: forward propagating the perturbation vector matrix and the vocabulary based on a forward propagation algorithm to calculate a loss function; calculating a gradient according to the loss function based on a back propagation algorithm; adjusting parameters of the perturbation vector matrix based on the gradient and an optimization algorithm to minimize the loss function, and using the parameters corresponding to the minimized loss function as the target vector matrix.

[0061] Exemplarily, the perturbation vector matrix and vocabulary are forward propagated based on the forward propagation algorithm (for example, after a series of linear transformations and nonlinear activation functions) to obtain the prediction result, and then the loss function (for example, the cross entropy loss function) is calculated to measure the gap between the prediction result and the true label. The gradient of the loss function to the output layer input is calculated, and according to the chain rule, the gradient of the loss function to each layer input and weight is calculated layer by layer, and finally the gradient of the loss function to the perturbation vector matrix is ​​calculated. Using the calculated gradient, the parameters of the motion vector matrix are updated according to the optimization algorithm (for example, Adam or SGD optimization algorithm) to minimize the loss function, and the parameters corresponding to the minimized loss function are used as the target vector matrix.

[0062] It should be noted that, considering the difficulty of fine-tuning training, interference can be added to some vectors first, and then interference can be added to other vectors after fine-tuning and fine-tuning training can be performed again. In addition, when the added interference is noise, it can be Gaussian distribution noise or noise that satisfies other specific distributions.

[0063] In some embodiments, the optimization algorithm includes an Adam (Adaptive Moment Estimation, an optimization algorithm based on a first-order gradient) or an SGD (Stochastic Gradient Descent, stochastic gradient descent) optimization algorithm.

[0064] In some embodiments, the above method also includes: shuffling the vocabulary in the first correspondence to obtain a first target correspondence and a second target correspondence, wherein the first target correspondence is the correspondence between the word segmentation and the index value after shuffling, and the second target correspondence is the correspondence between the index value after shuffling and each target vector in the target vector matrix.

[0065] Specifically, the word list in the first correspondence can also be shuffled to further enhance the privacy protection of user input data. On the local device, on the basis of ensuring the correspondence between each word segment and each target vector in the target vector matrix, the index values ​​are shuffled and finally reordered according to the sequence number of the index value.

[0066] In the following description, the word list includes 5 words as an example, but this is not a limitation to the present application. Figure 3 Before the shuffle, the index value T0 corresponds to the word segmentation W0 and the target vector V0', the index value T1 corresponds to the word segmentation W1 and the target vector V1', the index value T2 corresponds to the word segmentation W2 and the target vector V2', the index value T3 corresponds to the word segmentation W3 and the target vector V3', and the index value T4 corresponds to the word segmentation W4 and the target vector V4'. After the shuffle, the index value T0 corresponds to the word segmentation W1 and the target vector V1', the index value T1 corresponds to the word segmentation W4 and the target vector V4', the index value T2 corresponds to the word segmentation W0 and the target vector V0', the index value T3 corresponds to the word segmentation W2 and the target vector V2', and the index value T4 corresponds to the word segmentation W3 and the target vector V3'.

[0067] Further, refer to Figure 4 , determine the first target correspondence relationship according to the correspondence between the segmented words and the index values ​​after the disorder processing, for example, the index value T0 corresponds to the segmented word W1, the index value T1 corresponds to the segmented word W4, the index value T2 corresponds to the segmented word W0, the index value T3 corresponds to the segmented word W2, and the index value T4 corresponds to the segmented word W3.

[0068] The second target correspondence is determined based on the correspondence between the index value after shuffle processing and each target vector in the target vector matrix. For example, index value T0 corresponds to target vector V1', index value T1 corresponds to target vector V4', index value T2 corresponds to target vector V0', index value T3 corresponds to target vector V2', and index value T4 corresponds to target vector V3'.

[0069] In some embodiments, the first target correspondence relationship is stored in a local device, and the second target correspondence relationship is stored in a cloud server.

[0070] Specifically, continue to refer to Figure 4 , the second target correspondence is stored in the cloud server, the cloud server saves the second target correspondence, and uses the second target correspondence for search and calculation during reasoning. Since the index value has been out of order, the index value received by the cloud server cannot be directly associated with the original input data even if it is reversely inferred.

[0071] The first target correspondence is stored in the local device instead of the complete target vector matrix. Since the storage space occupied by word segmentation and index values ​​is much smaller than that of the target vector matrix, the storage requirements of the local device are greatly reduced, which not only maintains the privacy of user input data, but also reduces the additional local computing and storage costs, reduces the amount of data uploaded by the local device to the cloud server for processing, reduces communication costs, and improves processing efficiency and model performance.

[0072] After receiving the input data, the local device uses the word segmenter to obtain the correspondence between each word segment and the index value, and then converts the index value into a scrambled index value ID based on the first target correspondence, and uploads these scrambled index value IDs to the cloud server. Since the correspondence between the index value ID and the word segment is disrupted, the cloud server cannot infer the user input data through the index value ID. After receiving the scrambled index value ID, the cloud server converts these index value IDs into corresponding target vectors through the second target correspondence uploaded in advance. Then, the cloud server uses these target vectors for subsequent reasoning and calculation. Since the target vector has been perturbed and fine-tuned, the cloud server cannot infer the user's input data content through the generated target vector.

[0073] As a specific example, see Figure 5 The input protection method of the large language model of the embodiment of the present application may also include the following steps:

[0074] S201, downloading input layer data.

[0075] Download the input layer data of the large language model from the cloud server.

[0076] S202, after the vector matrix is ​​perturbed, a perturbation vector matrix is ​​obtained.

[0077] S203, fine-tuning training of the perturbation vector matrix.

[0078] Fine-tune the perturbation vector matrix according to the downstream task to obtain the target vector matrix to ensure that the accuracy of the calculation is not affected.

[0079] S204, index values ​​are processed out of order.

[0080] Maintain the one-to-one correspondence between word segmentation and target vector, and only disrupt the index value and the correspondence between the two.

[0081] S205, distributed deployment.

[0082] S206: The first target correspondence relationship is stored in the local device.

[0083] The first target correspondence is the correspondence between the word segments and the index values ​​after the out-of-order processing.

[0084] S207, the second target corresponding relationship is stored in the cloud server.

[0085] The second target correspondence is the correspondence between the index value after the shuffle processing and each target vector in the target vector matrix.

[0086] In summary, when using large language models, the privacy protection needs of users are becoming increasingly important. Traditional methods require a lot of calculations in the cloud, or use complex encryption techniques to protect input data, but these methods often have significant trade-offs between privacy protection effects, computational efficiency, and storage requirements. Therefore, the present application obtains a target vector matrix by perturbing and fine-tuning each vector of the input layer vector matrix on a local device, effectively breaking the one-to-one correspondence between the original input data and the cloud data, thereby significantly improving the privacy protection effect of the user, and the perturbation and fine-tuning process is completed on the local device without increasing the computational burden on the cloud; if the input layer of the large language model is stored entirely on the local device, it will cause a large storage burden on the local device, and the present application performs shuffle processing on the index values ​​to obtain the correspondence between the shuffled word segmentation and the index value and the correspondence between the shuffled index value and each target vector in the target vector matrix, and shuffles the vector matrix. The corresponding relationship between the out-of-order index values ​​and each target vector in the target vector matrix is ​​uploaded to the cloud server, reducing the storage requirements of the local device. Because there is no direct correlation between the out-of-order index values ​​and the input data, even if the cloud server receives these index values, it cannot infer the original input data, thereby ensuring data privacy and greatly reducing the storage space occupied by the local device. In this way, while ensuring the privacy of user input, it can reduce the storage space occupied by the local device and improve the computing efficiency of the cloud server, which not only reduces the overall latency of the system, but also improves the efficiency of the reasoning process, reduces communication costs, and meets the needs of efficient and secure computing.

[0087] Corresponding to the above embodiment, the present application also proposes an input protection device for a large language model.

[0088] In some embodiments, applied to a local device, the large language model is stored in a cloud server, and the large language model includes an input layer.

[0089] Reference Figure 6 The input protection device 600 of the large language model includes: a first acquisition module 610, a processing module 620, a training module 630 and a second acquisition module 640.

[0090] Among them, the first acquisition module 610 is used to obtain the input layer data of the large language model from the cloud server, the input layer data includes a vocabulary and an input layer vector matrix, and the vocabulary includes multiple word segments. The processing module 620 is used to add perturbations to each vector of the input layer vector matrix to obtain a perturbation vector matrix. The training module 630 is used to fine-tune the large language model based on the perturbation vector matrix to obtain a target vector matrix. The second acquisition module 640 is used to perform word segmentation processing on the input data to obtain a first corresponding relationship, and the first corresponding relationship and the target vector matrix obtain a second corresponding relationship, wherein the first corresponding relationship represents the corresponding relationship between the word segmentation and the index value, and the second corresponding relationship represents the corresponding relationship between each word segmentation in the vocabulary, the index value and each target vector in the target vector matrix, and the second corresponding relationship is stored in the local device.

[0091] According to one embodiment of the present application, the processing module 620 is specifically used to determine each disturbance vector based on the sum of each vector of the input layer vector matrix and a preset disturbance; determine the disturbance vector matrix based on each disturbance vector; wherein the preset disturbances are the same or different, and the preset disturbances include one of additive noise, random projection, and Gaussian noise.

[0092] According to one embodiment of the present application, the training module 630 is specifically used to perform forward propagation on the perturbation vector matrix and the vocabulary based on the forward propagation algorithm to calculate the loss function; calculate the gradient according to the loss function based on the back propagation algorithm; adjust the parameters of the perturbation vector matrix based on the gradient and optimization algorithm to minimize the loss function, and use the parameters corresponding to the minimized loss function as the target vector matrix.

[0093] According to one embodiment of the present application, the optimization algorithm includes an Adam or SGD optimization algorithm.

[0094] According to one embodiment of the present application, the vocabulary in the first correspondence is shuffled to obtain a first target correspondence and a second target correspondence, wherein the first target correspondence is the correspondence between the word segmentation and the index value after shuffle processing, and the second target correspondence is the correspondence between the index value after shuffle processing and each target vector in the target vector matrix.

[0095] According to one embodiment of the present application, the first target correspondence relationship is stored in a local device, and the second target correspondence relationship is stored in a cloud server.

[0096] It should be pointed out that the above-mentioned explanation of the embodiment and beneficial effects of the input protection method for a large language model is also applicable to the input protection device for a large language model of the embodiment of the present application. To avoid redundancy, it will not be elaborated here.

[0097] Corresponding to the above embodiments, the present application also proposes a computer-readable storage medium.

[0098] The computer-readable storage medium of the present application stores a program thereon, and when the program is executed by a processor, the input protection method of the aforementioned large language model is implemented.

[0099] It should be pointed out that the above-mentioned explanation of the embodiment and beneficial effects of the input protection method for a large language model is also applicable to the computer-readable storage medium of the embodiment of the present application. To avoid redundancy, it will not be elaborated here.

[0100] Corresponding to the above embodiment, the present application also proposes an electronic device.

[0101] See also Figure 7 As shown, the electronic device 700 of the present application includes a memory 710, a processor 720, and a program stored in the memory 710 and executable on the processor 720. When the processor executes the program, the aforementioned input protection method for the large language model is implemented.

[0102] It should be pointed out that the above-mentioned explanation of the embodiment and beneficial effects of the input protection method for a large language model are also applicable to the electronic device of the embodiment of the present application. To avoid redundancy, they will not be elaborated here.

[0103] Corresponding to the above embodiments, the present application also proposes an artificial intelligence system.

[0104] See also Figure 8 As shown, the artificial intelligence system 800 of the present application includes the aforementioned electronic device 700 and cloud server 100.

[0105] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0106] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0107] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0108] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0109] In this application, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0110] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for input protection of a large language model, characterized in that: Applied to a local device, the large language model is stored in a cloud server, the large language model includes an input layer, and the method includes: Acquire input layer data of the large language model from the cloud server, the input layer data including a vocabulary and an input layer vector matrix, the vocabulary including a plurality of segmented words; Adding perturbation to each vector of the input layer vector matrix to obtain a perturbation vector matrix; Fine-tune the large language model based on the perturbation vector matrix to obtain a target vector matrix; Perform word segmentation processing on the input data to obtain a first correspondence relationship, and obtain a second correspondence relationship between the first correspondence relationship and the target vector matrix, wherein the first correspondence relationship represents the correspondence between each word segment in the vocabulary and the index value, and the second correspondence relationship represents the correspondence between each word segment in the vocabulary, the index value and each target vector in the target vector matrix, and the second correspondence relationship is stored in the local device.

2. The input protection method for a large language model according to claim 1, characterized in that: Add perturbation to each vector of the input layer vector matrix to obtain a perturbation vector matrix, including: Determine each disturbance vector based on the sum of each vector of the input layer vector matrix and a preset disturbance; The disturbance vector matrix is ​​determined based on each disturbance vector; wherein the preset disturbances are the same or different, and the preset disturbances include one of additive noise, random projection, and Gaussian noise.

3. The input protection method for a large language model according to claim 1, characterized in that: Fine-tuning the large language model based on the perturbation vector matrix to obtain a target vector matrix includes: Performing forward propagation on the perturbation vector matrix and the vocabulary based on a forward propagation algorithm to calculate a loss function; Calculate the gradient according to the loss function based on the back propagation algorithm; Based on the gradient and optimization algorithm, the parameters of the perturbation vector matrix are adjusted to minimize the loss function, and the parameters corresponding to the minimized loss function are used as the target vector matrix.

4. The input protection method for a large language model according to claim 3, characterized in that: The optimization algorithm includes Adam or SGD optimization algorithm.

5. The input protection method for a large language model according to claim 1, characterized in that: The method further comprises: The vocabulary in the first correspondence is shuffled to obtain a first target correspondence and a second target correspondence, wherein the first target correspondence is the correspondence between the word segmentation and the index value after shuffle, and the second target correspondence is the correspondence between the index value and each target vector in the target vector matrix after shuffle.

6. The input protection method for a large language model according to claim 5, characterized in that: The first target correspondence relationship is stored in the local device, and the second target correspondence relationship is stored in the cloud server.

7. An input protection device for a large language model, characterized in that: Applied to a local device, the large language model is stored in a cloud server, the large language model includes an input layer, and the device includes: A first acquisition module is used to acquire input layer data of the large language model from the cloud server, wherein the input layer data includes a vocabulary and an input layer vector matrix, and the vocabulary includes a plurality of segmented words; A processing module, used for adding disturbance to each vector of the input layer vector matrix to obtain a disturbance vector matrix; A training module, used for fine-tuning the large language model based on the perturbation vector matrix to obtain a target vector matrix; The second acquisition module is used to perform word segmentation processing on the input data to obtain a first corresponding relationship, and the first corresponding relationship and the target vector matrix obtain a second corresponding relationship, wherein the first corresponding relationship represents the corresponding relationship between the word segmentation and the index value, the second corresponding relationship represents the corresponding relationship between each word segmentation, index value in the vocabulary and each target vector in the target vector matrix, and the second corresponding relationship is stored in the local device.

8. A computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the input protection method for a large language model according to any one of claims 1 to 6.

9. An electronic device, characterized in that: The invention comprises a memory, a processor and a program stored in the memory and executable on the processor. When the processor executes the program, the input protection method for a large language model according to any one of claims 1 to 6 is implemented.

10. An artificial intelligence system, characterized in that: It comprises the electronic device and the cloud server as described in claim 9.

Images