Language model-based reasoning method and system and computer equipment

Abstract

The invention provides a method for reasoning based on a language model, the method is executed by a server, and the server is deployed with a target language model; the target language model comprises a target vocabulary and a target weight matrix, the target vocabulary is obtained by multiplying each embedded vector in an original vocabulary of the original language model by a preset rotation matrix, and a corresponding relationship between the embedded vectors and the input data is stored in the target vocabulary; the target weight matrix is obtained by multiplying the inverse matrix of the rotation matrix by the original weight matrix of the original language model. In the reasoning process, receiving input data corresponding to an input text from a client, and then obtaining a first intermediate result corresponding to the input data; the first intermediate result comprises a plurality of embedded vectors; processing the first intermediate result according to the target weight matrix to obtain a second intermediate result; and obtaining a reasoning result of the target language model according to the second intermediate result, and returning the reasoning result to the client.

Description

(19) State Intellectual Property Office (12) Invention Patent Application (10) Application Publication Number (43) Application Publication Date (21) Application Number 202511221849.6 (22) Application Date 2025.08.28 (71) Applicant Ant Blockchain Technology (Shanghai) Co., Ltd. Address Room 803, 8th Floor, No. 618 Waima Road, Huangpu District, Shanghai 200010 (72) Inventors Yao Kai, Li Lichun, Zhao Yuan (74) Patent Agency Beijing Yiteng Intellectual Property Agency (General Partnership) 11309 Patent Attorney Zhang Jingjuan, Zhou Liangyu (51) Int.Cl. G06N 5 / 04 (2023.01) G06F 21 / 60 (2013.01) G06F 17 / 16 (2006.01) G06N 3 / 096 (2023.01) G06N 3 / 045 (2023.01) (54) Invention Title: A Method, System, and Computer Device for Reasoning Based on a Language Model (57) Abstract: This specification proposes a method for reasoning based on a language model, executed by a server. The server deploys a target language model. The target language model includes a target vocabulary and a target weight matrix. The target vocabulary is obtained by multiplying each embedding vector in the original vocabulary of the original language model with a preset rotation matrix. The target vocabulary stores the correspondence between the embedding vectors and the input data. The target weight matrix is ​​obtained by multiplying the inverse of the rotation matrix with the original weight matrix of the original language model. During the reasoning process, input data corresponding to the input text is received from the client, and then a first intermediate result corresponding to the input data is obtained. The first intermediate result includes several embedding vectors. According to the target weight matrix, the first intermediate result is processed to obtain a second intermediate result. According to the second intermediate result, the reasoning result of the target language model is obtained and the reasoning result is returned to the client. Claims (2 pages), Description (21 pages), Drawings (4 pages), CN 121119135 A, 2025.12.12, CN 1 21 11 91 35 A. 1. A method for reasoning based on a language model, the method being executed by a server, the server deploying a target language model; the target language model including a target weight matrix and a target vocabulary, the target vocabulary being obtained by multiplying each embedding vector in the original vocabulary of the original language model with a preset rotation matrix, the target vocabulary storing the correspondence between the encryption result of each word and its embedding vector; the target weight matrix being obtained by multiplying the inverse of the rotation matrix with the original weight matrix of the original language model, the rotation matrix being used to rotate the vectors in the corresponding vector space; the method comprising: receiving input data corresponding to user input text from a client; the input data including the encryption result of each word in the input text;According to the target vocabulary, a first intermediate result corresponding to the input data is obtained; the first intermediate result includes the embedding vectors corresponding to each word of the input text; the first intermediate result is processed according to the target weight matrix to obtain a second intermediate result; the inference result of the target language model is obtained according to the second intermediate result, and the inference result is returned to the client. 2. According to the method of claim 1, the original weight matrix includes a first matrix and a second matrix, and the target weight matrix includes a third matrix and a fourth matrix; the third matrix is ​​obtained by multiplying the inverse of the rotation matrix, the first matrix, and the permutation matrix; the fourth matrix is ​​obtained by multiplying the inverse of the permutation matrix and the second matrix; the permutation matrix is ​​used to adjust the order of rows or columns in the matrix; the step of processing the first intermediate result according to the target weight matrix to obtain the second intermediate result includes: processing the first intermediate result according to the first matrix to obtain a third intermediate result; and processing the third intermediate result through the fourth matrix to obtain the second intermediate result. 3. The method according to claim 2, wherein the original weight matrix is ​​the weight matrix of the feature transformation layer, a normalization layer is further included before the feature transformation layer, and the target language model further includes a target weight vector of the normalization layer; the target weight vector is a vector with the same dimension as the original weight vector of the normalization layer of the original language model; the first value is obtained by multiplying the inverse of the target weight vector, the inverse of the rotation matrix, and the first matrix; before processing the first intermediate result according to the target weight matrix to obtain the second intermediate result, the method further includes: processing the first intermediate result according to the target weight vector to obtain a new first intermediate result. 4. The method according to claim 3, wherein the normalization layer is a root mean square normalized (RMS) normalization layer, and the rotation matrix is ​​used to rotate the vector around the origin in the corresponding vector space. 5. The method according to claim 1, wherein the original weight matrix includes a first matrix and a second matrix, and the target weight matrix includes a third matrix and a fourth matrix; the third matrix is ​​obtained by multiplying the inverse of the rotation matrix by the first matrix, and the fourth matrix is ​​obtained by multiplying the second matrix by the rotation matrix; the step of obtaining the inference result of the target language model based on the second intermediate result includes: using the sum of the first intermediate result and the second intermediate result as the input of the next layer corresponding to the target weight matrix for inference, thereby obtaining the inference result of the target language model. 6. The method according to claim 1, wherein the generation method of the rotation matrix includes: obtaining a random matrix, performing QR decomposition on the random matrix to obtain an orthogonal rotation matrix Q; changing the orthogonal rotation...The sign in matrix Q is adjusted so that its determinant is 1 to obtain the rotation matrix; (Claim 1 / 2, page 2, CN 121119135 A) Alternatively, a Hadamard matrix is ​​generated, and the Hadamard matrix is ​​normalized to obtain the rotation matrix. 7. A system for reasoning based on a language model, the system comprising a server and a client, wherein the server deploys a target language model; the target language model includes a target vocabulary and a target weight matrix, the target vocabulary is obtained by multiplying each embedding vector in the original vocabulary of the original language model with a preset rotation matrix, the target vocabulary stores the correspondence between the embedding vectors and the encryption results of each word; the target weight matrix is ​​obtained by multiplying the inverse of the rotation matrix with the original weight matrix of the original language model, the rotation matrix being used to rotate the vectors in the corresponding vector space; the client sends input data corresponding to the user's input text to the server; the input data includes the encryption results of each word in the input text; the server obtains a first intermediate result corresponding to the input data according to the target vocabulary; the first intermediate result includes the embedding vectors corresponding to each word in the input text; the first intermediate result is processed according to the target weight matrix to obtain a second intermediate result; the reasoning result of the target language model is obtained according to the second intermediate result, and the reasoning result is returned to the client. 8. In the system according to claim 7, the order of the embedding vectors corresponding to each word in the original vocabulary and the target vocabulary is different; the client also performs word segmentation processing on the user's input text, and performs hash operation on each word after word segmentation to obtain input data; the client also obtains the output text corresponding to the reasoning result according to the parameters of the hash operation. 9. In the system according to claim 7, the target vocabulary corresponding to different users is different. 10. A computing device, including a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method of any one of claims 1-9. Claims 2 / 2 Page 3 CN 121119135 A A method, system and computer device for reasoning based on a language model Technical Field

[0001] The embodiments of this specification belong to the field of artificial intelligence technology, and particularly relate to a method, system and computer device for reasoning based on a language model. Background Art

[0002] With the development of technology, models for generating text / images have been widely used, such as large-scale language models (LLM). For models, they are generally deployed on servers accessed via the Internet. If a user needs to access the model for inference, they need to communicate with the server through their user device.The data is sent to the server. The model deployed on the server performs inference based on the received user data, obtains the inference result, and sends the inference result to the user device.

[0003] In the event of an attack on the server, the attacker can obtain the intermediate result of the model inference, and may then infer the user data based on the intermediate result. The intermediate result refers to the output result of a certain layer of the model during the inference process based on the user data. In order to solve the problem of intermediate result leakage in the related technologies, the following two methods exist in the related technologies.

[0004] In the first method, the model parameters are obfuscated by using an obfuscation matrix Pi. Specifically, the identity matrix I is obtained first. The identity matrix I refers to a matrix in which only the main diagonal elements are 1 and the rest are 0. For example, the 3rd order identity matrix is ​​I. Then the order of the rows or columns in the identity matrix can be changed to obtain the matrix Pi. For example, changing the order of the second and third rows of a third-order identity matrix can yield [a result] or [a result]. Changing the order of the second and third columns of a third-order identity matrix can yield [a result]. Then, obtaining the transpose of matrix Pi, we get [a result]. Since each row vector of matrix Pi is a unit vector (a unit vector is a vector with a magnitude of 1), and the dot product between any two row vectors is 0, it conforms to the characteristics of an orthogonal matrix. Therefore, matrix Pi is an orthogonal matrix. According to another characteristic of orthogonal matrices, we can obtain PiPiT = I.

[0005] Furthermore, multiplying matrix Pi with the model's weight matrix can change the order of rows or columns in the weight matrix. For example, if the model weight matrix is ​​W1 = [a1 a2 a3], using the previous example, then W1Pi = [a1 a3 a2]. Where a1, a1, and a1 are column vectors. It can be seen that multiplying the two matrices changes the order of the second and third columns of W1. For example, if the model weight matrix follows the previous example, then b1, b2, and b3 are row vectors. It can be seen that multiplying the two matrices changes the order of the second and third rows in W2. Furthermore, similar to the previous example, multiplying PiT with the model's weight matrix can change the order of rows or columns in the weight matrix, which will not be elaborated here.

[0006] Assume the model operation can be equivalent to Y = W1W2X, where X represents the model input, Y represents the model output, and W1 and W2 represent the weight matrices of the two layers of the model. Then the model operation can be equivalent to Y = W1IW2X. Since PiPiT = 1, it can be equivalent to Y = (W1Pi)(PiTW2)X. Then a new model weight matrix can be obtained, W1' = W1Pi, and W2' = PiTW2.

[0007] The two weight matrices (i.e., W1' and W2') obtained by the above method are different from the original weight matrices (i.e., W1 and W2).This makes the intermediate results calculated based on the new weight matrix W2' (e.g., (PiTW2)X) in the server different from the original intermediate results based on the original weight matrix W2 (e.g., W2X), thus achieving the effect of hiding the original intermediate results.

[0008] However, in this method, the matrix Pi can only change the order of rows or columns of the weight matrix, without changing the statistical information of rows or columns in the model weight matrix. For example, if X is a scalar, following the previous example, in the case of converting the original weight matrix into a new weight matrix, the statistical information (e.g., mean or variance) of each element in the second row of W2 is the same as the statistical information of each element in the third row of PiTW2. Similarly, the statistical information of each element in the third row of W2 is the same as the statistical information of each element in the second row of PiTW2. Therefore, after obtaining the original weight matrix (e.g., W2), the attacker can compare the rows of the original weight matrix and the weight matrix processed by Pi (e.g., PiTW2) to determine how to obfuscate the model. For example, in the previous example, the server obtained PiTW2 by changing the order of the second and third rows of W2. For the corresponding intermediate result (PiTW2)X, Pi is multiplied left by (PiTW2)X, i.e., Pi(PiTW2)X = IW2X = W2X, thus obtaining the original intermediate result. Specifically, an attacker can determine PiT using the statistical information method described above, and obtain the permutation matrix Pi based on PiT. The attacker can then determine the original intermediate result based on how the model obfuscates. Continuing with the previous example where X is a scalar, if the attacker obtains (PiTW2)X and determines that the obfuscation method of the weight matrix is ​​to swap the second and third rows, they can swap the second and third rows of (PiTW2)X to obtain the original intermediate result. This still leaks the model's intermediate result. After obtaining the original intermediate result, and with prior knowledge of the weight matrix W2, the attacker can deduce the user data X based on the weight matrix W2 and the original intermediate result W2X. Instruction manual 2 / 21 page 5 CN 121119135 A

[0009] In the second method, for LLM, the vectors of the vocabulary can be encrypted, and the model can be fine-tuned based on the encrypted vocabulary. The vocabulary records the correspondence between each word and the vector. For example, the matrix composed of the vectors of each word in the input text of the model based on the original vocabulary is A, and the matrix composed of the vectors of each word in the output text of the model based on the original vocabulary is B. The matrix corresponding to the above input text based on the encrypted vocabulary is A1, and the matrix corresponding to the above output text based on the encrypted vocabulary is B1. Then, A1 and B1 can be used to fine-tune the model. This allows the model to learn how to reason about encrypted data. After fine-tuning, the user device can encrypt the user's data and send the encrypted data to the encrypted device.The data is sent to the server. The model deployed on the server performs inference based on the received encrypted data and obtains the inference result.

[0010] Although the above method can also change the intermediate results generated by the model, firstly, the method requires fine-tuning the model, which incurs application costs. Secondly, the method is only applicable to classification models and not to language models used for text generation. Specifically, the final output of a classification model is one of a few labels, while a language model needs to output text. For example, LLM needs to determine the next word from hundreds of thousands of words for each inference. It can be seen that the richness of the output content of a language model is much greater than that of a classification model. This makes the language model more sensitive to changes in the input vector. For a language model, a slight change in the input vector will change the output content, while for a classification model, a change in the input vector within a certain range may not change the specific category of the output. Therefore, even if the language model is fine-tuned using encrypted training samples, it is difficult to achieve the original output accuracy of the language model. Summary of the Invention

[0011] The purpose of this specification is to provide a method, system, and computer device for inference based on a language model.

[0012] A first aspect of this specification provides a method for reasoning based on a language model, the method being executed by a server, the server having deployed a target language model; the target language model includes a target weight matrix and a target vocabulary, the target vocabulary being obtained by multiplying each embedding vector in the original vocabulary of the original language model by a preset rotation matrix, the target vocabulary storing the correspondence between the encryption result of each word and the embedding vector; the target weight matrix being obtained by multiplying the inverse of the rotation matrix by the original weight matrix of the original language model, the rotation matrix being used to rotate the vectors in the corresponding vector space; the method includes:

[0013] receiving input data corresponding to user input text from a client; the input data including the encryption result of each word in the input text;

[0014] obtaining a first intermediate result corresponding to the input data according to the target vocabulary; the first intermediate result including the embedding vectors corresponding to each word in the input text;

[0015] processing the first intermediate result according to the target weight matrix to obtain a second intermediate result;

[0016] obtaining the reasoning result of the target language model according to the second intermediate result and returning the reasoning result to the client.

[0017] A second aspect of this specification provides a system for reasoning based on a language model, the system involving a server and a client, the server deploying a target language model; the target language model includes a target vocabulary and a target weight matrix, the target vocabulary is obtained by multiplying each embedding vector in the original vocabulary of the original language model with a preset rotation matrix, the target vocabulary stores the correspondence between the embedding vectors and the encryption results of each word; the target weight matrix...The rotation matrix is ​​obtained by multiplying the inverse of the rotation matrix with the original weight matrix of the original language model. The rotation matrix is ​​used to rotate the vector in the corresponding vector space.

[0018] The client sends input data corresponding to the user's input text to the server. The input data includes the encryption results of each word in the input text.

[0019] The server obtains a first intermediate result corresponding to the input data according to the target vocabulary. The first intermediate result includes the embedding vectors corresponding to each word in the input text. The first intermediate result is processed according to the target weight matrix to obtain a second intermediate result. The inference result of the target language model is obtained according to the second intermediate result, and the inference result is returned to the client.

[0020] A third aspect of this specification provides a computer-readable storage medium storing a computer program that, when executed in a computer, causes the computer to perform the above-described language model deployment method or language model inference method.

[0021] A fourth aspect of this specification provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the above-described language model deployment method or language model inference method.

[0022] A fifth aspect of this specification provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-described language model deployment method or language model inference method.

[0023] This specification proposes a method for inference based on a language model. This method is executed by a server, which deploys a target language model; the target language model includes a target vocabulary and a target weight matrix. The target vocabulary is obtained by multiplying each embedding vector in the original vocabulary of the original language model with a preset rotation matrix. The target vocabulary stores the correspondence between embedding vectors and input data; the target weight matrix is ​​obtained by multiplying the inverse of the rotation matrix with the original weight matrix of the original language model. The rotation matrix is ​​used to rotate the vectors in the corresponding vector space. During the inference process, input data corresponding to the input text is received from the client, and then a first intermediate result corresponding to the input data is obtained; the first intermediate result includes several embedding vectors; according to the target weight matrix, the first intermediate result is processed to obtain a second intermediate result; the inference result of the target language model is obtained according to the second intermediate result, and the inference result is returned to the client.

[0024] The above method changes the embedding vectors in the vocabulary, the words in the vocabulary, and the weight matrix of the model. In the presence of a white-box attack (i.e., the attacker controls the server), even if the attacker obtains any of the above intermediate results andThe model's weight matrix can only be derived from the intermediate results to obtain the input data of the target language model. The input data is the result of encrypting each word of the input text, so even if an attacker obtains the input data, they cannot deduce the user's input text. This protects the security of user data.

[0025] Furthermore, the method in this specification uses a rotation matrix to obfuscate the embedding vectors in the original vocabulary. Even if an attacker obtains the embedding vectors corresponding to each word in the original vocabulary, they cannot determine which word each embedding vector in the first intermediate data corresponds to.

[0026] Moreover, compared to the first method in related technologies, the method in this specification not only prevents attackers from deducing the user's original data but also changes the statistical information of the weight matrix using the rotation matrix, making it difficult for attackers to decipher the original weight matrix and protecting the security of the weight matrix. Compared to the second method in related technologies, the method in this specification requires no fine-tuning and has a lower cost.

[0027] Brief Description of the Drawings: In order to more clearly illustrate the technical solutions of the embodiments of this specification, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained from these drawings on page 4 / 21 of this specification without creative effort.

[0028] Figure 1 is a schematic diagram of a method for deploying a language model in one embodiment;

[0029] Figure 2 is a schematic diagram of an MLP segment in one embodiment;

[0030] Figure 3 is a schematic diagram of a residual connection in one embodiment;

[0031] Figure 4 is a schematic diagram of a client deploying a language model on a server and using the language model deployed on the server for inference in one embodiment;

[0032] Figure 5 is a schematic diagram of deploying a language model and using the deployed language model for inference in one embodiment;

[0033] Figure 6 is a flowchart of a method for inference based on a language model in one embodiment;

[0034] Figure 7 is a schematic diagram of a target language model in one embodiment;

[0035] Figure 8 is a block diagram of a method for inference based on a language model in one embodiment. Detailed Description

[0036] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without inventive effort should fall within the scope of protection of this specification.

[0037] In this specification, Large Language ModelLarge Language Models (LLMs), also known as big models, are natural language processing models based on deep learning techniques. Their parameter count typically ranges from billions to hundreds of billions or even higher, possessing powerful language understanding and generation capabilities. LLMs can employ the Transformer architecture or its variants (such as GPT and BERT), which utilizes an attention mechanism to globally model sequential data, efficiently handling long-distance dependencies and thus performing exceptionally well in natural language tasks. By pre-training on large-scale corpora, LLMs learn the statistical features and semantic relationships of language, enabling them to generalize effectively. The core capabilities of LLMs include, but are not limited to: understanding contextual semantics, generating coherent and grammatically correct text, performing logical reasoning, and handling multi-task scenarios. Their usage typically includes two modes: direct inference and fine-tuning. In direct inference mode, users design prompts to guide the LLM in generating specific outputs. Prompts can be textual descriptions of the task or instructions used to stimulate the LLM's semantic understanding and generation capabilities. In fine-tuning mode, large language models are further trained on small-scale datasets in specific domains to optimize their performance on specific tasks. The powerful generalization ability and flexibility of large language models make them an important tool in the field of artificial intelligence technology, providing efficient and accurate solutions for automated text generation and understanding.

[0038] In some embodiments, large language models can also have the ability to understand and generate data from other modalities (such as visual, audio, etc.). In this case, large language models can also be called multimodal large language models (MLLMs). MLLMs provide a richer and more natural interactive experience by integrating multiple types of inputs and outputs such as text, images, and sound. The core advantage of MLLMs is that they can process and understand information from different modalities and fuse this information to complete complex tasks. For example, MLLMs can analyze an image and generate descriptive text, or generate a corresponding image based on the text description. This cross-modal understanding and generation capability makes MLLMs have broad application prospects in multiple fields.

[0039] It should be noted that the key technologies of large language models can be found in the paper "A Survey of Large Language Models" (Paper No.: arXiv:2303.18223v16, Publication Date: March 11, 2025, Publication Link Specification 5 / 21 pages 8 CN 121119135 A).The detailed description in https: / / doi.org / 10.48550 / arXiv.2303.18223 is not repeated here.

[0040] The method provided in this specification will be described in detail below. The method provided in this specification involves two stages: the model deployment process and the inference process using the deployed model. First, the model deployment process will be described.

[0041] First, the application scenarios of this method will be described. This method can be applied to servers, that is, servers modify the models they already hold to protect user privacy. In addition, this method can also be applied to user devices. Due to insufficient computing power, user devices deploy models in cloud servers. Correspondingly, this method can be executed by user devices, that is, user devices obfuscate the model and then deploy it in cloud servers.

[0042] The model deployed by the method in this specification is a language model (hereinafter referred to as the model or language model). A language model refers to a model used to generate corresponding output text / images based on the text input by the user. In one alternative embodiment, the language model can be a large language model; in another alternative embodiment, the language model can be a small language model. As mentioned above, language models are not suitable for related technologies due to their high sensitivity to input changes. However, the method provided in this specification does not require fine-tuning, overcoming the problem of insufficient output accuracy after fine-tuning in related technologies.

[0043] Next, a language model deployment method provided in this specification will be described with reference to FIG1. ​​This method can be executed, for example, by a computing device used for deployment.

[0044] As shown in FIG1, the method includes:

[0045] Step 101, obtaining the original vocabulary of the language model and the original weight matrix of each feature transformation layer.

[0046] Wherein, the original vocabulary includes the embedding vectors corresponding to each word.

[0047] Step 103, multiplying each embedding vector in the original vocabulary with a preset rotation matrix to obtain the target vocabulary.

[0048] Wherein, the rotation matrix is ​​used to rotate the vector in the corresponding vector space;

[0049] Step 105, multiply the inverse matrix of the rotation matrix with the original weight matrix to obtain the target weight matrix.

[0050] Here, the processing of the vocabulary and the processing of the weight matrix will be explained respectively.

[0051] First, the processing of the vocabulary will be explained. For ease of understanding, the process of how the language model in the related technology obtains the embedding vector based on the user input text during the inference process will be explained here. First, the user input text will pass through the tokenizer, which can map the input text into the form of an index according to the preset mapping relationship. For example, the input text "good morning, i am"good", the tokenizer outputs [19045, 6693, 11, 602, 1097, 1695]. The mapping can be done according to a preset vocabulary, which stores the correspondence between each word segmented by the tokenizer and its index.

[0052] After obtaining the index, it can be converted into an embedding vector. The server can store a vector table containing the embedding vectors corresponding to each index. For example, if the embedding vector for each word includes 4096 elements, and the tokenizer output includes 120,000 words, then the vector table can be a matrix of size 120,000 × 4096, and the index is the row number of the corresponding embedding vector in the vector table. Accordingly, after obtaining the index [19045, 6693, 11, 602, 1097, 1695], the vector at the corresponding position can be retrieved from the vector table according to the index to form an input matrix of size 6 × 4096.

[0053] For ease of explanation, the original vocabulary without rotation matrix processing will be referred to as the vocabulary in the following text. The inclusion of embedding vectors corresponding to each word in the vocabulary means that the vocabulary includes at least this vector table. In addition to the vector table, the vocabulary may also include the mapping relationship between word roots (hereinafter referred to as words) and indices in text form. Therefore, the vocabulary may include a vector table and the mapping relationship between words and indices. Specification 6 / 21 page 9 CN 121119135 A

[0054] The method provided in this specification can process the vector table as shown in step 103. Optionally, the mapping relationship stored in the vocabulary can also be encrypted and shuffled to further enhance the security of user data. The two processes will be described separately below.

[0055] First, for the words included in the mapping relationship, the text-form words in the mapping relationship can be encrypted, and the encryption result can replace the corresponding words in the word list. For example, "good" is encrypted into a 32-bit string, which can be used to replace the original word "good" in the mapping relationship. That is, the words in the word list are replaced with the data obtained by encrypting the words. Correspondingly, corresponding encryption transformation is also required during inference. For example, the user needs to segment the input text through a word segmenter, encrypt the segmentation result, and then send it to the server. The specific processing method of the user device will be described below and will not be detailed here.

[0056] Through encryption processing, even if the attacker intercepts the data sent by the user device to the server during inference, the original input text cannot be obtained, thus ensuring the user's data security.

[0057] As for the specific method of encrypting words, the encryption needs to ensure that: the encryption result is the same when the same data is encrypted multiple times; and the original data cannot be deduced from the encryption result alone; and the encryption results of different data are different.Different. This ensures the correctness of the reasoning.

[0058] Considering the above characteristics, in an optional embodiment, the above encryption process can be completed by hash operation. Specifically, if the vocabulary includes the above mapping relationship, that is, if the vocabulary also includes each word and its corresponding index value, a hash operation can be performed on each word, and the hash operation result can replace the corresponding word in the vocabulary. Of course, other encryption methods that can meet the above conditions can also be used as the encryption method of this specification.

[0059] In some embodiments, the deployed language model may need to serve different users. If all users use the same encryption method, it may still cause leakage of user data. For example, if encryption is performed by hash operation, and user A and user B have the same hash operation result for the same data, then user A can intercept the data sent by user B's device to the server and crack the data sent by user B by means of rainbow table, etc. Among them, rainbow table is a pre-computed set of hash chains used to efficiently crack passwords encrypted by hash function.

[0060] Based on this, in the process of word encryption in this specification, different keys can be set for different users, and different keys correspond to different encryption results. That is, different users correspond to different target vocabulary. The different target vocabularies here could be due to different encryption results for words in the target vocabularies of different users; different rotation matrices for different users, resulting in different embedding vectors for different users; different index values ​​for different users, resulting in different order of embedding vectors in the target vocabularies; or different encryption results, embedding vectors, and index values ​​for words in the target vocabularies of different users.

[0061] In addition, multiple keys can be set for the same user, with each key corresponding to a different encryption result. That is, multiple vocabularies are generated for the same user, with each vocabulary corresponding to a key. In this way, during inference, the user device can select the corresponding encryption method to encrypt the words according to the key input by the user and send the encrypted words to the server. Correspondingly, the server can also use the vocabulary corresponding to the key to obtain the index. This can further increase user privacy and security, and increase the difficulty for attackers to crack the code.

[0062] Although multiple vocabularies are generated for one user in the above method, the storage space occupied by the vocabulary is small, and even if multiple vocabularies are generated for one user, it will not occupy a lot of storage space.

[0063] Optionally, the BLAKE2 algorithm can be used for encryption when using a hash algorithm. The BLAKE2 algorithm supports generating different random seeds based on the user-input key. For the same data to be hashed, different random seeds can yield different hash results. (Specification 7 / 21, page 10, CN 121119135 A)

[0064] Using the BLAKE2 algorithm, the encryption process of words in the vocabulary can be flexibly implemented.

[0065] Secondly, for the vector table, the vector table can be processed as shown in step 103. Specifically, each embedded vector in the original vocabulary is multiplied by a preset rotation matrix, and the result of the multiplication is used to replace the corresponding vector in the original vocabulary to obtain the target vocabulary.

[0066] As mentioned above, the rotation matrix is ​​used to rotate the vector in the corresponding vector space. For ease of understanding, the meaning of "rotating in the corresponding vector space" will be explained here through an example of a three-dimensional vector. The vector space can also be Euclidean space. For example, for a 3-dimensional vector, the three elements of the vector can represent the values ​​on the x, y, and z axes respectively. A three-dimensional vector represents a point in the vector space. Assuming that the origin (0,0,0) of the vector space is taken as the rotation center, then a sphere can be formed with the line connecting the point corresponding to the vector and the origin as the radius and the origin as the center. The points on the surface of the sphere are the points corresponding to the vector that the vector can be rotated in the vector space.

[0067] By rotating, the embedding vectors corresponding to each word can be changed, thus preventing attackers from reconstructing the original input data through the embedding vectors and protecting user privacy. In addition, although the value of the embedding vector is changed here, the weight matrix will also be processed using the inverse of the rotation matrix later. Since the result of multiplying any matrix and its inverse is the identity matrix I, the multiplication of the rotation matrix and its inverse is also the identity matrix I. This eliminates the influence of the rotation matrix from the final output, so that the vector corresponding to the inference result is not changed. This allows the input data and the parameters of the model to be obfuscated by the rotation matrix and its inverse, respectively, without changing the final output of the model, thereby changing the size of the intermediate result of the model and protecting user privacy.

[0068] It can be seen that, since this method does not change the final output of the model, does not change the mapping relationship between the input (embedded vectors not multiplied with the rotation matrix) and the output, does not require additional fine-tuning, and does not change the original inference process of the model, the deployment cost of the model is reduced.

[0069] Moreover, the above rotation causes the embedding vector to rotate in the corresponding vector space, which causes a significant change in the statistical information of the target weight matrix after multiplying with the inverse of the rotation matrix. For example, the statistical information of a certain row of the original weight matrix is ​​no longer the same as the statistical information of a certain row of the target weight matrix. Therefore, even if an attacker obtains the original weight matrix, they will not know how to obfuscate the original weight matrix, and thus will not be able to deduce the intermediate results of the model using the original weight matrix for inference.

[0070] The specific processing procedure of step 103 will be explained next. For details, please refer to the following formula (1):

[0071] We' = WeRe (1)

[0072] Where Re represents the rotation matrix, and We represents the matrix corresponding to the original vector table, which is composed of the embedding vectors corresponding to each word in the order represented by the index value. For example, if there are 120,000 words, each row of the matrix corresponds to an embedding vector, and an embedding vector includes 4096 elements. Then the dimension of the matrix corresponding to the vector table can be 120,000 × 4096. Under this dimension, the dimension of the rotation matrix Re can be 4096 × 4096. We' is the vector table included in the target word table obtained after processing in step 103.

[0073] As for the specific form of the rotation matrix, if the origin of the vector space is taken as the rotation center, in this case, the rotation matrix can ensure that each row / column vector forms a set of orthogonal bases (that is, the magnitude of each row / column vector is 1, and they are orthogonal to each other). Then the rotation matrix conforms to the characteristics of an orthogonal matrix and is an orthogonal matrix. The rotation matrix can be generated by any of the following methods:

[0074] First, obtain a random matrix, perform QR decomposition on the random matrix to obtain an orthogonal rotation matrix Q; change the sign in the orthogonal rotation matrix Q so that its determinant value is 1, and obtain the rotation matrix. Specification 8 / 21 Page 11 CN 121119135 A

[0075] Here, by performing QR decomposition on the random matrix, an orthogonal matrix, that is, an orthogonal rotation matrix Q, can be obtained. Among them, QR decomposition is a process of decomposing a matrix into the product of two specific matrices. For example, for a random matrix A, A = QR can be obtained by QR decomposition. Among them, matrix Q is an orthogonal matrix. R is an upper triangular matrix, that is, the elements below the main diagonal of matrix R are all 0.

[0076] For the orthogonal rotation matrix Q, the elements contained therein may be negative numbers or positive numbers, which may make the final determinant value not 1. In this case, using this matrix to rotate is to take the mirror image of the original point and then rotate. If there are normalization layers between the layers of the original weight matrix, since the normalization layer can only pass on rotations around the origin, such rotations may not be passed on by the subsequent normalization layers. This will make it impossible to eliminate the influence of the rotation matrix after the data processed by the normalization layer is multiplied by the target weight matrix, thus changing the inference result of the model. Therefore, it is necessary to change the sign of the elements so that its determinant value is 1.

[0077] Second, generate the Hadamard matrix and normalize the Hadamard matrix to obtain the rotation matrix.

[0078] The Hadamard matrix is ​​composed of 1 and -1 elements and satisfies that the product of the matrix and its transpose is n times the identity matrix. As can be seen from the above definition of the Hadamard matrix, the normalized Hadamard matrix is ​​an orthogonal matrix with a determinant value of 1. This matrix also satisfies the characteristics of the rotation matrix required by this specification.

[0079] The above two examples are not intended to limit this specification. Any orthogonal matrix with a determinant of 1 can be used as the rotation matrix of this application.

[0080] Finally, for the index values ​​representing the order of each embedding vector in the vocabulary, the order of the words in the vocabulary can be further shuffled, that is, the index values ​​corresponding to each word can be changed. In other words, the order of different words in the original vocabulary can be changed.

[0081] This is done considering that if an attacker obtains the original unprocessed vocabulary of the model, even if the words are encrypted and the embedding vectors are rotated (as shown in step 103), but the index values ​​are not changed, that is, the position of each word's embedding vector in the vector table is not changed, then the attacker can still compare the order of each embedding vector in the processed vector table with the mapping relationship in the unprocessed vocabulary to determine the word corresponding to each processed embedding vector, causing the leakage of user privacy. Therefore, by shuffling the order of different words in the vocabulary and changing the index values ​​of each word, the leakage of user privacy caused by the attacker obtaining the unprocessed vocabulary can be prevented.

[0082] Regarding the specific processing method for changing the order of different words in the vocabulary, it can be that for each word, its index value is randomly changed, and the position of the embedding vector corresponding to each word in the vector table is adjusted according to the newly changed index value.

[0083] Through the above steps, the vocabulary can be encrypted to obtain the processed target vocabulary. This prevents attackers from obtaining user privacy through data sent by user devices, or prevents attackers from obtaining the user's original input data through the matrix input in the model when attacking the server.

[0084] In addition, the vocabulary can be used not only in the processing of input data, but also in the processing of output data. The language model outputs in the form of embedding vectors. In order to restore the original output text according to the embedding vectors, it is necessary to obtain the word corresponding to each embedding vector of the output according to the vocabulary. Therefore, the vocabulary can also be applied to the processing of the output embedding vectors.

[0085] The processing method of the model's weight matrix will be explained next. For the language model, it can include three types of layers, namely feature transformation layer, normalization layer and output layer.

[0086] The feature transformation layer refers to a layer that can transform the features input to that layer using a weight matrix. For example, as described on page 9 / 21 of the specification (CN 121119135 A), it may include a multi-head attention block (MHA), a multilayer perceptron block (MLP), and a linear layer. The normalization layer is used to standardize the input of each layer. The output layer is used to finally obtain the output data.

[0087] To facilitate understanding, the processing procedures of various feature transformation layers in related technologies are first explained here.

[0088] For MHA, each head is processed through Q, K, and V matrices, and finally the calculation results of multiple heads are linearly concatenated through the output projection matrix.

[0089] For MLP, as shown in Figure 2(a), the input data is first processed through weight matrix 1, and then the processed data is input into the activation function. Finally, the data processed by the activation function is processed through weight matrix 2 to obtain the output data.

[0090] For gated MLP, as shown in Figure 2(b), it is first processed through two parallel paths, namely the gated path and the upscaling path. In the gated path, the input data is multiplied by the weight matrix of the gated path (referred to as the gated weight in the figure) and then input into the activation function. In the upscaling path, the data is multiplied by the weight matrix of the upscaling path (referred to as the upscaling weight in the figure). Then the output data of the two paths are multiplied element by element to fuse the data of the two paths. Finally, the fused data is input into the lower projection path and multiplied by the weight matrix of the lower projection path (referred to as the lower projection weight in the figure) to obtain the output data.

[0091] For the linear layer, the processing can be summarized as Y = xW + b, where x is the input of the linear layer, Y is the output of the linear layer, W is the weight matrix of the linear layer, and b is the bias coefficient.

[0092] For the calculation method of the normalization layer, it is generally scaled using the mean or variance of the data. In some normalization layers, a specific vector can be used for further scaling.

[0093] The output layer is the last layer of the neural network model. In the language model, it is generally a linear layer.

[0094] Step 105 shows the improvement of the feature transformation layer using the inverse of the rotation matrix. Optionally, in addition to using the inverse of the rotation matrix, the weight matrix of the feature transformation layer can also be further processed using the permutation matrix. In addition, the weight matrix of the feature transformation layer can be processed by combining the vector of the normalization layer. The weight matrix of the output layer can also be processed. The above process will be explained in detail below.

[0095] First, regarding the specific implementation of step 105, the target weight matrix can be obtained by multiplying the inverse of the rotation matrix with the original weight matrix. Specifically, the target weight matrix can be calculated, where is the inverse of the rotation matrix and W is the original weight matrix.

[0096] In the linear layer, the original weight matrix is ​​the weight matrix W of the linear layer. In some types of layers, the input data has undergone processing of multiple matrices, such as in MLP and MHA, as mentioned above, multiple weight matrices are included. Correspondingly, the original weight matrix may include a first matrix and a second matrix, wherein the first matrix is ​​used to combine with the input of the feature transformation layer.Linear operations are performed, and the second matrix is ​​used to obtain the output of the feature transformation layer. In other words, the first matrix is ​​the matrix that comes first in the processing of this layer, and the second matrix is ​​the matrix that comes last. Correspondingly, the first matrix is ​​the one that is transformed using the inverse of the rotation matrix in step 105. That is, step 105 specifically involves multiplying the inverse of the rotation matrix with the first matrix.

[0097] Regarding the specific forms of the first and second matrices in various feature transformation layers, in MLP, the first matrix can be the first weight matrix, such as weight matrix 1 in Figure 2. The second matrix can be the second weight matrix, such as weight matrix 2 in Figure 2. In gated MLP, the first matrix can include the weight matrices of the gated path and the upward expansion path, and the second matrix can be the weight matrix of the downward projection path. In MHA, the first matrix can include Q, K, and V matrices, and the second matrix can be the output projection matrix mentioned above.

[0098] Here, we will take the feature transformation layer as the first linear layer of the model as an example to explain why the above process does not affect the output of the model specification page 10 / 21, 13 CN 121119135 A.

[0099] As mentioned above, a rotation matrix can be used to process the vector table. Correspondingly, the embedding matrix obtained after processing the user input text is WeRe. Here, We is the original embedding matrix corresponding to the user input text. The weight matrix of the linear layer after processing is Y = xW + b. Since the calculation formula of the linear layer is Y = xW + b, substituting the processed data above, according to the property that the product of a matrix and its inverse matrix equals the identity matrix, we can obtain that the result of the operation is the same as the original calculation result of the linear layer.

[0100] It can be seen that by processing the vocabulary with a rotation matrix and processing the weights of the feature transformation layer with the inverse matrix of the rotation matrix, the influence of the rotation matrix can be eliminated. This changes the size and statistical information of the input data and weights without changing the final inference result.

[0101] In addition to using a rotation matrix to process the weights of the feature transformation layer, a permutation matrix can also be used to change the order of rows or columns in the matrix processed by the rotation matrix to achieve a better obfuscation effect.

[0102] Specifically, the inverse of the rotation matrix can be multiplied by the original weight matrix to obtain a first value; then the first value can be multiplied by the permutation matrix to obtain the target weight matrix. The permutation matrix is ​​used to adjust the order of rows or columns in the matrix. Whether the permutation matrix adjusts the order of rows or columns depends on the position of the permutation matrix when multiplied by the first value. The first value is P, as mentioned above. The target weight matrix can be P or P, where the first value is the calculated matrix.

[0103] Based on this, to ensure that the final reasoning result is not affected by the permutation matrix, the inverse of the permutation matrix can be multiplied by the original weight matrix.The matrix is ​​multiplied with other weight matrices to eliminate the influence of the permutation matrix. For example, if the feature transformation layer is a linear layer and the target weight matrix is ​​W, the inverse of the permutation matrix is ​​multiplied with the weight matrix of the next layer corresponding to this layer. For example, if the weight matrix of the next layer is W, then P-1W can be obtained as the weight matrix of the next layer.

[0104] In addition, if the original weight matrix of the feature transformation matrix includes the first matrix and the second matrix mentioned above, the permutation matrix can also be multiplied with the first matrix, and the inverse of the permutation matrix can be multiplied with the second matrix to eliminate the influence of the permutation matrix while transforming both the first matrix and the second matrix.

[0105] Specifically, the inverse of the rotation matrix can be multiplied with the first matrix to obtain the first value; the first value can be multiplied with the permutation matrix to obtain the third matrix. Specifically, the third matrix can be where W1 is the first matrix.

[0106] Then, the inverse of the permutation matrix is ​​multiplied with the second matrix to obtain the fourth matrix. The second matrix is ​​W2, and the fourth matrix can be P-1W2.

[0107] Correspondingly, the obtained third and fourth matrices can constitute the target weight matrix corresponding to this layer.

[0108] In addition, for MHA, the permutation matrix needs to be designed according to the following characteristics: according to the characteristics of the head (attention head), the channel order is shuffled inside each head, and then the order between heads is shuffled outside to ensure the computational equivalence of each head.

[0109] For example, based on the feature transformation layer being MHA, according to the characteristics of MHA, the permutation matrices corresponding to the QK matrices need to be the same. Although the permutation matrix is ​​multiplied by the QK matrix, the influence of the permutation matrix can be canceled out when the data dot product of the two matrices is processed, so there is no need to process the permutation matrix. Correspondingly, in MHA, the inverse matrix of the permutation matrix of the projection matrix (i.e., the second matrix) on page 11 / 21 of the specification 14 CN 121119135 A can be the permutation matrix corresponding to the V matrix.

[0110] Then, according to the above description, the third matrix obtained corresponding to the Q matrix is ​​the third matrix obtained corresponding to the K matrix, where WQ and WK are the first matrices. It can be seen that the permutation matrices in the third matrices corresponding to the QK matrices are the same, both being PQK. The third matrix obtained corresponding to the V matrix is ​​where WV is the first matrix. The fourth matrix corresponding to the output projection matrix is ​​where WO is the original output projection matrix, i.e., the second matrix. It can be seen that the permutation matrix PVO used by the V matrix and the output projection matrix are different forms of the same permutation matrix.

[0111] In addition, the method for generating the permutation matrix may include: based on the number of columns n of the original weight matrix,Generate an n-dimensional identity matrix; randomly shuffle the rows of the n-dimensional identity matrix to obtain the permutation matrix. This generates a permutation matrix that can perform row or column permutations on the weight matrix.

[0112] In addition, there may be a normalization layer in the language model. Some normalization layers cannot pass the influence of the rotation matrix on the input data to the next layer, and such normalization layers will affect the final output of the model. Therefore, a rotation-invariant normalization layer can be used, which can pass the influence of the rotation matrix to the next layer without change. For example, root mean square normalization (RMSnorm) can be used as a normalization layer.

[0113] In addition, considering that RMSnorm can only pass the influence of rotation with the rotation center at the origin of space, in the case of the normalization layer being an RMSnorm layer, the rotation matrix is ​​used to rotate the vector around the origin of space in the corresponding vector space. The method for generating the corresponding rotation matrix that meets this requirement is detailed above and will not be repeated here.

[0114] As mentioned above, some normalization layers need to multiply the data with a specific vector after normalization. For example, in RSMnorm, after normalizing the data based on the variance and other data of the input layer, it is necessary to use the scale vector (hereinafter referred to as the original weight vector of the normalization layer) for processing. Since the normalization layer is generally before the feature transformation layer, if the input data is directly input into the normalization layer first and then into the feature transformation layer, the influence of the rotation matrix Re may not be eliminated.

[0115] Specifically, assuming the scale vector is Vn, after normalization by the normalization layer, but before the scale vector processing, the data is WeRe (here WeRe and the rotated embedding vector have the same label, and the influence of the normalization layer on the change of matrix size is not shown. The normalization layer changes the size of each element of the matrix in the same way, which is equivalent to WeRe being multiplied by a scalar), and the target weight matrix of the feature transformation layer is, then the input data after processing by the normalization layer and the feature transformation layer can be seen, which cannot eliminate the influence of the rotation matrix on the input data.

[0116] Therefore, when processing the weight data of each layer of the model, the order of processing of each layer needs to be considered. Based on this, if the original weight vector Vn of the normalization layer is processed, the original vector Vn will be changed into a matrix, changing the dimension of the weight vector of the normalization layer. Moreover, the attacker can also obtain the original weight vector Vn and infer the rotation matrix Re based on the obfuscation.

[0117] In order to solve the above problems, this specification also obfuscates the weight vector of the feature transformation layer by using the weight vector of the normalization layer before the feature transformation layer (if any) for each feature transformation layer. Specifically, step 105 may include: obtaining the original weight vector of the feature transformation layer before the normalization layer before the feature transformation layer.The inverse of the rotation matrix is ​​multiplied by the target weight vector, and the result of the multiplication is multiplied by the original weight matrix of the feature transformation layer to obtain the target weight matrix. Specification 12 / 21 Page 15 CN 121119135 A

[0118] Specifically, if the rotation matrix is ​​Re, the original weight vector is Vn, and the original weight matrix is ​​W, then the obtained target weight matrix can be

[0119] In addition, a random weight vector (hereinafter referred to as the target weight vector) can be generated for the normalization layer before the feature transformation layer. Furthermore, in subsequent feature transformation layers, the inverse of the target weight vector is used to obfuscate the original weight vector to counteract the influence of the random target weight vector on the result.

[0120] Specifically, step 105 can specifically include: for any feature transformation layer, obtaining the original weight vector of the normalization layer adjacent to and before the feature transformation layer; generating a target weight vector with the same dimension as the original weight vector; the target language model includes the target weight vector. This target weight vector serves as the weight vector of the normalization layer.

[0121] Then, multiply the inverse of the target weight vector by the inverse of the rotation matrix, and multiply the result by the original weight vector to obtain a second value; multiply the second value by the original weight matrix (or the first matrix if a first matrix and a second matrix exist) to obtain the target weight matrix.

[0122] For example, the target weight vector is V, and the inverse of the target weight vector is V-1. The notation of other matrices follows the notation above, so the second value can specifically be the target weight matrix W.

[0123] Furthermore, when using the permutation matrix mentioned above for obfuscation, this step can obtain the first value mentioned above. The first value can then be further obfuscated using the permutation matrix to obtain the target weight matrix.

[0124] Thus, even if the original weight vector is removed from the normalization layer and used to obfuscate the original weight matrix of the feature transformation layer, the original reasoning method is not changed because the target weight vector is used in the normalization layer. Moreover, this method can also use the target weight vector to confuse attackers. Even if the attacker discovers that the specific values ​​of the target weight vector and the original weight vector are different, they will not be able to know the confusion method of the model.

[0125] In addition, for the generated target weight vector, it can be a randomly generated vector with the same dimension as the original weight vector. This specification does not limit this.

[0126] Regarding the size of the target weight matrix, if the values ​​of the target weight matrix and its inverse vector (reciprocal) are too large or too small, it will cause overflow or underflow of the calculation result, affecting the accuracy of the calculation result. Therefore, it is necessary to ensure that the values ​​of the target weight matrix and its inverse are within a certain range. Therefore, if the target weight matrix is ​​randomly generated, it can be pre-generated.Random values ​​are taken within the set range. The preset range can be [epsilon, 1 / epsilon], where epsilon can be 1×10⁻³ or 1×10⁻⁶, and the specific size can be determined according to the numerical type of the data used in the inference process.

[0127] In some embodiments, residual connections may also exist in the model. Specifically, for any feature transformation layer, the result of adding the output of the feature transformation layer to the input is used as the input of the next layer of the feature transformation layer. In the case where there is a normalization layer before the feature transformation layer, the output of the feature transformation layer is added to the input of the normalization layer as the input of the next layer of the feature transformation layer. A schematic diagram of residual connections is shown in Figure 3.

[0128] In the case of residual connections, for the target weight matrix, it is necessary not only to multiply the inverse matrix of the rotation matrix with the original weight matrix, but also to multiply the result of the multiplication with the rotation matrix, so that the output of the feature transformation layer and the input of the feature transformation layer introduced in the residual connection correspond to each other, and are both data that have been rotated by the rotation matrix.

[0129] Furthermore, since the residual connection makes the input data of the next layer of the feature transformation layer actually the data obtained by multiplying the original input data by the rotation matrix, it is necessary to perform the processing shown in step 105 on each feature transformation layer so that the rotation matrix does not affect the final output result.

[0130] Here, taking the original weight matrix including the first matrix and the second matrix as an example, we will explain how to calculate the target weight matrix on page 13 / 21 of the basic specification 16 CN 121119135 A with residual connection. The meaning of the first matrix and the second matrix is ​​detailed above and will not be repeated here. Step 105 may include: multiplying the inverse matrix of the rotation matrix with the first matrix to obtain the third matrix; multiplying the second matrix with the rotation matrix to obtain the fourth matrix; wherein the target weight matrix includes the third matrix and the fourth matrix.

[0131] For example, the first matrix is ​​W1, the second matrix is ​​W2, and the rotation matrix is ​​Re, then the third matrix may specifically be W2Re.

[0132] Furthermore, this not only ensures accurate model inference results even with residual connections, but also makes the input of each feature transformation layer equivalent to being obfuscated using a rotation matrix, thereby obfuscating all intermediate results and protecting user privacy.

[0133] In addition, based on the presence of residual connections, not only can the original weight matrix be obfuscated using a rotation matrix, but also the permutation matrix, target weight vector, etc., mentioned above can be used for obfuscation. Specific obfuscation methods can be found in the example shown in Figure 6 below.

[0134] It should also be noted that, based on the presence of residual connections, if multiple normalization layers exist, then multiple normalization layers...The target weight vectors corresponding to the unification layers can be different. This can increase the obfuscation level of the model.

[0135] Step 107: Deploy the target language model on the server; the target language model includes a target vocabulary and a target weight matrix.

[0136] Specifically, after obtaining the target vocabulary and target weight matrix, a target language model including the target vocabulary and target weight matrix can be deployed on the server for use in subsequent inference processes.

[0137] In the case where the method shown in Figure 1 is executed by the server, step 107 can be the server deploying the target language model locally on its own.

[0138] In addition, as shown in Figure 4, the method can also be executed by the client, and the content included in the dashed box in the figure is executed by the client. The client is the user device mentioned above. As shown in Figure 4, the client can use rotation matrix and permutation matrix for obfuscation (referred to as rotation obfuscation and permutation obfuscation in the figure), as well as encrypt and rotate the vocabulary. After obtaining the target language model, the client uploads the target language model to the server to deploy the target language model on the server. Correspondingly, during the inference process, the client sends a request containing user input data to the server via the Internet. As shown in Figure 4, when encrypting the words in the vocabulary, the request may specifically include the encrypted strings corresponding to each word in the user-input text, rather than the user-input text in related technologies.

[0139] As shown in Figure 5, the above process describes the model inference deployment process, which involves processing the original model (i.e., the language model) into a confused model (i.e., the target language model) and processing the original tokenizer (segmenter) into an encrypted tokenizer. This process corresponds to the vocabulary processing process described above.

[0140] The above method only changes the model's weight data and does not change the model's inference method. When using the model for inference, the inference method used remains unchanged. Next, a method for inference based on a language model will be described. This method can be executed by the server. The server is deployed with a target language model; the target language model includes a target vocabulary and a target weight matrix. The target vocabulary is obtained by multiplying each embedding vector in the original vocabulary of the original language model with a preset rotation matrix. The target vocabulary stores the correspondence between the embedding vectors and the encryption results of each word. The target weight matrix is ​​obtained by multiplying the inverse of the rotation matrix with the original weight matrix of the original language model. The rotation matrix is ​​used to rotate the vectors in the corresponding vector space.

[0141] Wherein, the original language model is a language model without obfuscation, and the target language model is a language model with obfuscation. (See page 14 / 21 of the specification, 17 CN 121119135 A)The obtained language model. The method for obtaining the target vocabulary and target weight matrix can be found in the previous description of the method for deploying the language model, and will not be repeated here.

[0142] As shown in Figure 6, the method for reasoning based on the language model shown in this specification may include the following steps:

[0143] Step 601, receiving input data corresponding to the user's input text from the client.

[0144] In an optional embodiment, the above-mentioned input data may be the input text entered by the user.

[0145] In another optional embodiment, the above-mentioned input data may also be data obtained by encrypting the input text. As mentioned above, the original vocabulary also includes multiple words, and the target vocabulary includes the result of encrypting each word in the original vocabulary. In other words, the input data includes the encryption result of each word in the input text. Optionally, the order of the embedding vectors corresponding to each word in the original vocabulary and the target vocabulary is different.

[0146] Specifically, when encrypting the words in the vocabulary as described above, the client and server need to change the method of obtaining the embedding matrix during the process of using the model for reasoning. The embedding matrix is ​​the matrix of multiple word embedding vectors corresponding to the input model. The embedding matrix will also be referred to as the first intermediate data in the following text. As shown in Figure 5, during the model inference process, the user's plaintext input needs to be encrypted to obtain the user's ciphertext input, and the model's ciphertext output needs to be decrypted to obtain the model's plaintext output.

[0147] In related technologies, the client may directly send the user's input text to the client. In the method provided in this specification, the text form of words in the vocabulary can be encrypted. Correspondingly, during the inference process, the client needs to perform word segmentation on the user's input text and divide it into multiple words. And encrypt each word. The encryption method is the same as the method of encrypting the vocabulary, for example, both are encrypted through a specific hash operation. Through the above processing, the client obtains the input data and sends the input data to the server.

[0148] Step 603, according to the target vocabulary, obtain the first intermediate result corresponding to the input data; the first intermediate result includes several embedding vectors.

[0149] Specifically, the server compares the input data with words / encrypted words in the target vocabulary to obtain the embedding matrix (i.e., the first intermediate result) corresponding to each word in the user-input text. In subsequent steps, this embedding matrix is ​​used to input the target language model for inference, resulting in the final inference result.

[0150] Step 605: Based on the target weight matrix, the first intermediate result is processed to obtain the second intermediate result;

[0151] Step 607: Based on the second intermediate result, the inference result of the target language model is obtained, and the inference result is returned to the client.

[0152] Specifically, the server multiplies the target weight matrix with the first intermediate result to obtain the second intermediate result. The second intermediate result is then used to continue the inference for subsequent layers.

[0153] In an optional embodiment, the original weight matrix includes a first matrix and a second matrix. The original weight matrix can be obfuscated using a permutation matrix as described above. Specifically, the target weight matrix includes a third matrix and a fourth matrix. The third matrix is ​​obtained by multiplying the inverse of the rotation matrix, the first matrix, and the permutation matrix; the fourth matrix is ​​obtained by multiplying the inverse of the permutation matrix with the second matrix; the permutation matrix is ​​used to adjust the order of rows or columns in the matrix.

[0154] The method of obfuscating using a permutation matrix is ​​detailed above and will not be repeated here.

[0155] Accordingly, step 605 can specifically include processing the first intermediate result according to the first matrix to obtain a third intermediate result; and processing the third intermediate result using the fourth matrix to obtain the second intermediate result. Instruction manual, pages 15 / 21, 18 CN 121119135 A

[0156] The third intermediate result is obtained by multiplying the first matrix and the first intermediate result, and the second intermediate result is obtained by multiplying the fourth matrix and the third intermediate result.

[0157] In an optional embodiment, when a normalization layer exists, the normalization layer can also be obfuscated, and the original weight matrix can be obfuscated using the original weight vector of the normalization layer. Specifically, the original weight matrix is ​​the weight matrix of the feature transformation layer, and the feature transformation layer is further preceded by a normalization layer. The target language model also includes the target weight vector of the normalization layer; the target weight vector is a vector with the same dimension as the original weight vector of the normalization layer of the original language model; the first value is obtained by multiplying the inverse of the target weight vector, the inverse of the rotation matrix, and the first matrix.

[0158] The method of obfuscating the normalization layer and obfuscating the original weight matrix using the original weight vector of the normalization layer is detailed in the foregoing description and will not be repeated here.

[0159] Before the feature transformation layer exists and before executing step 605, the first intermediate needs to be processed according to the target weight vector to obtain a new first intermediate result. In step 605, the inference is completed using the new first intermediate result.

[0160] In an optional embodiment, in the case of residual connections, the fourth matrix can also be obfuscated using a rotation matrix. Specifically, the original weight matrix includes a first matrix and a second matrix, and the target weight matrix includes a third matrix and a fourth matrix; the third matrix is ​​obtained by multiplying the inverse of the rotation matrix by the first matrix, and the fourth matrix is ​​obtained by multiplying the second matrix by the rotation matrix. The specific processing method is detailed above and will not be repeated here.

[0161] Accordingly, step 607 specifically includes: using the sum of the first intermediate result and the second intermediate result as the input of the next layer corresponding to the target weight matrix for inference, to obtain the inference result of the target language model.

[0162] In this case, each original weight matrix of the original language model can be obfuscated as described above, so that all intermediate results cannot be obtained.

[0163] In addition, the inference result obtained by the server can specifically be: for the vector output by the language model, the words obtained by restoring from the target vocabulary / encrypting the words.

[0164] This specification also provides a system for inference based on a language model, the system involving a server and a client, the server deploying a target language model; the target language model includes a target vocabulary and a target weight matrix, the target vocabulary is obtained by multiplying each embedded vector in the original vocabulary of the original language model with a preset rotation matrix, the target vocabulary stores the correspondence between the embedded vector and the encryption result of each word; the target weight matrix is ​​obtained by multiplying the inverse matrix of the rotation matrix with the original weight matrix of the original language model, the rotation matrix is ​​used to rotate the vector in the corresponding vector space.

[0165] The steps performed by each device in this system include:

[0166] The client sends input data corresponding to the user's input text to the server; the input data includes the encryption results of each word in the input text;

[0167] The server obtains a first intermediate result corresponding to the input data according to the target vocabulary; the first intermediate result includes the embedding vectors corresponding to each word in the input text; the first intermediate result is processed according to the target weight matrix to obtain a second intermediate result; the inference result of the target language model is obtained according to the second intermediate result, and the inference result is returned to the client.

[0168] For the specific implementation of the above steps, please refer to the above description, which will not be repeated here. Specification 16 / 21 pages 19 CN 121119135 A

[0169] In addition, when the input data is the encryption results of each word in the input text, the client can also perform word segmentation processing on the user's input text, and perform hash operation on each word after word segmentation to obtain input data. After receiving the inference result, the client also needs to restore the inference result to text. In other words, the client also decrypts the inference result to obtain the output text corresponding to the inference result. For example, when using a hash method for encryption, the client can also obtain the output text corresponding to the inference result based on the parameters of the hash operation.

[0170] The obfuscation model process of this specification will be described next with reference to FIG7. FIG7 shows obfuscation oneThe process of processing the weight data of a language model is shown in the figure. The process of processing six layers is shown in the figure: the normalization layer Norm in the figure, the MHA in the first row, the gated MLP in the second row, and the output layer in the third row. The process of processing the vector table is also shown in the figure.

[0171] For the vector table, the original embedding matrix it contains is We. Its dimension can be 120,000 × 4096.

[0172] For the normalization layer, it can be RSMnorm, and the original weight vector it contains is Vn. The weight vector can contain 4096 elements, corresponding to the dimension of the embedding matrix. It should be noted that although the original weight vectors corresponding to the three normalization layers are all referred to as Vn, different Vn can be used in the actual model.

[0173] The MHA following the first row of normalization layers contains four original weight matrices: Q matrix Wq, K matrix Wk, V matrix Wv (Q, K, and V matrices are the first matrix mentioned earlier), and output projection matrix WO (corresponding to the second matrix mentioned earlier). Detailed explanations of these matrices are provided above and will not be repeated here. RoPE (Rotary Position Embedding) in the figure is a technique for injecting positional information using a self-attention mechanism; its specific implementation can be found in related technologies. Softmax is an activation function, identical to RoPE, and is a structure inherent in the MHA.

[0174] The MLP following the second row of normalization layers contains three original weight matrices: the gated path weight matrix Wgate (corresponding to the first matrix mentioned earlier), the upward projection path weight matrix Wup (corresponding to the first matrix mentioned earlier), and the downward projection path weight matrix Wdown (corresponding to the second matrix mentioned earlier). Swish is a common activation function in gated MLPs.

[0175] The normalization layer in the third row is followed by the output layer, which contains the original weight matrix Whead.

[0176] The yellow squares in the figure represent the original weight vectors, weight matrices, or embedding matrices corresponding to the original vector tables of the vocabulary. The specific meanings are as described above. The meaning within the dashed box is that all matrices or vectors contained within the dashed box are merged into one matrix through multiplication. The order of the elements in the dashed box is the order of multiplication.

[0177] The processing performed on each yellow square will be explained next.

[0178] For the embedding matrix We, it is multiplied by a random rotation matrix Re to obtain the embedded matrix WeRe included in the processed target vocabulary. Among them, the purple squares and purple shaded squares in the figure represent the rotation matrix Re and the inverse matrix Re-1 of the rotation matrix, respectively. As shown in the figure, the dimension of the rotation matrix is ​​4096×4096.

[0179] For the normalization layer, a random scale vector V (also called the target weight vector) is used to replace the original weight vector Vn. In the figure, the blue squares and blue reference squares represent the target weight vector V and its inverse matrix V-1, respectively.

[0180] For MHA, the Q matrix and K matrix are processed in the same way, and the permutation matrices multiplied by the two matrices are also the same. For example, for the Q matrix, the final weight fusion matrix (with the same meaning as the third matrix above) is the corresponding matrix, and the final fusion weight matrix corresponding to the K matrix is ​​

[0181] . Here, V-1 is used to eliminate the influence of V from the previous normalization layer, and Pqk is used to eliminate the influence of Re in the embedding matrix. Pqk is the permutation matrix. The subscript qk indicates that the permutation matrices corresponding to the Q matrix and the K matrix are the same. The other green squares and green shaded squares in the figure represent the permutation matrix and its inverse matrix, respectively. Instruction manual 17 / 21 pages 20 CN 121119135 A

[0182] For the V matrix, the processing method is similar to that of the Q matrix and K matrix, except that the permutation matrix used is different. The fused matrix corresponding to the V matrix (with the same meaning as the third matrix above) is For WO, as mentioned above, the inverse matrix of the permutation matrix needs to be used to eliminate the influence of the permutation matrix on the V matrix. In addition, since there is a residual connection, it needs to be processed using the rotation matrix Re. The final fused matrix (with the same meaning as the fourth matrix above) is

[0183] The MLP processing method corresponding to the second row is similar to the MHA processing method, except that the target weight vector V is different (not shown in the figure), and the permutation matrix is ​​different. In addition, for the gated path and the up-expansion path, since the data of the two paths need to be fused at the end, the two paths use the same permutation matrix Pf. Correspondingly, the down-projection path uses the inverse matrix of the permutation matrix Pf

[0184] For the output layer, unlike the aforementioned layers, since there are no other layers after this layer, there is no need to use the rotation matrix Re for obfuscation. Since there are no other layers after this layer, the influence of the permutation matrix cannot be eliminated, and therefore, the permutation matrix is ​​not used to obfuscate it. Therefore, the final target weight matrix obtained by this layer is

[0185] . In other words, the processing of the original weight matrix of the output layer can include: obtaining the original weight vector Vn of the normalized layer adjacent to the output layer, and generating a target weight vector V with the same dimension as the original weight vector Vn; the target language model includes the target weight vector; multiplying the inverse matrix V-1 of the target weight vector V with the inverse matrix of the rotation matrix, and multiplying the multiplication result with the original weight vector Vn to obtain a third intermediate result; multiplying the third intermediate result with the original weight matrix Whead to obtain the target weight matrix

[0186] .Furthermore, in the multiple layers shown in the figure, due to the presence of residual connections, the influence of the rotation matrix used in the previous layer will be passed to the next layer. Therefore, the rotation matrix Re used in different layers is the same, so it will not affect the final inference result. The random scale vector (also known as the target weight vector) V used in each layer can be different.

[0187] It should also be noted that for different users using the model, different rotation matrices, etc., can be used to generate different target language models of the same language model to protect the privacy and security of each user and prevent a participant from using the model parameters used by itself to reverse-engineer the original data corresponding to the intermediate results of other users.

[0188] Through the above method, the parameters such as the weights of the model can be obfuscated, and the parameters of the model can be changed, so that it is impossible to reverse-engineer the user's input text based on the intermediate data, thus protecting the security of the user's intermediate results and input text. Moreover, in some embodiments, this method can hide the real scale vector and generate a fake scale vector (target weight vector), which can confuse attackers.

[0189] Furthermore, the above method utilizes equivalent weight operations and normalization layers that ensure rotation-invariant operation, ensuring that while the obfuscation method changes model parameters, it does not affect the final inference result or the model's accuracy. This method also employs weight fusion without altering the size or number of weights, preventing the introduction of additional computation during inference and guaranteeing computational efficiency. Even after weight fusion, the original inference method remains unchanged; even if an attacker gains control of the server, they cannot determine how the model is obfuscated.

[0190] This specification also provides an apparatus for reasoning based on a language model, as shown in FIG8, comprising:

[0191] a receiving module 810, configured to receive input data corresponding to the user's input text from a client; the input data includes the encryption results of each word in the input text;

[0192] a first acquisition module 820, configured to acquire a first intermediate result corresponding to the input data according to the target vocabulary; the first intermediate result includes the embedding vectors corresponding to each word in the input text;

[0193] a second acquisition module 830, configured to process the first intermediate result according to the target weight matrix to obtain a second intermediate result;

[0194] a third acquisition module 840, configured to acquire the reasoning result of the target language model according to the second intermediate result and return the reasoning result to the client.

[0195] For the specific implementation of the above apparatus, please refer to the description of the method, which will not be repeated here.

[0196] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (e.g.,Improvements can be made to the circuit structure (such as diodes, transistors, and switches) or to the software (such as improvements to the methodology). However, with technological advancements, many improvements to the methodology can now be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that an improvement to the methodology cannot be implemented using a hardware entity module. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program a digital system themselves to "integrate" it onto a PLD, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0197] The controller can be implemented in any suitable manner; for example, the controller can take the form of a microprocessor or a processor.The controller can take the form of a computer-readable medium, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers that store computer-readable program code (e.g., software or firmware) executable by the (micro)processor. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0198] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, as shown on pages 19 / 21 of the specification, CN 121119135 A, or by products with certain functions. A typical implementation device is a server system. Of course, this specification does not exclude the possibility that, with the future development of computer technology, the computer implementing the functions of the above embodiments can be, for example, a personal computer, a laptop computer, an in-vehicle human-computer interaction device, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0199] Although one or more embodiments of this specification provide the method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-inventive means. The order of steps listed in the embodiments is merely one of many possible execution orders and does not represent a unique execution order. When executed in actual devices or terminal products, the methods shown in the embodiments or drawings can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment, or even a distributed data processing environment). The terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements provided for such a process, method, product, or apparatus.Elements inherent in a method, product, or device. Without further limitations, it is not excluded that other identical or equivalent elements may exist in a process, method, product, or device that includes the aforementioned elements. For example, the use of terms such as "first," "second," etc., to indicate names does not imply any specific order.

[0200] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, when implementing one or more of this specification, the functions of each module can be implemented in one or more software and / or hardware, or the module implementing the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative. For example, the division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0201] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams.

[0202] These computer program instructions may also be stored in a computer-readable storage medium capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowchart illustrations and / or one or more block diagrams.

[0203] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0204] In a typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory. Specification 20 / 21 pages 23 CN 121119135 A

[0205] Memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0206] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information may be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage, graphene storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media do not include transient media, such as modulated data signals and carrier waves.

[0207] Those skilled in the art will understand that one or more embodiments of this specification may be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0208] One or more embodiments of this specification may be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may reside in local and remote computer storage media, including storage devices.

[0209] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the method embodiments.The description is limited to a portion. In the description of this specification, the reference to the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., means that the specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0210] The above description is only an embodiment of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. For those skilled in the art, one or more embodiments of this specification can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this specification should be included within the scope of the claims. Instruction Manual 21 / 21 Page 24 CN 121119135 A Figure 1 Figure 2 Instruction Manual Appendix 1 / 4 Page 25 CN 121119135 A Figure 3 Figure 4 Instruction Manual Appendix 2 / 4 Page 26 CN 121119135 A Figure 5 Figure 6 Instruction Manual Appendix 3 / 4 Page 27 CN 121119135 A Figure 7 Figure 8 Instruction Manual Appendix 4 / 4 Page 28 CN 121119135 A Abstract Abdominal ultrasound examination method, system and device computer equipment The invention provides a method for reasoning based on a language model, the method is executed by a server, and the server is deployed with a target language model; the target language model comprises a target vocabulary and a target weight matrix, the target vocabulary is obtained by multiplying each embedded vector in anoriginal vocabulary of the original language model by a preset rotation matrix, and a corresponding relationship between the embedded vectors and the input data is stored in the target vocabulary; the target weight matrix is obtained by multiplying the inverse matrix of the rotation matrix by the original weight matrix of the original language model. In the reasoning process, receiving input data corresponding to an input text from a client, and then obtaining a first intermediate result corresponding to the input data; the first intermediate result comprises a plurality of embedded vectors; processing the first intermediate result according to the target weight matrix to obtain a second intermediate result; and obtaining a reasoning result of the target language model according to the second intermediate result, and returning the reasoning result to the client.

Claims

1. A method for reasoning based on a language model, the method being executed by a server, the server having a target language model deployed thereon; the target language model including a target weight matrix and a target vocabulary, the target vocabulary being obtained by multiplying each embedding vector in the original vocabulary of the original language model with a preset rotation matrix, the target vocabulary storing the correspondence between the encryption result of each word and its embedding vector; The target weight matrix is ​​obtained by multiplying the inverse of the rotation matrix by the original weight matrix of the original language model, wherein the rotation matrix is ​​used to rotate the vector in the corresponding vector space; the method includes: Receive input data from the client corresponding to the user's input text; the input data includes the encryption results of each word in the input text; Based on the target vocabulary, a first intermediate result corresponding to the input data is obtained; the first intermediate result includes the embedding vectors corresponding to each word of the input text. Based on the target weight matrix, the first intermediate result is processed to obtain the second intermediate result; The inference result of the target language model is obtained based on the second intermediate result, and the inference result is returned to the client.

2. The method according to claim 1, wherein the original weight matrix comprises a first matrix and a second matrix, and the target weight matrix comprises a third matrix and a fourth matrix; the third matrix is ​​obtained by multiplying the inverse of the rotation matrix, the first matrix, and the permutation matrix; the fourth matrix is ​​obtained by multiplying the inverse of the permutation matrix and the second matrix; the permutation matrix is ​​used to adjust the order of rows or columns in the matrix; The step of processing the first intermediate result according to the target weight matrix to obtain the second intermediate result includes: The first intermediate result is processed based on the first matrix to obtain the third intermediate result; The second intermediate result is obtained by processing the third intermediate result using the fourth matrix.

3. The method according to claim 2, wherein the original weight matrix is ​​the weight matrix of the feature transformation layer, the feature transformation layer is further preceded by a normalization layer, and the target language model further includes a target weight vector of the normalization layer; the target weight vector is a vector with the same dimension as the original weight vector of the normalization layer of the original language model; the first value is obtained by multiplying the inverse of the target weight vector, the inverse of the rotation matrix, and the first matrix. Before processing the first intermediate result according to the target weight matrix to obtain the second intermediate result, the method further includes: The first intermediate result is processed according to the target weight vector to obtain a new first intermediate result.

4. The method according to claim 3, the method according to claim 1, wherein the normalization layer is a root mean square normalization (RMS) normalization layer, and the rotation matrix is ​​used to rotate the vector around the origin in the corresponding vector space.

5. The method according to claim 1, wherein the original weight matrix comprises a first matrix and a second matrix, and the target weight matrix comprises a third matrix and a fourth matrix; the third matrix is ​​obtained by multiplying the inverse of the rotation matrix by the first matrix, and the fourth matrix is ​​obtained by multiplying the second matrix by the rotation matrix; The step of obtaining the inference result of the target language model based on the second intermediate result includes: The sum of the first intermediate result and the second intermediate result is used as the input to the next layer corresponding to the target weight matrix for inference, thereby obtaining the inference result of the target language model.

6. The method according to claim 1, wherein the rotation matrix is ​​generated in the following manner: Obtain a random matrix, perform QR decomposition on the random matrix to obtain an orthogonal rotation matrix Q; By changing the sign of the orthogonal rotation matrix Q to make its determinant value 1, the rotation matrix is ​​obtained. Alternatively, a Hadamard matrix can be generated, and the Hadamard matrix can be normalized to obtain the rotation matrix.

7. A system for reasoning based on a language model, the system involving a server and a client, wherein the server deploys a target language model; the target language model includes a target vocabulary and a target weight matrix, the target vocabulary is obtained by multiplying each embedding vector in the original vocabulary of the original language model with a preset rotation matrix, and the target vocabulary stores the correspondence between the embedding vectors and the encryption results of each word; The target weight matrix is ​​obtained by multiplying the inverse of the rotation matrix with the original weight matrix of the original language model. The rotation matrix is ​​used to rotate the vector in the corresponding vector space. The client sends input data corresponding to the user's input text to the server; the input data includes the encryption results of each word in the input text; The server obtains a first intermediate result corresponding to the input data based on the target vocabulary; the first intermediate result includes the embedding vectors corresponding to each word of the input text; the server processes the first intermediate result according to the target weight matrix to obtain a second intermediate result; the server obtains the inference result of the target language model based on the second intermediate result and returns the inference result to the client.

8. In the system according to claim 7, the order of the embedding vectors corresponding to each word in the original vocabulary and the target vocabulary is different; The client also performs word segmentation on the user's input text and performs hash operation on each word after word segmentation to obtain the input data; The client also obtains the output text corresponding to the reasoning result based on the parameters of the hash operation.

9. In the system according to claim 7, different users have different target word lists.

10. A computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-9.

Images