Large model protection method for Transform architecture
By preprocessing the large model, using spatial transformation to protect the model weights, and combining the large model service inference engine and client to process the inference process data in segments, the problem of large models being illegally copied and used in the server environment is solved, ensuring the rights and interests of the model producers, and providing higher-intensity security protection when deploying in a third-party computing environment.
Patent Information
- Application Number
- CN202411986283.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-31
- Publication Date
- 2025-04-25
AI Technical Summary
Existing large models are easily illegally copied and used in server environments, resulting in the inability to protect the intellectual property rights of model producers. At the same time, in the external computing environment, large models deployment may be accessed by unauthorized clients, infringing on the rights and interests of the service provider.
A large-model protection method for Transformer architecture is adopted, and the model weights are spatially transformed through large-model preprocessing tools. Combined with the large-model service inference engine and large-model inference client, the inference process data is processed in segments to ensure that the rights and interests of the model producers are not violated.
It effectively protects the rights and interests of large model producers and prevents unauthorized client-side access and replication. When deploying large models in third-party computing environments, they no longer worry about the model theft of structural parameters manufactured, and protection measures have little impact on model inference performance.
Smart Images
Figure CN119885117A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of artificial intelligence models, and specifically relates to a large model protection method for a Transformer architecture. Background Art
[0002] The existing large models are provided to users by providing model structure and model parameters. The number of parameters of language or multimodal large models, commonly used language large models have more than 7 billion parameters, and the model size is about 16G or more, so the encrypted model loading process has a greater impact. In addition, if the above large models are used for full-precision reasoning, the video memory needs to be more than 26 GB, and the computing power is at least 10TFLOPs, which requires high computing resources. Therefore, large models are deployed on the service side and provide access interfaces for end-side use. This model requires large models to be deployed on the computing power server side, which is easy to be illegally copied in the server environment. Once the large model is released, the model can be easily copied and used, thereby failing to protect the intellectual property rights of the model producer. At the same time, if the large model service provider uses an external computing power environment, the large model deployed as a service may be accessed by unauthorized clients, and the large model service provider is illegally accessed and infringed, infringing the rights and interests of the service provider.
[0003] Commonly used methods for model protection: 1. Model encryption solution: First use AES and other encryption algorithms to fully encrypt the model file, and then store the encrypted model in the disk space. After starting the AI reasoning process, the encrypted model will be transferred to the running memory for decryption. The decryption delay of the model in the memory is dozens of times the delay of importing the plaintext model into the framework, which affects the reasoning performance and user experience on the end side; in addition, the model is in plaintext in the memory during operation, and the model can be exported in the running memory through the memory export method; 2. Privacy-preserving computing solutions: A secret computing solution that uses technologies such as secure multi-party computing and homomorphic encryption. This type of solution allows the model to be inferred in ciphertext form. However, the disadvantage of privacy computing is that the computing or communication overhead is very high, which will cause the model inference delay to be more than 100 times that of ordinary models, and at the same time require more secret computing power.
[0004] 3. Model obfuscation scheme: Use pseudo-branch algorithm and flattening algorithm to insert pseudo-branches into the original calculation graph to obfuscate the real structure of the original model; at the same time, use weight obfuscation algorithm to obfuscate the operator weights of the original model. The obfuscation scheme can prevent the structure and parameters of the model from being copied, but it cannot prevent the model from being copied and used. Summary of the invention
[0005] The present invention provides an intellectual property protection method for a large model of a Transformer architecture. The large model protection mechanism includes three parts: a large model preprocessing tool, a large model service reasoning engine, and a large model reasoning client. The model preprocessing tool performs spatial transformation processing on the model weights to protect the weights when the model is released. The secure reasoning engine loads the protected model weights into the reasoning engine; the client input data is transformed by the client and then transmitted to the secure reasoning engine for reasoning calculations. The correct reasoning result can only be obtained through the client, thereby effectively protecting the rights and interests of the model producer. Through the above three technologies: model preprocessing tool, large model service reasoning engine and large model reasoning client, the data processing of the reasoning process is segmented. Under the condition that the protection measures do not affect the reasoning performance, the rights and interests of the model producer can be ensured not to be infringed.
[0006] The technical solution of the present invention is as follows: A large model protection method for Transformer architecture, comprising the following steps: Step 1, preprocessing stage: preprocess the large model to be protected to generate a distributable large model and DNY client (including DRM data parameter file); Step 2, inference use phase: the server starts the inference engine and loads the protected large model (including protection operators and parameters). The distributed client loads the protection operators and parameters defined in the DRM data parameter file and connects to the server.
[0007] Based on steps one and two, the pre-processed DRM protected parameters and operators are separated, and the operator calculation protection strategy is implemented on the server and client respectively to protect the weight parameters of the server-side large model.
[0008] The present invention realizes protection by spatial transformation of model parameters in the process of matrix calculation using large model parameters, and the whole protection process does not introduce new matrix calculation steps compared with the unprotected process. Therefore, the protection measures have very little impact on the performance of the model reasoning process because the algorithm complexity introduced in the whole process is minimized.
[0009] Preferably, the preprocessing stage analyzes the structure of the pre-trained large model, the network weights of each layer of the model, the matrix calculation operators used, performs protection transformation on the weights and operator parameters, and saves the transformation matrix and equivalent operator parameters in the DRM data file required by the client. The inference engine uses the transformed large model, so that user data vectorization must use the parameters and operators in the DRM data file to perform matrix calculations on the model client before executing equivalent model reasoning on the server side.
[0010] Preferably, in the inference use stage: first, the client side uses the DRM data file parameters and operators to perform vectorization and protection operator transformation on the user input required by the model, and then sends the transformed vector to the server. When the server performs inference, the large model after protection processing is loaded through the inference engine, the new weight parameters of the large model and the retention element matrix of the input vector are calculated, and the nonlinear technologies such as Softmax are operated using spatially equivalent operators, so as to obtain an overall inference result that is equivalent to the result of the overall spatial change of the original inference, and then the service-side result vector is returned to the client. The client uses the protection operator and parameters of the DRM data structure to perform reverse transformation to obtain an equivalent inference result of the original model space.
[0011] Preferably, the large model service provider model is deployed in a third-party computing environment.
[0012] Preferably, when protecting a large model, the client DRM file parameters generated by the preprocessing process are bound to the hardware unique identifiers of the client and the server.
[0013] Through the above method, the large model, inference engine operator parameters, and client DRM authorization data operator parameters must be processed. The three must come from the issuer at the same time to obtain the correct inference content. Moreover, if the large model parameters obtained by the server are converted, the client without the authorized DRM file cannot use it, preventing unauthorized customers from misappropriating it.
[0014] With this method, large model service providers can deploy the model in a third-party computing environment without worrying about model theft caused by duplication of large model structure parameters.
[0015] When protecting a large model, this method binds the parameters in the client DRM file and the unique hardware identifiers of the client and server to provide higher security protection.
[0016] Preferably, the step 1 includes using a large model protection processing tool on the client to perform the following processing, specifically the following steps: S1: Get the original pre-trained large model that has been trained; S2: Generate a normalized random matrix S for embedding layer conversion, which requires the model to have the same dimension as the original embedding layer; S3: Generate the defined protection matrix D through S, the vectorized transformation operator Fi from the user input to the embedding space of the large model, and the transformation operator Fo from the user receiving the output vector of the large model to the user-readable output; The S in S2 is randomly generated and may not be a full-rank matrix or may not satisfy the reversible transformation matrix B, that is, B cannot be found so that S*T*D=T. In this case, it is necessary to re-execute the S2 step to generate a new S. S4: Since the original embedding vector I is protected and then processed by the large model, the parameters of each layer after the large model processing; the parameters are also converted and saved in the new model weights; S5: Replace the Oper operator of the original large model with the protection operator SOper using nonlinear operators; S6: Convert the A parameters, use the DRM distribution parameters to protect the Fi and Fo operators, and distribute the operators and parameters, as well as the client, to the user side; S7: Generate a protected large model and an inference engine including the protection operator program SOper and distribute them to the service provider.
[0017] Preferably, the step 2 includes initiation by the client, inference to the server, and return to the content client, specifically the following steps: S0: The server starts the inference engine and loads the security protection operator SOper. The inference engine loads the large model that has been protected. S1: Client-side reasoning client loads DRM data structure and accepts user input I; S2: The inference client performs raw vectorization on the input; S3: The inference client uses Fi to perform security protection vector transformation; S4: The inference engine receives the client’s input vector Is; S5: The inference engine loads the corresponding security weights of each layer of the large model to calculate the equivalent space; S6: The new SOper operator is used in the inference engine to replace the original marked operator Oper for equivalent calculation; S7: The inference engine performs calculations to obtain the inference result Vs and returns it to the client; S8: The inference client performs a spatial inverse transformation operator to obtain a vector Vv=Fo(Vs); S9: Vv performs raw vector conversion and outputs the content O required by the user.
[0018] Compared with the prior art, the beneficial effects of the present invention are: The present invention can solve the problem of overall encryption or publishing efficiency of large models by preprocessing the large model, separate the preprocessed DRM-protected parameters and operators, and implement parameter protection strategies on the server and client to protect the weight parameters of the large model on the server from being stolen. The model service mode separates the client and the service party, and protects the model service party and the model generator, thereby ensuring that the intellectual property rights of the large model producer are protected; Only the client holding the DRM file is allowed to use it correctly. Compared with the existing common protection methods, the present invention has the characteristic of limited use for users, ensuring that the owner of the big model will not be stolen by unauthorized parties and protecting the rights and interests of the big model service provider; The present invention realizes protection by using spatial transformation of matrix multiplication parameters for large model parameters, and the entire protection process does not introduce new matrix calculation steps compared with unprotected process. Therefore, the protection measures have very little impact on the performance of the model reasoning process because the new complexity introduced in the entire process is minimized. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 This is a schematic diagram of the pre-processing stage of the protection method in this embodiment; Figure 2 This is a flow chart of the model reasoning operation phase in the protection method of this embodiment. DETAILED DESCRIPTION
[0020] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.
[0021] Embodiment: The following Table 1 is a description of the relevant terms in this embodiment; Table 1 noun Chinese explain DRM Digital Rights Management DRM (Digital Rights Management) is a technical means to protect digital content (such as music, video, e-books, etc.) from being illegally copied and distributed. In this article, DRM refers to the data file that stores digital copyright protection information, which is used to store security parameters and operator configuration information related to the protection of large models. Oper Operator An algorithm used for a specific operation in a model, often referring to the machine code implementation of an algorithmic function of the model like Figure 1 As shown, in this embodiment, a large model protection method for the Transformer architecture specifically includes the following steps: P1: preprocessing stage; Use the large model protection processing tool on the client to perform the following processing; S1: Get the original pre-trained large model that has been trained; S2: Generate a normalized random matrix S for embedding layer conversion, which requires the model to have the same dimension as the original embedding layer; S3: Generate the defined protection matrix D through S, the vectorized transformation operator Fi from the user input to the embedding space of the large model, and the transformation operator Fo from the user receiving the output vector of the large model to the user-readable output; The S in S2 is randomly generated and may not be a full-rank matrix or may not satisfy the reversible transformation matrix B, that is, B cannot be found so that S*T*D=T. In this case, it is necessary to re-execute the S2 step to generate a new S. S4: Since the original embedding vector I is protected and processed by the large model as X=S*I, the parameters of each layer after the large model processing, such as Wqs, Wvs, and Wks, need to maintain spatial equivalence, that is, Q=Wq*X≡Wq*S*I=Wqs*I, that is, Wqs=Wq*S; Similarly, Wvs and Wks are also converted and saved in the new model weights.
[0022] S5: For the use of nonlinear operators to replace the Oper operator of the original large model with the protection operator SOper, such as A=softmax(Q, K), use the equivalent marking operator As= S*A =S*softmax(Q, K)= S*SSoftmax(Qs, Ks), marking the SSoftmax operator as the runtime model space conversion equivalent operator; S6: Convert the A parameters, use the DRM distribution parameters to protect the Fi and Fo operators, and distribute the operators and parameters, as well as the client, to the user side; S7: Generate a protected large model and a reasoning engine including the protection operator program SOper and distribute them to the service provider; P2: Model inference operation phase, such as Figure 2 As shown; The client initiates, the server infers, and the content is returned to the client.
[0023] S0: The server starts the inference engine and loads the security protection operator SOper. The inference engine loads the large model that has been protected. S1: Client-side reasoning client loads DRM data structure and accepts user input I; S2: The inference client performs raw vectorization on the input; S3: The inference client uses Fi to perform security protection vector transformation; S4: The inference engine receives the client’s input vector Is; S5: The inference engine loads the corresponding security weights of each layer of the large model, such as Qs Ks Vs, to calculate the equivalent space; S6: The new SOper operator is used in the inference engine to replace the original marked operator Oper for equivalent calculation; S7: The inference engine performs calculations to obtain the inference result Vs and returns it to the client; S8: The inference client performs a spatial inverse transformation operator to obtain a vector Vv=Fo(Vs); S9: Vv performs raw vector conversion and outputs the content O required by the user.
[0024] It should be noted that the above content only illustrates the technical idea of the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.
Claims
1. A large model protection method for Transformer architecture, characterized in that: The following steps are involved: Step 1, preprocessing stage: preprocess the large model to be protected to generate a distributable large model and client and DRM data parameter files; Step 2, inference use phase: the server starts the inference engine, loads the protected large model and protection operators and parameters, and the distributed client loads the protection operators and parameters defined in the DRM data parameter file and connects to the server; Based on steps one and two, the pre-processed DRM protected parameters and operators are separated, and the operator calculation protection strategy is implemented on the server and client respectively to protect the weight parameters of the server-side large model.
2. The large model protection method for Transformer architecture according to claim 1 is characterized in that: The preprocessing stage analyzes the structure of the pre-trained large model, the network weights of each layer of the model, the matrix calculation operators used, performs protection transformation on the weights and operator parameters, and saves the transformation matrix and equivalent operator parameters in the DRM data file required by the client. The inference engine uses the transformed large model, so that user data vectorization must use the parameters and operators in the DRM data file to perform matrix calculations on the model client before executing equivalent model reasoning on the server side.
3. The large model protection method for Transformer architecture according to claim 1 is characterized in that: The reasoning use stage: first, the client side uses the DRM data file parameters and operators to perform vectorization and protection operator transformation on the user input required by the model, and then sends the transformed vector to the server. When the server is reasoning, the large model after protection processing is loaded through the reasoning engine, the new weight parameters of the large model and the retention element matrix of the input vector are calculated, and the nonlinear technologies such as Softmax are operated using spatially equivalent operators, so as to obtain an overall reasoning result that is equivalent to the result of the overall spatial change of the original reasoning, and then the service-side result vector is returned to the client. The client uses the protection operator and parameters of the DRM data structure to perform reverse transformation to obtain an equivalent reasoning result of the original model space.
4. The large model protection method for Transformer architecture according to claim 2 is characterized in that: When protecting a large model, the client DRM file parameters generated by the preprocessing process are bound to the hardware unique identifiers of the client and server.
5. The large model protection method for Transformer architecture according to claim 1 is characterized in that: The step 1 includes using a large model protection processing tool on the client to perform the following processing, specifically the following steps: S1: Get the original pre-trained large model that has been trained; S2: Generate a normalized random matrix S for embedding layer conversion, which requires the model to have the same dimension as the original embedding layer; S3: Generate the defined protection matrix D through S, the vectorized transformation operator Fi from the user input to the embedding space of the large model, and the transformation operator Fo from the user receiving the output vector of the large model to the user-readable output; The S in S2 is randomly generated and may not be a full-rank matrix or may not satisfy the reversible transformation matrix B, that is, B cannot be found so that S*T*D=T. In this case, it is necessary to re-execute the S2 step to generate a new S. S4: Since the original embedding vector I is protected and then processed by the large model, the parameters of each layer after the large model processing; the parameters are also converted and saved in the new model weights; S5: Replace the Oper operator of the original large model with the protection operator SOper using nonlinear operators; S6: Convert the A parameters, use the DRM distribution parameters to protect the Fi and Fo operators, and distribute the operators and parameters, as well as the client, to the user side; S7: Generate a protected large model and an inference engine including the protection operator program SOper and distribute them to the service provider.
6. The large model protection method for Transformer architecture according to claim 1 is characterized in that: The step 2 includes initiation by the client, inference to the server, and return to the content client, which is specifically the following steps: S0: The server starts the inference engine and loads the security protection operator SOper. The inference engine loads the large model that has been protected. S1: Client-side reasoning client loads DRM data structure and accepts user input I; S2: The inference client performs raw vectorization on the input; S3: The inference client uses Fi to perform security protection vector transformation; S4: The inference engine receives the client’s input vector Is; S5: The inference engine loads the corresponding security weights of each layer of the large model to calculate the equivalent space; S6: The new SOper operator is used in the inference engine to replace the original marked operator Oper for equivalent calculation; S7: The inference engine performs calculations to obtain the inference result Vs and returns it to the client; S8: The inference client performs a spatial inverse transformation operator to obtain a vector Vv=Fo(Vs); S9: Vv performs raw vector conversion and outputs the content O required by the user.