Method and device for verifiable generative pre-training model based on zero-knowledge proof
By decomposing the text generation and inference process of the generative pre-trained model into constraint relationships and verifying these constraint relationships using zero-knowledge proof technology, the problem of computational complexity and inefficiency of large-scale text generation models in the existing technology is solved, and more efficient and safe model verification is achieved.
Patent Information
- Application Number
- CN202510167921.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-14
- Publication Date
- 2025-06-10
AI Technical Summary
The existing zero-knowledge proof solution is complex and inefficient in the calculation process of large-scale text generation generative pre-trained models, resulting in poor reliability and security of the calculation.
The text generation and inference process of the generative pretrained model are decomposed into constraint relationships between input and output, and the zero-knowledge proof technology is used to verify the constraint relationship of each sub-proof task, thereby completing the verification of the inference results of the generative pretrained model.
By reducing the required number of constraints, reducing the computational burden, improving the security and accuracy of model calculations, significantly improving the proof speed and security of generative pre-trained models.
Smart Images
Figure CN120124677A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of neural network security, and in particular, to a method and device for a verifiable generative pre-trained model based on zero-knowledge proof. Background Art
[0002] A Generative Pre-trained Transformer (GPT) is a deep learning model widely used in natural language processing and generation tasks. GPT is pre-trained using large-scale text data and has the ability to understand context and generate coherent text. Its architecture is based on a multi-layer stacked self-attention mechanism and a feed-forward neural network, demonstrating excellent performance from the construction of language models to the processing of complex generative tasks.
[0003] The pre-training process of the GPT model utilizes a large amount of text data and, through unsupervised learning, enables the model to learn the structure and context information of the language. Subsequently, through a fine-tuning step, the model can be adaptively adjusted for specific tasks (such as text generation, machine translation, sentiment analysis, etc.). Such a design makes GPT perform excellently in generating fluent and context-related text.
[0004] In recent years, with the rapid development of cloud computing, Machine Learning as a Service (MLaaS) has gradually become a convenient way for enterprises and developers to access advanced AI technologies. Cloud service providers such as AWS, Google Cloud, and Azure have successively launched platforms based on models such as GPT. Users can obtain powerful text processing capabilities through simple API calls without having to deeply understand the internal structure and algorithm details of the models. Through MLaaS, enterprises can apply the GPT model to various scenarios such as chatbots, content generation, virtual assistants, and document analysis to improve business efficiency and user experience. For example, in the field of customer service, GPT can generate responses in real time to improve the response speed; in content creation, GPT can generate high-quality articles or creative content based on a given topic. In addition, MLaaS also lowers the technical threshold, enabling small and medium-sized enterprises to easily access advanced NLP technologies. This service model not only promotes the popularization of artificial intelligence technologies but also facilitates cross-industry application innovation, enabling various enterprises to enhance their core competitiveness using AI in the fierce market competition.
[0005] Zero-Knowledge Proof (ZKP) is an advanced cryptographic technique that allows the model owner to prove to users that the returned results are indeed the inference results of the model while protecting the model's privacy. By leveraging mathematical puzzles, the GPT model owner can prove the truth of a certain assertion to an external verifier without revealing the specific information of the GPT model itself. This method is particularly applicable to scenarios where user data and model details need to be protected, such as chatbots, content generation, virtual assistants, document analysis, or personalized recommendations. Through ZKP, model owners can ensure that their models are accurate when performing calculations and returning results, while avoiding disclosing any key information, thereby enhancing the credibility and security of the service under the premise of protecting privacy.
[0006] Although there have been many advancements in the development of ZKP for neural networks, the high computational cost makes it difficult to deploy in practice. Especially for existing ZKP schemes for generative pre-training models for large-scale text generation, the calculation process is very complex, requiring multiple interactions between the prover and the verifier to ensure the correctness and security of the calculation. The overall efficiency is low, resulting in poor credibility and security of the calculation. Summary of the Invention
[0007] The object of the present invention is to provide a method and device for a verifiable generative pre-training model based on zero-knowledge proof. By decomposing the text generation and inference processes in the generative pre-training model into the constraint relationship between the input and the output, and then using zero-knowledge proof technology, it realizes the efficient verification of the correctness of the generation results of the generative pre-training model, while protecting the privacy of the model parameters and improving the security of model calculation.
[0008] To achieve the above object of the invention, a method for a verifiable generative pre-training model based on zero-knowledge proof provided by an embodiment includes the following steps:
[0009] Divide the generative pre-training model for text generation into several neural network layers according to the hierarchical structure. Each neural network layer is regarded as a sub-proof task. Calculate the output based on the input of each neural network layer, and store the input and output of each neural network layer.
[0010] Classify the sub-proof tasks according to the type of calculation operation of each neural network layer, generate the constraint relationship of each sub-proof task based on the type of calculation operation, and ensure the accuracy of the calculation of each neural network layer based on the constraint relationship.
[0011] Use the zero - knowledge proof method to verify the constraint relationships of each sub - proof task in sequence, obtain the proof results of each constraint relationship, and reduce the proof results of each constraint relationship to the proof results of the generative pre - training model for text generation, thus completing the verification of the inference results of the generative pre - training model.
[0012] In one embodiment, the generative pre - training model for text generation is divided into several neural network layers according to a hierarchical structure, including: a hierarchical structure based on a linear layer, a GELU activation layer, a Softmax layer, and a Layer Norm layer, and the generative pre - training model is divided into several neural network layers according to the hierarchical structure.
[0013] In one embodiment, the types of the computing operations include: linear layer computation, GELU activation layer computation, Softmax layer computation, and Layer Norm layer computation;
[0014] The linear layer computation is expressed as y L = Ax L + B, where x L represents the input of the linear layer, y L represents the output of the linear layer, A represents the weight matrix of the linear layer, and B represents the bias vector;
[0015] The GELU activation layer computation is expressed as where erf is the Gaussian error function, and through erf(x G )≈sign(x G )·L(min(|x G |, - B)) for approximate calculation, x G represents the input of the GELU activation layer, and y G represents the output of the GELU activation layer;
[0016] The Softmax layer computation is expressed as where exp is the natural exponential function, The is decomposed into p s in the range (-ln2, 0], z s is a non - negative integer, and then through approximate calculation of the natural exponential function, x s represents the input of the Softmax layer, and y s represents the output of the Softmax layer;
[0017] The Layer Norm layer computation is expressed as where, is x iThe average value of is x i The standard deviation of i ranges from [1, N], where N is the number of x, and x i represents the input of the Layer Norm layer, and y i represents the output of the Layer Norm layer.
[0018] In one embodiment, the constraint relationships include the constraint relationships of the linear layer, the GELU activation layer, the Softmax layer, and the Layer Norm layer;
[0019] The constraint relationship of the linear layer is expressed as Ax L +B - y L = 0, where x L represents the input of the linear layer, y L represents the output of the linear layer, A represents the weight matrix of the linear layer, and B represents the bias vector;
[0020] The constraint relationships of the GELU activation layer include: s = sign(x G ), |x G | = abs(x G ), and where x G represents the input of the GELU activation layer, sign(·) represents the digital sign function, abs(·) represents the absolute value function, mmin(·) represents the minimum value function, L(·) represents the approximation function, and y G represents the output of the GELU activation layer;
[0021] The constraint relationships of the Softmax layer include: t s = L(p s ) >> z s and where, x s represents the input of the Softmax layer, p s is in the range (-ln 2, 0], z s is a non - negative integer, L(·) represents the approximation function, and y s represents the output of the Softmax layer;
[0022] The constraint relationships of the Layer Norm layer include: and x i -μ - σy i = 0, where x i represents the input of the Layer Norm layer, and μ is xi The average value, where σ is x i The standard deviation of, and the value of i ranges from [1, N], where N is the number of x, and y i represents the output of the Layer Norm layer.
[0023] In one embodiment, the method of using zero-knowledge proof is used to verify the constraint relationships of each sub-proof task in sequence, including: authenticating all the constraint relationships of each sub-proof task through an information-theoretic message authentication code, so that all the constraint relationships of each sub-proof task satisfy the proof relationship of m = k - ΔX, where the prover's private message X includes all variables in the constraint relationships corresponding to the linear layer, GELU activation layer, Softmax layer, and LayerNorm layer, m is the authentication tag held by the prover corresponding to X, k is the local key held by the verifier corresponding to X, and Δ is the global key held by the verifier and remains fixed.
[0024] In one embodiment, making all the constraint relationships of each sub-proof task satisfy the proof relationship of m = k - ΔX includes: selecting an appropriate proof type for verification according to the type of constraint relationship of each sub-proof task, and the proof types include: the identity proof type of X = 0 and the range proof type of X ≥ 0.
[0025] In one embodiment, for the identity proof type of X = 0, by regarding the variables of the constraint relationship in each sub-proof task as the prover's private message X, based on the proof relationship m = k - ΔX, the prover obtains m and the verifier obtains k, and the prover proves that m is equal to k through the identity proof type of X = 0;
[0026] For the range proof type of X ≥ 0, by regarding the variables of the constraint relationship in each sub-proof task as the prover's private message X, based on the proof relationship m = k - ΔX, the prover obtains m and the verifier obtains k, and the prover constrains the message X' held by the verifier within a specific range through the range proof type of X ≥ 0 to satisfy the proof relationship.
[0027] In one embodiment, reducing the proof result of each constraint relationship to the proof result of the generative pre-training model for text generation includes: representing the zero-knowledge proof results of all constraint relationships in the sub-proof task as B 1 , B 2 ,..., B n , and through the formula B = B 1 &B 2 &...&B n reduce the proof results of the constraint relationships of each sub-proof task to the proof result of the generative pre-training model, where B is the proof result of the generative pre-training model, and & represents the reduction calculation operation.
[0028] To clearly demonstrate the method of a verifiable generative pre-training model based on zero-knowledge proof, a device for a verifiable generative pre-training model based on zero-knowledge proof is provided, including a memory and a processor. The memory is used to store a computer program, and the processor is used to implement the method of the verifiable generative pre-training model based on zero-knowledge proof when executing the computer program.
[0029] Compared with the prior art, the beneficial effects of the present invention at least include:
[0030] The method and device for a verifiable generative pre-training model based on zero-knowledge proof provided by the present invention, compared with the prior art, decompose the text generation and reasoning processes in the generative pre-training model into constraint relationships between inputs and outputs, requiring fewer constraints and reducing the computational burden; use zero-knowledge proof to verify the constraint relationships of each sub-proof task, thereby improving the security and accuracy of model calculation, reducing the proof results of the constraint relationships of the sub-proof tasks to the results of the original generative pre-training model, significantly improving the proof speed and security of the generative pre-training model without revealing model parameters, and being more efficient in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art.
[0032] Figure 1 It is a flowchart of the method for a verifiable generative pre-training model (GPT) based on zero-knowledge proof provided by the present invention;
[0033] Figure 2 It is a schematic diagram of dividing the simplified verifiable GPT model into sub-proof tasks according to the hierarchical structure in this embodiment;
[0034] Figure 3 It is a schematic diagram of the chatbot system structure of the verifiable GPT model provided in this embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0035] To make the objectives, technical solutions, and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the protection scope of the present invention.
[0036] To enhance the credibility and security of artificial intelligence services such as chatbots while protecting user privacy, the embodiment provides a method for a verifiable generative pre-training model based on zero-knowledge proof. This method decomposes the text generation and reasoning processes in the GPT model into constraint relationships between the input and output, and then uses zero-knowledge proof technology to efficiently verify the correctness of the model reasoning results without exposing the model parameters, thereby protecting user privacy and enhancing security.
[0037] The GPT model used in the present invention is stacked by multiple hierarchical structures, including a linear layer (implementing operations such as matrix multiplication, dot product, and residual connection), a GeLU activation layer, a Softmax layer, and a Layer Norm layer, etc. This neural network architecture encompasses a wide range of Transformer models, such as GPT-2 and BERT. Taking the GPT-2 text generation system based on Machine Learning as a Service (MLaaS) as an example, in the traditional framework, the user application sends a text generation request to the cloud server, and the server returns the generation result. However, if the cloud service platform is not trustworthy, the application cannot verify the authenticity and accuracy of the generation result.
[0038] Therefore, by introducing the zero-knowledge proof mechanism, it is ensured that the cloud platform can efficiently provide users with a credible proof of the GPT model reasoning result while protecting the model privacy. In the embodiment, based on the chatbot, a method for a verifiable generative pre-training model based on zero-knowledge proof is provided, as Figure 1 shown, the method for the verifiable generative pre-training model includes the following steps:
[0039] S1. Divide the generative pre-training model for text generation into several neural network layers according to the hierarchical structure. Each neural network layer is regarded as a sub-proof task. Calculate the output based on the input of each neural network layer, and store the input and output of each neural network layer.
[0040] In the embodiment, as Figure 2 shown, the generative pre-training model for text generation is divided into several neural network layers according to the hierarchical structure. A typical GPT model usually includes: a linear layer (including matrix multiplication, matrix dot product, residual connection, etc.), a GeLu activation layer, a Softmax layer, and a Layer Norm layer; for the user's initial input, the outputs of all neural network layers are obtained according to the calculation order in the original GPT model. Since the output of the previous neural network layer is the input of the next neural network layer, it is only necessary to store the user's initial input and the calculation results of each neural network layer.
[0041] S2. Classify the sub-proof tasks according to the type of calculation operation of each neural network layer, and generate the constraint relationship of each sub-proof task based on the type of calculation operation.
[0042] In the embodiments, as Figure 2 shown, the sub-proof tasks are converted into the constraint relationships between the inputs and outputs of each neural network layer according to the calculation operation types of the GPT model, including: classifying the sub-proof tasks through four types of calculation operations, namely, the linear layer calculation operation, the GELU activation layer calculation operation, the Softmax layer calculation operation, and the Layer Norm layer calculation operation, determining the constraint relationships of each sub-proof task based on the types of calculation operations, verifying the constraint relationships of each sub-proof task in sequence, and using the same constraint relationships for the same neural network layer.
[0043] The calculation types of each layer are respectively represented as follows:
[0044] (1) Linear layer calculation: y L = Ax L + B, where x L represents the input of the linear layer, y L represents the output of the linear layer, A represents the weight matrix of the linear layer, and B represents the bias vector;
[0045] (2) GELU activation layer calculation: where erf is the Gaussian error function, and the approximate calculation is performed through erf(x G )≈sign(x G )·L(min(|x G |, -B)), x G represents the input of the GELU activation layer, and y G represents the output of the GELU activation layer;
[0046] (3) Softmax layer calculation: where exp is the natural exponential function, is decomposed into ps is in the range (-ln 2, 0], zs is a non-negative integer, and then the natural exponential function is approximately calculated through x s represents the input of the Softmax layer, and y s represents the output of the Softmax layer;
[0047] (4) LayerNorm layer calculation: where, is the average of x i , is the standard deviation of x i , the value of i is [1, N], N is the number of x, x i represents the input of the Layer Norm layer, and yi Represents the output of the Layer Norm layer.
[0048] Based on the calculation types of the above neural network layers, the following constraints are used to convert the sub-proof tasks into the constraint relationships between the inputs and outputs of the linear layer, generating the constraint relationships for each sub-proof task to ensure the accuracy of the calculations of each neural network layer:
[0049] (1) The constraint relationship of the linear layer is expressed as Ax L + B - y L = 0, where x L represents the input of the linear layer, y L represents the output of the linear layer, A represents the weight matrix of the linear layer, and B represents the bias vector;
[0050] (2) The constraint relationships of the GELU activation layer include the following five: 1) s = sign(x G ), 2) |x G | = abs(x G ), 3) 4) 5) where x G represents the input of the GELU activation layer, sign(·) represents the digital sign function, abs(·) represents the absolute value function, mmin(·) represents the minimum value function, L(·) represents the approximation function, and y G represents the output of the GELU activation layer;
[0051] (3) The constraint relationships of the Softmax layer include: 1) 2) 3) 4) t s = L(p s ) >> z s , 5) where, x s represents the input of the Softmax layer, p s is in the range (-ln 2, 0], z s is a non-negative integer, L(·) represents the approximation function, and y s represents the output of the Softmax layer;
[0052] (4) The constraint relationships of the Layer Norm layer include: 1) 2) 3) x i - μ - σy i = 0, where x i represents the input of the Layer Norm layer, μ is x iThe average value, and σ is x i The standard deviation of, and the value of i is [1, N], where N is the number of x, and y i Represents the output of the Layer Norm layer.
[0053] S3. Use the zero-knowledge proof method to verify the constraint relationships of each sub-proof task in turn, obtain the proof results of each constraint relationship, and reduce the proof results of each constraint relationship to the proof results of the generative pre-training model for text generation to complete the verification of the inference results of the generative pre-training model.
[0054] In the embodiment, the message X private to the prover is authenticated through Information Theoretic Message Authentication Codes (IT-MACs) so that the authentication result satisfies the proof relationship of m = k - ΔX, where the message X private to the prover includes all variables in the corresponding constraint relationships of the linear layer, GELU activation layer, Softmax layer, and Layer Norm layer, m is the authentication tag held by the prover corresponding to X, k is the local key held by the verifier corresponding to X, and Δ is the global key held by the verifier and is fixed.
[0055] The specific process is as follows: Based on the message X private to the prover, the verifier generates and securely stores the global key Δ. Then, the verifier generates the corresponding local key k based on the message X private to the prover and the global key Δ. The prover securely obtains the local key k corresponding to X from the verifier and calculates the authentication tag m so that the authentication tag satisfies the proof relationship of m = k - ΔX; select an appropriate proof type for verification according to the constraint relationship type of each sub-proof task.
[0056] The proof types include: the identity proof type of X = 0 and the range proof type of X ≥ 0; for the identity proof type of X = 0, by regarding the variables in the constraint relationship of each sub-proof task as the message X private to the prover, based on the proof relationship m = k - ΔX, the prover obtains m and the verifier obtains k, and the prover proves that m is equal to k through the identity proof type of X = 0;
[0057] For the range proof type of X ≥ 0, by regarding the variables in the constraint relationship of each sub-proof task as the message X private to the prover, based on the proof relationship m = k - ΔX, the prover obtains m and the verifier obtains k, and the prover constrains the message X' held by the verifier within a specific range through the range proof type of X ≥ 0 to satisfy the proof relationship.
[0058] In the embodiment, taking the constraint relationship of the Softmax layer and t s = L(p s ) >> zs For example:
[0059] (1) Verify the constraint relationship through IT-MACs. The prover's private message X includes variables x x s and x 1 ,..., x n . Based on the proof relationship m = k - ΔX, select the identity proof type with X = 0. The verifier generates and securely stores the global key Δ. Then, the verifier generates the corresponding local key k based on the prover's private message X and the global key Δ. The prover securely obtains the local key k corresponding to X from the verifier, calculates the authentication tag m, and proves that m is equal to k;
[0060] (2) Verify the constraint relationship t s = L(p s ) >> z s through IT-MACs. The prover's private message X includes variables t s , p s and z s . Based on the proof relationship m = k - ΔX, select the range proof type with X ≥ 0. The verifier generates and securely stores the global key Δ. Then, the verifier generates the corresponding local key k based on the prover's private message X and the global key Δ. The prover securely obtains the local key k corresponding to X from the verifier. The prover constrains the message X' held by the verifier within a specific range through the range proof type with X ≥ 0, satisfying the proof relationship.
[0061] The constraint relationships of all sub-proof tasks need to be verified using the zero-knowledge proof method, thereby proving that the output of each neural network layer is indeed completed by the corresponding computational operation.
[0062] Then, reduce the proof results of each constraint relationship to the proof results of the generative pre-training model for text generation. Among them, for all the zero-knowledge proof results B 1 of the constraint relationships in each sub-proof task, B 2 ,..., B n , the proof result B of the original GPT model is B = B 1 & B 2 &... & B n , where & is the reduction calculation operation.
[0063] According to the method of the verifiable generative pre-training model based on zero-knowledge proof proposed by the present invention, combined with the chatbot system based on MLaaS, the structure of the verifiable generative pre-training model based on zero-knowledge proof is as Figure 3 shown, including:
[0064] Server GPT neural network, chatbot application, prover of zero-knowledge proof, and verifier of zero-knowledge proof.
[0065] The server GPT neural network is a machine learning model of a cloud platform provider in the traditional MLaaS (Machine Learning as a Service) framework, responsible for providing various services to client applications.
[0066] The chatbot application interacts with the server GPT neural network by sending a text generation request to the cloud server. The server GPT neural network returns the generated result, which is verified by the chatbot application.
[0067] The zero-knowledge prover converts the input, output, and weights of the GPT neural network (i.e., the constraint relationships mentioned in the embodiments) into a proof, which can prove the validity of the inference result to the verifier without revealing the neural network weights of the cloud platform provider, improving the credibility of the service.
[0068] The zero-knowledge verifier checks the proof and returns the request result and the corresponding proof result to the application.
[0069] To clearly demonstrate the method of the verifiable generative pre-training model based on zero-knowledge proof, the embodiments also provide a device for the verifiable generative pre-training model based on zero-knowledge proof, including a memory and a processor. The memory is used to store a computer program, and the processor is used to implement the method of the verifiable generative pre-training model based on zero-knowledge proof when executing the computer program.
[0070] Based on the above method and device of the verifiable generative pre-training model based on zero-knowledge proof, by decomposing the text generation and inference processes in the generative pre-training model into the constraint relationships between the input and output, the efficiency of zero-knowledge proof is improved. Using zero-knowledge proof technology, the efficient verification of the correctness of the generated results of the GPT model is realized, while protecting the privacy of the model parameters and enhancing the security, making it widely promoted in practical applications.
[0071] The above specific embodiments have detailed the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not used to limit the present invention. Any modifications, supplements, equivalent replacements, etc. made within the scope of the principles of the present invention shall be included in the protection scope of the present invention.
Claims
1. A method for verifiable generative pre-training model based on zero-knowledge proof, characterized in that: The following steps are involved: The generative pre-trained model for text generation is divided into several neural network layers according to the hierarchical structure. Each neural network layer is regarded as a sub-proof task. The output is calculated based on the input of each neural network layer, and the input and output of each neural network layer are stored. Classify the sub-proof tasks according to the type of computational operation of each neural network layer, and generate constraint relationships for each sub-proof task based on the type of computational operation; The zero-knowledge proof method is used to verify the constraint relationship of each sub-proof task in turn, and the proof result of each constraint relationship is obtained. The proof result of each constraint relationship is reduced to the proof result of the generative pre-trained model used for text generation, and the verification of the inference result of the generative pre-trained model is completed.
2. The method of verifiable generative pre-training model according to claim 1, characterized in that: The generative pre-trained model for text generation is divided into several neural network layers according to a hierarchical structure, including: based on a hierarchical structure of a linear layer, a GELU activation layer, a Softmax layer and a LayerNorm layer, the generative pre-trained model is divided into several neural network layers according to a hierarchical structure.
3. The method of verifiable generative pre-training model according to claim 1, characterized in that: The types of calculation operations include: linear layer calculation, GELU activation layer calculation, Softmax layer calculation and Layer Norm layer calculation; The linear layer calculation is represented as y L =Ax L +B, where x L represents the input of the linear layer, y L represents the output of the linear layer, A represents the weight matrix of the linear layer, and B represents the bias vector; The GELU activation layer calculation is expressed as Where erf is the Gaussian error function, through erf(x G )≈sign(x G )·L(min(|x G |, -B)) for approximate calculation, x G represents the input of the GELU activation layer, and yG represents the output of the GELU activation layer; The Softmax layer calculation is expressed as Where exp is the natural exponential function, Will Decompose into p s In the range (-ln 2, 0], z s is a non-negative integer, and then Approximate the natural exponential function, x s Represents the input of the Softmax layer, y s Represents the output of the Softmax layer; The Layer Norm calculation is expressed as in, is x i The average of is x i The standard deviation of , i is in [1, N], N is the number of x, x i Represents the input of the Layer Norm layer, y i Represents the output of the Layer Norm layer.
4. The method of verifiable generative pre-training model according to claim 1, characterized in that: The constraint relationship includes the constraint relationship of the linear layer, the constraint relationship of the GELU activation layer, the constraint relationship of the Softmax layer and the constraint relationship of the Layer Norm layer; The constraint relationship of the linear layer is expressed as Ax L +By L =0, where x L represents the input of the linear layer, y L represents the output of the linear layer, A represents the weight matrix of the linear layer, and B represents the bias vector; The constraint relationship of the GELU activation layer includes: s=sign(x G ),|x G |=abs(x G ), and where x G represents the input of the GELU activation layer, sign(·) represents the digital sign function, abs(·) represents the absolute value function, min(·) represents the minimum value function, L(·) represents the approximate function, y G Represents the output of the GELU activation layer; The constraints of the Softmax layer include: t s =L(p s )>>z s and Among them, x s represents the input of the Softmax layer, p s In the range (-ln 2, 0], z s is a non-negative integer, L(·) represents the approximate function, y s Represents the output of the Softmax layer; The constraints of the Layer Norm layer include: and x i -μ-σy i =0, where x i Represents the input of the Layer Norm layer, μ is x i The average of x i The standard deviation of , i is [1, N], N is the number of x, y i Represents the output of the Layer Norm layer.
5. The method of verifiable generative pre-training model according to claim 1, characterized in that: The method of using zero-knowledge proof to sequentially verify the constraint relationship of each sub-proof task includes: authenticating all constraint relationships of each sub-proof task through information-theoretic message authentication codes, so that all constraint relationships of each sub-proof task satisfy the proof relationship of m=k-ΔX, wherein the prover's private message X includes all variables in the constraint relationships corresponding to the linear layer, the GELU activation layer, the Softmax layer, and the LayerNorm layer, m is the authentication tag held by the prover corresponding to X, k is the local key held by the verifier corresponding to X, and Δ is the global key held by the verifier and is fixed.
6. The method of verifiable generative pre-training model according to claim 5, characterized in that: The method of making all constraint relationships of each sub-proof task satisfy the proof relationship of m=k-ΔX includes: selecting a suitable proof type for verification according to the constraint relationship type of each sub-proof task, and the proof type includes: an identity proof type of X=0 and a range proof type of X≥0.
7. The method of verifiable generative pre-training model according to claim 6, characterized in that: The identity proof type of X=0 regards the variables of the constraint relationship in each sub-proof task as the prover's private message X. Based on the proof relationship m=k-ΔX, the prover obtains m and the verifier obtains k. The prover proves that m is equal to k through the identity proof type of X=0. The range proof type of X≥0 regards the variables of the constraint relationship in each sub-proof task as the prover's private message X. Based on the proof relationship m=k-ΔX, the prover obtains m and the verifier obtains k. The prover constrains the message X′ held by the verifier to be within a specific range through the range proof type of X≥0, satisfying the proof relationship.
8. The method of verifiable generative pre-training model according to claim 1, characterized in that: The method of reducing the proof result of each constraint relationship to the proof result of the generative pre-trained model for text generation includes: based on Boolean values, representing the zero-knowledge proof results of all constraint relationships in the sub-proof task as B1, B2, ..., B n , through the formula B = B1 & B2 & ... & B n The proof result of the constraint relationship of each sub-proof task is reduced to the proof result of the generative pre-trained model, where B is the proof result of the generative pre-trained model and & represents the reduction calculation operation.
9. A device for a verifiable generative pre-training model based on zero-knowledge proof, comprising a memory and a processor, wherein the memory is used to store a computer program, characterized in that: The processor is used to implement the method of verifiable generative pre-training model based on zero-knowledge proof as described in any one of claims 1 to 8 when executing the computer program.