Model training integrity verification method based on trusted execution environment

Abstract

The invention discloses a model training integrity verification method based on a trusted execution environment. The method comprises the following steps: acquiring a model offline training data set; constructing an operator training set to train an operator operation time prediction model; obtaining the overall expected time of each training round of the corresponding to-be-trained model; quantifying model parameters after single-round training of each model in the GPU equipment to obtain a parameter group of the corresponding model; corresponding model training is carried out in the target GPU, TEE training and timing are started according to the pseudo-random seeds, and quantitative check points and complete check points are generated; and repeatedly verifying the integrity of model training in the TEE according to the quantitative check points and the complete check points, and regarding the model with the integrity as a qualified model. According to the method, abnormity can be found in time in the training process, waste of computing resources and reduction of model performance are avoided, the storage burden can be effectively relieved, robustness and expandability are improved, and correctness and safety of training are guaranteed.

Description

Technical Field

[0001] This invention belongs to the field of computer security and artificial intelligence technology, specifically relating to a method for verifying the integrity of model training based on a trusted execution environment. Background Technology

[0002] In recent years, deep neural networks have continued to break performance limits in fields such as image recognition, medical image analysis, and machine translation. However, the computational demands for training cutting-edge large-scale models are increasing exponentially. Taking DeepSeek-V3, which employs a breakthrough Hybrid Expert (MoE) architecture, as an example, this model achieves up to 90% improvement in computational efficiency through sparse activation mechanisms, but the total number of parameters remains as high as 671 billion (671B). Its complete training process consumes approximately 2.788 million H800 GPU hours, and running on a heterogeneous computing cluster consisting of 2048 H800s takes up to 55 days, with an average daily computational scheduling intensity of 144,000 GPU hours. To alleviate the bottleneck of hardware resources for developers and SMEs, various companies have launched managed training and inference services for large models.

[0003] Recent studies have shown that while cloud-hosted training alleviates local computing power bottlenecks to some extent, it also renders the training process almost entirely a "black box." Tenants cannot verify in real time whether the cloud is executing all agreed-upon epochs, batch size, and learning rate settings. They also find it difficult to detect whether the service provider is unilaterally reducing the number of iterations, dynamically lowering training device performance, saving GPU computing time, or even secretly tampering with the model structure. More worryingly, if training data or model weights are maliciously altered, it can lead to a sharp drop in model accuracy, or even the implantation of backdoors, privacy leaks, and serious compliance risks. The following details three types of attacks that can result:

[0004] (1) Poisoning attack: Attackers can implement label flipping, backdoor injection or weight poisoning, etc., to accurately destroy key categories, implant triggers or increase inference energy consumption without significantly reducing the overall indicators. It is highly concealed. Once deployed to the production environment, it is often only discovered when the error output or resource consumption surges. At this time, the loss of computing power and data security is difficult to recover.

[0005] (2) Training skip attack: Cloud service providers or malicious operators may significantly shorten the training time by reducing epochs, skipping gradient updates, or not reporting local models in federated scenarios, but charge for the full training. This kind of "cutting corners" is difficult to detect in the early stages through the final accuracy, causing users to suffer both performance risks and economic losses.

[0006] (3) Training equipment performance fraud attack: Cloud service providers or malicious actors within them may exaggerate equipment performance (e.g., claiming an A100 / H100 GPU but using a consumer-grade GTX 1080) or dynamically adjust computing power (forced downclocking, disabling computing cores, limiting memory bandwidth) to prolong the time spent on single-step training. Under the performance-based payment model, the actual cost paid by users far exceeds the true value of the computing power.

[0007] To defend against the aforementioned attacks, ensuring the integrity of model training is considered a practical protection strategy. Currently, mainstream solutions can be broadly categorized into three types: ① Closed training: encapsulating the entire training process within a Trusted Execution Environment (TEE); ② Training proof mechanism: generating non-repudiable training proofs using verifiable computation or zero-knowledge proofs; ③ Offline repeated verification: after training, randomly sampling and re-executing parts of the training task to compare the results. These three solutions each seek a balance between security and system overhead, but all suffer from key bottlenecks: the TEE mode is limited by CPU-level computational performance, making it difficult to support large-scale training loads; while the training proof mechanism provides strong formal guarantees, the generation and verification processes are extremely costly; and offline repeated verification suffers from detection lag and untimely response. Therefore, existing methods for protecting the integrity of model training generally suffer from the following problems:

[0008] Lack of performance evaluation mechanism: Current methods cannot effectively monitor the performance of training equipment. Cloud service providers may provide hardware with reduced performance and dynamically adjust resources, resulting in a significant decrease in training speed. The planned number of training epochs to be completed by the TEE cannot be completed within the specified time, ultimately causing insufficient model convergence and affecting the final accuracy. At the same time, under the performance-based billing model, the user's payment is not commensurate with the computing power obtained, resulting in economic losses.

[0009] Significant bottlenecks exist in efficiency: On the one hand, deploying training within a TEE is limited by resource constraints, making it difficult to support large-scale model training; on the other hand, zero-knowledge proof-based solutions still have limitations in model scalability and automated circuitry, making efficiency difficult to guarantee. Furthermore, offline repeated verification requires storing a large number of checkpoints and frequent I / O operations, resulting in high verification costs; while reducing the sampling rate can improve efficiency, it increases the risk of attack escape. In addition, verification methods based on checkpoint hash values ​​are difficult to accurately reproduce the original training results due to differences in hardware parallel computing or low-level library implementations. The hash values ​​of the original training output and the re-execution result often cannot match due to reproduction errors, leading to verification failure.

[0010] Insufficient real-time performance: Existing verification mechanisms are generally lagging, performing verification only after training is complete. This means that if the cloud reduces the number of training rounds or injects poisoned gradients in the early stages of training, the user must consume all training computing resources to complete training before triggering an alert, severely delaying anomaly detection and potentially rendering the training results completely invalid. Furthermore, methods based on end-to-end trusted proofs or training within a TEE consume significant computing and storage resources, further exacerbating resource waste. Summary of the Invention

[0011] The purpose of this invention is to address the above-mentioned problems by proposing a model training integrity verification method based on a trusted execution environment. This method enables real-time integrity verification of the model training process, allowing for timely detection of anomalies during training. It avoids wasting computational resources and degrading model performance due to delayed verification, minimizes computational waste and data risks, effectively alleviates the storage burden caused by repeated verification, greatly improves the robustness and scalability of the verification mechanism, and ensures the correctness and security of the training process.

[0012] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0013] The present invention proposes a method for verifying the integrity of model training based on a trusted execution environment, comprising the following steps:

[0014] S1. Obtain the offline training dataset of the model, which consists of several multi-dimensional variables, including different models, the hyperparameters of the corresponding models, and the running time of each operator on different GPU devices;

[0015] S2. Construct an operator training set based on the offline training dataset of the model to train the operator runtime prediction model;

[0016] S3. Based on the operator training set, the model to be trained and the corresponding hyperparameters, perform reliability verification in the cloud, and based on the results of the reliability verification, use the trained operator runtime prediction model to obtain the overall expected time for each training round of the corresponding model to be trained. The cloud includes TEE and target GPU, and TEE is a trusted execution environment.

[0017] S4. Collect model parameters after single-round training of different models from various GPU devices, and quantize the model parameters of each model in the GPU device to obtain the parameter set of the corresponding model.

[0018] S5. Using the training set, initial weights, and hyperparameters of the model to be trained, train the corresponding model on the target GPU. Simultaneously, start TEE training according to a pseudo-random seed and time it. During training, each... Each iteration generates a quantization checkpoint, recording the quantization values ​​and parameter sets of the model parameters at the corresponding quantization checkpoint. A complete checkpoint is generated in each training epoch, recording the model parameters at the corresponding complete checkpoint. The preset checkpoint interval number;

[0019] S6. Submit the quantization checkpoints corresponding to each training round to the TEE as a training commitment through the target GPU. When the deviation between the actual timing time of the TEE and the overall expected time of each training round of the corresponding model to be trained does not exceed the predicted time threshold, sample in the training commitment according to the preset sampling number and combine the complete checkpoints to repeatedly verify the integrity of the model training in the TEE. The model with integrity after repeated verification is regarded as a qualified model.

[0020] Preferably, the operator samples in the operator training set are represented as a quadruple of <type, configuration parameters, micro-baseline time, runtime>. The micro-baseline time is the time that the performance index of the corresponding operator is mapped, that is, the ratio of the operator's computational load to the theoretical computational speed of the GPU device or the ratio of the operator's read / write load to the theoretical read / write speed of the GPU device. The type and configuration parameters are derived from the corresponding type and configuration parameters of the operator in the model parameters.

[0021] Preferably, the operator runtime prediction model is an MLP model or a random forest model.

[0022] Preferably, the operators include matrix multiplication, convolutional layers, ReLU activation function, and LSTM module.

[0023] Preferably, reliability verification is performed in the cloud based on the operator training set, the model to be trained, and the corresponding hyperparameters. Based on the results of the reliability verification, the overall expected time for each training round of the corresponding model to be trained is obtained using the trained operator runtime prediction model, as follows:

[0024] S31. Extract the corresponding operators based on the model to be trained, and match the operator samples with the highest cosine similarity in the operator training set under TEE.

[0025] S32. Randomly generate the input parameters of the operators of the model to be trained as the micro-benchmark test task of the corresponding operators;

[0026] S33. Distribute the micro-benchmark test tasks of each operator to the target GPU for testing, and at the same time use TEE to obtain the execution results and actual timing of the micro-benchmark test tasks of the corresponding operators.

[0027] S34. The test results of the target GPU for each operator are sent back to the TEE for reliability verification. When the target GPU is reliable, the model to be trained, its corresponding hyperparameters, and the micro-reference time of the corresponding operator are input into the trained operator runtime prediction model to obtain the predicted runtime of the corresponding operator. The predicted runtime of all operators of the model to be trained is added together to obtain the overall expected time of each training round of the model to be trained. When the target GPU is unreliable, the model to be trained, its corresponding hyperparameters, and the micro-reference time of the corresponding operator set to zero are input into the input parameters of the trained operator runtime prediction model to obtain the predicted runtime of the corresponding operator. The predicted runtime of all operators of the model to be trained is added together to obtain the overall expected time of each training round of the model to be trained.

[0028] Preferably, the reliability verification is as follows:

[0029] S341. Determine whether the error between the execution result of the micro-benchmark test task of the corresponding operator and the test result of the target GPU is within the first preset threshold. If yes, the corresponding operator is considered to have correct execution and step S342 is executed. Otherwise, the corresponding operator is considered not to have correct execution and the target GPU is not reliable.

[0030] S342. Determine whether the ratio of the micro-reference time of the corresponding operator to the actual timing time is within the second preset threshold. If it is, it is considered to reflect the true performance of the corresponding operator and the target GPU is reliable. Otherwise, there is fraud, and it is considered not to reflect the true performance of the corresponding operator and the target GPU is not reliable.

[0031] Preferably, the model parameters of each model in the GPU device are quantized to obtain the parameter set of the corresponding model, as follows:

[0032] S41. Based on the model parameters collected from each GPU device after a single round of training, obtain the quantization hash value of the corresponding network layer for each model, as follows:

[0033] S411. For each network layer, generate multiple candidate quantization parameter pairs, then the... Each candidate quantization parameter pair The formula is as follows:

[0034] , ;

[0035] in, To quantize the step size, Zero point , Let be the total number of candidate quantization parameter pairs, and b be a variable ranging from 2 to 32, representing the quantization bit width of the corresponding network layer. This represents the numerical range for the current network layer. This is the minimum value for the current network layer. This represents the maximum value for the current network layer.

[0036] S412. Calculate the quantized value of each candidate quantization parameter for the corresponding network layer on each GPU device. Then, in the... The first network layer under the first GPU device Quantization values ​​of each parameter The formula is as follows:

[0037] ;

[0038] in, This represents the rounding function. For the corresponding model in the first The first network layer under the first GPU device There are parameters, among which, , This represents the total number of network layers in the corresponding model. , Total number of GPU devices , This represents the total number of parameters for the corresponding network layer.

[0039] S413, Calculate the pair of candidate quantization parameters at the 1st... The first network layer under the first GPU device Quantized hash value of each parameter The formula is as follows:

[0040] ;

[0041] in, This represents the SHA-3 cryptographic hash function;

[0042] S42. Determine whether there exists a candidate quantization parameter pair that makes the quantization hash values ​​of the corresponding network layers of the same model completely consistent across all GPU devices. If so, use the candidate quantization parameter pair as the parameter set of the corresponding model. In other words, global quantization configuration parameters are used; otherwise, the model is considered incomplete, and the process ends.

[0043] Preferably, the quantization checkpoints corresponding to each training epoch are submitted to the TEE as training commitments via the target GPU. When the deviation between the actual timing of the TEE and the overall expected time of each training epoch of the corresponding model to be trained does not exceed the predicted time threshold, sampling is performed in the training commitments according to a preset sampling number, and the integrity of the model training is repeatedly verified in the TEE in conjunction with complete checkpoints. The repeated verification process for each training epoch is as follows:

[0044] S61. Generate a training commitment by collecting the hash values ​​of all quantization checkpoints in the current training round using the target GPU. It is expressed as follows:

[0045] ;

[0046] in, This refers to the SHA-3 cryptographic hash function. This represents the model parameters corresponding to the m-th quantization checkpoint, where m=1~ , This represents the iteration number of the current training round.

[0047] S62. In the TEE, compare the deviation between the actual timing of the TEE and the overall expected time of each training round of the corresponding model to be trained, and whether it exceeds the predicted time threshold. If not, the performance of the corresponding model is considered compliant. According to the preset sampling number, randomly select quantization checkpoints in the training commitment based on the random number generator, and perform dequantization based on the quantization value of the model parameters and the parameter group of the selected quantization checkpoints to restore the model parameters of the corresponding training round. Load the subset of the training set of the model to be trained associated with the selected quantization checkpoints, and execute step S63 according to the order and hyperparameters determined by the pseudo-random seed. Otherwise, the performance of the corresponding model is considered non-compliant, and step S65 is executed.

[0048] S63. In the TEE, the training process starts from the previous quantization checkpoint of each extracted quantization checkpoint and calculates the hash value of the new quantization checkpoint. As proof, the formula is as follows:

[0049] ;

[0050] in, This represents the model parameters corresponding to the s-th quantization checkpoint. This represents the model parameters corresponding to the t-th quantization checkpoint. ;

[0051] S64. Compare the hash values ​​of the new quantization checkpoints respectively. With training commitment If the hash values ​​of the corresponding indices are consistent, the integrity of the current training epoch of the corresponding model is confirmed through remote proof signature, i.e., the model training is complete. Otherwise, the previous complete checkpoint before the currently sampled quantization checkpoint is selected as the starting point to start the training process, and the cosine distance difference between the model parameters before and after training is compared. Does it meet the requirements? , This indicates the upper bound of the global error. If it is, the model training is considered complete, and the process ends. Otherwise, the model training is considered incomplete, and step S65 is executed.

[0052] S65. Pause the target GPU training, retain the complete checkpoint of the previous training round, then restore the model parameters and the order determined by the pseudo-random seed to the model parameters corresponding to the complete checkpoint of the previous training round, and record the cloud default evidence, which includes the runtime of the current training round, the training commitment, and the hash value of the new quantization checkpoint, and end the process.

[0053] Preferably, the upper bound of the global error The formula is as follows:

[0054] ;

[0055]

[0056] Among them, the The vector of model parameters after a single round of training on a GPU device =[a1,a2,…,an,…,aN], the vector of model parameters after a single round of training on the j-th GPU device. =[b1,b2,…,bn,…,bN], where an is the nth... The nth parameter in the model parameters after a single round of training on a GPU device, where bn is the nth parameter. The nth parameter in the model parameters after a single round of training on a GPU device, where N is the total dimension of the corresponding model parameters. for and cosine similarity, This represents the total number of GPU devices.

[0057] Preferably, after recording the evidence of breach of contract in the cloud, it is determined whether to continue training. If so, the preset sampling number is increased, and the process returns to step S63 to restart the target GPU training. Otherwise, the process ends.

[0058] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0059] This method achieves real-time integrity verification of the model training process by periodically repeating key calculations in a trusted execution environment (TEE) during training. Compared to traditional post-training verification methods, this method can detect anomalies in a timely manner during training, avoiding wasted computational resources and model performance degradation caused by delayed verification. Simultaneously, by utilizing micro-baseline time and the actual timing of the TEE, it achieves accurate evaluation of the target GPU-trained model performance, thus precisely measuring whether the device performance meets the standards and avoiding model accuracy degradation and economic losses caused by falsely reporting device performance. By combining encrypted hash functions and random sampling strategies, it achieves efficient protection of the correctness and integrity of the training process. Once inconsistencies are detected, it immediately rolls back to the nearest trusted checkpoint to stop the damage, minimizing wasted computational power and data risk. Furthermore, the introduction of model quantization technology effectively alleviates the storage burden caused by repeated verification and, to some extent, eliminates reproduction errors caused by device differences, further improving the robustness and scalability of the verification mechanism. In summary, this study utilizes a micro-benchmark-enhanced operator runtime prediction model for performance evaluation. Combined with reliability verification in a trusted execution environment, real-time quantization fault tolerance, and repetitive verification mechanisms, it aims to address the performance bottlenecks, high proof generation overhead, and detection lag issues inherent in existing TEE closed training, zero-knowledge proofs, and offline repetitive verification. This enables efficient prediction and evaluation of model and device performance, and provides low-overhead, real-time alarm assurance for large-scale model training in the cloud. It ensures the correctness and security of the training process in cloud computing scenarios and reduces the risks and costs associated with dishonest computation or model poisoning. It is particularly suitable for cloud computing and outsourced training applications. Attached Figure Description

[0060] Figure 1 This is a flowchart of the model training integrity verification method based on a trusted execution environment according to the present invention;

[0061] Figure 2 This is a flowchart of step S3 of the present invention;

[0062] Figure 3 This is a flowchart of step S6 of the present invention. Detailed Implementation

[0063] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0064] It should be noted that, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application.

[0065] To address the issue of inflated hardware performance or dynamic performance degradation in cloud services that harms user interests, this paper proposes a hardware performance evaluation method based on multi-source data fusion and trusted execution. This method constructs a high-precision operator runtime prediction model and combines it with trusted timing via a Trusted Execution Environment (TEE) to evaluate the performance of the model and its mounted hardware. The method operates within a trusted execution environment system, which includes several GPU devices and a cloud platform. The cloud platform comprises a TEE and the target GPU. The process is divided into offline and online flows. The offline flow involves executing steps S1 and S2 via the GPU devices, while the remaining steps are performed online.

[0066] like Figures 1-3 As shown, a method for verifying the integrity of model training based on a trusted execution environment includes the following steps:

[0067] S1. Obtain the offline training dataset of the model, which consists of several multi-dimensional variables, including different models, the hyperparameters of the corresponding models, and the running time of each operator on different GPU devices.

[0068] In one embodiment, the operator includes matrix multiplication, convolutional layers, ReLU activation function, and LSTM module.

[0069] The offline data collection phase needs to cover multi-dimensional variables, including model architecture (different models, such as CNN, RNN, Transformer, and other deep learning models), hyperparameters of the corresponding models (such as batch size, learning rate, optimizer type, etc.), and, to improve data quality and model generalization ability, the PyTorchProfiler tool can be used to collect the runtime of different operators of the corresponding model (such as matrix multiplication, convolutional layers, ReLU activation function, LSTM module, etc.) on different hardware platforms (GPU devices) as training data, ensuring adaptability to different hardware platforms and model scenarios. LSTM module stands for Long Short-Term Memory. The specific model and operator selection can be adjusted according to actual needs.

[0070] S2. Construct an operator training set based on the offline training dataset of the model to train the operator runtime prediction model.

[0071] In one embodiment, the operator samples in the operator training set are represented as a quadruple of <type, configuration parameters, micro-baseline time, runtime>. The micro-baseline time is the time that the performance index of the corresponding operator is mapped, that is, the ratio of the operator's computational load to the theoretical computational speed of the GPU device or the ratio of the operator's read / write load to the theoretical read / write speed of the GPU device. The type and configuration parameters are derived from the corresponding type and configuration parameters of the operator in the model parameters.

[0072] In one embodiment, the operator runtime prediction model is an MLP model or a random forest model.

[0073] The offline training dataset can be used to construct a high-precision operator runtime prediction model. This embodiment uses an MLP (Multilayer Perceptron) model, but other models, such as random forest models, can be selected according to actual needs. Specifically, to address the heterogeneity of unknown cloud training devices, micro-benchmarks (i.e., operators and their corresponding configuration parameters that can effectively map time through performance metrics) are used as training data for the operator runtime prediction model. The focus is on core operators commonly used in the model (such as matrix multiplication, convolutional layers, ReLU activation functions, LSTM modules, etc.) as micro-benchmark tests, which greatly improves the accuracy of the operator runtime prediction model. Due to their computational and memory sensitivity, these operators can be directly mapped to computation time through performance metrics, such as operator computational load (flops per second) and operator read / write load (memory read / write bandwidth). For example, for the matrix multiplication operator, the computation time can be estimated by calculating the ratio of floating-point operations per second (FLOPS) to the GPU's floating-point computing power (the theoretical computing speed of GPU devices, in fLOPS), which serves as the micro-benchmark time for the corresponding operator. In the actual construction of the operator training set, micro-benchmark performance data is systematically collected for different types of GPU devices. The focus is on selecting operators and their corresponding configuration parameters that can effectively map time through performance metrics, such as a computation time / uptime deviation of ±5%. This significantly improves the adaptability of the operator runtime prediction model to heterogeneous hardware. Combining the runtime data of each operator collected in step S1 under different GPU devices, an operator sample containing the quadruple <type, configuration parameters, micro-benchmark time, runtime> is constructed to form the operator training set (specifically, for operators whose time cannot be mapped through performance metrics or are not included in the micro-benchmark, the corresponding micro-benchmark time is left empty), used to train the MLP model. The type in the operator sample can directly be the operator itself.

[0074] S3. Based on the operator training set, the model to be trained and the corresponding hyperparameters, perform reliability verification in the cloud, and based on the results of the reliability verification, use the trained operator runtime prediction model to obtain the overall expected time for each training round of the corresponding model to be trained. The cloud includes TEE and target GPU, and TEE is a trusted execution environment.

[0075] In one embodiment, reliability verification is performed in the cloud based on the operator training set, the model to be trained, and the corresponding hyperparameters. Based on the results of the reliability verification, the overall expected time for each training round of the corresponding model to be trained is obtained using the trained operator runtime prediction model, as detailed below:

[0076] S31. Extract the corresponding operators based on the model to be trained, and match the operator samples with the highest cosine similarity in the operator training set under TEE.

[0077] S32. Randomly generate the input parameters of the operators of the model to be trained as the micro-benchmark test task of the corresponding operators;

[0078] S33. Distribute the micro-benchmark test tasks of each operator to the target GPU for testing, and at the same time use TEE to obtain the execution results and actual timing of the micro-benchmark test tasks of the corresponding operators.

[0079] S34. The test results of the target GPU for each operator are sent back to the TEE for reliability verification. When the target GPU is reliable, the model to be trained, its corresponding hyperparameters, and the micro-reference time of the corresponding operator are input into the trained operator runtime prediction model to obtain the predicted runtime of the corresponding operator. The predicted runtime of all operators of the model to be trained is added together to obtain the overall expected time of each training round of the model to be trained. When the target GPU is unreliable, the model to be trained, its corresponding hyperparameters, and the micro-reference time of the corresponding operator set to zero are input into the input parameters of the trained operator runtime prediction model to obtain the predicted runtime of the corresponding operator. The predicted runtime of all operators of the model to be trained is added together to obtain the overall expected time of each training round of the model to be trained.

[0080] In one embodiment, the reliability verification is as follows:

[0081] S341. Determine whether the error between the execution result of the micro-benchmark test task of the corresponding operator and the test result of the target GPU is within the first preset threshold. If yes, the corresponding operator is considered to have correct execution and step S342 is executed. Otherwise, the corresponding operator is considered not to have correct execution and the target GPU is not reliable.

[0082] S342. Determine whether the ratio of the micro-reference time of the corresponding operator to the actual timing time is within the second preset threshold. If it is, it is considered to reflect the true performance of the corresponding operator and the target GPU is reliable. Otherwise, there is fraud, and it is considered not to reflect the true performance of the corresponding operator and the target GPU is not reliable.

[0083] The process involves obtaining the user-provided model to be trained and its corresponding hyperparameters (such as batch size, learning rate, optimizer type, etc.). The model is then analyzed to extract corresponding operators. The model can be any model type within the model architecture. The operator sample with the highest cosine similarity in the operator training set is matched within the TEE (Training Equipment). The input parameters of the operators for the model to be trained are randomly generated as a micro-benchmark task for that model's operators. This micro-benchmark task is then sent to the target training device (target GPU) for execution. The testing process uses the TEE for precise timing (utilizing the TEE's secure clock register to prevent system time tampering) to ensure the reliability of the time data. After completing the micro-benchmark test, the output results of the corresponding operators (the test results of the target GPU) are sent back to the TEE. The TEE verifies the validity of the test results through two dimensions: execution correctness and performance metric reflectivity. Execution correctness verification is performed by comparing the execution results of the corresponding operator's micro-benchmark task with the test results of the target GPU. If the numerical error between the two results is within a first preset threshold (e.g., relative error ≤ 1e-5), the corresponding operator is considered to have execution correctness; otherwise, the corresponding operator does not have execution correctness. Performance indicator reflectivity verification is performed by comparing the corresponding operator's micro-benchmark time with the actual timing time of the TEE. If the ratio of the corresponding operator's micro-benchmark time to the actual timing time is within a second preset threshold (e.g., 0.8-1.2 times), it is considered to reflect the true performance of the corresponding operator; otherwise, performance manipulation may exist, and the performance cannot be reflected.

[0084] Based on the reliability verification results, a tiered processing mechanism is adopted: Under reliable conditions (both execution correctness and performance responsiveness pass reliability verification), the model to be trained, its corresponding hyperparameters, and the micro-baseline time of the corresponding operators are input into the trained operator runtime prediction model, and the results are accumulated to obtain the overall expected time for each training epoch of the corresponding model. Under unreliable conditions (reliability verification fails), the model to be trained, its corresponding hyperparameters, and the micro-baseline time of the corresponding operators (all set to zero) are input into the input parameters of the trained operator runtime prediction model, and the results are accumulated to obtain the overall expected time for each training epoch of the corresponding model. The overall expected time for each training epoch is output through the trained MLP model, providing users with reliable performance predictions.

[0085] S4. Collect model parameters after single-round training of different models from various GPU devices, and quantize the model parameters of each model in the GPU device to obtain the parameter set of the corresponding model.

[0086] In one embodiment, the model parameters of each model in the GPU device are quantized to obtain the parameter set of the corresponding model, as follows:

[0087] S41. Based on the model parameters collected from each GPU device after a single round of training, obtain the quantization hash value of the corresponding network layer for each model, as follows:

[0088] S411. For each network layer, generate multiple candidate quantization parameter pairs, then the... Each candidate quantization parameter pair The formula is as follows:

[0089] , ;

[0090] in, To quantize the step size, Zero point , Let be the total number of candidate quantization parameter pairs, and b be a variable ranging from 2 to 32, representing the quantization bit width of the corresponding network layer. This represents the numerical range for the current network layer. This is the minimum value for the current network layer. This represents the maximum value for the current network layer.

[0091] S412. Calculate the quantized value of each candidate quantization parameter for the corresponding network layer on each GPU device. Then, in the... The first network layer under the first GPU device Quantization values ​​of each parameter The formula is as follows:

[0092] ;

[0093] in, This represents the rounding function. For the corresponding model in the first The first network layer under the first GPU device There are parameters, among which, , This represents the total number of network layers in the corresponding model. , Total number of GPU devices , This represents the total number of parameters for the corresponding network layer.

[0094] S413, Calculate the pair of candidate quantization parameters at the 1st... The first network layer under the first GPU device Quantized hash value of each parameter The formula is as follows:

[0095] ;

[0096] in, This represents the SHA-3 cryptographic hash function;

[0097] S42. Determine whether there exists a candidate quantization parameter pair that makes the quantization hash values ​​of the corresponding network layers of the same model completely consistent across all GPU devices. If so, use the candidate quantization parameter pair as the parameter set of the corresponding model. In other words, global quantization configuration parameters are used; otherwise, the model is considered incomplete, and the process ends.

[0098] Step S4 aims to develop a quantization-based fault-tolerance strategy. To address hash comparison failures in repeated verification due to differences in floating-point precision, quantization converts high-precision floating-point numbers into low-order integers, thus tolerating numerical deviations within a certain training epoch and reducing storage overhead from frequent save points (such as subsequent quantization checkpoints and full checkpoints). Before formal training, to construct a reasonable fault-tolerance strategy, a systematic pre-experiment is conducted to quantize the inherent errors between GPU devices. Model parameters after single-epoch training of different models are collected from various GPU devices. The model architecture covers different network layers and numerical distributions (i.e., the numerical range of the corresponding network layers). (Originating from model parameters), parameters from different GPU devices are collected for each network layer. Based on the numerical range of the corresponding network layer, multiple candidate quantization parameter pairs are generated for each layer. For each candidate quantization parameter pair, the quantized value of the parameter at the corresponding network layer on each GPU device is calculated, and the quantization hash value of the corresponding layer is calculated. Finally, the candidate quantization parameter pair that makes the quantization hash values ​​of the corresponding network layers of the same model completely consistent across all GPU devices is selected as the parameter group. In other words, as a unified global quantization configuration parameter, if it is impossible to find a candidate quantization parameter pair that makes the quantization hash values ​​of the corresponding network layers of the same model completely consistent across all GPU devices... If the model is incomplete, the process is terminated. The rounding function is used to discretize continuous floating-point values ​​into the nearest integer quantized value. Model parameters are configuration variables inside the model, while hyperparameters are configuration variables outside the model, meaning their values ​​are manually set. Model parameters and hyperparameters are well-known techniques to those skilled in the art and will not be elaborated upon here. Network layers are hierarchical units in a neural network, divided according to function or structure. Each layer is responsible for different functions. For example, the VGG16 model (a CNN model) has 16 layers, including 13 convolutional layers and 3 fully connected layers. The same applies to other models, which are also well-known techniques to those skilled in the art and will not be elaborated upon here.

[0099] S5. Using the training set, initial weights, and hyperparameters of the model to be trained, train the corresponding model on the target GPU. Simultaneously, start TEE training according to a pseudo-random seed and time it. During training, each... Each iteration generates a quantization checkpoint, recording the quantization values ​​and parameter sets of the model parameters at the corresponding quantization checkpoint. A complete checkpoint is generated in each training epoch, recording the model parameters at the corresponding complete checkpoint. This is the preset checkpoint interval.

[0100] The training process involves initiating model training on the target GPU. After training begins, a pseudo-random seed provided by the TEE (Training Environment) is used for data shuffling. Simultaneously, the TEE enables a trusted timer to ensure deterministic data selection and prevent replay attacks. The training process utilizes the training set of the model to be trained. Initial weights ( and hyperparameters Update the model parameters and optimizer parameters, and in each Each iteration generates a quantization checkpoint, recording the quantized values ​​and parameter sets of the model parameters at the corresponding checkpoint to support dequantization. A complete checkpoint is generated in each training epoch, recording the model parameters (including current weight parameters and optimizer parameters) at the corresponding complete checkpoint. The pseudo-random seed is V, where V is a positive integer.

[0101] S6. Submit the quantization checkpoints corresponding to each training round to the TEE as a training commitment through the target GPU. When the deviation between the actual timing time of the TEE and the overall expected time of each training round of the corresponding model to be trained does not exceed the predicted time threshold, sample in the training commitment according to the preset sampling number and combine the complete checkpoints to repeatedly verify the integrity of the model training in the TEE. The model with integrity after repeated verification is regarded as a qualified model.

[0102] In one embodiment, the target GPU submits the quantization checkpoints corresponding to each training epoch to the TEE as a training commitment. When the deviation between the actual timing of the TEE and the overall expected time of each training epoch of the corresponding model to be trained does not exceed the predicted time threshold, sampling is performed in the training commitment according to a preset sampling number, and the integrity of the model training is repeatedly verified in the TEE in conjunction with the complete checkpoints. The repeated verification process for each training epoch is as follows:

[0103] S61. Generate a training commitment by collecting the hash values ​​of all quantization checkpoints in the current training round using the target GPU. It is expressed as follows:

[0104] ;

[0105] in, This refers to the SHA-3 cryptographic hash function. This represents the model parameters corresponding to the m-th quantization checkpoint, where m=1~ , This represents the iteration number of the current training round.

[0106] S62. In the TEE, compare the deviation between the actual timing of the TEE and the overall expected time of each training round of the corresponding model to be trained, and whether it exceeds the predicted time threshold. If not, the performance of the corresponding model is considered compliant. According to the preset sampling number, randomly select quantization checkpoints in the training commitment based on the random number generator, and perform dequantization based on the quantization value of the model parameters and the parameter group of the selected quantization checkpoints to restore the model parameters of the corresponding training round. Load the subset of the training set of the model to be trained associated with the selected quantization checkpoints, and execute step S63 according to the order and hyperparameters determined by the pseudo-random seed. Otherwise, the performance of the corresponding model is considered non-compliant, and step S65 is executed.

[0107] S63. In the TEE, the training process starts from the previous quantization checkpoint of each extracted quantization checkpoint and calculates the hash value of the new quantization checkpoint. As proof, the formula is as follows:

[0108] ;

[0109] in, This represents the model parameters corresponding to the s-th quantization checkpoint. This represents the model parameters corresponding to the t-th quantization checkpoint. ;

[0110] S64. Compare the hash values ​​of the new quantization checkpoints respectively. With training commitment If the hash values ​​of the corresponding indices are consistent, the integrity of the current training epoch of the corresponding model is confirmed through remote proof signature, i.e., the model training is complete. Otherwise, the previous complete checkpoint before the currently sampled quantization checkpoint is selected as the starting point to start the training process, and the cosine distance difference between the model parameters before and after training is compared. Does it meet the requirements? , This indicates the upper bound of the global error. If it is, the model training is considered complete, and the process ends. Otherwise, the model training is considered incomplete, and step S65 is executed.

[0111] S65. Pause the target GPU training, retain the complete checkpoint of the previous training round, then restore the model parameters and the order determined by the pseudo-random seed to the model parameters corresponding to the complete checkpoint of the previous training round, and record the cloud default evidence, which includes the runtime of the current training round, the training commitment, and the hash value of the new quantization checkpoint, and end the process.

[0112] In one embodiment, the global error upper bound The formula is as follows:

[0113] ;

[0114]

[0115] Among them, the The vector of model parameters after a single round of training on a GPU device =[a1,a2,…,an,…,aN], the vector of model parameters after a single round of training on the j-th GPU device. =[b1,b2,…,bn,…,bN], where an is the nth... The nth parameter in the model parameters after a single round of training on a GPU device, where bn is the nth parameter. The nth parameter in the model parameters after a single round of training on a GPU device, where N is the total dimension of the corresponding model parameters. for and cosine similarity, This represents the total number of GPU devices.

[0116] The process involves repeated verification within a trusted execution environment. Specifically, after each training epoch, the training process is sampled and verified. This is done by submitting corresponding quantized checkpoints as training commitments via a GPU device. Then, sampling and combining these checkpoints with complete data are repeated in the trusted execution environment to ensure the integrity of the training. The process is as follows:

[0117] 1) Training commitment generation and submission:

[0118] After each training epoch, the GPU device collects the hash values ​​of all quantization checkpoints for that epoch to generate a training commitment. Training commitment Transmission to TEE ensures confidentiality and integrity. Training promises extremely low generation and transmission overhead, supporting efficient real-time verification.

[0119] 2) Quantitative checkpoint sampling and model parameter restoration:

[0120] TEE receives training commitment Next, the actual timing time (TEE timing time) is compared with the predicted time (the overall expected time for each training epoch of the corresponding model to be trained, output by the trained MLP model) to determine whether the performance is compliant. If the deviation does not exceed the predicted time threshold (±10% of the predicted time), the performance is considered compliant; otherwise, if the deviation exceeds the predicted time threshold, it is marked as abnormal, and the process proceeds to step 4) to ensure that the task has not been tampered with or slowed down. Then, based on a random number generator, quantization checkpoints are randomly selected from the training commitment according to a preset sampling number. Simultaneously, dequantization is performed based on the quantized values ​​of the model parameters and parameter sets of the selected quantization checkpoints to restore the model parameters (such as model weights and optimizer parameters) for the corresponding training epochs. The training set of the model to be trained associated with the selected quantization checkpoints is loaded. A subset of, in order determined by a pseudo-random seed and hyperparameters ζ Hyperparameters, including batch size, learning rate, and optimizer type, ensure the integrity and consistency of the training context.

[0121] 3) Duplicate verification and hash value comparison:

[0122] TEE from each sampled quantitative checkpoint Previous quantitative checkpoint Begin the training process and calculate the hash value of the new quantization checkpoint. As proof, and in conjunction with the training commitment Hash value comparison of the corresponding index in the middle. u is the training commitment The index of the corresponding quantization checkpoint.

[0123] If the hash value of the new quantization checkpoint With training commitment If the hash values ​​of the corresponding indices are consistent, the integrity of the current training round is confirmed through remote proof signature (TEE tool). If they are inconsistent, it is because there may still be error accumulation after quantization, leading to inconsistency. In this case, the previous complete checkpoint before the currently sampled quantization checkpoint is selected as the starting point to reproduce the training process, and the cosine distance difference of the model parameters before and after training is compared. Does it meet the requirements? Let's verify again. If it still doesn't match, then the current training round is considered incomplete, and we proceed to step 4). We use the cosine distance comparison mechanism to verify whether the condition is met. This avoids the failure of the quantization comparison mechanism. Under the same training configuration, after each GPU device performs a single round of training, the cosine distance comparison mechanism, in order to accurately characterize the systematic differences between GPU devices, iterates through all GPU device pairs and extracts the maximum cosine distance difference (cosine similarity) between the model parameters of each GPU device after training, as the upper bound of the global error. The value of the total dimension N of the corresponding model parameters is determined by the model structure. This metric (upper bound of global error) reflects the maximum degree of deviation caused by different hardware platforms, floating-point arithmetic precision, and parallel computing strategies.

[0124] 4) Rollback stop-loss and default records:

[0125] If the TEE detects a performance mismatch or hash value discrepancy in a certain round, it pauses the target GPU training, retains the complete checkpoint from the previous round, and then restores the model parameters and the order determined by the pseudo-random seed to the model parameters corresponding to the complete checkpoint of the previous training round. It also records evidence of default in the cloud, including the runtime of the current training round and the training commitment. and the hash value of the new quantization checkpoint ), thus ending the process.

[0126] In one embodiment, after recording evidence of breach of contract in the cloud, it is determined whether to continue training. If so, the preset sampling number is increased, and the process returns to step S63 to restart the target GPU training; otherwise, the process ends.

[0127] Users can choose whether to continue training, either by increasing the preset sampling number and restarting training, or by terminating the job immediately to avoid wasting computing power or allowing a contaminated model to enter the production environment, thereby controlling potential losses within the required training rounds.

[0128] This invention was also verified experimentally, as follows:

[0129] Four different GPUs (2080Ti, A6000, 3080, and V100) were used as the hardware environment to construct an offline training dataset containing 3000 multi-dimensional variables, covering different models (such as ResNet-50, VGG-16, LSTM modules, and BERT-based models). Training parameters included: batch size ranging from 32 to 256, learning rate ranging from 0.001 to 0.1, and optimizer type (SGD, Adam).

[0130] The PyTorch Profiler tool was used to perform end-to-end performance tracking of the model training process, accurately collecting the runtime of each operator. To ensure data reliability, the input parameters of the model to be trained were initialized using a normal distribution, and the model was run 25 times under each hyperparameter configuration, with the average value taken to measure latency. By analyzing the effectiveness of performance metrics for different operators, a micro-benchmark library (operator training set) containing 8249 operator samples was finally constructed.

[0131] The operator training set was divided into 70%:20%:10% portions for training, validation, and testing, respectively. The MLP model was trained using the AdamW optimizer with L2 regularization, with 100 training epochs and batch sizes ranging from 16 to 256. The initial learning rate was set to 5 × 10⁻⁻⁻⁶. 5 and in 1×10⁻ 6 Optimize within a range of 5×10⁻³ to obtain the best accuracy.

[0132] The performance of the MLP model with micro-reference time enhancement (i.e., the training set contains micro-reference time) of this invention was measured in predictions of different models and compared with existing MLP models (i.e., the micro-reference time in the training set is set to zero). The mean absolute error (MAE) was ultimately selected as the evaluation metric for model performance, as shown in Table 1. The formula is as follows:

[0133]

[0134] in, The predicted runtime for the MLP model. This represents the runtime of the MLP model on the target GPU. The total number of samples.

[0135] Table 1

[0136] ResNet-50 VGG-16 LSTM BERT-base MLP model of this invention 9.8% 4.3% 7.2% 13.2% Existing MLP models 11.9% 8.3% 10.5% 13.7%

[0137] Table 1 shows the performance index comparison results, with smaller values ​​indicating better performance. The results show that the method of the present invention has higher accuracy for various models.

[0138] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0139] The embodiments described above are merely specific and detailed examples of the embodiments described in this application, and should not be construed as limiting the scope of the application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. A method for verifying integrity of model training based on a trusted execution environment, characterized in that: The method comprises the following steps: S1, obtaining a model offline training data set composed of a plurality of multi-dimensional variables, the multi-dimensional variables comprising different models, hyperparameters corresponding to the models, and running times of operators under different GPU devices; S2, constructing an operator training set according to the model offline training data set to train an operator running time prediction model; S3, performing reliability verification in the cloud according to the operator training set, a model to be trained, and corresponding hyperparameters, and obtaining an overall expected time of each training round of the corresponding model to be trained based on a result of the reliability verification by using the trained operator running time prediction model, the cloud comprising a TEE and a target GPU, the TEE being a trusted execution environment; S4, collecting model parameters after a single round of training of different models from the GPU devices to quantize the model parameters of the models in the GPU devices to obtain a parameter group of the corresponding model; S5, training the corresponding model in the target GPU by using the training set, the initial weight and the hyperparameter of the to-be-trained model, starting TEE training and timing according to the pseudo-random seed, generating a quantization checkpoint every iteration during the training process, recording the quantization value and the parameter group of the model parameter of the model under the corresponding quantization checkpoint, and generating a complete checkpoint every training round, recording the model parameter of the model under the corresponding complete checkpoint, is a preset checkpoint interval number; S6, submitting a quantization checkpoint corresponding to each training round to the TEE as a training commitment by the target GPU, and when a deviation between an actual timing time of the TEE and the overall expected time of each training round of the corresponding model to be trained does not exceed a prediction time threshold, sampling according to a preset sampling number in the training commitment and repeatedly verifying the integrity of the model training in the TEE by combining a complete checkpoint, and regarding the model with integrity after repeated verification as a qualified model.

2. The trusted execution environment based model training integrity verification method of claim 1, wherein: An operator sample in the operator training set is represented as a four-tuple of <type, configuration parameter, micro-benchmark time, running time>, the micro-benchmark time being a time corresponding to a performance index mapping of the operator, that is, a ratio of an operator calculation amount to a theoretical calculation speed of the GPU device or a ratio of an operator read-write amount to a theoretical read-write speed of the GPU device, and the type and the configuration parameter being derived from a type and a configuration parameter of the operator corresponding to the model parameters. 3.The trusted execution environment based model training integrity verification method of claim 1, wherein: The operator running time prediction model is an MLP model or a random forest model.

4. The trusted execution environment based model training integrity verification method of claim 1, wherein: The operator comprises a matrix multiplication, a convolution layer, a ReLU activation function, and an LSTM module.

5. The trusted execution environment based model training integrity verification method of claim 1, wherein: The reliability verification in the cloud according to the operator training set, the model to be trained, and the corresponding hyperparameters, and the obtaining of the overall expected time of each training round of the corresponding model to be trained based on the result of the reliability verification by using the trained operator running time prediction model are as follows: S31, extracting corresponding operators according to the model to be trained, and matching an operator sample with the highest cosine similarity in the operator training set under the TEE; S32, randomly generating input parameters of the operators of the model to be trained as micro-benchmark test tasks of the corresponding operators; S33, assigning the micro-benchmark test tasks of the operators to the target GPU for testing, and simultaneously obtaining execution results of the micro-benchmark test tasks of the corresponding operators and actual timing times by using the TEE; S34, return the test result of the target GPU of each operator to the TEE for reliability verification, when the target GPU has reliability, input the to-be-trained model and corresponding hyperparameters, micro-benchmark time of the corresponding operator into the trained operator running time prediction model, obtain the predicted running time of the corresponding operator, add the predicted running time of all operators of the to-be-trained model to obtain the overall expected time of each training round of the corresponding to-be-trained model; When the target GPU does not have reliability, input the to-be-trained model and corresponding hyperparameters, and all-zero micro-benchmark time of the corresponding operator into the input parameters of the trained operator running time prediction model, obtain the predicted running time of the corresponding operator, and add the predicted running time of all operators of the to-be-trained model to obtain the overall expected time of each training round of the corresponding to-be-trained model.

6. The trusted execution environment based model training integrity verification method of claim 5, wherein: The reliability verification is specifically as follows: S341, judge whether the error between the execution result of the micro-benchmark test task of the corresponding operator and the test result of the target GPU is within a first preset threshold, if yes, consider that the corresponding operator has execution correctness, and execute step S342, otherwise, consider that the corresponding operator does not have execution correctness, and the target GPU does not have reliability; S342, judge whether the ratio of the micro-benchmark time of the corresponding operator to the actual timing time is within a second preset threshold, if yes, consider that the real performance of the corresponding operator can be reflected, and the target GPU has reliability, otherwise, there is fraud, and it is considered that the real performance of the corresponding operator cannot be reflected, and the target GPU does not have reliability.

7. The trusted execution environment based model training integrity verification method of claim 1, wherein: The model parameters of each model in the GPU device are quantized to obtain a parameter group of the corresponding model, and the specific process is as follows: S41, according to the model parameters collected from each GPU device after single round training of different models, the quantization hash values of the network levels corresponding to each model are obtained respectively, and the operation is as follows: S411. For each network layer, generate multiple candidate quantization parameter pairs, then the... Each candidate quantization parameter pair The formula is as follows: , ； wherein, is a quantization step size, is a zero point, , is a total number of candidate quantization parameter pairs, b is a variable, and has a value range of 2-32, indicating a quantization bit width of a corresponding network level, is a numerical range of a current network level, is a minimum value of a current network level, is a maximum value of a current network level; S412、calculate the quantization value of each candidate quantization parameter to the parameter of the corresponding network level under each GPU device, then under the first GPU device, the quantization value of the first parameter of the first network level The formula is as follows: ； wherein, denotes a rounding function, is the l-th parameter of the j-th network level of the corresponding model at the i-th GPU device, wherein, , is the total number of parameters of the corresponding network level, , is the total number of network levels of the corresponding model, , is the total number of GPU devices, , is the total number of parameters of the corresponding network level. S413, Calculate the pair of candidate quantization parameters at the 1st... The first network layer under the first GPU device Quantized hash value of each parameter The formula is as follows: ； wherein denotes a SHA-3 cryptographic hash function; S42, judge whether there is a candidate quantization parameter pair that makes the quantization hash values of the corresponding network levels of the same model in all GPU devices completely consistent, if yes, take the candidate quantization parameter pair as the parameter group of the corresponding model , that is, the global quantization configuration parameter, if not, consider that the model is incomplete, and end the process.

8. The trusted execution environment based model training integrity verification method of claim 1, wherein: The target GPU submits the corresponding quantization checkpoint of each training round to the TEE as a training commitment, and when the deviation between the actual timing time of the TEE and the overall expected time of each training round of the corresponding to-be-trained model does not exceed the prediction time threshold, the sampling number is sampled in the training commitment according to the preset sampling number and the integrity of the model training is repeatedly verified in the TEE combined with the complete checkpoint, wherein the repeated verification process of each training round is as follows: S61, collect, by the target GPU, hash values of all quantization checkpoints of the current training round to generate a training commitment, the training commitment is represented as follows: ； wherein, denotes a SHA-3 cryptographic hash function, denotes the model parameters of the corresponding model at the mthquantization checkpoint, m = 1 , is the number of iterations for the current training epoch; S62, compare in the TEE whether the deviation between the actual timing time of the TEE and the overall expected time of each training round of the corresponding to-be-trained model exceeds the prediction time threshold, if not, consider that the performance of the corresponding model is compliant, randomly extract the quantization checkpoint based on the random number generator in the training commitment according to the preset sampling number, and restore the model parameters of the corresponding training round according to the quantization values and parameter groups of the model parameters of the extracted quantization checkpoint, load the subset of the training set of the to-be-trained model associated with the extracted quantization checkpoint, and execute step S63 in the order determined by the pseudo-random seed and the hyperparameters, otherwise, consider that the performance of the corresponding model is not compliant, and execute step S65; S63, in the TEE, starting from a previous quantization checkpoint of each extracted quantization checkpoint, performing a training process to calculate a hash value of a new quantization checkpoint As a proof, the formula is as follows: ； wherein, denotes the model parameters of the corresponding model at the s-th quantization checkpoint, denotes the model parameters of the corresponding model at the t-th quantization checkpoint, ; S64, compare the hash values of the new quantization checkpoint respectively With training commitment The hash value of the corresponding index in the middle is consistent, if consistent, confirm the integrity of the current training round of the corresponding model through the remote proof signature, that is, the model training has integrity, otherwise, select the previous complete checkpoint of the current sampled quantization checkpoint as the starting point to start the training process, compare the cosine distance gap of the model parameters before and after training Whether to meet , Indicates the upper bound of the global error, if yes, it is considered that the model training has integrity, and the process is ended, otherwise, it is considered that the model training does not have integrity, and step S65 is executed; S65, suspending the target GPU training, reserving a complete checkpoint of the last training round, then restoring the model parameters and the order determined by the pseudo-random seed to the model parameters corresponding to the complete checkpoint of the last training round, and recording cloud default evidence, the cloud default evidence including the running time of the current training round, the training commitment and the hash value of the new quantitative checkpoint, ending the process.

9. The trusted execution environment based model training integrity verification method of claim 8, wherein: the global error upper bound , as follows: ； ； wherein the vector of model parameters after single round of training of the i-th GPU device is [ai, a2, …, an, …, aN], the vector of model parameters after single round of training of the j-th GPU device is [b1, b2, …, bn, …, bN], an is the n-th parameter in the vector of model parameters after single round of training of the i-th GPU device, bn is the n-th parameter in the vector of model parameters after single round of training of the j-th GPU device, N is the total dimension of parameters of the corresponding model, and the cosine similarity of a and b, is the total number of GPU devices.​​​​ 10. The trusted execution environment based model training integrity verification method of claim 8, wherein: After recording the cloud default evidence, it is further judged whether to continue training, if yes, the preset sampling number is increased, and step S63 is returned to restart the target GPU training, otherwise, the process is ended.

Images