A method for confidential inference of large models based on TDX and GPU collaboration
By establishing an authenticated and protected communication channel and an adaptive scheduling mechanism between Intel TDX and GPU, the problems of cross-domain data transmission and operator scheduling are solved, enabling efficient data transmission and computation for confidential inference of large models, significantly reducing latency and increasing throughput.
Patent Information
- Application Number
- CN202611079229.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-20
- Publication Date
- 2026-11-17
AI Technical Summary
There are difficulties in performing efficient cross-domain data transfer and operator scheduling between the trusted domain and untrusted GPU of Intel TDX, resulting in low efficiency of TDX and GPU collaborative inference, especially in the process of large model inference, where it is difficult to ensure both data confidentiality and efficient computation at the same time.
By remotely authenticating trusted domains, establishing authentication protection communication channels, initializing cross-domain communication paths, and performing hardware profiling and scheduling parameter calibration, the execution domain is dynamically selected. Data transmission is protected using a one-time mask resource pool, achieving operator-level adaptive scheduling and reducing cross-domain communication overhead and encryption/decryption latency.
Significantly reduces cross-domain communication overhead, accelerates encryption and decryption stages, reduces unnecessary domain switching and intermediate result transport, enables load and hardware adaptive linear operator offloading decisions, improves end-to-end inference efficiency, reduces latency and increases throughput.
Smart Images

Figure CN123122791A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of large model inference acceleration technology under Confidential Computing and Trusted Execution Environment (TEE); more specifically, it relates to a method and system for establishing confidential and high-throughput cross-domain communication and performing cross-domain scheduling of large model operators between the Trusted Domain (TD) of Intel TDX and untrusted GPUs. Background Technology
[0002] With the deployment of large models, especially large language models (LLMs), in privacy-sensitive scenarios such as healthcare and finance, it is crucial to protect the confidentiality of user input prompts during inference, as well as the intellectual property of model parameters / weights—the model's own intellectual property—from being spied on or tampered with by the host platform. To address this, existing research generally employs CPU-side TEEs (such as Intel SGX, Intel TDX, AMD SEV / SEV-SNP, etc.) to provide hardware isolation, preventing privileged software such as the host OS / Hypervisor from directly inspecting or tampering with the code and data within the TEE, thereby achieving confidential inference.
[0003] On the other hand, the large parameter scale of LLM makes GPUs crucial for LLM inference, but traditional CPU-GPU communication relies on Direct Memory Access (DMA) to access host memory. In Trust Domain Extensions (TDX), the trusted domain memory is encrypted and isolated, making it impossible for untrusted peripheral devices (such as non-confidential GPUs) to directly access the private memory data of the trusted domain via DMA as in the conventional way. This makes direct and efficient collaboration between TDX and GPU very difficult.
[0004] The current approach to using TEE and GPU for collaborative inference of deep neural networks (DNN) is usually as follows: security-critical but computationally lightweight nonlinear operators are kept inside the TEE for execution, while computationally intensive linear operators (matrix multiplication / projection) are offloaded to an untrusted GPU for execution. Lightweight intermediate data protection measures (such as using one-time pad encryption and decryption algorithms, OTP) are used to protect intermediate activation and weight-related data transmitted across domains, so as to achieve inference that balances security and performance.
[0005] Compared to earlier Intel SGX (Software Guard Extensions), Intel TDX expands the scope of memory protection to the entire machine's memory, extending the trusted computing base from the "secure enclave" of SGX to support "confidential virtual machines." Moreover, unlike SGX, which requires users to manually write adaptation code, TDX ensures that programs run transparently internally (with virtually no need for user modification of application logic), theoretically making it more suitable as the foundation for modern LLM confidential inference. However, there is currently a lack of system frameworks that can efficiently achieve TDX-GPU collaboration, especially in terms of efficient cross-domain data transfer and operator scheduling between GPU and TDX, where key obstacles remain. Summary of the Invention
[0006] The present invention aims to at least partially solve one of the technical problems existing in the prior art.
[0007] Therefore, the purpose of this invention is to provide a large-scale confidential inference method based on TDX and GPU collaboration, which achieves efficient cross-domain data transmission and operator-level adaptive scheduling while ensuring that data is always protected, so as to reduce end-to-end inference latency and improve batch processing throughput.
[0008] To achieve the above objectives, the present invention adopts the following technical solution:
[0009] The first aspect of this invention provides a method for confidential inference of large models based on TDX and GPU collaboration, comprising:
[0010] Remote authentication is performed on the trusted domain and an authentication protection communication channel is established. The inference parameters of the large model to be inferred are securely transmitted to the private memory of the trusted domain. A cross-domain communication path for data interaction between the trusted domain and the untrusted GPU is initialized. The underlying hardware computing power and cross-domain transmission characteristics are profiled offline and the scheduling parameters are calibrated. A one-time mask resource for GPU offload protection is initialized.
[0011] After receiving the user's input request, the input request is sent to the trusted domain via the authentication protection communication channel; during the execution inference of the large model, the operators in the computation graph are identified by type and structural constraint analysis is performed. Among them, nonlinear operators are determined to be executed in the trusted domain, and linear operators are selected based on the calibrated hardware parameters under the premise of satisfying the structural same-domain constraint, so as to determine whether the linear operators are executed in the trusted domain or on the untrusted GPU.
[0012] For operators determined to be executed within the trusted domain, the computation is performed directly within the trusted domain. For operators determined to be offloaded to the untrusted GPU, the operands involved in the offloading are protected by a one-time mask within the trusted domain, and then sent to the untrusted GPU for mask computation via the cross-domain communication channel. After the result is returned to the trusted domain, recovery processing is performed to obtain the plaintext intermediate result and continue subsequent inference until the final output is generated. The final output is then returned to the user through the authentication protection communication channel.
[0013] In some embodiments, the remote authentication of the trusted domain and the establishment of an authentication-protected communication channel includes:
[0014] Before the inference task begins, TDX authentication evidence is generated from the trusted domain, and a temporary communication key pair is generated. The large model owner verifies whether the authentication evidence is consistent with the expected trusted domain metric. After the verification is successful, an authentication-protected communication channel is established based on the temporary communication key.
[0015] In some embodiments, the initialization of the cross-domain communication path for data interaction between the trusted domain and the untrusted GPU includes:
[0016] A shared relay buffer is established between the trusted domain and the untrusted GPU, and runtime resources for the communication path corresponding to the shared relay buffer are initialized.
[0017] The shared relay buffer serves as a data relay area between the trusted domain and the untrusted GPU, carrying protected input data written from the trusted domain and protected result data returned from the untrusted GPU.
[0018] In some embodiments, the offline profiling and scheduling parameter calibration of the underlying hardware computing power and cross-domain transmission characteristics includes:
[0019] Offline tests were conducted on the computational capabilities of the trusted domain, the cross-domain transmission bandwidth between the shared relay buffer and the untrusted GPU, the GPU-side computing power throughput, and the execution overhead of different types of operators to obtain hardware profile parameters. Combined with the operator morphology and tensor dimension in the large model, the scheduling parameters used for execution domain selection were calibrated to obtain the scheduling configuration for use in the online inference stage.
[0020] In some embodiments, the one-time mask resource pool is deployed within the trust domain, and the one-time mask resource pool stores random masks and corresponding recovery items, correction items, or equivalent recovery information.
[0021] In some embodiments, the selection of the execution domain for the linear operator based on calibrated hardware parameters, while satisfying structural co-domain constraints, includes:
[0022] The linear operators in the computation graph are used as candidate operators, and the structural coupling relationship between the candidate operators and adjacent nonlinear operators is analyzed.
[0023] If a candidate operator and its adjacent nonlinear operators form a strongly coupled computational graph subgraph where runtime splitting would significantly increase latency, then the candidate operator and the related nonlinear operators are constrained to execute within the trusted domain.
[0024] If a candidate operator does not form a strongly coupled subgraph with its adjacent nonlinear operators, based on the hardware profile parameters obtained from offline calibration and the inference load characteristics at runtime, the total latency of the candidate operator when executed in the trusted domain and when it is offloaded to the untrusted GPU via the cross-domain communication path is estimated, and the execution domain with the minimum total latency is selected.
[0025] In some embodiments, obtaining the plaintext intermediate result based on an operator determined to be offloaded to the untrusted GPU includes:
[0026] Within the trusted domain, the one-time mask resource pool is invoked to perform one-time masking on the input activation, weight matrix, or corresponding operands involved in the calculation of the current operator, thereby obtaining protected operands.
[0027] The protected operand is written from the private memory of the trusted domain to the cross-domain communication path. The untrusted GPU initiates a DMA access to the cross-domain communication path, transfers the protected operand to the buffer or video memory of the untrusted GPU, and the untrusted GPU performs the corresponding mask linear calculation to obtain the protected output result.
[0028] The untrusted GPU transmits the protected output result back to the trusted domain via the cross-domain communication path. The trusted domain performs demasking or recovery processing on the protected output result based on the recovery term, correction term, or equivalent recovery information associated with the current operator to obtain the plaintext intermediate result.
[0029] In some embodiments, during the data transfer and protection processing stage between the trusted domain and the cross-domain communication path, single-core or multi-core parallel processing is dynamically selected according to the size of the data to be transferred, and a copy strategy that bypasses the untrusted GPU cache is adopted for streaming transfer of data blocks larger than the hardware L1 cache.
[0030] A second aspect of the present invention provides a large-model confidential inference device based on TDX and GPU collaboration, comprising:
[0031] The environment configuration module is configured to remotely authenticate the trusted domain and establish an authentication protection communication channel, securely transmit the inference parameters of the large model to be inferred to the private memory of the trusted domain, initialize the cross-domain communication path for data interaction between the trusted domain and the untrusted GPU, perform offline profiling and scheduling parameter calibration of the underlying hardware computing power and cross-domain transmission characteristics, and initialize a one-time mask resource for GPU offload protection.
[0032] The execution domain decision module is configured to send the received input request into the trusted domain via the authentication protection communication channel; during the large model inference process, the operators in the computation graph are subjected to type identification and structural constraint analysis, wherein nonlinear operators are determined to be executed in the trusted domain, and linear operators are selected based on the calibrated hardware parameters under the premise of satisfying the structural same-domain constraint, so as to determine whether the linear operator is executed in the trusted domain or on the untrusted GPU;
[0033] The inference and output module is configured to perform calculations directly within the trusted domain for operators determined to be executed within the trusted domain; for operators determined to be offloaded to the untrusted GPU for execution, the module performs one-time mask protection on the operands involved in the offloading within the trusted domain, sends them to the untrusted GPU for mask calculation via the cross-domain communication channel, and performs recovery processing after the result is returned to the trusted domain to obtain plaintext intermediate results and continue subsequent inference until the final output is generated; the final output is returned to the user through the authentication protection communication channel.
[0034] A third aspect of the present invention provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes a large-scale confidential inference method according to any embodiment of the first aspect of the present invention.
[0035] This invention has the following characteristics and beneficial effects:
[0036] 1) Significantly reduced cross-domain communication overhead: By adopting the above technical solution, efficient confidential large-scale model inference across trusted and untrusted hardware can be achieved without relying on dedicated trusted GPU hardware. The host bypass zero-copy communication path significantly shortens the cross-domain transmission link and reduces CPU involvement by eliminating redundant CPU-side memory copies and retaining only the necessary encryption / decryption steps, thereby alleviating the high transmission latency and I / O serialization bottlenecks caused by the traditional host proxy path.
[0037] 2) Accelerate the unavoidable encryption / decryption phase: The hybrid parallel cross-domain secure transmission mechanism accelerates the necessary encryption / decryption transfer process by dynamically parallelizing the number of cores and adopting a copy strategy with a bypass cache, thereby reducing encryption / decryption latency and minimizing interference with the TDX's internal inference computing cache, thus providing more stable high-throughput cross-domain transmission capabilities under different workload scales.
[0038] 3) Reduce unnecessary domain switching and intermediate result transfer: In terms of static operator scheduling, the structure-aware collaborative scheduling proposed in this invention allocates the attention-tightly coupled related structures to the TDX for execution as a whole, reducing unnecessary frequent cross-domain data transfers before and after nonlinear operations, while avoiding task scheduling to GPU execution due to online computation of decryption operations under a lightweight intermediate data protection mechanism.
[0039] 4) Implement load / hardware adaptive linear operator offloading decision: Hardware-aware dynamic scheduling decomposes the total latency of linear operators into three parts: loading, computation, and storage, and makes decisions based on bandwidth and throughput estimates. This enables the system to select the better execution domain under different scales of computational tasks such as GEMM and GEMV, thereby improving end-to-end inference efficiency.
[0040] 5) In real TDX and GPU hardware evaluation, the present invention can achieve a maximum reduction of approximately 80.9% in end-to-end generation latency in online single-request inference scenarios and a maximum acceleration of approximately 3.7 times in offline batch inference scenarios, thus demonstrating that confidentiality and high efficiency can be achieved simultaneously under heterogeneous trusted and untrusted hardware conditions. Attached Figure Description
[0041] Figure 1 This is an overall flowchart of a large model confidential inference method based on TDX and GPU collaboration provided in the first aspect embodiment of the present invention;
[0042] Figure 2 yes Figure 1 The diagram illustrates the specific implementation process of the method shown.
[0043] Figure 3 yes Figure 1 A schematic diagram of the TDX-GPU cross-domain communication path constructed in the method shown;
[0044] Figure 4 This is a comparison chart of end-to-end latency for online small-batch inference. Green represents our method, and other colors represent the baseline method. The horizontal axis represents the batch size, and the vertical axis represents the latency (unit: seconds).
[0045] Figure 5 This is a comparison chart of end-to-end latency for large-scale inference. Green represents our method, and other colors represent the baseline method. The horizontal axis represents the batch size, and the vertical axis represents the latency (unit: seconds).
[0046] Figure 6 This is a schematic diagram of the structure of an electronic device provided in a third aspect embodiment of the present invention. Detailed Implementation
[0047] To make the objectives, technical solutions, and advantages of this application clearer, the application will be described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application.
[0048] Conversely, this application covers any alternatives, modifications, equivalent methods, and schemes made within the spirit and scope of this application as defined by the claims. Furthermore, to provide the public with a better understanding of this application, certain specific details are described in detail below. However, this application can be fully understood by those skilled in the art even without these detailed descriptions.
[0049] TDX, based on Trusted Domain (TD), Secure Arbitration Mode (SEAM), memory encryption and access control, and Remote Attestation, is theoretically more suitable as the foundation for modern LLM confidential inference. However, extending TEE and GPU to LLM for collaborative inference presents the following significant problems:
[0050] 1. The communication path between TDX and GPU is infeasible or inefficient.
[0051] TDX's private memory has a DMA access protection mechanism for peripheral devices, making data transfer impossible using the conventional CPU-GPU communication path. A straightforward alternative, using the host machine as a proxy, involves copying data from TDX's private memory to shared memory, then to pinned host memory, and finally via DMA to the GPU; the return transfer also requires a reverse multi-level copy. This path introduces significant CPU involvement and redundant memory copying, leading to I / O serialization and consequently, substantial latency. Experiments show that this simple proxy design, compared to performing large model inference directly within the TEE, not only fails to achieve the expected acceleration using the GPU but also introduces approximately 6–7 times the original latency, almost entirely due to data transfer.
[0052] 2. The scheduling strategy of offloading all linear operators to the GPU is not the optimal strategy.
[0053] On the one hand, LLM inference involves complex coupling of linear and nonlinear operators. If all linear operators are unloaded onto the GPU indiscriminately, frequent cross-domain data transfers will occur, thereby weakening or offsetting the benefits of scheduling to the GPU and adding cross-domain transfer costs. In some schemes, this transfer strategy can account for a considerable proportion of the total inference time (up to 95%), and may even bring higher additional overhead than performing large model inference within TDX.
[0054] On the other hand, the benefits of linear operator offloading are strongly input- and scale-dependent. In large-scale matrix multiplication operations (such as GEMM), the time saved by GPU computation is more advantageous than the overhead caused by cross-domain data transfer. In small-scale matrix-vector multiplication operations (such as GEMV), the computational overhead caused by cross-domain transfer may dominate the overall inference latency, thus offsetting the computational speedup benefits of scheduling computation to the GPU. Therefore, the optimal execution domain (TDX or GPU) should be dynamically adjusted according to workload characteristics, rather than adopting static execution rules.
[0055] In view of this, this invention addresses the privacy-preserving inference scenario of large models using a CPU-side Trusted Execution Environment (TEE) and an untrusted, non-confidential GPU. It employs Intel TDX as the Trusted Domain (TD) to protect the confidentiality and integrity of prompts, model parameters, and intermediate inference data. Simultaneously, it leverages the high throughput of the unconfidential GPU to reduce latency in the large model inference process. Since TD memory has visibility limitations on DMA access to peripheral devices, the traditional CPU-GPU path cannot be directly reused in the TDX scenario. Therefore, this invention establishes a TDX-GPU collaborative computing framework and data transmission path. While ensuring data protection at all times, it achieves efficient cross-domain data transmission and operator-level adaptive scheduling to reduce end-to-end inference latency and improve batch processing throughput.
[0056] See Figure 1 This invention provides a method for confidential inference of large models based on TDX and GPU collaboration, the method comprising the following steps:
[0057] Step S100: Perform confidential inference runtime environment initialization
[0058] Remote authentication is performed on the target trusted domain (hereinafter referred to as the "trusted domain") and an authentication protection communication channel is established. The parameters of the model to be inferred are securely transmitted to the private memory of the trusted domain. A cross-domain communication path for data interaction between the trusted domain and the untrusted GPU is initialized (hereinafter also named the cross-domain communication path as the "shared relay buffer"). The underlying hardware computing capabilities and cross-domain transmission characteristics are offline profiled and the scheduling parameters are calibrated. A one-time mask resource for GPU offload protection is initialized.
[0059] Step S200: Perform operator-level execution domain decision for the input request.
[0060] After receiving the user's input request, the input request is sent to the trusted domain via the authentication-protected communication channel. During the model inference process, the operators in the computation graph are identified by type and analyzed by structural constraints. Nonlinear operators are determined to be executed in the trusted domain. For linear operators, the execution domain is selected based on the calibrated hardware parameters under the premise of satisfying the structural co-domain constraints, so as to determine whether the linear operator is executed in the trusted domain or on the untrusted GPU.
[0061] Step S300: Complete collaborative confidential reasoning according to the execution domain decision and return the output result.
[0062] For operators determined to be executed within the trusted domain, computation is performed directly within the trusted domain. For linear operators determined to be offloaded to an untrusted GPU, the operands involved in the offloading are protected by a one-time mask within the trusted domain, sent to the untrusted GPU for mask computation via a cross-domain communication channel, and restored after the result is returned to the trusted domain to obtain the plaintext intermediate result and continue subsequent inference until the final output is generated. The final output is then returned to the user through an authentication-protected communication channel.
[0063] In some embodiments, see Figure 2 In the upper flowchart, step S100 runs in the offline phase, specifically including:
[0064] Step S110: Perform remote authentication of the trusted domain and establish an authentication-protected communication channel.
[0065] Before the inference task begins, the deployer or the owner of the large model performs remote attestation on the trusted domain to be used for the inference task. Specifically, the trusted domain generates TDX authentication evidence and a temporary communication key pair. The owner of the large model verifies whether the authentication evidence matches the expected trusted domain metric. Upon successful verification, an authentication-protected communication channel is established based on the temporary communication key, for example, a secure transmission channel between the trusted domain and an external entity based on RA-TLS (Remote Attestation Transport Layer Security). This authentication-protected communication channel ensures that subsequent transmission of large model parameters and user input is only directed to the verified trusted execution instance.
[0066] Step S120: Securely transmit large model parameters to trusted domain private memory through the authentication-protected communication channel to complete model provisioning.
[0067] After successful remote authentication, the inference parameters of the large model to be inferred are transmitted to the trusted domain through the authentication-protected communication channel and stored in the private memory of the trusted domain. This ensures that the large model parameters are protected by TDX hardware during loading, residence, and subsequent inference. As a result, the host operating system, virtual machine monitor, and other software or devices outside the trusted domain cannot directly access the plaintext content of the large model parameters.
[0068] Step S130: Initialize the TDX-GPU cross-domain communication path.
[0069] See Figure 3 ( Figure 3 The gray-covered area represents the simplest method for implementing confidential inference in large models, using the host machine as a GPU-TDX communication relay path. In this embodiment, this path is replaced with the shared relay buffer shown below. A shared relay buffer is established between the trusted domain and the untrusted GPU, and the runtime resources of the communication path corresponding to this shared relay buffer are initialized. The shared relay buffer serves as a data relay area between the trusted domain and the untrusted GPU, carrying protected input data written from the trusted domain and protected result data returned from the GPU. The communication path runtime resources support cross-domain data flow via trusted domain → shared relay area → GPU and GPU → shared relay area → trusted domain. Preferably, the shared relay buffer only serves as a temporary storage and forwarding area and does not participate in the inference computation of large models.
[0070] Step S140: Perform offline profiling of the underlying hardware characteristics and calibrate scheduling parameters.
[0071] Before the large model service officially processes input requests, offline tests are conducted on the computing power of the trusted domain side, the cross-domain transmission bandwidth between the shared relay buffer and the GPU, the GPU-side computing power throughput, and the execution overhead of different types of operators to obtain hardware profiling parameters. Then, combined with the operator morphology in the target large model (such as the projection operator in the attention module, matrix multiplication related to the calculation of attention scores) and tensor dimensions (such as the sequence length in the pre-filling stage and the decoding stage), the scheduling parameters (data transmission bandwidth parameters and computing throughput parameters) used for execution domain selection are calibrated to obtain the scheduling configuration for use in the online inference stage.
[0072] Optionally, calibrating the scheduling parameters includes:
[0073] 1) Extract operator description information of the linear operators to be scheduled in the target model, including operator type, input tensor dimension, output tensor dimension, batch size, sequence length, weight scale, etc.
[0074] 2) Construct test tasks corresponding to the operator description information;
[0075] 3) Execute test tasks on both the trusted domain side and the GPU side to obtain execution latency and effective throughput;
[0076] 4) Perform data transfer tests on cross-domain transmission paths to obtain the effective bandwidth and fixed overhead under different directions and data scales;
[0077] 5) Based on execution latency, effective throughput, and effective bandwidth, establish latency estimation models for various linear operators under trusted domain execution and GPU offload execution conditions;
[0078] 6) Based on the estimated total latency difference of the same operator in different execution domains, generate execution domain selection rules, lookup tables, threshold configurations, or segmented scheduling parameters. During the online inference phase, determine the load of the inference task. If it is less than the threshold in the scheduling configuration, execute directly in the trusted domain; if it is greater than the threshold, schedule it to execute on an untrusted GPU.
[0079] Step S150: Initialize the one-time mask pool (OTP Mask Pool Initialization).
[0080] To protect sensitive tensors involved in the offloading of linear operators when they are subsequently offloaded to untrusted GPUs for execution, a streaming one-time mask resource pool is initialized within the trusted domain. For large model weights or related operands suitable for offloading, random masks and corresponding recovery terms, correction terms, or equivalent recovery information are pre-generated and stored in the one-time mask resource pool for on-demand access during online inference. The one-time mask resource pool can be maintained using a combination of pre-generation and runtime incremental expansion to ensure continuous availability while reducing online preparation overhead.
[0081] In some embodiments, see Figure 2 The lower flowchart, step S200, runs in the online phase and specifically includes:
[0082] Step S210: Send the user input to the authenticated and protected communication channel into the trusted domain.
[0083] During the online inference phase, the user sends an input request to the trusted domain. The input request can be a prompt, a text sequence, audio input, image input, or a combination thereof. The input request enters the trusted domain in encrypted form via the aforementioned authentication-protected communication channel, and after preprocessing within the trusted domain, it is sent to the computation graph of the target large model.
[0084] Step S220: Classify the operators in the computation graph by type.
[0085] During the execution of the large model, the operators in the computation graph are identified by type. Operators that satisfy linear transformation forms, such as matrix multiplication, linear projection, and fully connected mapping, are identified as linear operators. Normalization operators, activation function operators, softmax operators, and other nonlinear operators are identified as nonlinear operators. For nonlinear operators, their execution is fixed within the trusted domain based on safety constraints. For linear operators, they are entered as candidate operators into the subsequent execution domain determination process.
[0086] Step S230: Determine the non-unloadable strongly coupled operator cluster based on the structure-aware co-domain constraints.
[0087] For linear operators that are identified as candidate operators in the computation graph, further analyze their structural coupling relationship with adjacent nonlinear operators. If a candidate operator and its adjacent nonlinear operators form a strongly coupled computation graph subgraph that would cause a significant increase in latency due to runtime splitting, then the candidate operator and the related nonlinear operators are constrained to execute within the trusted domain.
[0088] For example, in the Transformer attention module, the linear operators QK (attention score calculation) and AV (attention weighted calculation) are separated by the nonlinear operator Softmax. If all linear operators are simply scheduled to the GPU for execution, Softmax needs to be executed within the trusted domain, which causes the intermediate results before and after Softmax to frequently travel between the trusted domain and the GPU, generating additional cross-domain transfers. At the same time, under OTP protection, some operators (such as attention score-related calculations) may introduce online correction terms or recovery-related additional computational burdens, making it possible that scheduling them to the GPU does not necessarily yield the expected benefits. Therefore, the strongly coupled links such as QK–Softmax–AV are kept as a strongly coupled computation graph subgraph and executed within the trusted domain to reduce the number of domain switching and the amount of cross-domain data transfer.
[0089] Step S240: Perform hardware- and load-aware dynamic scheduling on the remaining candidate operators.
[0090] For candidate operators not subject to the aforementioned strong coupling and co-domain constraints, based on the hardware profile parameters obtained from offline calibration and the inference load characteristics during large-scale model runtime, the total latency of the candidate operator during execution in the trusted domain and offloaded to GPU execution is estimated, and the execution domain with the smaller total latency is selected. The total latency includes data loading latency, operator computation latency, and result write-back latency. The data loading latency and result write-back latency are determined by the amount of data in the relevant tensors and the effective bandwidth of cross-domain links or local memory links, while the operator computation latency is determined by the amount of operator computation and the effective throughput on the trusted domain side or GPU side. This enables dynamic adaptive selection of the execution domain for linear operators.
[0091] Step S250: Generate the execution domain decision results for each operator.
[0092] After completing the operator type classification, structural constraint analysis, and latency estimation, execution domain decision results are generated for each operator in the large model computation graph corresponding to the current input request. These execution domain decision results indicate whether each operator is executed locally within the trusted domain or offloaded to an untrusted GPU via the TDX-GPU communication path.
[0093] In some embodiments, see Figure 2 The lower flowchart, step S300, specifically includes:
[0094] S310: Proceed model execution step by step according to the execution domain decision.
[0095] Based on the execution domain decision results obtained in step S200, operators are executed sequentially according to the large model computation graph or decoding step order. Wherein:
[0096] If the current operator is determined to be executed within a trusted domain, the computing resources within the trusted domain are directly invoked to complete the computation of the current operator, and the output is used as the input of subsequent operators.
[0097] If the current linear operator is determined to be unloaded to an untrusted GPU for execution, then steps S320 to S340 are executed sequentially.
[0098] Step S320: Perform one-time mask protection on the linear operators determined to be offloaded to the untrusted GPU.
[0099] Within the trusted domain, a one-time mask resource pool is invoked to perform one-time masking on the input activation, weight matrix, or corresponding operands involved in the computation of the current linear operator, resulting in protected operands. This one-time masking can be implemented using a combination of online lightweight transformation and offline pre-computation of correction terms, enabling the untrusted GPU to perform corresponding linear computations on the protected operands without accessing the plaintext.
[0100] S330: Sends protected operands to an untrusted GPU via the TDX-GPU cross-domain communication path and performs mask calculation.
[0101] The protected operands obtained in step S320 are written from the trusted domain private memory to the shared relay buffer. Then, the host-side transmission component controls the protected operands to be transferred from the shared relay buffer to the untrusted GPU-side buffer or video memory via DMA. The GPU then performs the corresponding mask linear calculation to obtain the protected output result. Preferably, in the data transfer and protection processing stage between the trusted domain private memory and the shared relay buffer, single-core or multi-core parallel processing is dynamically selected according to the size of the data to be transferred. For data blocks larger than the hardware L1 cache size (such as data blocks larger than the L1 cache size), a copy strategy that bypasses the cache is adopted for streaming transfer to reduce CPU cache pollution and improve cross-domain transmission throughput.
[0102] Step S340: The protected output is sent back to the trusted domain and restored.
[0103] After the untrusted GPU completes the masked linear computation, it sends the protected output back to the trusted domain via a shared relay buffer. The trusted domain performs demasking or recovery processing on the protected output based on the recovery term, correction term, or equivalent recovery information associated with the current linear operator to obtain the plaintext intermediate result, and then inputs the plaintext intermediate result into the subsequent operator to continue inference.
[0104] S350: Repeat the above process S310-S340 until the final output is generated and return safely.
[0105] During autoregressive generation or other stepwise inference processes, the above-described operator-by-operator progression, local execution, offloading protection, cross-domain transfer, GPU computation, and result recovery processes are repeatedly executed until the final output is obtained. The generated final output is returned to the user via an authentication-protected communication channel. During inference, intermediate activations, temporary buffers, and shared relay resources that are no longer dependent on subsequent computations can be reclaimed and reused to reduce trusted domain memory pressure and cross-domain resource consumption.
[0106] Validation of the embodiments of the present invention:
[0107] To verify the effectiveness of the large-model confidential inference method based on TDX trusted domain and GPU collaboration provided in this invention, the scheme of this invention was implemented on a real hardware platform and comparative tests were conducted. The test platform included an Intel Xeon Silver 4514Y processor (32 cores) and 128GB DDR5 memory (5th generation Double Data Rate Synchronous Dynamic Random Access Memory), equipped with an NVIDIA GeForce RTX 4090 graphics card (24GB) connected via a 16-channel high-speed serial bus interface PCIe 4.0 x16; the trusted domain was deployed in a TDX environment, configured with 8 virtual CPUs and 64GB of private memory. The test environment ran the large-model inference program while ensuring data protection within the trusted domain, thereby verifying the feasibility and performance benefits of this invention in a real-world confidential inference deployment scenario.
[0108] Regarding the tested models and inference accuracy, large language models of different sizes and architectures were selected, including Mistral-7B, Mistral-small-24B, Qwen2.5-7B, and Qwen2.5-14B, and BFloat16 precision was used for inference to cover representative real-world inference workloads.
[0109] In terms of workload construction, to reflect real-world application scenarios, the tests used prompt word trajectories sampled from the ChatGPT Prompts dataset and combined with public inference statistics to set up two typical verification scenarios: The first was an online single-request inference scenario, with a batch size of 1, prompt length varying from 32 to 2048 tokens, and generation length varying from 32 to 128 tokens; the second was an offline batch processing inference scenario, with a batch size of 64 or 512, a prompt length of 32 tokens, and a generation length of 128 tokens. The test metric used was end-to-end generation time, reflecting user-perceived latency and system throughput.
[0110] To ensure that the verification results reflect the contribution of each component technique of this invention to the overall performance, multiple comparative schemes were set up for testing, corresponding to: a scheme that performs inference only within the trusted domain (TDX-Only), a collaborative scheme using a non-optimized cross-domain communication path (NaivePath w. Dispatch), a scheme using only communication optimization without employing the scheduling strategy of this invention (OptimizedPath w. Dispatch), a scheme using only the scheduling strategy of this invention without communication optimization (NaivePath w. Dispatch), and the complete scheme of this invention (TrustBridge). Through these comparative settings, the comprehensive improvement effect of this invention on end-to-end inference performance in terms of both "cross-domain communication optimization" and "operator scheduling strategy" can be verified.
[0111] In an online single-request inference verification scenario, test results show that the embodiments of the present invention can effectively reduce end-to-end generation latency under confidential inference constraints: compared with the scheme that only executes within the trusted domain, the end-to-end generation latency of the present invention is reduced by an average of approximately 18.1%, and by up to approximately 80.9% under certain model and input settings. This result verifies that the embodiments of the present invention, by constructing a TDX-GPU cross-domain communication path and combining it with operator-level execution domain decision-making, can improve confidential inference efficiency in latency-sensitive scenarios. See the experimental results below. Figure 4 .
[0112] In offline batch processing inference verification scenarios, test results further demonstrate that the embodiments of the present invention can achieve stable acceleration in throughput-oriented scenarios: with batch sizes of 64 and 512, compared to schemes that execute only within the trusted domain, the average end-to-end generation latency of the embodiments of the present invention is reduced by approximately 29.8% (corresponding to approximately 1.43 times speedup), and the maximum end-to-end generation latency is reduced by approximately 73.5% (corresponding to approximately 3.7 times speedup) under certain model settings. These results verify that the embodiments of the present invention can fully utilize the computational throughput of the GPU side in batch processing scenarios and amortize cross-domain transmission overhead, thereby achieving a more significant overall performance improvement. See the experimental results below. Figure 5 .
[0113] Furthermore, ablation analysis demonstrates the necessity and synergy of the key technical measures in this invention's embodiments for performance improvement. Test results show that, keeping other conditions consistent, using only the cross-domain communication optimization mechanism of this invention achieves a maximum speedup of approximately 1.71 times, verifying the significant effect of cross-domain communication path optimization on reducing transmission overhead. Introducing the operator-level execution domain decision of this invention only in the non-optimized system yields an average speedup of approximately 26 times and a maximum of approximately 64.8 times, verifying that this invention's embodiments can significantly improve end-to-end performance by reducing unnecessary cross-domain switching and selecting the execution domain of linear operators. When cross-domain communication optimization and operator-level execution domain decision are enabled simultaneously, the maximum combined benefit is achieved, reaching a maximum speedup of approximately 71.27 times, verifying that this invention's embodiments can achieve significantly better overall results than a single method in confidential inference collaboration scenarios through the combination of communication optimization and scheduling strategies. Ablation experiment results are shown in Table 1.
[0114]
[0115] Table 1 illustrates that the cross-domain communication optimization mechanism and operator-level execution domain decision-making in this embodiment work synergistically to improve system performance. Retaining either one can achieve performance optimization, but the combined use of the two has a more significant optimization effect.
[0116] In summary, the verification results based on real TDX and GPU platforms in both online and offline typical scenarios show that the present invention can significantly reduce end-to-end generation latency and improve batch processing inference efficiency under confidential inference constraints. Specifically, in the online single-request scenario, it can achieve a maximum end-to-end latency reduction of approximately 80.9%, and in the offline batch processing scenario, it can achieve a maximum speedup of approximately 3.7 times. Ablation verification further proves that cross-domain communication optimization and operator-level execution domain decision-making together constitute the key sources of the performance gains obtained by the present invention, thereby verifying the effectiveness and practical value of the embodiments of the present invention.
[0117] A second aspect of the present invention provides a large model confidential inference device based on TDX and GPU collaboration, comprising:
[0118] The environment configuration module is configured to remotely authenticate the trusted domain and establish an authentication protection communication channel, securely transmit the inference parameters of the large model to be inferred to the private memory of the trusted domain, initialize the cross-domain communication path for data interaction between the trusted domain and the untrusted GPU, perform offline profiling and scheduling parameter calibration of the underlying hardware computing power and cross-domain transmission characteristics, and initialize a one-time mask resource for GPU offload protection.
[0119] The execution domain decision module is configured to send the received input request into the trusted domain via the authentication protection communication channel; during the large model inference process, the operators in the computation graph are subjected to type identification and structural constraint analysis, wherein nonlinear operators are determined to be executed in the trusted domain, and linear operators are selected based on the calibrated hardware parameters under the premise of satisfying the structural same-domain constraint, so as to determine whether the linear operator is executed in the trusted domain or on the untrusted GPU;
[0120] The inference and output module is configured to perform calculations directly within the trusted domain for operators determined to be executed within the trusted domain; for operators determined to be offloaded to the untrusted GPU for execution, the module performs one-time mask protection on the operands involved in the offloading within the trusted domain, sends them to the untrusted GPU for mask calculation via the cross-domain communication channel, and performs recovery processing after the result is returned to the trusted domain to obtain plaintext intermediate results and continue subsequent inference until the final output is generated; the final output is returned to the user through the authentication protection communication channel.
[0121] It should be noted that the foregoing explanation of an embodiment of a large model confidential inference method based on TDX and GPU collaboration also applies to the large model confidential inference device based on TDX and GPU collaboration in this embodiment, and will not be repeated here.
[0122] To implement the above embodiments, this invention also proposes a computer-readable storage medium storing a computer program that is executed by a processor to perform a large model confidential inference method based on TDX and GPU collaboration as described in the above embodiments.
[0123] The following is for reference. Figure 6 The diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of the present invention. It should be noted that the electronic device in the embodiments of the present invention may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs, desktop computers, and servers. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.
[0124] like Figure 6As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 102 or a program loaded from a storage device 108 into a random access memory (RAM) 103. The RAM 103 also stores various programs and data required for the operation of the electronic device. The processing unit 101, ROM 102, and RAM 103 are interconnected via a bus 104. An input / output (I / O) interface 105 is also connected to the bus 104.
[0125] Typically, the following devices can be connected to I / O interface 105: input devices 106 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, etc.; output devices 107 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 108 including, for example, magnetic tapes, hard disks, etc.; and communication devices 109. Communication device 109 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have instead.
[0126] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, this embodiment includes a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device 109, or installed from a storage device 108, or installed from a ROM 102. When the computer program is executed by the processing device 101, it performs the functions defined above in the methods of the embodiments of the present invention.
[0127] It should be noted that the computer-readable medium described above in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.
[0128] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.
[0129] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the aforementioned large-model confidential inference method based on TDX and GPU collaboration.
[0130] Computer program code for performing the operations of this disclosure can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and Python, as well as conventional procedural programming languages such as the "C-" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0131] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0132] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.
[0133] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this application pertain.
[0134] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.
[0135] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.
[0136] Those skilled in the art will understand that implementing all or part of the steps of the methods in the above embodiments can be accomplished by instructing related hardware through a program. The developed program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.
[0137] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.
[0138] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.
Claims
1. A method for confidential inference of large models based on TDX and GPU collaboration, characterized in that, include: Remote authentication is performed on the trusted domain and an authentication protection communication channel is established. The inference parameters of the large model to be inferred are securely transmitted to the private memory of the trusted domain. A cross-domain communication path for data interaction between the trusted domain and the untrusted GPU is initialized. The underlying hardware computing power and cross-domain transmission characteristics are profiled offline and the scheduling parameters are calibrated. A one-time mask resource for GPU offload protection is initialized. Upon receiving a user's input request, the input request is sent to the trusted domain via the authentication-protected communication channel; During the inference process of the large model, the operators in the computation graph are identified by type and structural constraint analysis is performed. Nonlinear operators are determined to be executed in the trusted domain. For linear operators, the execution domain is selected based on the calibrated hardware parameters under the premise of satisfying the structural co-domain constraint, so as to determine whether the linear operator is executed in the trusted domain or on the untrusted GPU. For operators determined to be executed within the trusted domain, the computation is performed directly within the trusted domain. For operators determined to be offloaded to the untrusted GPU, the operands involved in the offloading are protected by a one-time mask within the trusted domain, and then sent to the untrusted GPU for mask computation via the cross-domain communication channel. After the result is returned to the trusted domain, recovery processing is performed to obtain the plaintext intermediate result and continue subsequent inference until the final output is generated. The final output is then returned to the user through the authentication protection communication channel.
2. The large-scale confidential reasoning method according to claim 1, characterized in that, The process of remotely authenticating the trusted domain and establishing an authentication-protected communication channel includes: Before the inference task begins, TDX authentication evidence is generated from the trusted domain, and a temporary communication key pair is generated. The large model owner verifies whether the authentication evidence is consistent with the expected trusted domain metric. After the verification is successful, an authentication-protected communication channel is established based on the temporary communication key.
3. The large-scale confidential reasoning method according to claim 1, characterized in that, The initialization of the cross-domain communication path for data interaction between the trusted domain and the untrusted GPU includes: A shared relay buffer is established between the trusted domain and the untrusted GPU, and runtime resources for the communication path corresponding to the shared relay buffer are initialized. The shared relay buffer serves as a data relay area between the trusted domain and the untrusted GPU, carrying protected input data written from the trusted domain and protected result data returned from the untrusted GPU.
4. The large-scale confidential reasoning method according to claim 3, characterized in that, The offline profiling and scheduling parameter calibration of the underlying hardware computing capabilities and cross-domain transmission characteristics include: Offline tests were conducted on the computational capabilities of the trusted domain, the cross-domain transmission bandwidth between the shared relay buffer and the untrusted GPU, the GPU-side computing power throughput, and the execution overhead of different types of operators to obtain hardware profile parameters. Combined with the operator morphology and tensor dimension in the large model, the scheduling parameters used for execution domain selection were calibrated to obtain the scheduling configuration for use in the online inference stage.
5. The large-scale confidential reasoning method according to claim 1, characterized in that, The one-time mask resource pool is deployed within the trust domain, and the one-time mask resource pool stores random masks and corresponding recovery items, correction items, or equivalent recovery information.
6. The large-scale confidential reasoning method according to claim 1, characterized in that, The process of selecting the execution domain for linear operators based on calibrated hardware parameters, while satisfying structural co-domain constraints, includes: The linear operators in the computation graph are used as candidate operators, and the structural coupling relationship between the candidate operators and adjacent nonlinear operators is analyzed. If a candidate operator and its adjacent nonlinear operators form a strongly coupled computational graph subgraph where runtime splitting would significantly increase latency, then the candidate operator and the related nonlinear operators are constrained to execute within the trusted domain. If a candidate operator does not form a strongly coupled subgraph with its adjacent nonlinear operators, based on the hardware profile parameters obtained from offline calibration and the inference load characteristics at runtime, the total latency of the candidate operator when executed in the trusted domain and when it is offloaded to the untrusted GPU via the cross-domain communication path is estimated, and the execution domain with the minimum total latency is selected.
7. The large-scale confidential reasoning method according to claim 1, characterized in that, The plaintext intermediate results are obtained based on the operators determined to be offloaded to the untrusted GPU for execution, including: Within the trusted domain, the one-time mask resource pool is invoked to perform one-time masking on the input activation, weight matrix, or corresponding operands involved in the calculation of the current operator, thereby obtaining protected operands. The protected operand is written from the private memory of the trusted domain to the cross-domain communication path. The untrusted GPU initiates a DMA access to the cross-domain communication path, transfers the protected operand to the buffer or video memory of the untrusted GPU, and the untrusted GPU performs the corresponding mask linear calculation to obtain the protected output result. The untrusted GPU transmits the protected output result back to the trusted domain via the cross-domain communication path. The trusted domain performs demasking or recovery processing on the protected output result based on the recovery term, correction term, or equivalent recovery information associated with the current operator to obtain the plaintext intermediate result.
8. The large-scale confidential reasoning method according to claim 7, characterized in that, In the trusted domain, which is the data transfer and protection processing stage between private memory and the cross-domain communication path, single-core or multi-core parallel processing is dynamically selected according to the size of the data to be transferred, and a copy strategy that bypasses the untrusted GPU cache is adopted for streaming transfer of data blocks larger than the hardware level 1 cache.
9. A large-model confidential inference device based on TDX and GPU collaboration, characterized in that, include: The environment configuration module is configured to remotely authenticate the trusted domain and establish an authentication protection communication channel, securely transmit the inference parameters of the large model to be inferred to the private memory of the trusted domain, initialize the cross-domain communication path for data interaction between the trusted domain and the untrusted GPU, perform offline profiling and scheduling parameter calibration of the underlying hardware computing power and cross-domain transmission characteristics, and initialize a one-time mask resource for GPU offload protection. The execution domain decision module is configured to send the received input request into the trusted domain via the authentication protection communication channel; during the large model inference process, the operators in the computation graph are subjected to type identification and structural constraint analysis, wherein nonlinear operators are determined to be executed in the trusted domain, and linear operators are selected based on the calibrated hardware parameters under the premise of satisfying the structural same-domain constraint, so as to determine whether the linear operator is executed in the trusted domain or on the untrusted GPU; The inference and output module is configured to perform calculations directly within the trusted domain for operators determined to be executed within the trusted domain; for operators determined to be offloaded to the untrusted GPU for execution, the module performs one-time mask protection on the operands involved in the offloading within the trusted domain, sends them to the untrusted GPU for mask calculation via the cross-domain communication channel, and performs recovery processing after the result is returned to the trusted domain to obtain plaintext intermediate results and continue subsequent inference until the final output is generated; the final output is returned to the user through the authentication protection communication channel.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the large model confidential reasoning method according to any one of claims 1 to 8.
