A heterogeneous perception scheduling optimization method and system for efficient and secure inference service of LLM

Abstract

The application discloses a heterogeneous perception scheduling optimization method and system for LLM efficient and safe reasoning service. The method comprises the following steps: constructing an asymmetric multi-instance parallel mechanism, thereby relieving the performance marginal diminishing effect caused by protected memory access, encryption and decryption overhead and security context isolation in the cross-processor safe execution process of TEE; constructing a reasoning execution time estimator perceiving the TEE instance state and the LLM reasoning stage characteristics, thereby providing reliable decision basis for the load balancing and task scheduling among the safe instances; and realizing a heterogeneous perception adaptive request scheduler, thereby relieving the head-of-line blocking problem caused by the heterogeneity of LLM request, controlling the request delay within the service level target range, and improving the effective throughput of the overall safe reasoning service. The application realizes the "available invisible" LLM safe reasoning based on the TEE security boundary in the untrusted environment, and effectively improves the overall performance of the reasoning service system through safe perception scheduling optimization.

Description

Technical Field

[0001] This application relates to the fields of computer technology and privacy-preserving machine learning, and in particular to a heterogeneous perceptual scheduling optimization method and system for efficient and secure inference services in LLM. Background Technology

[0002] With the widespread application of Large Language Models (LLMs) in natural language processing, code generation, and decision support, Machine Learning as a Service (MLA) has become the mainstream deployment model. However, ensuring data security during the inference process when processing data involving user privacy or core corporate secrets in the cloud has become a core bottleneck restricting the further popularization of LLMs.

[0003] Currently, using hardware-assisted Trusted Execution Environments (TEEs) for confidential computation in LLM inference is one of the mainstream data protection solutions. However, in real-world production environments with multiple concurrent requests, existing secure inference service systems face severe performance challenges. First, in multi-socket platforms, the expansion of TEEs is constrained by the bandwidth of inter-processor interconnect buses (such as UPI) and the performance bottleneck of the memory encryption / decryption engine (TME-MT), resulting in a significant marginal effect on computing power improvement. The additional latency caused by cross-processor communication significantly reduces inference efficiency. Second, LLM inference requests are highly heterogeneous, with vastly different input sequence lengths and estimated output lengths for different requests. Under the traditional first-come, first-served (FCFS) scheduling strategy, long-sequence requests are highly susceptible to severe head-of-line blocking.

[0004] Existing GPU scheduling algorithms primarily focus on allocating general-purpose computing resources, failing to fully consider the security boundary constraints and cross-domain collaboration overhead specific to TEE hardware. Therefore, how to improve system throughput and reduce request latency while ensuring LLM inference privacy and security, by being aware of hardware characteristics and implementing dynamic scheduling optimization for heterogeneous requests, has become a critical issue that urgently needs to be addressed to enhance secure inference performance. Summary of the Invention

[0005] To address the problems of low efficiency in secure inference, limited TEE performance scalability in multi-processor environments, and head-of-line blocking caused by heterogeneous requests in existing technologies, this invention provides a heterogeneous-aware adaptive scheduling optimization method and system for efficient secure inference services for large language models. First, by analyzing the marginal benefit of TEE performance with increasing CPU core count, this invention implements an asymmetric multi-instance parallel strategy. This involves asymmetrically partitioning computational cores to construct differentiated strong and weak security execution instances, aiming to mitigate performance losses caused by cross-processor communication. Based on this, the invention establishes a time complexity-based inference execution time prediction model. By analyzing the time complexity of the prefill and decoding stages in LLM inference, it provides accurate data support for scheduling decisions. Finally, this invention constructs a heterogeneous-aware adaptive scheduler. By combining earliest-to-complete load balancing, high-response-ratio priority scheduling, and service level objective (SLO)-aware preemptive scheduling strategies, it achieves load balancing and scheduling order optimization for heterogeneous requests across different security instances. This invention significantly reduces the average response latency of requests and optimizes the effective throughput of the system under complex load environments while strictly protecting the privacy of inference data.

[0006] In a first aspect, the present invention provides a heterogeneous awareness scheduling optimization method for efficient and secure inference services in LLM, comprising the following steps: Step S1: Construct an asymmetric multi-instance parallel mechanism to address the encryption overhead of cross-processor data movement in the Trusted Execution Environment (TEE). Analyze the marginal benefit of TEE performance with increasing CPU core count, and apply a preset marginal benefit threshold. Determine the CPU core allocation ratio and asymmetrically allocate all CPU cores to multiple TEE instances. Step S2: Construct an inference execution time predictor, analyze the time complexity of inference tasks in the Prefill and Decoding stages respectively, construct an inference execution time prediction function based on a trusted execution environment, and fit the specific parameters of the inference execution time prediction function based on historical inference execution time data; Step S3: Heterogeneous perception adaptive request scheduling. Based on the asymmetric multi-instance parallel mechanism and inference execution time estimator, the scheduling of secure inference services for large language models is achieved through a hybrid scheduling strategy, secure computing through TEE / GPU collaboration is realized, load balancing among multiple TEE instances is achieved by estimating the earliest completion time, inference task scheduling in the Prefill stage is achieved by a priority scheduling strategy, and inference task scheduling in the Decoding stage is achieved by a preemptive scheduling strategy. Inference tasks are sent to the target TEE instance according to the corresponding scheduling strategy. Step S4: Task execution and response feedback. Each TEE instance monitors newly arriving inference tasks in real time, performs calculations according to the TEE / GPU collaborative scheduling order, and feeds back the generated token sequence to the user upon completion.

[0007] Furthermore, the asymmetric CPU core allocation logic in step S1 is as follows: Asymmetric CPU core allocation reduces the additional data movement and encryption / decryption overhead caused by specific encryption units in the TEE cross-processor data exchange process; for servers equipped with dual processors, a fixed-step increase in the number of CPU cores is provided. The corresponding TEE inference execution time is marginal revenue The calculation formula is: when At that time, Each CPU core is assigned to one TEE instance, and the remaining cores are assigned to another instance. At the same time, cores of the same processor are preferentially assigned to a single TEE instance. The instance with more cores is called the strong instance, and the other instance is called the weak instance.

[0008] Furthermore, the logic for constructing the inference execution time estimation function in the Prefill stage described in step S2 is as follows: given a user input sequence Corresponding length The hidden dimension of LLM is LLM captures semantic information of user input sequences through multi-layer Transformer network blocks, which include attention layers and linear layer computations; wherein, the attention layer computation involves Matrix computations between tensors, where the dimensions of each tensor are... Therefore, the time complexity of the attention layer is . The time complexity of the linear layer is... Therefore, the inference execution time prediction function for the inference task in the Prefill phase... The function form is: in, These are the specific parameters to be fitted.

[0009] Furthermore, the logic for constructing the inference execution time prediction function in the Decoding stage described in step S2 is as follows: given the length of the sequence already generated by the model... Corresponding length and LLM hidden dimensions LLM outputs tokens one by one through autoregressive generation, including computation in attention and linear layers; autoregressive generation takes a single token as input each time, therefore... The dimension of the vector is The time complexity of the attention layer is The inference execution time prediction function for the inference task in the Decoding phase. The function form is: in, These are the specific parameters to be fitted.

[0010] Furthermore, in step S3, the strategy for achieving load balancing among multiple TEE instances based on the estimated earliest completion time includes: given a set of TEE instances Given a user input sequence The earliest completion time includes the time spent waiting for instance scheduling. and instance execution time Therefore, specific examples The earliest completion time is: Load balancing will schedule requests to the earliest estimated completion time. The smallest TEE instance, thus achieving load balancing among multiple TEE instances; in, TEE Example Input sequence under conditions At that time, the Prefill phase predicts the inference execution time; correspondingly, the Decoding phase is... .

[0011] Furthermore, in step S3, the prefill phase inference task scheduling based on the priority scheduling strategy includes: in a certain TEE instance In the example, given a set of inference tasks in the Prefill phase, the input sequence for each task is as follows: The waiting time for each task is... The priority of a certain Prefill stage inference task is as follows: By priority The scheduling order of the inference tasks in the Prefill stage is obtained by sorting in descending order, thereby avoiding the head-of-line blocking problem caused by long input tasks.

[0012] Furthermore, in step S3, the scheduling of inference tasks in the Decoding phase based on a preemptive scheduling strategy includes scheduling tasks within a specific TEE instance. In the example, given a set of inference tasks in the Decoding phase, the output sequence for each task is as follows: Given the Service Level Objective (SLO) for the inference task in the Prefill phase, Given the waiting time of the highest priority task in the Prefill phase. and predictive inference execution time The relaxation time corresponding to the service level target is calculated as follows: For inference tasks in the Decoding phase, the scheduler prioritizes executing the task with the shortest current output sequence length. At the same time, when If the output is in the specified condition, the token is continuously iterated and output; otherwise, the task in the Decoding phase is interrupted and the task in the Prefill phase is executed, thereby avoiding queue blocking caused by long output tasks.

[0013] Secondly, the present invention provides a heterogeneous awareness scheduling optimization system for efficient and secure inference services in LLM, characterized in that it includes: The core allocation module is used to construct an asymmetric multi-instance parallel mechanism. Addressing the encryption overhead of cross-processor data movement in Trusted Execution Environments (TEEs), it analyzes the marginal performance gain of TEEs with increasing CPU core count, based on a preset marginal gain threshold. Determine the CPU core allocation ratio and asymmetrically allocate all CPU cores to multiple TEE instances; The time estimation module is used to analyze the time complexity of inference tasks in the Prefill and Decoding stages respectively, construct an inference execution time estimation function based on a trusted execution environment, and fit the specific parameters of the inference execution time estimation function based on historical inference execution time data. The task scheduling module is used to schedule secure inference tasks for large language models based on the asymmetric multi-instance parallel mechanism and inference execution time predictor through a hybrid scheduling strategy: the earliest completion time prediction strategy is used to achieve load balancing among multiple TEE instances, the priority scheduling strategy is used to schedule inference tasks in the Prefill stage, and the preemptive scheduling strategy is used to schedule inference tasks in the Decoding stage. The inference tasks are sent to the target TEE instance according to the corresponding scheduling strategy. The inference execution module controls each TEE instance to monitor newly arrived inference tasks in real time, executes calculations according to the TEE / GPU collaborative scheduling order, and feeds back the generated token sequence to the user upon completion.

[0014] Thirdly, the present invention provides an electronic device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the above-described method.

[0015] Fourthly, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described method.

[0016] This invention first analyzes the marginal benefit of TEE performance with increasing CPU core count and implements an asymmetric multi-instance parallel strategy. By asymmetrically partitioning computing cores to construct differentiated strong and weak security execution instances, it aims to mitigate the performance loss caused by cross-processor communication. Based on this, the invention establishes a time complexity-based inference execution time prediction model. By analyzing the time complexity of the prefill and decoding stages in LLM inference, it provides accurate data support for scheduling decisions. Finally, the invention constructs a heterogeneous-aware adaptive scheduler. By combining earliest-completion-time load balancing, high-response-ratio priority scheduling, and service level objective (SLO)-aware preemptive scheduling strategies, it achieves load balancing and scheduling order optimization for heterogeneous requests across different security instances. This invention significantly reduces the average response latency of requests and optimizes the effective throughput of the system under complex load environments while strictly protecting the privacy of inference data. Attached Figure Description

[0017] Figure 1 This is a flowchart of a heterogeneous perception scheduling optimization method for efficient and secure inference services in LLM according to the present invention; Figure 2 This is a framework diagram of a heterogeneous perception scheduling optimization system for efficient and secure inference services of LLM according to the present invention. Detailed Implementation

[0018] The specific embodiments of this application will be described in detail below with reference to the accompanying drawings. It should be noted that the embodiments shown in this document are only a typical application of this application, and the technical solutions involved in this application should be protected until those skilled in the art can propose better improvement solutions.

[0019] In one implementation example, such as Figure 1As shown in the figure, this invention provides a heterogeneous awareness scheduling optimization method for efficient and secure inference services in LLM (Liquidity Management System). Step S1: Construct an asymmetric multi-instance parallel mechanism to address the encryption overhead of cross-processor data movement in the Trusted Execution Environment (TEE). Analyze the marginal benefit of TEE performance with increasing CPU core count, and apply a preset marginal benefit threshold. Determine the CPU core allocation ratio and asymmetrically allocate all CPU cores to multiple TEE instances. Step S2: Construct an inference execution time predictor, analyze the time complexity of inference tasks in the Prefill and Decoding stages respectively, construct an inference execution time prediction function based on a trusted execution environment, and fit the specific parameters of the inference execution time prediction function based on historical inference execution time data; Step S3: Heterogeneous perception adaptive request scheduling. Based on the asymmetric multi-instance parallel mechanism and inference execution time estimator, the scheduling of secure inference services for large language models is achieved through a hybrid scheduling strategy, secure computing through TEE / GPU collaboration is realized, load balancing among multiple TEE instances is achieved by estimating the earliest completion time, inference task scheduling in the Prefill stage is achieved by a priority scheduling strategy, and inference task scheduling in the Decoding stage is achieved by a preemptive scheduling strategy. Inference tasks are sent to the target TEE instance according to the corresponding scheduling strategy. Step S4: Task execution and response feedback. Each TEE instance monitors newly arriving inference tasks in real time, performs calculations according to the TEE / GPU collaborative scheduling order, and feeds back the generated token sequence to the user upon completion.

[0020] In one implementation, the asymmetric CPU core allocation logic in step S1 is as follows: Asymmetric CPU core allocation reduces the additional data movement and encryption / decryption overhead caused by specific encryption units in the TEE cross-processor data exchange process; for servers equipped with dual processors, given a fixed-step increase in the number of CPU cores... The corresponding TEE inference execution time is marginal revenue The calculation formula is: when At that time, Each CPU core is assigned to one TEE instance, and the remaining cores are assigned to another instance. At the same time, cores of the same processor are preferentially assigned to a single TEE instance. The instance with more cores is called the strong instance, and the other instance is called the weak instance.

[0021] In one implementation, the logic for constructing the inference execution time estimation function in the Prefill stage of step S2 is as follows: given a user input sequence Corresponding length The hidden dimension of LLM is LLM captures semantic information of user input sequences through multi-layer Transformer network blocks, which include attention layers and linear layer computations; wherein, the attention layer computation involves Matrix computations between tensors, where the dimensions of each tensor are... Therefore, the time complexity of the attention layer is . The time complexity of the linear layer is... Therefore, the inference execution time prediction function for the inference task in the Prefill phase... The function form is: in, These are the specific parameters to be fitted.

[0022] In one implementation, the logic for constructing the inference execution time prediction function in the Decoding stage in step S2 is as follows: given the length of the sequence already generated by the model... Corresponding length and LLM hidden dimensions LLM outputs tokens one by one through autoregressive generation, including computation in attention and linear layers; autoregressive generation takes a single token as input each time, therefore... The dimension of the vector is The time complexity of the attention layer is The inference execution time prediction function for the inference task in the Decoding phase. The function form is: in, These are the specific parameters to be fitted.

[0023] In one implementation, in step S3, the strategy for achieving load balancing among multiple TEE instances based on the estimated earliest completion time includes: given a set of TEE instances Given a user input sequence The earliest completion time includes the time spent waiting for instance scheduling. and instance execution time Therefore, specific examples The earliest completion time is: Load balancing will schedule requests to the earliest estimated completion time. The smallest TEE instance, thus achieving load balancing among multiple TEE instances; Among them, it means TEE Example Input sequence under conditions At that time, the Prefill phase predicts the inference execution time; correspondingly, the Decoding phase is... .

[0024] In one implementation, step S3, which involves scheduling the inference task in the Prefill phase based on a priority scheduling strategy, includes: scheduling a specific TEE instance... In the example, given a set of inference tasks in the Prefill phase, the input sequence for each task is as follows: The waiting time for each task is... The priority of a certain Prefill stage inference task is as follows: By priority The scheduling order of the inference tasks in the Prefill stage is obtained by sorting in descending order, thereby avoiding the head-of-line blocking problem caused by long input tasks.

[0025] In one implementation, step S3, which involves scheduling inference tasks during the Decoding phase based on a preemptive scheduling strategy, includes scheduling tasks within a specific TEE instance. In the example, given a set of inference tasks in the Decoding phase, the output sequence for each task is as follows: Given the Service Level Objective (SLO) for the inference task in the Prefill phase, Given the waiting time of the highest priority task in the Prefill phase. and predictive inference execution time The relaxation time corresponding to the service level target is calculated as follows: For inference tasks in the Decoding phase, the scheduler prioritizes executing the task with the shortest current output sequence length. At the same time, when If the output is in the specified condition, the token is continuously iterated and output; otherwise, the task in the Decoding phase is interrupted and the task in the Prefill phase is executed, thereby avoiding queue blocking caused by long output tasks.

[0026] Figure 2 This invention provides a heterogeneous awareness scheduling optimization system for efficient and secure inference services in LLM, comprising: The 21-core allocation module is used to construct an asymmetric multi-instance parallel mechanism. Addressing the encryption overhead of cross-processor data movement in Trusted Execution Environments (TEEs), it analyzes the marginal performance gain of TEEs with increasing CPU core count, based on a preset marginal gain threshold. Determine the CPU core allocation ratio and asymmetrically allocate all CPU cores to multiple TEE instances; The 22-time prediction module is used to analyze the time complexity of inference tasks in the Prefill and Decoding stages respectively, construct an inference execution time prediction function based on a trusted execution environment, and fit the specific parameters of the inference execution time prediction function based on historical inference execution time data. 23. The task scheduling module is used to schedule secure inference tasks for large language models based on the asymmetric multi-instance parallel mechanism and inference execution time predictor through a hybrid scheduling strategy: the earliest completion time prediction strategy is used to achieve load balancing among multiple TEE instances, the priority scheduling strategy is used to schedule inference tasks in the Prefill stage, and the preemptive scheduling strategy is used to schedule inference tasks in the Decoding stage. The inference tasks are sent to the target TEE instance according to the corresponding scheduling strategy. The 24 Inference Execution Module controls each TEE instance to monitor newly arriving inference tasks in real time, executes calculations according to the TEE / GPU collaborative scheduling order, and feeds back the generated Token sequence to the user upon completion.

[0027] In one embodiment, the present invention provides an electronic device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the method described above.

[0028] In one embodiment, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

Claims

1. A heterogeneous perceptual scheduling optimization method for efficient and secure inference services in LLM, characterized in that, Includes the following steps: Step S1: Construct an asymmetric multi-instance parallel mechanism to address the encryption overhead of cross-processor data movement in the Trusted Execution Environment (TEE). Analyze the marginal benefit of TEE performance with increasing CPU core count, and apply a preset marginal benefit threshold. Determine the CPU core allocation ratio and asymmetrically allocate all CPU cores to multiple TEE instances. Step S2: Construct an inference execution time predictor, analyze the time complexity of inference tasks in the Prefill and Decoding stages respectively, and construct an inference execution time prediction function based on a trusted execution environment. and The specific parameters of the inference execution time prediction function are fitted based on historical inference execution time data; Step S3: Heterogeneous perception adaptive request scheduling. Based on the aforementioned asymmetric multi-instance parallel mechanism and inference execution time predictor, a hybrid scheduling strategy is used to schedule the secure inference service of the large language model, realize secure computation through TEE / GPU collaboration, and achieve load balancing among multiple TEE instances by estimating the earliest completion time. A priority scheduling strategy is used to schedule inference tasks in the Prefill stage, and a preemptive scheduling strategy is used to schedule inference tasks in the Decoding stage. Inference tasks are sent to the target TEE instance according to the corresponding scheduling strategy. Step S4: Task execution and response feedback. Each TEE instance monitors newly arriving inference tasks in real time, performs calculations according to the TEE / GPU collaborative scheduling order, and feeds back the generated token sequence to the user upon completion.

2. The method according to claim 1, characterized in that, The asymmetric CPU core allocation logic in step S1 is as follows: Asymmetric CPU core allocation reduces the additional data movement and encryption / decryption overhead caused by specific encryption units during TEE cross-processor data exchange; for servers equipped with dual processors, a fixed-step increase in the number of CPU cores is used. The corresponding TEE inference execution time is marginal revenue The calculation formula is: when At that time, Each CPU core is assigned to one TEE instance, and the remaining cores are assigned to another instance. At the same time, cores of the same processor are preferentially assigned to a single TEE instance. The instance with more cores is called the strong instance, and the other instance is called the weak instance.

3. The method according to claim 1, characterized in that, The logic for constructing the inference execution time estimation function in the Prefill stage described in step S2 is as follows: given a user input sequence Corresponding length The hidden dimension of LLM is LLM captures semantic information of user input sequences through multi-layer Transformer network blocks, which include attention layers and linear layer computations; wherein, the attention layer computation involves Matrix computations between tensors, where the dimensions of each tensor are... Therefore, the time complexity of the attention layer is O(n). The time complexity of the linear layer is... Therefore, the inference execution time prediction function for the inference task in the Prefill phase... The function form is: in, These are the specific parameters to be fitted.

4. The method according to claim 1, characterized in that, The logic for constructing the inference execution time prediction function in the Decoding stage described in step S2 is as follows: given the length of the sequence already generated by the model... Corresponding length and LLM hidden dimensions LLM outputs tokens one by one through autoregressive generation, including computation in attention and linear layers; autoregressive generation takes a single token as input each time, therefore... The dimension of the vector is The time complexity of the attention layer is The inference execution time prediction function for the inference task in the Decoding phase. The function form is: in, These are the specific parameters to be fitted.

5. The method according to claim 1, characterized in that, In step S3, the strategy of achieving load balancing among multiple TEE instances based on the estimated earliest completion time includes: given a set of TEE instances Given a user input sequence The earliest completion time includes the time spent waiting for instance scheduling. and instance execution time Therefore, specific examples The earliest completion time is: Load balancing will schedule requests to the earliest estimated completion time. The smallest TEE instance, thus achieving load balancing among multiple TEE instances; in, Indicates a TEE instance Input sequence under conditions At that time, the Prefill phase predicts the inference execution time; correspondingly, the Decoding phase is... .

6. The method according to claim 1, characterized in that, In step S3, scheduling the inference task in the Prefill phase based on the priority scheduling strategy includes: in a certain TEE instance In the example, given a set of inference tasks in the Prefill phase, the input sequence for each task is as follows: The waiting time for each task is... The priority of a certain Prefill stage inference task is as follows: By priority The scheduling order of the inference tasks in the Prefill stage is obtained by sorting in descending order, thereby avoiding the head-of-line blocking problem caused by long input tasks.

7. The method according to claim 1, characterized in that, In step S3, the scheduling of inference tasks in the Decoding phase based on a preemptive scheduling strategy includes scheduling tasks within a specific TEE instance. In the example, given a set of inference tasks in the Decoding phase, the output sequence for each task is as follows: Given the Service Level Objective (SLO) for the inference task in the Prefill phase, Given the waiting time of the highest priority task in the Prefill phase. and predictive inference execution time The relaxation time corresponding to the service level target is calculated as follows: For inference tasks in the Decoding phase, the scheduler prioritizes executing the task with the shortest current output sequence length. At the same time, when If the output is in the specified condition, the token is continuously iterated and output; otherwise, the task in the Decoding stage is interrupted and the task in the Prefill stage is executed, thereby avoiding queue blocking caused by long output tasks.

8. A heterogeneous perceptual scheduling optimization system for efficient and secure inference services of LLM as described in any one of claims 1-7, characterized in that, include: The core allocation module is used to construct an asymmetric multi-instance parallel mechanism. Addressing the encryption overhead of cross-processor data movement in Trusted Execution Environments (TEEs), it analyzes the marginal performance gain of TEEs with increasing CPU core count, based on a preset marginal gain threshold. Determine the CPU core allocation ratio and asymmetrically allocate all CPU cores to multiple TEE instances; The time estimation module is used to analyze the time complexity of inference tasks in the Prefill and Decoding stages respectively, construct an inference execution time estimation function based on a trusted execution environment, and fit the specific parameters of the inference execution time estimation function based on historical inference execution time data. The task scheduling module is used to schedule secure inference tasks for large language models based on the asymmetric multi-instance parallel mechanism and inference execution time predictor through a hybrid scheduling strategy: the earliest completion time prediction strategy is used to achieve load balancing among multiple TEE instances, the priority scheduling strategy is used to schedule inference tasks in the Prefill stage, and the preemptive scheduling strategy is used to schedule inference tasks in the Decoding stage. The inference tasks are sent to the target TEE instance according to the corresponding scheduling strategy. The inference execution module controls each TEE instance to monitor newly arrived inference tasks in real time, executes calculations according to the TEE / GPU collaborative scheduling order, and feeds back the generated token sequence to the user upon completion.

9. An electronic device, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the method according to any one of claims 1 to 8.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the method of any one of claims 1 to 8.

Images