A weighted scheduling and caching method for multi-tenant LoRA inference

Abstract

This invention provides a weighted scheduling and caching method for multi-tenant LoRA inference, including request access and initialization steps, dynamic priority adjustment steps, weighted token-level scheduling steps, adapter cache management steps, and execution feedback and optimization steps. This invention simultaneously features a request priority scheduling mechanism, a look-ahead adapter prefetching mechanism, and a multi-factor driven adapter value-aware eviction strategy. It can dynamically adjust the scheduling and caching strategy based on the real-time status of tenant requests, thereby achieving higher inference performance, more stable response latency, and stronger quality of service assurance capabilities with limited GPU memory resources.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence system optimization and inference service management technology, specifically to a weighted scheduling and caching method for multi-tenant LoRA inference. This method is suitable for the efficient deployment of Low-Rank Adaptation (LoRA) methods in shared model architectures and is widely used in natural language processing, personalized recommendation, code generation and other fields. It is particularly suitable for scenarios with service latency sensitivity and strong task variability. Background Technology

[0002] With the widespread adoption of Large Language Models (LLMs), the demand for multi-user, multi-task concurrent inference in cloud platforms has increased dramatically. To meet users' personalized inference needs and reduce computational resource consumption, efficient parameter fine-tuning methods have been widely adopted, among which LoRA has become a mainstream solution due to its lightweight structure and flexible deployment. LoRA allows each tenant to perform personalized inference through independent adapters by inserting trainable low-rank parameters into the frozen backbone model. In a multi-tenant inference system, all users share the LLM backbone model, while each is bound to a dedicated LoRA adapter. While this architecture improves model reusability, it also brings the following core challenges: GPU memory resources are limited: The LLM backbone model already occupies a large amount of video memory, making it difficult to keep all LoRA adapters resident. It needs to be frequently loaded from host memory or disk on demand, which can easily cause I / O bottlenecks and cold start delays.

[0003] Task heterogeneity and scheduling strategy mismatch: LoRA adapters vary significantly in parameter size and loading overhead, and user requests are also highly uneven in terms of input length, maximum decoding length, and latency sensitivity.

[0004] Runtime load is highly dynamic: tenant activity, request intensity, and adapter popularity fluctuate rapidly, posing a significant challenge to scheduling and caching strategies that rely on fixed thresholds or static rules.

[0005] To address the aforementioned shortcomings, existing technical solutions employ two main approaches. One approach attempts to mitigate latency caused by frequent adapter loading through caching mechanisms. However, with limited GPU memory, traditional Least Recently Used (LRU) or First In First Out (FIFO) eviction policies often only consider access frequency or time, neglecting adapter loading overhead and future reuse probability. This can lead to the mistaken eviction of high-value adapters, increasing the risk of redundant loading and performance fluctuations. Another approach focuses on scheduling optimization, such as batch processing strategies targeting maximum throughput or round-robin scheduling based on fairness. While these methods improve overall GPU utilization, they lack differentiated guarantees for latency-sensitive requests and cannot achieve priority awareness and quality of service control in multi-tenant scenarios.

[0006] In summary, the existing technical solutions have the following drawbacks: High adapter loading overhead: Frequent PCIe data transfers result in high cold start latency, affecting the first token response time.

[0007] The scheduling lacks priority awareness: existing FIFO, round-robin, or throughput-oriented scheduling cannot take into account the latency sensitivity and urgency of different requests.

[0008] Inefficient cache eviction: Traditional LRU or FIFO strategies do not take into account adapter size, loading cost and future reuse probability, resulting in low memory utilization efficiency.

[0009] Lack of adaptability: In highly dynamic multi-tenant scenarios, fixed rules are difficult to adapt to rapid changes in request patterns and tenant activity.

[0010] Therefore, there is an urgent need for a new approach that can comprehensively consider weighted scheduling, look-ahead adapter prefetching, and value-aware eviction strategies under the constraint of limited GPU memory resources, in order to achieve low latency, high throughput, and stable SLO for multi-tenant LoRA inference services. Summary of the Invention

[0011] To address the problems of frequent adapter loading and eviction, unstable request latency, low throughput, and poor Service Level Objective (SLO) achievement rate in existing multi-tenant LLM online inference services, this invention provides a weighted scheduling and caching method for multi-tenant LoRA inference. This method combines a request priority scheduling mechanism, a look-ahead adapter prefetching mechanism, and a multi-factor driven adapter value-aware eviction strategy. It can dynamically adjust the scheduling and caching strategy according to the real-time status of tenant requests, thereby achieving higher inference performance, more stable response latency, and stronger service quality assurance capabilities under limited GPU memory resources.

[0012] The present invention achieves the above objectives through the following technical solutions: A weighted scheduling and caching method for multi-tenant LoRA inference includes: Access request and initialization steps: Receive inference requests from multiple tenants, extract metadata for each request, including arrival time, maximum decoding length, adapter ID, service priority, and initialize the scheduling label for each request based on the extracted metadata; Priority dynamic adjustment steps: Use a sliding window statistics mechanism to record the achievement of service level targets corresponding to each service priority in the recent period, and dynamically adjust the token share of different priority tasks entering the waiting queue based on the statistical results; Weighted Token-level scheduling steps: Calculate the scheduling score for each request based on its priority weight, cumulative waiting time, and maximum decoding length. Perform fine-grained scheduling at the token level based on the scheduling score, and introduce a soft preemption mechanism and a credit-based scheduling control mechanism during the scheduling process. Adapter cache management steps: A multi-layered cache architecture including GPU cache, host memory cache and local / remote storage is adopted. A forward-looking adapter prefetching mechanism is used to predict and load LoRA adapters that may be used in the future. At the same time, a value-aware adapter eviction strategy is adopted to dynamically optimize the GPU memory usage based on the adapter loading cost, size and future access probability. Execution feedback and optimization steps: Collect key data during system operation, and feed the operational data back to the scheduling module and cache management module in a closed loop to drive the system to dynamically adjust the scheduling strategy and cache management strategy.

[0013] According to the weighted scheduling and caching method for multi-tenant LoRA inference provided by the present invention, in the request access and initialization steps, a scheduling label is generated for each request based on the extracted metadata. The scheduling label contains the following dynamic fields: Priority Classification: Maps service priorities to discrete priority categories for token quota allocation in the priority shaper; Initial scheduling score: The initial score is calculated using a weighted formula based on parameters such as priority weight and maximum decoding length; Time-related markers: Attach arrival timestamp and current system time to calculate cumulative waiting time; Adapter dependency information: Records the adapter ID and cache status, which is used by the adapter cache manager to make prefetching or eviction decisions; The initialized requests and their scheduling tags are stored in the candidate request pool, and a multi-level index structure is established.

[0014] According to the weighted scheduling and caching method for multi-tenant LoRA inference provided by the present invention, in the priority dynamic adjustment step, a fixed-size sliding window is maintained to record the service level target achievement status of each priority task in each scheduling cycle. For each priority task, count the number of SLO defaults and the total number of requests within the sliding window, and calculate the SLO achievement rate; Based on the SLO achievement rate of each priority task, the token share entering the waiting queue is dynamically adjusted, which is the upper limit of the number of tokens allowed to enter the waiting queue in each round of scheduling. Update the Token share configuration table for each priority task based on the dynamic quota calculation results; At the start of each round of scheduling, requests are selected from the candidate request pool in priority order according to the Token share limit in the configuration table and enter the waiting queue. If the number of requests for a priority task exceeds its token share limit, the excess requests will be temporarily stored in a buffer queue and will participate in the quota allocation again in the next round of scheduling.

[0015] According to the weighted scheduling and caching method for multi-tenant LoRA inference provided by the present invention, in the weighted token-level scheduling step, the priority weight of the request is obtained from the scheduling tag of the request. Cumulative waiting time Maximum decoding length The scheduling score for each request is calculated using a comprehensive scoring formula, expressed as follows:

[0016] in, α , β , γ These are preset weighting coefficients used to adjust the proportion of priority weights, cumulative waiting time, and factors related to the maximum decoding length in the overall score. α > β >γ ; The calculated scheduling score for each request Recorded in the dispatch tag of the request.

[0017] According to the weighted scheduling and caching method for multi-tenant LoRA inference provided by the present invention, before the start of each round of token decoding, the scheduling module accesses the waiting queue, which stores multiple requests to be processed after the scheduling score is calculated, and each request carries a corresponding scheduling score. Requests are selected from the waiting queue in descending order of scheduling scores and added to the activation queue. The filling process continues until the following conditions are met, at which point filling will stop: The number of tokens corresponding to the requests already populated in the activation queue has reached the preset batch token limit; Alternatively, the GPU memory usage of requests in the activation queue reaches the GPU memory capacity limit; Once the activation queue is filled, the system starts a parallel execution mechanism to perform single-token decoding operations on each request in the activation queue simultaneously. During parallel execution, the decoding progress of each request is monitored in real time to ensure that each request is decoded according to its own generation logic; During parallel decoding execution, if a request completes all the generation tasks, the system immediately reclaims the computing resources occupied by that request. The recycled resources will be released to the system resource pool. During the activation queue filling phase before the next round of token decoding begins, new requests will be selected from the waiting queue and filled into the activation queue, where the recycled resources will be used for decoding.

[0018] According to the weighted scheduling and caching method for multi-tenant LoRA inference provided by the present invention, the soft preemption mechanism is implemented as follows: Preemption trigger condition judgment steps: When the activation queue is full, the system continuously monitors the scheduling score of newly arriving requests; If a new request exists, its scheduling score is compared with the scheduling score of the lowest-scoring request currently in the active queue; When the scheduling score of a new request is significantly higher than the scheduling score of the current lowest-scoring request, and the difference between the two exceeds the preset preemption score threshold, the preemption trigger condition is determined to be met. One-time replacement operation execution steps: When the preemption trigger condition is met, the system performs a one-time replacement operation, removing the request with the lowest current score from the activation queue and releasing the GPU computing resources and video memory resources it occupies; at the same time, the new request is filled into the activation queue. Preemption constraint control steps: To avoid excessive preemption causing system performance fluctuations, a maximum preemption count attribute is set for each request to record the number of times it has been preempted; When a request is preempted a maximum number of times, it will no longer participate in the soft preemption mechanism during subsequent scheduling processes, ensuring that it can at least get some execution opportunities.

[0019] According to the present invention, a weighted scheduling and caching method for multi-tenant LoRA inference includes a credit-based scheduling control mechanism, the implementation of which includes: Each request entering the scheduling system is initialized with an accumulative credit value, with the initial credit value set to 0. Before each round of scheduling decisions, the priority weight of each request is determined. Their credit scores are updated according to priority weights. Linear growth, the growth formula is: C redit_i = C redit_i + ×Δ t ,in C redit_i For the first i The current credit value of each request, Δ t This is the time interval of the scheduling cycle.

[0020] For requests that are not scheduled, their credit score will continue to accumulate in each round of scheduling; The system monitors the credit score of each request in real time to determine whether the accumulated credit score of any request exceeds a preset scheduling threshold. Threshold ; When the accumulated credit value of a request exceeds the scheduling threshold Threshold In this case, the request will receive a priority scheduling opportunity in the next round of scheduling; After the request is given priority scheduling and the scheduling is completed, the system deducts the corresponding credit score, and the deducted score is the scheduling threshold. Threshold ,Right now C redit_i = C redit_i Threshold .

[0021] According to the present invention, a weighted scheduling and caching method for multi-tenant LoRA inference is provided, the multi-layer caching architecture comprising: A dedicated cache area is set up on the GPU as a level 0 cache to house the currently executing LoRA adapter; it is also used to store short-term, high-frequency access objects. A designated area in the host memory is allocated as the first-level cache, which serves as an intermediate cache layer to store candidate LoRA adapters that have been loaded but have not yet entered the GPU memory. When the GPU cache space is insufficient or when the prefetching strategy determines that certain adapters need to be preloaded to a location closer to the GPU, these adapters are stored in the host memory cache. Local storage devices or remote storage servers are used as the second layer of storage, serving as a persistent storage layer to store complete LoRA adapter weight files. All LoRA adapters are initially stored in this layer of storage. When the system needs to use a particular adapter, it is loaded from this layer of storage into the upper-layer cache according to the caching policy. Based on the frequency and importance of data access and the usage of cache space, the flow of data between multiple cache layers is dynamically managed. When the GPU needs to access a LoRA adapter, it first searches in the GPU's Level 0 cache. If it is not found, it searches in the Level 1 cache. If it is not found in the host memory cache either, it is loaded from the Level 2 storage into the host memory cache, and then further loaded into the GPU cache as needed. When the cache space is insufficient, a value-aware adapter eviction policy is used to evict some data from the corresponding layer cache.

[0022] According to the weighted scheduling and caching method for multi-tenant LoRA inference provided by the present invention, the implementation of the look-ahead adapter prefetch mechanism includes: Set a sliding window of size W, and extract the request data within the range of the sliding window from the front of the waiting queue; The frequency of occurrence of each LoRA adapter within the sliding window is counted, and the Top-K adapters are selected from high to low frequency to form a short-term prefetch candidate set; We model the historical access patterns of each adapter using an exponential decay and time-weighted approach, and update the hit probability of each adapter. Based on the hit probability, we construct a long-term trend candidate set from high to low. Design a hybrid scoring function: Represented as:

[0023] in, Represents adapter, This is a function related to the frequency of the adapter's appearance within the sliding window. This is the adapter hit probability correlation function obtained based on long-term trend modeling. λ This is a weighting coefficient used to balance the proportion of short-term frequency and long-term trend in the scoring; Before each round of scheduling in the system, a hybrid scoring function is applied to score all candidate adapters; Select the top - K adapters with the highest scores from all candidate adapters as the prefetch targets for the next round.

[0024] After the prefetching target is determined, the asynchronous loader executes the loading task in parallel, loading the relevant data of the selected Top - K adapters from the storage layer to the appropriate cache layer.

[0025] The weighted scheduling and caching method for multi-tenant LoRA inference provided by the present invention includes the following implementation of a value-aware adapter eviction policy: For each LoRA adapter in the cache The deportation score is calculated using the deportation scoring formula, expressed as:

[0026] in, For adapter The loading cost; Indicates adapter Size; For adapter The probability of future visits; ω , θ , κ These are preset weighting coefficients used to adjust the weighting ratios of loading cost, adapter size, and future access probability in the eviction score; The system monitors the usage of video memory in real time. When the video memory usage approaches the system's set capacity threshold, the adapter replacement mechanism is triggered. Based on the calculated deportation score, according to EScore Select several adapters as expulsion candidates in descending order of their quality; Remove the selected candidate adapter from video memory, freeing up the video memory resources it occupies and making room for the loading of the new adapter or other data processing operations.

[0027] Therefore, compared with the prior art, the weighted scheduling and caching method for multi-tenant LoRA inference proposed in this invention has the following advantages: 1. Reduce cold start latency: This invention introduces a look-ahead adapter prefetching mechanism, combined with sliding window statistics and a lightweight prediction model, to preload the LoRA adapter with high probability of use during the decoding stage, which significantly reduces PCIe transmission overhead and effectively shortens the generation latency of the first token.

[0028] 2. Improve the service quality of critical requests: This invention adopts a weighted token-level scheduling strategy, which integrates request priority, cumulative waiting time and decoding requirements into a comprehensive model to achieve rapid response to high-priority and latency-sensitive tasks. At the same time, it avoids starvation of low-priority tasks through soft preemption and credit-based scheduling control mechanisms, thereby improving the SLO achievement rate.

[0029] 3. Improve cache utilization and system stability: The value-aware adapter eviction strategy designed in this invention comprehensively considers adapter size, loading cost and future reuse probability, dynamically optimizes GPU memory usage, significantly improves cache hit rate, reduces unnecessary repeated loading, and ensures system stability under high concurrency.

[0030] 4. Enhanced Adaptability: In scenarios with multi-tenant request patterns and highly dynamic changes in adapter popularity, this invention can dynamically adjust prefetching and eviction strategies based on runtime statistics and scheduling feedback, exhibiting good adaptability and avoiding performance degradation caused by fixed rules.

[0031] 5. Balancing high throughput and low latency: This invention optimizes scheduling and caching mechanisms in a coordinated manner, thereby improving both system throughput and request completion rate under limited GPU resource constraints. It provides differentiated protection for different tenants and request types, making it suitable for the actual deployment of large-scale online LLM inference platforms.

[0032] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0033] Figure 1 This is a flowchart of an embodiment of a weighted scheduling and caching method for multi-tenant LoRA inference according to the present invention.

[0034] Figure 2 This is a system schematic diagram of the weighted scheduling and caching method for multi-tenant LoRA inference proposed in this invention.

[0035] Figure 3 This is a flowchart of the weighted token-level scheduling process in an embodiment of a weighted scheduling and caching method for multi-tenant LoRA inference according to the present invention.

[0036] Figure 4 This is a schematic diagram of the value-aware adapter eviction strategy in an embodiment of a weighted scheduling and caching method for multi-tenant LoRA inference according to the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0038] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0039] See Figures 1 to 4 This embodiment provides a weighted scheduling and caching method for multi-tenant LoRA inference, including: Access request and initialization steps: Receive inference requests from multiple tenants, extract metadata for each request, including arrival time, maximum decoding length, adapter ID, service priority, and initialize the scheduling label for each request based on the extracted metadata; Priority dynamic adjustment steps: Use a sliding window statistics mechanism to record the achievement of Service Level Objectives (SLOs) for each service priority in the recent period, and dynamically adjust the token share of different priority tasks entering the waiting queue based on the statistical results; Weighted Token-level scheduling steps: Calculate the scheduling score for each request based on its priority weight, cumulative waiting time, and maximum decoding length. Perform fine-grained scheduling at the token level based on the scheduling score. In the scheduling process, introduce a soft preemption mechanism and a credit-based scheduling control mechanism to ensure that high-priority requests are responded to in a timely manner while avoiding starvation for low-priority requests. Adapter cache management steps: A multi-layered cache architecture including GPU cache, host memory cache and local / remote storage is adopted. A forward-looking adapter prefetching mechanism is used to predict and load LoRA adapters that may be used in the future. At the same time, a value-aware adapter eviction strategy is adopted to dynamically optimize the GPU memory usage based on the adapter loading cost, size and future access probability, so as to reduce I / O operation latency and improve cache hit rate. Execution feedback and optimization steps: Collect key data during system operation, including but not limited to the number of SLO defaults, cache hit rate, and I / O operation latency. Feedback the runtime data in a closed loop to the scheduling module and cache management module to drive the system to dynamically adjust the scheduling strategy and cache management strategy, thereby achieving adaptive optimization of system performance.

[0040] During the request access and initialization steps, a scheduling tag is generated for each request based on the extracted metadata. The scheduling tag contains the following dynamic fields: Priority classification: Maps service priorities to discrete priority categories (such as high, medium, and low), used for token quota allocation in the priority shaper; Initial scheduling score: The initial score is calculated using a weighted formula based on parameters such as priority weight and maximum decoding length, such as initial score = priority weight × α + maximum decoding length × β, where α and β are weight coefficients and serve as the benchmark values ​​for subsequent scheduling; Time-related markers: Attach arrival timestamp and current system time to calculate cumulative waiting time, such as cumulative waiting time = current time - arrival time; Adapter dependency information: Records the adapter ID and cache status, such as whether it has been loaded into the GPU memory, for the adapter cache manager to make prefetching or eviction decisions; The initialized requests and their scheduling tags are stored in the candidate request pool, and a multi-level index structure is established.

[0041] In the priority dynamic adjustment step, a fixed-size sliding window is maintained, such as the request records of the most recent 100 scheduling cycles or the most recent 10 seconds, to record the service level target (SLO) achievement status of each priority task (high, medium, low) in each scheduling cycle. For each priority task, count the number of SLO defaults within the sliding window (e.g., first token delay exceeding the threshold, task completion time exceeding the deadline, etc.) and the total number of requests, and calculate the SLO achievement rate, such as achievement rate = (total number of requests - number of defaults) / total number of requests; Based on the SLO achievement rate of each priority task, the token share entering the waiting queue is dynamically adjusted, which is the upper limit of the number of tokens allowed to enter the waiting queue in each round of scheduling. The specific adjustment rules are as follows: High-priority tasks: If their SLO achievement rate is lower than the preset threshold (e.g., 90%), their quota will be increased by the preset step size (e.g., increasing the token share by 10%); if the achievement rate continues to be lower than the threshold for more than the preset period (e.g., three consecutive sliding windows), the adjustment will be further increased (e.g., doubling the step size). Medium / low priority tasks: If the SLO achievement rate of high priority tasks has met the threshold (e.g., ≥90%) and the overall system load is low (e.g., GPU utilization is below 60%), then some resources will be released to medium / low priority tasks according to preset rules (e.g., the remaining token share will be allocated proportionally). Based on the dynamic quota calculation results, update the Token share configuration table (such as a key-value pair structure stored in memory) for each priority task. At the start of each round of scheduling, requests are selected from the candidate request pool in order of priority according to the Token share limit in the configuration table and enter the waiting queue. For example, high-priority tasks are selected first until their Token share is used up, then medium-priority tasks are selected, and so on. If the number of requests for a priority task exceeds its token share limit, the excess requests will be temporarily stored in a buffer queue and will participate in the quota allocation again in the next round of scheduling. To avoid frequent adjustments to token allocation due to short-term fluctuations in SLO achievement rates, this embodiment also introduces a rollback delay window (such as 5 sliding window cycles) and a minimum allocation threshold (such as the minimum token allocation for high-priority tasks not being less than 30% of the total quota). If the SLO achievement rate of a high-priority task recovers to above the threshold within the rollback delay window, the increase in its token share will be suspended and gradually rolled back to the base quota (e.g., the quota will be reduced by a preset step size). If the system load is too high (e.g., GPU utilization consistently ≥90%), resources will be temporarily suspended for medium / low priority tasks, prioritizing the achievement of SLOs for high priority tasks.

[0042] As can be seen, this embodiment achieves adaptive adjustment of token share for tasks of different priorities in a multi-tenant scenario through mechanisms such as sliding window statistics, dynamic quota calculation, quota allocation execution, and default rate rollback suppression. This not only ensures the low latency requirements of critical tasks but also takes into account the overall system throughput and fairness.

[0043] In the weighted token-level scheduling step, the priority weight of the request is obtained from the scheduling tag of the request. Cumulative waiting time Maximum decoding length The scheduling score for each request is calculated using a comprehensive scoring formula, expressed as follows:

[0044] in, α , β , γ These are preset weighting coefficients used to adjust the proportion of priority weights, cumulative waiting time, and factors related to the maximum decoding length in the overall score.α > β > γ By setting these weighting coefficients appropriately, the emphasis of the scheduling strategy on different factors can be flexibly adjusted. The calculated scheduling score for each request Recorded in the dispatch tag of the request; In the subsequent token-level scheduling process, the scheduling score of each request is used as a basis. Sorting and other operations are performed to determine the execution order of requests, with higher-scoring requests having higher priority during scheduling.

[0045] Before each round of token decoding begins, the scheduling module accesses the waiting queue, which stores multiple requests to be processed after the scheduling score has been calculated. Each request carries a corresponding scheduling score. Requests are selected from the waiting queue in descending order of scheduling scores and added to the activation queue. The filling process continues until one of the following two conditions is met, at which point the filling will stop: The number of tokens corresponding to the requests that have been filled in the activation queue has reached the preset batch token limit. This batch token limit is preset according to the system's processing capacity and performance requirements to ensure parallel processing under reasonable computing resource consumption. The number of GPU memory requests in the activation queue reaches the GPU memory capacity limit to avoid system malfunctions due to memory overflow. Once the activation queue is filled, the system starts a parallel execution mechanism to perform single-token decoding operations on each request in the activation queue simultaneously. During parallel execution, the system monitors the decoding progress of each request in real time to ensure that each request is decoded according to its own generation logic; During parallel decoding, if a request completes all the generation tasks, the system immediately reclaims the computing resources occupied by that request, including GPU computing resources and video memory resources. The recycled resources will be released to the system resource pool. During the activation queue filling phase before the next round of token decoding begins, new requests will be selected from the waiting queue and filled into the activation queue, using the recycled resources for subsequent decoding processing. After completing one round of token decoding, resource reclamation, and replacement operations, the system enters the next round of token decoding, repeating the above steps of activation queue filling, parallel decoding execution, resource reclamation, and replacement until all requests have completed the generation task.

[0046] In the weighted token-level scheduling step, the soft preemption mechanism can be implemented in the following ways: Preemption trigger condition judgment steps: When the activation queue is full, the system continuously monitors the scheduling score of newly arriving requests; If a new request exists, its scheduling score is compared with the scheduling score of the lowest-scoring request currently in the active queue; When the scheduling score of a new request is significantly higher than the scheduling score of the current lowest-scoring request, and the difference between the two exceeds a preset preemption score threshold, the preemption trigger condition is determined to be met.

[0047] One-time replacement operation execution steps: When the preemption trigger condition is met, the system performs a one-time replacement operation, removing the request with the lowest current score from the activation queue and releasing the GPU computing resources and video memory resources it occupies. At the same time, new requests are populated into the activation queue so that they can participate in subsequent parallel single-token decoding operations, thereby improving the response timeliness of high-priority tasks.

[0048] Preemption constraint control steps: To avoid excessive preemption causing system performance fluctuations, the system sets a maximum preemption count attribute for each request and records the number of times it has been preempted. When a request is preempted a maximum number of times, it will no longer participate in the soft preemption mechanism during subsequent scheduling processes, ensuring that it can at least get a certain number of execution opportunities. In addition, the system continuously monitors the score difference between new requests and the lowest-scoring request in the activation queue. Even if the scheduling score of a new request is higher than that of the lowest-scoring request in the activation queue, the replacement operation will not be performed if the score difference does not reach the preset preemption score threshold, thereby ensuring the stability and fairness of the system.

[0049] In the weighted token-level scheduling step, the credit-based scheduling control mechanism can be implemented in the following ways: Each request entering the scheduling system is initialized with an accumulative credit value, with the initial credit value set to 0. Before each round of scheduling decisions, the priority weight of each request is determined. Their credit scores are updated according to priority weights. Linear growth, the growth formula is: C redit_i = C redit_i + ×Δ t ,in C redit_i For the first i The current credit value of each request, Δ t This is the time interval of the scheduling cycle.

[0050] For requests that are not scheduled, their credit score will continue to accumulate in each round of scheduling; The system monitors the credit score of each request in real time to determine whether the accumulated credit score of any request exceeds a preset scheduling threshold. Threshold The scheduling threshold is preset based on factors such as the overall system load and request priority distribution.

[0051] When the accumulated credit value of a request exceeds the scheduling threshold Threshold In this case, the request will receive a priority scheduling opportunity in the next round of scheduling; After the request is given priority scheduling and the scheduling is completed, the system deducts the corresponding credit score, and the deducted score is the scheduling threshold. Threshold ,Right now C redit_i = C redit_i Threshold This causes the credit score of the request to return to a lower level, and the credit score will continue to be accumulated according to the rules thereafter. Through the aforementioned mechanism of credit value accumulation, monitoring, priority scheduling opportunity acquisition, and point deduction, it is ensured that in the case of intense resource competition, low-priority requests can eventually obtain priority scheduling opportunities due to the continuous accumulation of credit values, thereby obtaining stable service opportunities. Meanwhile, high-priority requests have higher priority weights. w i The credit score is relatively high and grows rapidly, but points will be deducted after certain conditions are met. This avoids high-priority requests from monopolizing resources in the long term and achieves a balance between short-term response priority and long-term fairness.

[0052] In the adapter cache management process, the multi-tiered caching architecture includes: A dedicated cache area is set up on the GPU as a Level 0 cache to reside the currently executing LoRA adapter, ensuring fast access to relevant adapter data during request processing and reducing data transfer time. At the same time, short-term high-frequency access objects are stored. These objects are data that are frequently accessed in a short period of time, as determined by the system's access statistics. By residing them in the GPU cache, access to this part of the data is guaranteed to have extremely low latency, improving the overall response speed of the system. A designated area in the host memory is allocated as the first-level cache, serving as an intermediate cache layer to store candidate LoRA adapters that have been loaded but not yet entered the GPU memory. When the GPU cache space is insufficient or when the prefetching strategy determines that certain adapters need to be preloaded to a location closer to the GPU, these adapters are stored in the host memory cache. When the GPU needs to use these adapters, they can be quickly transferred from the host memory to the GPU memory, thereby accelerating repeated access to these adapters and reducing the latency caused by loading data from a more distant storage layer.

[0053] Local storage devices (such as hard drives) or remote storage servers are used as a second layer of storage, serving as a persistent storage layer to store complete LoRA adapter weight files. All LoRA adapters are initially stored in this layer of storage. When the system needs to use a particular adapter, it is loaded from this layer of storage into the upper-layer cache (host memory cache or GPU cache) according to the caching policy, providing basic support for data storage for the entire system and ensuring data persistence and integrity.

[0054] The system dynamically manages the flow of data between multiple layers of cache based on the frequency and importance of data access and the usage of cache space. When the GPU needs to access a LoRA adapter, it first searches in the GPU cache (layer 0). If it is not found, it searches in the host memory cache (layer 1). If it is not found in the host memory cache either, it loads it from local / remote storage (layer 2) into the host memory cache, and then loads it into the GPU cache as needed. When cache space is insufficient, a value-aware adapter eviction strategy is adopted to evict some data from the corresponding layer cache, freeing up space for loading new data, ensuring that the multi-layer cache architecture can run efficiently and meet the data access needs of multi-tenant LoRA inference tasks.

[0055] In the adapter cache management step, the implementation methods of the look-ahead adapter prefetch mechanism include: Set a sliding window of size W, and extract the request data within the range of the sliding window from the front of the waiting queue; The frequency of occurrence of each LoRA adapter within the sliding window is counted, and the Top - K adapters are selected from high to low frequency to form a short-term prefetch candidate set. This step can keenly capture recent load fluctuations, enabling the prefetch strategy to quickly respond to short-term request changes and have good short-term responsiveness.

[0056] The system continuously maintains a record of adapters that have been actually scheduled in the last N rounds; By employing exponential decay and time weighting, the historical access patterns of each adapter are modeled, and the hit probability of each adapter is updated. Based on the hit probability from high to low, a long-term trend candidate set is constructed. This method has stronger robustness to periodic, stable, and high-frequency access patterns and can effectively grasp long-term access trends.

[0057] Design a hybrid scoring function: Represented as:

[0058] in, Represents adapter, This is a function related to the frequency of the adapter's appearance within the sliding window. This is the adapter hit probability correlation function obtained based on long-term trend modeling. λ This is a weighting coefficient used to balance the proportion of short-term frequency and long-term trend in the scoring; Before each round of scheduling in the system, a hybrid scoring function is applied to score all candidate adapters (the union of the short-term prefetch candidate set and the long-term trend candidate set); Select the top - K adapters with the highest scores from all candidate adapters as the prefetch targets for the next round.

[0059] After the prefetching target is determined, the asynchronous loader executes the loading task in parallel, loading the relevant data of the selected Top - K adapters from the storage layer (such as local / remote storage) to the appropriate cache layer (such as host memory cache or GPU cache). By using parallel loading, the I / O and token decoding stages are fully overlapped, reducing waiting time caused by data loading and improving the overall processing efficiency of the system.

[0060] In the adapter cache management process, the implementation methods of the value-aware adapter eviction policy include: For each LoRA adapter in the cache The deportation score is calculated using the deportation scoring formula, expressed as:

[0061] in, For adapter The loading cost, which takes into account historical loading time and PCIe latency, reflects the time cost consumed during the adapter loading process; Indicates adapter The size reflects the space resources it occupies in the cache; For adapter The future access probability is used to predict the likelihood that the adapter will be accessed subsequently; ω , θ , κ These are preset weighting coefficients used to adjust the weighting of loading cost, adapter size, and future access probability in the eviction score.

[0062] The system monitors the usage of video memory in real time. When the video memory usage approaches the system's set capacity threshold, the adapter replacement mechanism is triggered. Based on the calculated deportation score, according to EScore Several adapters were selected as candidates for eviction in descending order of their scores. These high-scoring adapters are those with high loading costs, large memory footprints, and relatively low probability of future access, making them more suitable for being eliminated to free up video memory resources.

[0063] To ensure the correctness of the execution process, a reference count is maintained for each adapter in use; the adapter is only allowed to be evicted after its task is completed and the reference count is zero, thus avoiding the accidental removal of the adapter while it is still in use and ensuring the stability of the system and the consistency of the data.

[0064] Remove the selected candidate adapter from video memory, freeing up the video memory resources it occupies and making room for the loading of the new adapter or other data processing operations; As can be seen, the value-aware adapter eviction strategy provided in this embodiment effectively avoids the problem of high-reuse and high-cost adapters being mistakenly evicted in traditional strategies. By comprehensively considering factors such as loading cost, adapter size and future access probability, it improves memory utilization and I / O scheduling stability.

[0065] This embodiment also provides a weighted scheduling and caching system for multi-tenant LoRA inference. The system adopts a distributed modular architecture and mainly includes: a request accessor, a priority shaper, a weighted token scheduler, an adapter cache manager, and an execution feedback and tuner.

[0066] The request access unit is used to receive inference requests from each tenant, extract metadata including arrival time, maximum decoding length, adapter ID, service priority, and initialize scheduling labels; classify requests into high, medium and low priorities and insert them into the candidate pool for subsequent scheduling module processing; Among them, the priority shaper is used to dynamically adjust the token share of tasks with different priorities entering the waiting queue; the sliding window records the SLO achievement status of each priority in a recent period of time; if high-priority SLOs frequently default, the system increases their token quota. Among them, the weighted token-level scheduler is used to introduce a weight-aware token-level scheduling mechanism, which takes into account the real-time response, system fairness and throughput efficiency; it calculates a scheduling score for each request based on multiple factors such as request weight, cumulative waiting time and maximum decoding length, and finely controls the execution order; Before each round of token decoding begins, a group of requests is selected from the waiting queue in descending order of score and added to the activation queue, until the batch token limit or GPU memory capacity limit is reached. Subsequently, requests in the activation queue will perform single-token decoding in parallel. If a request completes its generation, its resources will be reclaimed and replaced by a new request in the next round.

[0067] To further accelerate the response to high-priority requests, this embodiment designs a soft preemption mechanism: when the activation queue is full, if a new request has a score significantly higher than the current lowest-scoring request and exceeds a preset threshold, the system will perform a one-time replacement operation to improve the timeliness of high-priority task responses. Meanwhile, to avoid performance fluctuations caused by excessive preemption, the system sets a maximum number of preemption attempts and a score threshold to constrain this process.

[0068] Furthermore, to alleviate the starvation problem of low-priority requests, this embodiment introduces a credit-based scheduling control mechanism, maintaining an accumulative credit value for each request and scheduling it according to its weight. w i Linear growth. The credit value of unscheduled requests accumulates round by round. When the accumulated value exceeds the scheduling threshold, the request will be given a priority scheduling opportunity and the corresponding points will be deducted. This ensures that low-priority requests can still get stable service opportunities in the event of intense resource competition, thereby achieving a balance between short-term response priority and long-term fairness.

[0069] At the system implementation level, the WTLS scheduler implementation of this invention runs entirely on the CPU side. All scheduling logic is completed before GPU decoding, without introducing any GPU kernel copying or switching overhead, thus maintaining the stability and efficiency of the execution flow. Furthermore, after each round of scheduling decisions, the scheduler synchronously sends the adapter usage prediction for the next round of requests to the adapter cache manager. This, combined with forward-looking adapter prefetching and cache management, minimizes the impact of loading latency on service performance.

[0070] The adapter cache manager includes: (1) Multi-layer caching architecture GPU cache (level 0): Used to reside the currently executing adapter and short-term, high-frequency access objects, ensuring extremely low access latency; Host memory (first layer): As an intermediate cache, it stores candidate adapters that have been loaded but have not yet entered video memory, in order to accelerate repeated access; Local / Remote Storage (Second Layer): As a permanent storage layer, it stores the complete LoRA adapter weight file.

[0071] (2) Look-ahead adapter prefetching mechanism Sliding window statistics (recent frequency): The system extracts a sliding window of size W from the front of the waiting queue, counts the frequency of each adapter's appearance within it, and selects the Top-K adapters as a short-term prefetch candidate set. This strategy can accurately reflect recent load fluctuations and has good responsiveness.

[0072] Lightweight historical access modeling (long-term probability): The system continuously maintains records of adapters actually scheduled over the past N rounds, and updates the hit probability of each adapter using an exponential decay and time-weighted approach to construct a long-term trend candidate set. This method exhibits stronger robustness to periodic, stable, and high-frequency access patterns.

[0073] Before each round of scheduling, the system selects the top-K adapters with the highest scores from all candidates as the prefetch targets for the next round. The asynchronous loader executes the loading tasks in parallel, thereby achieving full overlap between the I / O and token decoding stages.

[0074] When video memory usage approaches the system's set capacity threshold, the system selects several adapters as eviction candidates in descending order of EScore and releases the resources they occupy. To ensure the correctness of the execution process, the system maintains a reference count for each adapter in use, and eviction is only allowed after the task is completed and the reference count reaches zero. This eviction mechanism effectively avoids the problem of highly reused and high-cost adapters being mistakenly evicted in traditional strategies, improving video memory utilization and I / O scheduling stability.

[0075] The execution feedback and tuner is used to collect runtime data (SLO defaults, cache hit rate, I / O latency, etc.) and feed it back to the scheduling and caching module in a closed loop.

[0076] In practical applications, when a request arrives, the request accessor initializes its scheduling label and sends it to the candidate request pool; the priority shaper adjusts the token admission ratio of the three priority queues according to the SLO achievement rate in the sliding window to achieve a dynamic balance between critical request protection and overall throughput; subsequently, the scheduler in this embodiment selects a group of high-scoring requests to form a decoding batch at the token granularity, and prioritizes the satisfaction of high-priority or long-waiting-time requests. During the decoding execution phase, the adapter cache manager initiates the look-ahead adapter prefetching module to predict and load the LoRA adapters that may be used in the next round, while executing a value-aware eviction policy. After execution, the feedback and tuning module records key metrics (such as latency, hit rate, I / O pressure, etc.) to drive the adaptive adjustment of priority allocation, scheduling score parameters and cache update policies in the next round.

[0077] As can be seen, this implementation, through decoupled collaboration and feedback-driven system design, enables each functional module to be optimized independently while forming an efficient linkage closed loop, demonstrating excellent scalability and service stability in scenarios such as multi-tenancy, high dynamic load and heterogeneous tasks.

[0078] In terms of scheduling mechanism, the system introduces a dynamic priority shaping strategy. Unlike traditional static queue partitioning methods, the priority shaper module adjusts the token entry ratio for high, medium, and low priority requests in each scheduling round in real time based on the SLO achievement rate calculated using a sliding window. If high-priority requests frequently default, the system will increase their token admission cap and increase the resource allocation for high-priority requests. Conversely, if high-priority tasks have fully met their SLOs, the system will release some resources to medium and low-priority tasks to improve overall throughput. To prevent the system from prematurely reverting resource allocations for high-priority requests due to short-term improvements in SLO achievement rates, this embodiment designs a default rate rollback suppression strategy, setting a rollback delay window and a minimum allocation threshold to mitigate system fluctuations caused by frequent scheduling adjustments.

[0079] At the scheduling execution level, this embodiment constructs a weighted token-level scheduler, allocating resources based on a single token. The system dynamically calculates a scheduling score for each request to be scheduled, taking into account the request's priority weight, cumulative waiting time, and maximum decoding length. Before each round of token decoding, the scheduler selects a group of requests with the highest scores from the candidate pool to form an activation queue, maximizing the system's response timeliness and service fairness. To support elastic scheduling under resource constraints, this embodiment introduces a soft preemption mechanism, allowing currently executing low-score requests to be preempted by urgent high-priority requests. A credit-based scheduling control mechanism is used to avoid starvation: the system maintains an accumulative credit value for all requests. Unscheduled requests have their credit values ​​increase over time, and requests exceeding a set threshold gain a promotion opportunity, ensuring that even low-priority requests can receive stable service.

[0080] Meanwhile, this embodiment designs a full lifecycle caching mechanism for LoRA adapter management, covering the prefetching, loading, residency, and eviction processes of adapters. To avoid frequent cold loading delays, the system initiates a look-ahead adapter prefetching module during each round of decoding. Based on sliding window statistics and historical access modeling, it predicts the set of adapters most likely to be scheduled in the next round and loads them in parallel with the current batch, significantly reducing the risk of PCIe I / O blocking. Regarding the eviction strategy, this embodiment designs a value-aware eviction strategy based on value scoring. It comprehensively considers the adapter's parameter size, loading time, and future hit probability to assign learnable priority weights to cached objects, thereby prioritizing the release of low-value adapters under memory-constrained conditions. It also supports downgrading to CPU-level cache to enhance the efficiency of tiered storage collaboration.

[0081] In summary, this invention proposes a unified framework for multi-tenant LoRA inference, addressing key challenges such as adapter contention management under GPU memory constraints and SLO defaults caused by dynamic workloads. This framework achieves low-latency, high-throughput services under dynamic workloads. To advance urgent tasks promptly while avoiding starvation, this invention designs a weighted token-level scheduling strategy that integrates request priority, cumulative waiting time, and maximum decoding length, and introduces soft preemption and a credit-based scheduling control mechanism. This invention proposes a cache-aware adapter management strategy that combines look-ahead prefetching with value-aware eviction to reduce I / O latency and improve memory efficiency under strict GPU memory constraints.

[0082] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0083] The above embodiments are merely preferred embodiments of the present invention and should not be construed as limiting the scope of protection of the present invention. Any non-substantial changes and substitutions made by those skilled in the art based on the present invention shall fall within the scope of protection claimed by the present invention.

Claims

1. A weighted scheduling and caching method for multi-tenant LoRA inference, characterized in that, include: Access request and initialization steps: Receive inference requests from multiple tenants, extract metadata for each request, including arrival time, maximum decoding length, adapter ID, service priority, and initialize the scheduling label for each request based on the extracted metadata; Priority dynamic adjustment steps: Use a sliding window statistics mechanism to record the achievement of service level targets corresponding to each service priority in the recent period, and dynamically adjust the token share of different priority tasks entering the waiting queue based on the statistical results; Weighted Token-level scheduling steps: Calculate the scheduling score for each request based on its priority weight, cumulative waiting time, and maximum decoding length. Perform fine-grained scheduling at the token level based on the scheduling score, and introduce a soft preemption mechanism and a credit-based scheduling control mechanism during the scheduling process. Adapter cache management steps: A multi-layered cache architecture including GPU cache, host memory cache and local / remote storage is adopted. A forward-looking adapter prefetching mechanism is used to predict and load LoRA adapters that may be used in the future. At the same time, a value-aware adapter eviction strategy is adopted to dynamically optimize the GPU memory usage based on the adapter loading cost, size and future access probability. Execution feedback and optimization steps: Collect key data during system operation, and feed the operational data back to the scheduling module and cache management module in a closed loop to drive the system to dynamically adjust the scheduling strategy and cache management strategy.

2. The method according to claim 1, characterized in that: During the request access and initialization steps, a scheduling tag is generated for each request based on the extracted metadata. The scheduling tag contains the following dynamic fields: Priority Classification: Maps service priorities to discrete priority categories for token quota allocation in the priority shaper; Initial scheduling score: The initial score is calculated using a weighted formula based on parameters such as priority weight and maximum decoding length; Time-related markers: Attach arrival timestamp and current system time to calculate cumulative waiting time; Adapter dependency information: Records the adapter ID and cache status, which is used by the adapter cache manager to make prefetching or eviction decisions; The initialized requests and their scheduling tags are stored in the candidate request pool, and a multi-level index structure is established.

3. The method according to claim 1, characterized in that: In the priority dynamic adjustment step, a fixed-size sliding window is maintained to record the service level target achievement status of each priority task in each scheduling cycle; For each priority task, count the number of SLO defaults and the total number of requests within the sliding window, and calculate the SLO achievement rate; Based on the SLO achievement rate of each priority task, the token share entering the waiting queue is dynamically adjusted, which is the upper limit of the number of tokens allowed to enter the waiting queue in each round of scheduling. Update the Token share configuration table for each priority task based on the dynamic quota calculation results; At the start of each round of scheduling, requests are selected from the candidate request pool in priority order according to the Token share limit in the configuration table and enter the waiting queue. If the number of requests for a priority task exceeds its token share limit, the excess requests will be temporarily stored in a buffer queue and will participate in the quota allocation again in the next round of scheduling.

4. The method according to claim 1, characterized in that: In the weighted token-level scheduling step, the priority weight of the request is obtained from the scheduling tag of the request. Cumulative waiting time Maximum decoding length The scheduling score for each request is calculated using a comprehensive scoring formula, expressed as follows: in, α , β , γ These are preset weighting coefficients used to adjust the proportion of priority weights, cumulative waiting time, and factors related to the maximum decoding length in the overall score. α > β > γ ; The calculated scheduling score for each request Recorded in the dispatch tag of the request.

5. The method according to claim 4, characterized in that: Before each round of token decoding begins, the scheduling module accesses the waiting queue, which stores multiple requests to be processed after the scheduling score has been calculated. Each request carries a corresponding scheduling score. Requests are selected from the waiting queue in descending order of scheduling scores and added to the activation queue. The filling process continues until the following conditions are met, at which point filling will stop: The number of tokens corresponding to the requests already populated in the activation queue has reached the preset batch token limit; Alternatively, the GPU memory usage of requests in the activation queue reaches the GPU memory capacity limit; Once the activation queue is filled, the system starts a parallel execution mechanism to perform single-token decoding operations on each request in the activation queue simultaneously. During parallel execution, the decoding progress of each request is monitored in real time to ensure that each request is decoded according to its own generation logic; During parallel decoding execution, if a request completes all the generation tasks, the system immediately reclaims the computing resources occupied by that request. The recycled resources will be released to the system resource pool. During the activation queue filling phase before the next round of token decoding begins, new requests will be selected from the waiting queue and filled into the activation queue, where the recycled resources will be used for decoding.

6. The method according to claim 5, characterized in that, The implementation methods of soft preemption mechanisms include: Preemption trigger condition judgment steps: When the activation queue is full, the system continuously monitors the scheduling score of newly arriving requests; If a new request exists, its scheduling score is compared with the scheduling score of the lowest-scoring request currently in the active queue; When the scheduling score of a new request is significantly higher than the scheduling score of the current lowest-scoring request, and the difference between the two exceeds the preset preemption score threshold, the preemption trigger condition is determined to be met. One-time replacement operation execution steps: When the preemption trigger condition is met, the system performs a one-time replacement operation, removing the request with the lowest current score from the activation queue and releasing the GPU computing resources and video memory resources it occupies; at the same time, the new request is filled into the activation queue. Preemption constraint control steps: To avoid excessive preemption causing system performance fluctuations, a maximum preemption count attribute is set for each request to record the number of times it has been preempted; When a request is preempted a maximum number of times, it will no longer participate in the soft preemption mechanism during subsequent scheduling processes, ensuring that it can at least get some execution opportunities.

7. The method according to claim 1, characterized in that, Credit-based scheduling control mechanisms can be implemented in the following ways: Each request entering the scheduling system is initialized with an accumulative credit value, with the initial credit value set to 0. Before each round of scheduling decisions, the priority weight of each request is determined. Their credit scores are updated according to priority weights. Linear growth, the growth formula is: C redit_i = C redit_i + ×Δ t ,in C redit_i For the first i The current credit value of each request, Δ t The time interval for the scheduling cycle; For requests that are not scheduled, their credit score will continue to accumulate in each round of scheduling; The system monitors the credit score of each request in real time to determine whether the accumulated credit score of any request exceeds a preset scheduling threshold. Threshold ; When the accumulated credit value of a request exceeds the scheduling threshold Threshold In this case, the request will receive a priority scheduling opportunity in the next round of scheduling; After the request is given priority scheduling and the scheduling is completed, the system deducts the corresponding credit score, and the deducted score is the scheduling threshold. Threshold ,Right now C redit_i = C redit_i Threshold .

8. The method according to claim 1, characterized in that, Multi-layered caching architecture includes: A dedicated cache area is set up on the GPU as a level 0 cache to house the currently executing LoRA adapter; it is also used to store short-term, high-frequency access objects. A designated area in the host memory is allocated as the first-level cache, which serves as an intermediate cache layer to store candidate LoRA adapters that have been loaded but have not yet entered the GPU memory. When the GPU cache space is insufficient or when the prefetching strategy determines that certain adapters need to be preloaded to a location closer to the GPU, these adapters are stored in the host memory cache. Local storage devices or remote storage servers are used as the second layer of storage, serving as a persistent storage layer to store complete LoRA adapter weight files. All LoRA adapters are initially stored in this layer of storage. When the system needs to use a particular adapter, it is loaded from this layer of storage into the upper-layer cache according to the caching policy. Based on the frequency and importance of data access and the usage of cache space, the flow of data between multiple cache layers is dynamically managed. When the GPU needs to access a LoRA adapter, it first searches in the GPU's Level 0 cache. If it is not found, it searches in the Level 1 cache. If it is not found in the host memory cache either, it is loaded from the Level 2 storage into the host memory cache, and then further loaded into the GPU cache as needed. When the cache space is insufficient, a value-aware adapter eviction policy is used to evict some data from the corresponding layer cache.

9. The method according to claim 8, characterized in that, The implementation methods of the look-ahead adapter prefetch mechanism include: Set a sliding window of size W, and extract the request data within the range of the sliding window from the front of the waiting queue; The frequency of occurrence of each LoRA adapter within the sliding window is counted, and the Top-K adapters are selected from high to low frequency to form a short-term prefetch candidate set; We model the historical access patterns of each adapter using an exponential decay and time-weighted approach, and update the hit probability of each adapter. Based on the hit probability, we construct a long-term trend candidate set from high to low. Design a hybrid scoring function: Represented as: in, Represents adapter, This is a function related to the frequency of the adapter's appearance within the sliding window. This is the adapter hit probability correlation function obtained based on long-term trend modeling. λ This is a weighting coefficient used to balance the proportion of short-term frequency and long-term trend in the scoring; Before each round of scheduling in the system, a hybrid scoring function is applied to score all candidate adapters; Select the top - K adapters with the highest scores from all candidate adapters as the prefetching targets for the next round; After the prefetching target is determined, the asynchronous loader executes the loading task in parallel, loading the relevant data of the selected Top - K adapters from the storage layer to the appropriate cache layer.

10. The method according to claim 9, characterized in that, The implementation methods of value-aware adapter replacement strategies include: For each LoRA adapter in the cache The deportation score is calculated using the deportation scoring formula, expressed as: in, For adapter Loading cost; Indicates adapter Size; For adapter The probability of future visits; ω , θ , κ These are preset weighting coefficients used to adjust the weighting ratios of loading cost, adapter size, and future access probability in the eviction score; The system monitors the usage of video memory in real time. When the video memory usage approaches the system's set capacity threshold, the adapter replacement mechanism is triggered. Based on the calculated deportation score, according to EScore Select several adapters as expulsion candidates in descending order of their quality; Remove the selected candidate adapter from video memory, freeing up the video memory resources it occupies and making room for the loading of the new adapter or other data processing operations.

Images