A low-latency scheduling method, system and medium for real-time interactive AI services

Abstract

The application discloses a low-delay scheduling method and system for real-time interactive AI services and a medium, the method comprising: determining an initial priority coefficient according to each tenant level and task type; dynamically analyzing the priority coefficient of each task request according to the waiting time; predicting and dynamically allocating the video memory space capacity of a first cache area; obtaining the real-time residual amount of the video memory capacity of the predicted and allocated cache area, dynamically analyzing the upper limit value of the number of task requests processed in parallel, and based on the number of currently waiting task requests and the preset regulation mode, analyzing the number of currently optimal task requests processed in parallel to balance resource utilization and delay waiting time. The application effectively solves the delay problem of existing AI services when processing high real-time and high-interactive tasks, improves the service response speed and efficiency of real-time interactive tasks, and meets the strict performance index standards of various high-demand AI services.

Description

Technical Field

[0001] This invention belongs to the field of AI service technology and relates to a low-latency scheduling system, method and medium for real-time interactive AI services. Background Technology

[0002] With the rapid development of artificial intelligence technology, GPU-based online services have been widely applied in interactive scenarios such as real-time voice calls, real-time video generation, real-time game interaction, and AIGC dialogues. These AI services, designed for real-time interaction, have extremely high requirements for response latency, typically needing to complete request processing within milliseconds or even sub-milliseconds to ensure a smooth and immediate interactive experience for users. However, existing AI service systems generally suffer from the following technical shortcomings when handling mixed loads of high concurrency, multi-tenancy, and multiple task types: First, traditional scheduling methods typically employ a first-come, first-served (FFS) or a single fixed-priority strategy, failing to effectively differentiate the latency requirements of different tenant levels and task types (real-time tasks, regular tasks, background tasks). Second, existing systems usually dynamically allocate GPU memory space when a task starts. This process involves time-consuming operations such as memory allocator calls and address mapping, resulting in significant latency jitter in scenarios with extremely high real-time requirements. Particularly for high-priority real-time tasks, the unpredictability of dynamic GPU memory allocation becomes a bottleneck on the critical path. Existing technologies lack intelligent predictive allocation methods based on task priority and historical load, failing to proactively adjust cache capacity according to dynamic changes in task requests, leading to resource waste or under-allocation. Finally, existing service systems generally use batch processing, accumulating a certain number of task requests before executing computations uniformly. However, a fixed number of parallel task requests is difficult to adapt to dynamically changing task requests. Executing computations after accumulating a certain number of task requests increases the queuing time for individual task requests, which is detrimental to real-time interactive tasks. Conversely, processing a large number of real-time task requests leads to decreased throughput, failing to effectively balance latency and throughput.

[0003] Therefore, for real-time interactive AI services, there is an urgent need for a low-latency scheduling method to solve the problem of balancing latency and throughput, so as to meet the stringent performance standards of high real-time and high interactivity AI services. Summary of the Invention

[0004] The purpose of this invention is to provide a low-latency scheduling system, method, and medium for real-time interactive AI services, thereby solving the aforementioned technical problems.

[0005] The objective of this invention can be achieved through the following technical solutions: A low-latency scheduling method for real-time interactive AI services includes: Obtain task requests from each tenant and determine the initial priority coefficient based on each tenant's level and task type; The system obtains the waiting time of each task request under each tenant in real time, dynamically analyzes the priority coefficient of each task request, and adjusts the priority processing order of each task request according to the adjusted priority coefficient of each task request. The GPU memory space is pre-divided into a first cache area and a second cache area, and the memory space capacity of the first cache area and the second cache area is predicted and dynamically allocated to obtain memory capacity in a timely manner and reduce allocation time. The first cache area is used to store task requests with a priority coefficient not less than the set priority threshold, and the second cache area is used to store task requests with a priority coefficient less than the set priority threshold. The system obtains the real-time remaining amount of video memory in the predicted cache area, dynamically analyzes the upper limit of the number of parallel processing task requests, and analyzes the current optimal number of parallel processing task requests based on the number of currently waiting task requests and the preset control mode. Based on the current number of waiting task requests, the system determines the current optimal number of parallel processing task requests and performs batch processing calculations to balance resource utilization and latency.

[0006] Furthermore, obtain the tenant level and task type corresponding to each task request; The initial priority coefficient of each task request is dynamically evaluated by multiplying the priority value corresponding to the tenant level and the priority weight coefficient corresponding to the task type. Based on the dynamic evaluation of the initial priority coefficient of each task request, the task requests are prioritized and processed first.

[0007] Furthermore, based on the relationship between the priority coefficient of the task request and the set priority threshold, high priority is determined. Based on whether there are high-priority task requests, the video memory space of the first cache area is predictively allocated. The prediction allocation method includes the following steps: Determine whether there are any task requests whose priority coefficient is not less than the set priority threshold. If so, extract the maximum actual video memory space required by the high-priority task request in the previous detection period. Otherwise, do not reserve the video memory capacity of the first cache area in advance. Determine whether the maximum actual video memory space required by high-priority task requests in the previous detection period is greater than the video memory capacity of the first cache. If it is greater than the video memory capacity of the first cache, dynamically compensate the video memory capacity of the second cache to the video memory capacity missing in the first cache. The video memory capacity dynamically compensated by the second cache to the video memory capacity missing in the first cache is equal to the difference between the predicted maximum actual video memory space required and the current video memory capacity of the first cache. Based on the maximum actual video memory space required by high-priority task requests in the previous detection cycle, the video memory capacity of the first cache area is reserved and locked. Extract the sum of video memory required by low-priority task requests in the previous detection period, and determine whether the current video memory capacity of the second cache is less than the sum of video memory required by low-priority task requests. If it is less, dynamically increase the current video memory capacity of the second cache; if it is greater, dynamically decrease the current video memory capacity of the second cache, so as to realize dynamic prediction of the increase or decrease of video memory space according to task requests.

[0008] Furthermore, the method for predictively allocating the video memory space of the first cache includes: Determine whether the triggering condition for predicted allocation is met. If the triggering condition is met, extract the arrival rate of task requests, the length of the request queue, and historical load data. Based on the arrival rate of task requests, the length of the request queue, and historical load data, analyze the predicted memory demand values ​​of the first and second cache areas within a preset time window. The triggering condition is time or load. Compare the predicted memory demand of the first cache with the current memory capacity of the first cache, calculate the missing memory capacity of the current first cache, and if the missing memory capacity of the current first cache is greater than 0, generate an active memory capacity pre-allocation instruction to actively allocate memory capacity for the first cache. According to the memory capacity pre-allocation instruction, the memory capacity that matches the missing memory capacity is obtained from the GPU memory space to update the memory capacity of the current first cache, so that the memory capacity of the first cache after allocation is not less than the predicted memory demand value of the corresponding cache.

[0009] Furthermore, the method for predicting the triggering condition includes: Set a time window and count the arrival time of each high-priority task request within the time window; Calculate the arrival interval between adjacent task requests, and calculate the average arrival interval within the window; The idle determination threshold is dynamically calculated based on the average arrival interval; Obtain the arrival time of the latest task request and the current time. Determine whether the difference between the current time and the arrival time of the latest task request is greater than the idle threshold. If it is greater than the set idle threshold, the trigger condition for prediction allocation is met, and the memory demand prediction for the first cache area is started.

[0010] Furthermore, the number of high-priority task requests arriving within two adjacent time windows is obtained, and the average arrival rate of task requests within the window is calculated based on the number of task requests arriving within the time window. Determine whether the average arrival rate of task requests in the next time window reaches the average arrival rate of task requests in the previous time window. If it does, the prediction allocation trigger condition is met, and prediction allocation of the video memory capacity of the first cache area is initiated.

[0011] Furthermore, based on the arrival rate of task requests, the length of the request queue, and historical load data, the predicted values ​​of video memory demand for the first and second caches within the time window are analyzed, and based on the currently allocated video memory capacity of the first and second caches, the increase in video memory demand required by the first and second caches is analyzed.

[0012] Furthermore, based on the relationship between the number of task requests of the same priority level currently waiting to be processed and the set lower and upper limits of the number of task requests of that priority level, the number of task requests to be processed in parallel is dynamically processed to optimize the current optimal number of task requests to be processed in parallel.

[0013] This invention provides a low-latency scheduling system for real-time interactive AI services, comprising: The priority determination module is used to obtain the task requests from each tenant and determine the initial priority coefficient based on the tenant's level and task type. The priority update module is used to obtain the waiting time of each task request under each tenant in real time, dynamically analyze the priority coefficient of each task request, and adjust the priority processing order of each task request according to the adjusted priority coefficient of each task request. The resource allocation module is used to pre-divide the GPU memory space, divide it into a first cache area and a second cache area, and predict and dynamically allocate the memory space capacity of the first cache area and the second cache area to obtain the memory capacity in a timely manner and reduce the allocation time. The optimization processing module obtains the real-time remaining amount of the video memory capacity of the predicted allocated cache area, dynamically analyzes the upper limit of the number of parallel processing task requests, and analyzes the current optimal number of parallel processing task requests based on the number of currently waiting task requests and the preset control mode. Based on the current number of waiting task requests, it determines the current optimal number of parallel processing task requests and performs batch processing calculations to balance resource utilization and latency.

[0014] The present invention provides a computer-readable storage medium including a computer program, which, when executed by a processor, implements the low-latency scheduling method described in any of the preceding claims.

[0015] The beneficial effects of this invention are: When faced with high-priority task requests requiring real-time interaction, this invention analyzes the priority of each tenant's task requests, classifying them into high and low priorities. Task requests requiring real-time interaction are designated as high priority, and a cache for storing these high-priority requests is pre-allocated. This cache is dynamically adjusted based on historical data, and the number of parallel processing requests for high-priority tasks is adjusted at the computational level. This achieves collaborative optimization from request scheduling and resource allocation to execution computation, effectively solving the latency problem existing in current AI services when handling highly real-time and highly interactive tasks. It improves the service response speed and efficiency of real-time interactive tasks, meeting the stringent performance standards of various demanding AI services.

[0016] This invention employs two dimensions—tenant level and task type—for quantitative analysis to initially quantify priorities. Compared to a single priority adjustment strategy, this approach can satisfy real-time interactive task requests while also fulfilling diverse service needs, ensuring that real-time interactive task requests are processed first and reducing response latency from the source.

[0017] This invention dynamically adjusts the initial quantified priority of each task request by measuring the waiting time of each task request under each tenant, thereby obtaining an updated priority quantified value. Based on the updated priority quantified value, task requests with higher priority quantified values ​​are processed first, so that the priority coefficient of each task request increases with the waiting time. This ensures that tasks with higher priority coefficients are processed first, avoiding continuous priority service for high-priority task requests and preventing low-priority task requests from being interrupted. While ensuring low latency for high-priority tasks, it also takes into account the fairness and executability of all tasks, effectively balancing high-priority and low-priority task requests and improving the overall service quality of the system.

[0018] This invention determines the memory capacity of a first cache by judging the relationship between the maximum actual GPU memory required by a high-priority task request in the previous detection period and the GPU memory capacity of a first cache pre-allocated to store high-priority task requests when a high-priority task request exists. Based on historical data, the invention reserves and locks the GPU memory capacity of the first cache according to the maximum actual GPU memory required by high-priority task requests. It also allows the GPU memory capacity of a second cache to dynamically compensate for the missing GPU memory capacity in the first cache, thereby meeting the resource requirements of high-priority task requests. At the same time, it can reserve resources for high-priority task requests in advance, avoiding the latency problem of GPU memory allocation for high-priority task requests, and achieving zero latency for high-priority task requests, thus providing low latency guarantee for real-time interactive AI services.

[0019] This invention constructs a memory demand prediction model based on the arrival rate of high-priority task requests, request queue length, and historical load data. This model analyzes the predicted memory demand of the first cache within a preset time window. Based on the relationship between the predicted memory demand and the pre-allocated memory capacity of the first cache for storing high-priority task requests, it compensates for any missing memory capacity in the first cache. During the compensation process, the time window and / or load are used as triggering conditions to determine whether to predict and allocate memory capacity in the first cache. This proactively expands the first cache to address sudden trends or periodic patterns in memory capacity required by high-priority task requests, avoiding allocation delays due to resource shortages and improving the system's robustness and throughput efficiency in dynamic, high-concurrency scenarios.

[0020] This invention calculates the upper limit of the number of parallel processing task requests based on the real-time remaining capacity of the pre-allocated first cache memory and the safe capacity threshold of the first cache memory. It then determines the relationship between the number of task requests at the same priority level and the upper and lower limits of the number of parallel processing task requests. Using a preset control model, it determines the number of parallel processing task requests at the same priority level. When the number of task requests at the same priority level is between the upper and lower limits, it calculates the waiting time corresponding to the task requests at the same priority level. This enables dynamic adjustment of the number of parallel processing task requests to balance the efficiency and latency of task request processing, achieving the optimal dynamic balance between computing resource utilization and task response latency, and improving the overall throughput of the system. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating the low-latency scheduling method of the present invention; Figure 2 This is a schematic diagram of the low-latency scheduling system in this invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] Real-time interactive AI services place extreme demands on response latency, and the tenant experience is highly dependent on the system's ability to provide low-latency real-time feedback. In real-time interactive scenarios, any perceptible stutter or delay can significantly degrade the interactive experience, impair fluency, and in severe cases, even cause interaction interruption, thus affecting service quality. However, existing AI service systems struggle to achieve both high throughput and low latency and high stability when dealing with real-time interactive AI services, necessitating a low-latency scheduling method.

[0025] like Figure 1 As shown, this invention discloses a low-latency scheduling method for real-time interactive AI services, comprising the following steps: Step 1: Obtain the task requests from each tenant and determine the initial priority coefficient based on the tenant's level and task type; For VIP tenants and regular tenants, the corresponding priority values ​​are P1 and P2, respectively, with P1 > P2. Under the same tenant, different priority weights are assigned to different task types based on their required real-time conditions. The priority weight coefficient for real-time tasks is A1, for regular tasks it is A2, and for background tasks it is A3. The specific values ​​of priority weight coefficients A1, A2, and A3 are set as needed, with priority weight coefficient 1 > A1 > A2 > A3 > 0, and A1 + A2 + A3 = 1.

[0026] The real-time task type is a real-time interactive request, which requires millisecond-level response, such as real-time voice calls, real-time video generation, and real-time game interaction. The ordinary task type is an ordinary interactive request, which allows for latency, such as text dialogue and image generation. The background task type is a background processing request, which involves a large amount of computation and is not sensitive to latency, such as offline model analysis and background log processing.

[0027] The priority values ​​P1 and P2 are both constants less than a set threshold value, and are set as needed. To avoid VIP's ordinary task types and various task types under ordinary tenants remaining in a waiting state under continuous real-time task types, the tenant level and task type corresponding to each task request are obtained. The initial priority coefficient of each task request is dynamically evaluated by multiplying the priority value corresponding to the tenant level and the priority weight coefficient corresponding to the task type. Based on the dynamically evaluated initial priority coefficient of each task request, the task requests are prioritized and processed first, with the task request with the highest priority coefficient being processed first.

[0028] Specifically, the initial priority coefficient for each task request is obtained based on the tenant level and the tenant's task type. The formula for calculating the initial priority coefficient is: K j =p j *A ij , where K j p represents the initial priority coefficient corresponding to the j-th task request. j This represents the priority value corresponding to the tenant level that issued the j-th task request, taking the value P1 or P2, A ij A represents the priority weight coefficient of the j-th task request issued. ij It equals one of the priority weight coefficients A1, A2, and A3.

[0029] Step 2: Obtain the waiting time of each task request under each tenant in real time, dynamically analyze the priority coefficient of each task request, and adjust the priority order of each task request according to the adjusted priority coefficient of each task request. To prevent task requests with lower initial priority coefficients from always being placed after task requests with higher initial priority coefficients, thus causing the lower priority task requests to be in a "starved" state, the waiting time of each task request under each tenant is obtained. Based on the waiting time of each task request, the priority coefficient of the task requests is dynamically adjusted so that each task request is processed in sequence according to the adjusted priority coefficient.

[0030] Based on the waiting time of each task request under each tenant, the priority coefficient of each task request is dynamically updated. The dynamic update model adopted is as follows: K △tj =K j +g*△tj, where K △tjThis represents the updated priority coefficient of each task request. △tj represents the waiting time for the j-th task request, which is equal to the difference between the current time and the arrival time of that task request (the scheduler's reception time). g is a coefficient reflecting the rate at which the priority coefficient increases with the waiting time, wherein the coefficient g ≥ △K / T0. △P represents the maximum difference in the initial priority coefficients, i.e., △K = K. max -K min T0 represents the maximum allowed fair waiting time.

[0031] By analyzing the waiting time of each task request, the priority coefficient of each task request is dynamically updated. This allows for priority adjustment based on the updated task requests, preventing low-priority task requests from being indefinitely interrupted by high-priority task requests, thus avoiding continuous delays for low-priority tasks.

[0032] Step 3: Pre-divide the GPU memory space into a first cache area and a second cache area, and predict and dynamically allocate the memory space capacity of the first cache area and the second cache area to obtain the memory capacity in a timely manner and reduce the allocation time.

[0033] The first cache is used to store task requests with high priority values, and the second cache is used to store task requests with dynamically changing high priority values ​​or task requests with priority values ​​lower than high priority values. The sum of the video memory space of the first cache and the second cache is less than the GPU video memory space.

[0034] The first and second caches are pre-allocated, ready-to-use pools of GPU memory resources. They are used to eliminate memory allocation latency when the GPU inference service starts, achieving near-zero-wait resource acquisition and shortening allocation time.

[0035] This embodiment discloses a predictive allocation method that predictively allocates video memory space in the first and second cache areas based on the existence of high-priority task requests. The predictive allocation method includes the following steps: Step A31: Determine whether there are any task requests whose priority coefficient is not less than the set priority threshold. If so, extract the maximum actual video memory space required by the high-priority task request in the previous detection period. Otherwise, do not reserve the video memory capacity of the first cache area in advance. Task requests with a priority coefficient not less than the set priority threshold are designated as high priority, while task requests with a priority coefficient less than the set priority threshold are designated as low priority.

[0036] Step A32: Determine whether the maximum actual video memory space required by the high-priority task request in the previous detection period is greater than the video memory capacity of the first cache. If it is greater than the video memory capacity of the first cache, dynamically compensate the video memory capacity of the second cache to the video memory capacity missing in the first cache. Wherein, the second cache dynamically compensates for the missing video memory capacity of the first cache, which is equal to the difference between the predicted maximum actual video memory space and the current video memory capacity of the first cache.

[0037] Step A33: Based on the maximum actual video memory space required by high-priority task requests in the previous detection period, reserve the video memory capacity of the first cache area and lock the video memory capacity of the first cache area so that the video memory capacity of the first cache area is only used for the resources required by high-priority task requests, and prohibit task requests other than high-priority values ​​from occupying it. Based on the maximum actual video memory space required by the tenant corresponding to the high-priority task request in step A31 during the previous detection period, the video memory capacity of the first cache is reserved to ensure that high-priority task requests do not need to wait when allocating video memory space, reducing time latency. The first cache is a pre-defined dedicated video memory area, exclusively for tasks with a priority threshold not lower than the set priority threshold, to achieve advance reservation of video memory resources. The second cache is a pre-established and dynamically adjusted video memory area in the GPU video memory, providing a fast-allocation video memory area for task requests with a priority threshold lower than the set priority threshold, to achieve advance warm-up of video memory resources. At the same time, based on the video memory capacity of the first cache.

[0038] Step A34: Extract the sum of video memory capacity required by low-priority task requests in the previous detection period, and determine whether the current video memory capacity of the second cache is less than the sum of video memory capacity required by low-priority task requests. If it is less, dynamically increase the current video memory capacity of the second cache; if it is greater, dynamically decrease the current video memory capacity of the second cache, so as to realize dynamic prediction of increase or decrease of video memory space according to task requests.

[0039] When the current memory capacity of the second cache is less than the sum of memory required by low-priority task requests, the GPU memory space is dynamically compensated to the second cache based on the missing memory capacity of the second cache, so as to dynamically adjust the current memory capacity of the second cache. Conversely, when the current memory capacity of the second cache is greater than the sum of memory required by low-priority task requests, the excess memory capacity of the second cache is allocated to the GPU memory space to maximize the memory space required by high-priority task requests.

[0040] By using the relationship between the maximum actual video memory space required by high-priority task requests in the previous detection cycle and the video memory capacity of the first cache, the video memory capacity of the first cache is reserved and dynamically adjusted. The video memory capacity of the second cache is used to dynamically compensate for the missing video memory capacity in the first cache to meet the resource requirements of high-priority task requests. At the same time, the video memory capacity of the second cache in the preheating state is dynamically increased or decreased by using the sum of the video memory capacity required by low-priority task requests to meet the resource requirements of low-priority task requests. This avoids the latency problem of video memory allocation between high-priority and low-priority task requests and achieves the zero-latency characteristic of high-priority task requests.

[0041] When predictively allocating video memory space in the first and second cache regions, it is necessary to monitor the resource status of the GPU's video memory space in real time. This resource status includes used video memory capacity, free video memory capacity, and the video memory capacity of the first and second cache regions. This invention also discloses another method for predictively allocating video memory space in the first and second cache regions, comprising: Step B31: Determine whether the triggering conditions for prediction allocation are met. If the triggering conditions are met, extract the arrival rate of task requests, the length of the request queue, and historical load data. Based on the arrival rate of task requests, the length of the request queue, and historical load data, analyze the predicted memory demand values ​​of the first and second cache areas within the preset time window. The triggering conditions for prediction allocation can be time or load.

[0042] Step B32: Compare the predicted memory demand of the first cache with the current memory capacity of the first cache, calculate the missing memory capacity of the current first cache, and if the missing memory capacity of the current first cache is greater than 0, generate an active memory capacity pre-allocation instruction; actively allocate memory capacity for the first cache so that the memory capacity of the allocated first cache is greater than the predicted memory demand of the first cache. The missing memory capacity of the first cache can be allocated to the first cache through the GPU memory capacity, or the missing memory capacity of the first cache can be allocated through the memory capacity of the second cache to meet the resource requirements of the first cache.

[0043] Step B33: Compare the predicted memory requirement of the second cache with the current memory capacity of the second cache, calculate the missing memory capacity of the current second cache, and if the missing memory capacity of the current second cache is greater than 0, generate an active memory capacity pre-allocation instruction; actively allocate memory capacity for the second cache so that the allocated memory capacity of the second cache is greater than the predicted memory requirement of the second cache. For the missing memory capacity of the second cache, allocate the missing memory capacity of the second cache from the GPU memory capacity to the second cache to meet the resource requirements of the second cache. Step B34: According to the memory capacity pre-allocation instruction, obtain the memory capacity from the GPU memory space that matches the missing memory capacity, so as to update the memory capacity of the current first cache and / or second cache, so that the memory capacity of the first cache and the second cache after allocation is not less than the predicted memory demand value of the corresponding cache.

[0044] In this embodiment, taking the predicted allocation of the first buffer as an example, specifically, when the selection time is used as the triggering condition, the prediction method for the triggering condition includes: Step B311: Set a time window and count the arrival times of each high-priority task request within the time window, denoted as t1, t2, ..., tk; Step B312: Calculate the arrival interval between adjacent task requests and calculate the average arrival interval Δt within the window. avg ; Step B313, based on the average arrival interval Δt avg The idle determination threshold is dynamically calculated, wherein the idle determination threshold TL = Q × Δt avg Q is a constant used to eliminate the interference of task request arrival time jitter on triggering conditions. The preferred value is 3, which is determined according to the fluctuation of task request arrival interval.

[0045] Step B314: Obtain the arrival time t corresponding to the latest task request. last and the current time t c Determine the current time t c The arrival time t corresponding to the latest task request last Is the difference between them greater than the idle threshold T? L If it is greater than the set idle threshold T L If the predicted allocation trigger condition is met, the memory demand prediction for the first cache area will be initiated.

[0046] By checking whether the difference between the arrival time of the latest task request and the current time is greater than the set idle threshold, it is determined whether the predicted allocation conditions are met, thereby determining that there is a high probability that a new task request will arrive in the next window, thus preventing delays caused by GPU memory allocation.

[0047] This embodiment discloses that, when load is selected as the trigger condition, the number of high-priority task requests arriving within two adjacent time windows is obtained, and the average arrival rate (v) of task requests within the window is calculated based on the number of task requests arriving within the time window. t and v t-1 ); Determine whether the average arrival rate of task requests in the next time window reaches the average arrival rate of task requests in the previous time window (i.e., v). t Is it greater than v? t-1 If the value is greater than 0, the predictive allocation trigger condition is met, and predictive allocation of the video memory capacity of the first cache area is initiated.

[0048] Using time or load as the triggering condition, the memory capacity of the second cache can also be predicted and allocated.

[0049] Based on the arrival rate of task requests, the length of the request queue, and historical load data, the predicted memory demand of the first cache area within a preset time window is analyzed. The calculation model for the predicted memory demand of the first cache area is as follows: M=λ1×M1+λ2×M2+λ3×M3+M4; Wherein, λ1, λ2, and λ3 are weighting coefficients, and λ1+λ2+λ3=1. M1 represents the memory requirement predicted based on the arrival rate of high-priority task requests, used to cope with the transient change trend of task requests and to handle sudden task requests. M2 represents the memory requirement predicted based on the status of the high-priority request queue, reflecting the current cumulative task request situation. M3 represents the memory requirement predicted based on the changes in historical load data of high-priority tasks. M4 represents the memory capacity required for the GPU inference service to run. This memory capacity is relatively fixed and does not change dynamically with the number of task requests.

[0050] The predicted memory requirement value M1 is based on the arrival rate of high-priority task requests, M1 = (v t +Y×△v t )×t1×q,v t This represents the average arrival rate of high-priority task requests within the current time window (corresponding to time t), Δv. t This represents the difference between the average arrival rate of high-priority task requests within the current time window (corresponding to time t) and the average arrival rate of task requests within the previous time window (corresponding to time t-1). When Δvt A value greater than 0 indicates an increase in high-priority task requests, requiring an increase in the memory capacity of the first cache. t A value not greater than 0 indicates that the number of high-priority task requests is stable or decreasing, and the cache capacity of the cache area needs to be maintained. Y represents the safety factor, which takes a value of 1.1-1.2. Based on the safety factor, the average arrival rate of task requests in the next window is predicted. t1 represents the prediction time interval, and q represents the average memory usage of a single high-priority task request.

[0051] The memory requirement value M2 is predicted based on the state of the high-priority request queue, where M2 = L * q, and L represents the number of high-priority task requests.

[0052] Methods for obtaining the predicted video memory requirement value M3 based on high-priority historical load data changes include: Get the peak memory usage of the first cache region within the same time period over the past n days. ; Extract the high-priority task request data within the previous T time period and the number of task requests in the same period in history, and calculate the correlation coefficient β between the current time and the data in the same period in history. In the process of calculating the correlation coefficient β, the previous T time period is divided into several time sampling intervals. The number of task requests in each time sampling interval and the number of task requests in each time sampling interval in the same period in history are used to analyze the degree of correlation between the current time and the data in the same period in history. Based on correlation coefficient β and peak memory usage Calculate the memory requirement value predicted by changes in historical load data.

[0053] The memory requirement value M3 is predicted based on changes in high-priority historical load data. , , This indicates the current memory usage, where x represents the number of time sampling intervals divided into the previous T time period. This represents the number of task requests within the j-th time sampling interval in the preceding T time period. This represents the average number of task requests over the preceding T time period. This represents the number of task requests within the same time period (equal to the length of time period T) corresponding to the peak video memory usage over the previous x days. This represents the average number of task requests within the same time period over the previous x days. The value of β ranges from [-1, 1]. When β approaches 1, it indicates that the current task request volume is highly consistent with the historical task request volume data for the same period. When β approaches 0, it indicates that the current task request volume is not correlated with the historical task request volume data for the same period. When β approaches -1, it indicates that the current task request volume is negatively correlated with the historical task request volume data for the same period.

[0054] The memory requirement is predicted by the rate at which task requests that are not less than the set priority threshold arrive within the time window. However, when calculating the memory requirement prediction for the second cache, the memory requirement is predicted by the rate at which task requests that are less than the set priority threshold arrive within the window, based on the average arrival rate of such requests.

[0055] The memory requirement predicted by the status of the high-priority request queue can also be used to predict the memory requirement of the second cache based on the number of low-priority task requests.

[0056] The calculation of the memory demand value M3 predicted by the historical load data changes of high priority is also based on the peak value of memory usage of task requests with a priority threshold less than the set priority threshold in the same time period within the previous n days, and the correlation is determined by combining the number of task requests with a priority threshold less than or greater than the set priority threshold in the current time period T with the number of task requests in each time sampling interval of the same period in history. Thus, the memory demand value predicted by the historical load data changes corresponding to the second cache area is calculated.

[0057] The predicted memory demand for the first cache is calculated using the arrival rate of high-priority task requests, the request queue length, and historical load data. Similarly, the predicted memory demand for the second cache can be determined using the same calculation model as the predicted memory demand for the first cache. During the calculation of the predicted memory demand for the second cache, it is necessary to analyze the request speed of low-priority tasks, the request queue length, and historical load data to determine the predicted memory demand for the second cache.

[0058] Using the above model, the predicted memory demand of the first cache is calculated. Combined with the currently allocated memory capacity of the first cache, the increase in memory demand of the first cache within the time window is analyzed.

[0059] Increase in video memory requirements of the first cache area within the window , This indicates the memory capacity of the first allocated cache area.

[0060] Similarly, based on the above memory demand increase model, combined with the predicted memory demand of the second cache and the currently allocated memory capacity of the second cache, the increase in memory demand of the second cache within the time window is analyzed.

[0061] By using the above method, the arrival rate of task requests, the length of the request queue, and historical load data are used to analyze the predicted memory requirements of the first and second caches within the time window. Based on the currently allocated memory capacity of the first and second caches, the increase in memory requirements required by the first and second caches is analyzed. This allows for adjustments to the memory capacity of the first cache corresponding to task requests that meet or exceed a set priority threshold, and the second cache corresponding to task requests that fall below the set priority threshold, thereby meeting the memory capacity requirements of real-time task request data changes.

[0062] The resource allocation module pre-allocates memory to the first cache area to predict the required cache memory capacity. It then adjusts the pre-allocated cache memory capacity based on dynamic changes in task requests, achieving low-latency response and preventing the predicted allocated memory capacity from being insufficient to meet the changing needs of task requests, thus avoiding increased memory allocation latency. Using the cache memory capacity allocated by the resource allocation module provides a smoother and more immediate interactive experience in high-concurrency or real-time application scenarios, significantly improving service response speed and overall throughput efficiency.

[0063] Step 4: Obtain the real-time remaining amount of the video memory capacity of the predicted allocated cache area, dynamically analyze the upper limit of the number of parallel processing task requests, and analyze the current optimal number of parallel processing task requests based on the current number of task requests waiting to be processed and the preset control mode. Then, determine the current optimal number of parallel processing task requests based on the current number of task requests waiting to be processed, and perform batch processing calculations to balance resource utilization and latency waiting time.

[0064] The preset control mode selects the corresponding number of parallel processing task requests based on the relationship between the number of task requests of the same priority level currently waiting to be processed and the upper and lower limits of the number of parallel processing task requests corresponding to that priority level.

[0065] The actual number of high-priority or low-priority task requests currently being processed in parallel varies depending on the number of task requests of the same priority level currently waiting to be processed.

[0066] Furthermore, based on the relationship between the number of task requests of the same priority level currently waiting to be processed and the set lower and upper limits of the number of task requests of that priority level, the number of task requests to be processed in parallel is dynamically processed to optimize the current optimal number of task requests to be processed in parallel.

[0067] Calculate the upper limit of the number of task requests that can be processed in parallel in the first buffer or the second buffer. The formula for calculating the upper limit of the number of task requests that can be processed in parallel is as follows: ,in, This represents the upper limit of the number of task requests that can be processed in parallel in the first buffer. This represents the real-time remaining amount of video memory in the first cache corresponding to the high-priority task requests predicted for allocation. q represents the safe capacity threshold for the video memory capacity of the first cache corresponding to the predicted high-priority task request, and q represents the average video memory usage of a single high-priority task request.

[0068] The above formula for calculating the upper limit of the number of parallel processing task requests can be used to calculate the upper limit of the number of parallel processing task requests in the first cache. Similarly, the above formula can be used to calculate the upper limit of the number of parallel processing task requests in the second cache. Specifically, the real-time remaining amount of video memory capacity in the second cache corresponding to the predicted low-priority task requests and the real-time remaining amount of video memory capacity in the second cache corresponding to the predicted low-priority task requests are extracted. Combined with the average video memory usage of a single low-priority task request, the upper limit of the number of parallel processing task requests in the second cache is obtained.

[0069] In this embodiment, low-priority task requests are processed using the video memory capacity of the second cache, while high-priority task requests are processed using the video memory capacity of the first cache.

[0070] By combining the real-time remaining memory capacity of the first cache with the safe capacity threshold of the first cache and the average memory usage of a single high-priority task request, the upper limit of the number of task requests that can be processed in parallel in the first cache can be obtained. The safe capacity threshold refers to the situation where the task request cannot be processed when the memory capacity of the first cache or the second cache is less than the safe capacity threshold corresponding to that cache.

[0071] It should be noted that, according to the priority order of task requests, the cache corresponding to the task requests with higher priority are selected in sequence, and the task requests corresponding to each cache are processed. The security capacity thresholds of the first cache and the second cache may be different. The security capacity thresholds of the first cache and the second cache are set according to actual needs. Since the second cache processes low-priority task requests, there is a problem of a large number of task requests being processed. Generally, the security capacity threshold of the second cache is set to be greater than that of the first cache.

[0072] The method for analyzing the optimal number of parallel processing requests based on the current number of pending request requests includes the following steps: Step 41: Compare the currently pending task requests with the set priority threshold to determine the number of high-priority task requests, that is, the number of high-priority (not less than the set priority threshold) task requests and the number of low-priority (less than the set priority threshold) task requests. Step 42: Compare the number of high-priority task requests among the currently pending task requests with the upper and lower limits of the number of high-priority task requests to be processed in parallel, and determine whether the number of high-priority task requests is less than the lower limit of the number of high-priority task requests. If the number of high-priority task requests is less than 1, then the number of high-priority task requests to be processed in parallel is equal to 1, so as to process the incoming task requests in a timely manner and reduce waiting delays; otherwise, step 43 is executed. The upper limit of the number of task requests at the same priority level is greater than the lower limit of the number of task requests at that priority level. The number of task requests at any priority level is greater than 1.

[0073] Step 43: Determine whether the number of high-priority task requests among the currently pending task requests exceeds the upper limit for the number of high-priority task requests. If the value is greater than the limit, then the upper limit of the number of task requests allowed for parallel processing by the higher priority will be used. To achieve full-load operation and improve processing efficiency, the process is executed; otherwise, step 44 is executed. Step 44: When the number of high-priority task requests among the waiting task requests equals the lower limit, the cumulative number of high-priority task requests processed in parallel reaches the upper limit. Maximum required waiting time Combined with the number of high-priority task requests currently waiting to be processed and the upper limit of the number of task requests processed in parallel. Calculate the waiting scheduling time corresponding to the adjusted number of high-priority task requests. .

[0074] Step 45: Determine whether the waiting time corresponding to the number of high-priority task requests in a single parallel processing is equal to the waiting scheduling time. If it is equal to the waiting scheduling time, then process the number of high-priority task requests waiting to be processed in parallel within the waiting scheduling time.

[0075] The adjusted waiting time calculation formula is as follows: Based on the dynamic changes in the number of high-priority task requests among the currently waiting task requests, the waiting scheduling time for high-priority task requests to be processed in parallel is dynamically adjusted.

[0076] The method described above, which analyzes the optimal number of parallel processing task requests based on the number of currently waiting task requests, is for the number of high-priority task requests in parallel processing. Similarly, the number of low-priority task requests, determined by the number of currently waiting task requests being less than a set priority threshold, can be compared with the upper and lower limits of the number of low-priority task requests in parallel processing. This allows for the use of a predictive control mode to regulate the number of low-priority task requests or the waiting time for scheduling in parallel processing, thereby balancing resource utilization and waiting time.

[0077] By comparing the number of currently pending task requests with the upper and lower limits of the corresponding number of parallel task requests, the range corresponding to the number of currently pending task requests is determined, so as to select the corresponding number of parallel task requests. When the number of currently pending task requests is between the upper and lower limits, the waiting time is dynamically adjusted based on the number of currently pending task requests to make full use of the waiting time to improve GPU utilization and throughput, balance resource utilization and waiting time, prevent task request backlog, and achieve optimal control of latency and throughput.

[0078] Based on the same inventive concept, such as Figure 2 As shown, this application also proposes a low-latency scheduling system for real-time interactive AI services, comprising: The priority determination module is used to obtain the task requests from each tenant and determine the initial priority coefficient based on the tenant's level and task type. The priority update module is used to obtain the waiting time of each task request under each tenant in real time, dynamically analyze the priority coefficient of each task request, and adjust the priority processing order of each task request according to the adjusted priority coefficient of each task request. The resource allocation module is used to pre-divide the GPU memory space, dividing it into a first cache area and a second cache area, and dynamically allocating the memory space capacity of the first cache area and the second cache area based on prediction.

[0079] The optimization processing module obtains the real-time remaining amount of the video memory capacity of the predicted allocated cache area, dynamically analyzes the upper limit of the number of parallel processing task requests, and analyzes the current optimal number of parallel processing task requests based on the number of currently waiting task requests and the preset control mode. Based on the current number of waiting task requests, it determines the current optimal number of parallel processing task requests and performs batch processing calculations to balance resource utilization and latency.

[0080] The specific implementation of the low-latency scheduling system for real-time interactive AI services provided in this application can be referred to the low-latency scheduling method for real-time interactive AI services described above, and will not be repeated here.

[0081] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the aforementioned low-latency scheduling methods for real-time interactive AI services.

[0082] In practical applications, a computer-readable storage medium can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this embodiment, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0083] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0084] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0085] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0086] In the description of this specification, references to terms such as "an embodiment," "example," and "specific example" indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0087] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A low-latency scheduling method for real-time interactive AI services, characterized in that, include: Obtain task requests from each tenant and determine the initial priority coefficient based on each tenant's level and task type; The system obtains the waiting time of each task request under each tenant in real time, dynamically analyzes the priority coefficient of each task request, and adjusts the priority processing order of each task request according to the adjusted priority coefficient of each task request. A dynamic update model is adopted to dynamically update the priority coefficients of each task request. The dynamic update model is as follows: K △tj =K j +g*△tj, where K △tj This represents the priority coefficient of each task request after the update, Δtj represents the waiting time of the j-th task request, which is equal to the difference between the current time and the arrival time of the task request, and g is a coefficient, wherein the coefficient g≥ΔK / T0, and ΔK represents the maximum difference of the initial priority coefficients, i.e., ΔK=K. max -K min T0 represents the maximum allowed fair waiting time; The GPU memory space is pre-divided into a first cache area and a second cache area, and the memory space capacity of the first cache area and the second cache area is predicted and dynamically allocated to obtain memory capacity in a timely manner and reduce allocation time. The first cache area is used to store task requests with a priority coefficient not less than the set priority threshold, and the second cache area is used to store task requests with a priority coefficient less than the set priority threshold. The system obtains the real-time remaining capacity of the predicted allocated cache area, dynamically analyzes the upper limit of the number of parallel processing task requests, and analyzes the optimal number of parallel processing task requests based on the current number of waiting task requests and the preset control mode. Based on the current number of waiting task requests, it determines the optimal number of parallel processing task requests and performs batch processing calculations to balance resource utilization and latency. The formula for calculating the upper limit of the number of parallel processing task requests is as follows: ,in, This represents the upper limit of the number of task requests that can be processed in parallel in the first buffer. This represents the real-time remaining amount of video memory in the first cache corresponding to the predicted high-priority task request. The safe capacity threshold for the first cache area corresponding to the predicted high-priority task request is q, where q represents the average memory usage of a single high-priority task request. The method for analyzing the optimal number of parallel processing requests based on the current number of pending task requests includes the following steps: Step 41: Compare the currently pending task requests with the set priority threshold to determine the number of high-priority task requests; Step 42: Compare the number of high-priority task requests in the current waiting task requests with the upper and lower limits of the number of high-priority task requests to be processed in parallel. Determine whether the number of high-priority task requests is less than the lower limit of the number of high-priority task requests. If it is less, select 1 high-priority task request to be processed in parallel to process the incoming task requests in a timely manner and reduce waiting delay. Otherwise, proceed to step 43. Step 43: Determine whether the number of high-priority task requests among the currently pending task requests exceeds the upper limit for the number of high-priority task requests. If the value is greater than the limit, then the upper limit of the number of task requests allowed for parallel processing by the higher priority will be used. If the condition is not met, proceed to step 44; Step 44: Obtain the maximum waiting time required to reach the upper limit of the number of high-priority parallel processing task requests when the number of high-priority task requests equals the lower limit. Combined with the number of high-priority task requests currently waiting to be processed and the upper limit of the number of task requests processed in parallel. Calculate the waiting scheduling time corresponding to the adjusted number of high-priority task requests. ; Step 45: Determine whether the waiting time corresponding to the number of high-priority task requests in a single parallel processing is equal to the waiting scheduling time. If it is equal to the waiting scheduling time, then process the number of high-priority task requests waiting to be processed in parallel within the waiting scheduling time. Adjusted formula for calculating waiting time for scheduling: .

2. The low-latency scheduling method for real-time interactive AI services according to claim 1, characterized in that, Obtain the tenant level and task type corresponding to each task request; The initial priority coefficient of each task request is dynamically evaluated by multiplying the priority value corresponding to the tenant level and the priority weight coefficient corresponding to the task type. Based on the dynamic evaluation of the initial priority coefficient of each task request, the task requests are prioritized and processed first.

3. The low-latency scheduling method for real-time interactive AI services according to claim 1, characterized in that, Based on the relationship between the priority coefficient of the task request and the set priority threshold, high priority is determined. Based on whether there are high-priority task requests, the video memory space of the first cache area is predicted and allocated. The prediction allocation method includes the following steps: Determine whether there are any task requests whose priority coefficient is not less than the set priority threshold. If so, extract the maximum actual video memory space required by the high-priority task request in the previous detection period. Otherwise, do not reserve the video memory capacity of the first cache area in advance. Determine whether the maximum actual video memory space required by high-priority task requests in the previous detection period is greater than the video memory capacity of the first cache. If it is greater than the video memory capacity of the first cache, dynamically compensate the video memory capacity of the second cache to the video memory capacity missing in the first cache. The video memory capacity dynamically compensated by the second cache to the video memory capacity missing in the first cache is equal to the difference between the predicted maximum actual video memory space required and the current video memory capacity of the first cache. Based on the maximum actual video memory space required by high-priority task requests in the previous detection cycle, the video memory capacity of the first cache area is reserved and locked. Extract the sum of video memory required by low-priority task requests in the previous detection period, and determine whether the current video memory capacity of the second cache is less than the sum of video memory required by low-priority task requests. If it is less, dynamically increase the current video memory capacity of the second cache; if it is greater, dynamically decrease the current video memory capacity of the second cache, so as to realize dynamic prediction of the increase or decrease of video memory space according to task requests.

4. The low-latency scheduling method for real-time interactive AI services according to claim 1, characterized in that, Methods for predictive allocation of video memory space in the first cache region include: Determine whether the triggering condition for predicted allocation is met. If the triggering condition is met, extract the arrival rate of task requests, the length of the request queue, and historical load data. Based on the arrival rate of task requests, the length of the request queue, and historical load data, analyze the predicted memory demand values ​​of the first and second cache areas within a preset time window. The triggering condition is time or load. Compare the predicted memory demand of the first cache with the current memory capacity of the first cache, calculate the missing memory capacity of the current first cache, and if the missing memory capacity of the current first cache is greater than 0, generate an active memory capacity pre-allocation instruction to actively allocate memory capacity for the first cache. According to the memory capacity pre-allocation instruction, the memory capacity that matches the missing memory capacity is obtained from the GPU memory space to update the memory capacity of the current first cache, so that the memory capacity of the first cache after allocation is not less than the predicted memory demand value of the corresponding cache.

5. A low-latency scheduling method for real-time interactive AI services according to claim 4, characterized in that, The method for predicting the triggering condition includes: Set a time window and count the arrival time of each high-priority task request within the time window; Calculate the arrival interval between adjacent task requests, and calculate the average arrival interval within the window; The idle determination threshold is dynamically calculated based on the average arrival interval; Obtain the arrival time of the latest task request and the current time. Determine whether the difference between the current time and the arrival time of the latest task request is greater than the idle threshold. If it is greater than the set idle threshold, the trigger condition for prediction allocation is met, and the memory demand prediction for the first cache area is started.

6. A low-latency scheduling method for real-time interactive AI services according to claim 4, characterized in that, Get the number of high-priority task requests arriving within two adjacent time windows, and calculate the average arrival rate of task requests within the window based on the number of task requests arriving within the time window. Determine whether the average arrival rate of task requests in the next time window reaches the average arrival rate of task requests in the previous time window. If it does, the prediction allocation trigger condition is met, and prediction allocation of the video memory capacity of the first cache area is initiated.

7. A low-latency scheduling method for real-time interactive AI services according to claim 5, characterized in that, Based on the arrival rate of task requests, the length of the request queue, and historical load data, we analyze the predicted memory requirements of the first and second caches within the time window, and based on the currently allocated memory capacity of the first and second caches, we analyze the increase in memory requirements required by the first and second caches.

8. A low-latency scheduling method for real-time interactive AI services according to claim 1, characterized in that, Based on the relationship between the number of task requests of the same priority level currently waiting to be processed and the set lower and upper limits of the number of task requests of that priority level, the number of task requests to be processed in parallel is dynamically processed to optimize the current optimal number of task requests to be processed in parallel.

9. A low-latency scheduling system for real-time interactive AI services, characterized in that, include: The priority determination module is used to obtain the task requests from each tenant and determine the initial priority coefficient based on the tenant's level and task type. The priority update module is used to obtain the waiting time of each task request under each tenant in real time, dynamically analyze the priority coefficient of each task request, and adjust the priority processing order of each task request according to the adjusted priority coefficient of each task request. A dynamic update model is adopted to dynamically update the priority coefficients of each task request. The dynamic update model is as follows: K △tj =K j +g*△tj, where K △tj This represents the priority coefficient of each task request after the update, Δtj represents the waiting time of the j-th task request, which is equal to the difference between the current time and the arrival time of the task request, and g is a coefficient, wherein the coefficient g≥ΔK / T0, and ΔK represents the maximum difference of the initial priority coefficients, i.e., ΔK=K. max -K min T0 represents the maximum allowed fair waiting time; The resource allocation module is used to pre-divide the GPU memory space, divide it into a first cache area and a second cache area, and predict and dynamically allocate the memory space capacity of the first cache area and the second cache area to obtain the memory capacity in a timely manner and reduce the allocation time. The optimization processing module obtains the real-time remaining amount of the video memory capacity of the predicted allocated cache area, dynamically analyzes the upper limit of the number of parallel processing task requests, and analyzes the current optimal number of parallel processing task requests based on the number of currently waiting task requests and the preset control mode. Based on the current number of waiting task requests, it determines the current optimal number of parallel processing task requests and performs batch processing calculations to balance resource utilization and latency. The formula for calculating the upper limit of the number of parallel processing task requests is as follows: ,in, This represents the upper limit of the number of task requests that can be processed in parallel in the first buffer. This represents the real-time remaining amount of video memory in the first cache corresponding to the predicted high-priority task request. The safe capacity threshold for the first cache area corresponding to the predicted high-priority task request is q, where q represents the average memory usage of a single high-priority task request. The method for analyzing the optimal number of parallel processing requests based on the current number of pending task requests includes the following steps: Step 41: Compare the currently pending task requests with the set priority threshold to determine the number of high-priority task requests; Step 42: Compare the number of high-priority task requests in the current waiting task requests with the upper and lower limits of the number of high-priority task requests to be processed in parallel. Determine whether the number of high-priority task requests is less than the lower limit of the number of high-priority task requests. If it is less, select 1 high-priority task request to be processed in parallel to process the incoming task requests in a timely manner and reduce waiting delay. Otherwise, proceed to step 43. Step 43: Determine whether the number of high-priority task requests among the currently pending task requests exceeds the upper limit for the number of high-priority task requests. If the value is greater than the limit, then the upper limit of the number of task requests allowed for parallel processing by the higher priority will be used. If the condition is not met, proceed to step 44; Step 44: Obtain the maximum waiting time required to reach the upper limit of the number of high-priority parallel processing task requests when the number of high-priority task requests equals the lower limit. Combined with the number of high-priority task requests currently waiting to be processed and the upper limit of the number of task requests processed in parallel. Calculate the waiting scheduling time corresponding to the adjusted number of high-priority task requests. ; Step 45: Determine whether the waiting time corresponding to the number of high-priority task requests in a single parallel processing is equal to the waiting scheduling time. If it is equal to the waiting scheduling time, then process the number of high-priority task requests waiting to be processed in parallel within the waiting scheduling time. Adjusted formula for calculating waiting time for scheduling: .

10. A computer-readable storage medium comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the low-latency scheduling method as described in any one of claims 1-8.

Images