Elastic computing power dynamic scheduling and allocation method for large-scale advertising model inference tasks
By breaking down background tasks into computational subtasks and controlling their execution using data-ready flags, the latency problem caused by resource contention in advertising inference tasks is resolved, improving resource utilization and overall efficiency.
Patent Information
- Application Number
- CN202611377358.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-07
- Publication Date
- 2026-11-17
AI Technical Summary
In advertising inference tasks, graphics processor resources are prone to becoming idle while waiting for recall data. If background tasks are directly distributed in whole segments, they will compete for resources with the main computing flow, affecting the fine-sorting recovery latency and making it difficult to improve the overall inference efficiency.
The background task is split into multiple computational subtasks with an execution time not exceeding the preset operator duration limit. The background computation flow is controlled by a data ready flag to continue execution until the recalled data arrives. When the recalled data arrives, the subsequent subtasks are stopped and the main computation flow is resumed.
It improved resource utilization efficiency, reduced fine sorting recovery latency, and enhanced overall inference efficiency.
Smart Images

Figure CN123114679A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, specifically to a method for dynamic scheduling and allocation of elastic computing power for large-scale advertising model inference tasks. Background Technology
[0002] In advertising inference tasks, recall, ranking, and various background computing tasks usually need to be handled simultaneously. While the ranking task is waiting for the recall data to return, the graphics processor resources are prone to being idle. If the background tasks are sent out in whole segments, they are prone to resource contention with the main computing flow, affecting the ranking recovery latency. In traditional methods, background tasks are often executed continuously according to a fixed queue or fixed priority, lacking dynamic judgment on waiting window duration, task interruptibility, and resource usage. Therefore, after the recalled data arrives, the main computing flow often cannot preempt and recover in time, and the overall inference efficiency is difficult to improve steadily. Summary of the Invention
[0003] The purpose of this invention is to provide a method for dynamic scheduling and allocation of elastic computing power for large-scale advertising model inference tasks. During the period when the fine-ranking task is waiting for the recall data, the background task is split and executed, which avoids the delay in the recovery of the main computing flow caused by resource competition, thereby improving the utilization rate of computing power and the overall inference efficiency.
[0004] The objective of this invention can be achieved through the following technical solutions: A method for dynamic scheduling and allocation of elastic computing power for large-scale advertising model inference tasks, applied to servers configured with graphics processors and network adapters, including: Receive ad inference requests and trigger corresponding ad recall and ranking tasks. The ranking task depends on the recall data returned by the ad recall task. Determine the background tasks to be scheduled in parallel with the ad inference requests, and establish the main computing flow for the ranking task and the background computing flow for the background tasks. Set a data-ready flag in the graphics processor's video memory that is directly updated by the network adapter; The background task is divided into multiple computational subtasks whose execution time does not exceed the preset operator duration limit. The preset operator duration limit is determined based on the idle window duration, which is the time difference between the main computation flow entering the waiting state and the arrival of the recalled data. While the main computing stream completes the local preprocessing of the fine sorting task and waits for the recalled data, the control background computing stream executes multiple computing subtasks in sequence, and reads the data ready flag after each computing subtask is completed. If the read result indicates that the recalled data has not been received, the control background computing flow continues to execute subsequent computing subtasks; When the read result indicates that the recalled data has arrived, the submission of subsequent computational subtasks that have not yet been issued to the graphics processor is stopped, and the corresponding instructions of the main computation stream are written into the high-priority execution queue of the graphics processor, triggering the main computation stream to resume execution of the fine-ordering task.
[0005] In some possible implementations, the step of setting a data-ready flag in the graphics processor's video memory, which is directly updated by the network adapter, includes: Allocate a predetermined data buffer and flag address for the data batch corresponding to the current fine sorting task, and initialize the data ready flag to the not ready state; The control network adapter writes the recall data to a predetermined data buffer via remote direct memory access; after the network adapter completes the remote direct memory access write to the predetermined data buffer, the control network adapter writes the ready value to the flag address via remote direct memory access. When the value read from the flag address is equal to the ready value, it is determined that the recall data has arrived and the main computing stream is allowed to resume execution.
[0006] In some possible implementations, the step of assigning a flag address to the data batch corresponding to the current fine sorting task includes: Assign array indices to the data batches corresponding to the current fine sorting task in multiple flag arrays; Bind the data batch corresponding to the current fine sorting task to the flag address corresponding to the array index; Before the main computing stream enters the waiting state, record the expected request identifier and expected version number; before the main computing stream resumes execution, obtain the request identifier and version number of the data batch corresponding to the current fine sorting task, and compare the request identifier and version number with the expected request identifier and expected version number respectively; If the comparison results are consistent, the main computation flow is allowed to resume execution; If the comparison results are inconsistent, the main computing flow recovery should not be triggered based on the flag address, and the data ready flag value at the flag address should be cleared to wait for the next batch of recalled data.
[0007] In some possible implementations, the steps of breaking down the background task into multiple computational subtasks with execution times not exceeding a preset operator duration limit include: Obtain the candidate task pool and the corresponding computation data for each candidate task from the preset background task set; according to the data type of the candidate task, split the tasks in the candidate task pool into data blocks, word blocks, or sample blocks; Set a task identifier, batch identifier, and version identifier for each computational subtask; Each computational subtask is controlled to save its intermediate computational state to a state buffer corresponding to the task identifier upon completion of its execution, and to write the execution result to a result buffer corresponding to the batch identifier.
[0008] In some possible implementations, the tasks in the candidate task pool include advertising feature calculation tasks that are divisible, idempotent, and interruptible, and have result caching or reuse indexing mechanisms. Background computation streams can use independent buffers, share read-only parameter areas, or not request specified mutual exclusion resources.
[0009] In some possible implementations, prior to the step of controlling the background computation stream to execute multiple computational subtasks sequentially, the following is also included: The remaining available time for advertising inference requests is determined based on business latency constraints and the minimum guaranteed execution time of the main computing flow; the graphics processor's memory bandwidth utilization and L2 cache utilization are obtained; Background computing streams are allowed to start when the remaining available time is greater than zero and the video memory bandwidth utilization rate does not exceed the video memory bandwidth utilization rate threshold and the L2 cache utilization rate does not exceed the L2 cache utilization rate threshold; otherwise, background computing streams are prohibited from starting.
[0010] In some possible implementations, the steps of controlling the background computation stream to execute multiple computational subtasks sequentially include: Obtain the total resources of the graphics processor and the preset background task scheduling ratio, and determine the upper limit of streaming multiprocessor resources, upper limit of video memory bandwidth and upper limit of L2 cache for the background computing stream accordingly. Before each computational subtask is executed, the resource usage information stored in the computational subtask is used to determine whether the streaming multiprocessor usage, video memory bandwidth usage, and L2 cache usage of the computational subtask do not exceed the corresponding streaming multiprocessor resource limit, video memory bandwidth limit, and L2 cache limit, respectively. If none of the above conditions are met, the computational subtasks will be issued; if any one condition is met, the background computation stream will be suspended.
[0011] In some possible implementations, the steps for determining the upper limit of the preset operator duration include: Candidate values for the preset operator duration upper limit are determined based on the statistical results of the window duration of previous tasks; The remaining available time for the ad inference request is obtained, and the maximum execution time of a single operator allowed by the graphics processor to ensure preemptive response is used as the hardware time limit. The minimum value among the candidate value, remaining available time, and hardware duration limit is determined as the preset operator duration limit.
[0012] Among some possible implementations, the following also applies: obtaining the 99.9th percentile latency of the fine-ranking task after enabling background computing streams and the 99.9th percentile latency of the fine-ranking task without enabling background computing streams; The tail latency jitter rate is determined by the ratio of the difference between the 99.9th percentile latency of the fine-ranking task after enabling background computing flow and the 99.9th percentile latency of the fine-ranking task without enabling background computing flow to the 99.9th percentile latency without enabling background computing flow. Obtain the duration of the idle window and the execution time of each computational subtask completed during the idle window; The window fill rate is determined by summing the execution times of all computational subtasks completed during the window period and the window duration. When the tail latency jitter rate does not exceed the preset jitter threshold and the window fill rate is lower than the target fill rate, the background task scheduling ratio is increased by a preset step size to no more than the upper limit threshold; when the tail latency jitter rate exceeds the preset jitter threshold, the background task scheduling ratio is decreased by a preset step size.
[0013] In some possible implementations, background tasks are obtained from a pre-defined pool of candidate tasks; other methods include: Obtain the reuse rate, duration matching, and resource consumption of multiple candidate tasks in the candidate task pool; Based on the reusability, duration matching, and resource consumption of each candidate task, the scheduling priority score of each candidate task is determined. Candidate tasks are selected as background tasks in descending order of scheduling priority score. In continuous When the tail latency jitter rate exceeds the preset jitter threshold within a statistical window, the background task scheduling ratio will be adjusted to the preset lower limit threshold. It is a preset positive integer.
[0014] The beneficial effects of this invention are: This invention improves resource utilization efficiency and reduces fine-tuning recovery latency by splitting background tasks according to the idle window duration and coordinating scheduling with the data ready flag while the main computing stream is waiting for recalled data. This allows the graphics processor to continuously fill idle window resources when the recalled data has not arrived, and to promptly stop the issuance of subsequent subtasks and resume the execution of the main computing stream after the recalled data arrives. Attached Figure Description
[0015] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating the elastic computing power dynamic scheduling and allocation method for advertising large model inference tasks provided in this application embodiment. Detailed Implementation
[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0017] Please see Figure 1 A method for dynamic scheduling and allocation of elastic computing power for advertising large model inference tasks is applied to servers configured with graphics processors and network adapters. The method includes receiving advertising inference requests and triggering corresponding advertising recall tasks and fine ranking tasks. The fine ranking task depends on the recall data returned by the advertising recall task. Identify background tasks that are scheduled in parallel with advertising inference requests, and establish a main computation flow for fine ranking tasks and a background computation flow for background tasks. Set a data-ready flag in the graphics processor's video memory that is directly updated by the network adapter; The background task is divided into multiple computational subtasks whose execution time does not exceed the preset operator duration limit. The preset operator duration limit is determined based on the idle window duration, which is the time difference between the main computation flow entering the waiting state and the arrival of the recalled data. While the main computing stream completes the local preprocessing of the fine sorting task and waits for the recalled data, the control background computing stream executes multiple computing subtasks in sequence, and reads the data ready flag after each computing subtask is completed. If the read result indicates that the recalled data has not been received, the control background computing flow continues to execute subsequent computing subtasks; When the reading result indicates that the recalled data has arrived, the submission of subsequent computational subtasks that have not yet been issued to the graphics processor is stopped, and the corresponding instructions of the main computational stream are written into the high-priority execution queue of the graphics processor, triggering the main computational stream to resume execution of the fine-ordered task; In one alternative approach, the step of setting a data-ready flag in the graphics processor's video memory, which is directly updated by the network adapter, includes: Allocate a predetermined data buffer and flag address for the data batch corresponding to the current fine sorting task, and initialize the data ready flag to the not ready state; The control network adapter writes the recall data to a predetermined data buffer via remote direct memory access; after the network adapter completes the remote direct memory access write to the predetermined data buffer, the control network adapter writes the ready value to the flag address via remote direct memory access. When the value read from the flag address is equal to the ready value, it is determined that the recall data has arrived and the main computing stream is allowed to resume execution. The steps for assigning flag addresses to the data batch corresponding to the current fine sorting task include: Assign array indices to the data batches corresponding to the current fine sorting task in multiple flag arrays; Bind the data batch corresponding to the current fine sorting task to the flag address corresponding to the array index; Before the main computing stream enters the waiting state, record the expected request identifier and expected version number; before the main computing stream resumes execution, obtain the request identifier and version number of the data batch corresponding to the current fine sorting task, and compare the request identifier and version number with the expected request identifier and expected version number respectively; If the comparison results are consistent, the main computing stream is allowed to resume execution; if the comparison results are inconsistent, the main computing stream is prohibited from being triggered to resume based on the flag address, and the data ready flag value at the flag address is cleared to wait for the next batch of recalled data.
[0018] After receiving the inference request from the online business side, the advertising inference node first completes the request parsing, feature assembly and queue establishment on the host side, and then triggers the series execution of the recall phase and the fine ranking phase. The specific process of feature assembly is as follows: extract the user attribute feature vector and the current device environment feature vector carried in the ad inference request, concatenate the user attribute feature vector and the device environment feature vector to generate a feature dimension of . The dense feature vectors are generated and loaded into a contiguous memory space on the host side as the model input for the fine ranking task. After the local preprocessing is completed in the fine-sorting stage, it is necessary to wait for the recalled data to be returned. At this time, the graphics processor enters the waiting interval. The system selects background tasks that run in parallel with the current request and organizes them into the main computing stream and the background computing stream. The main computing stream is used to carry out the subsequent execution of fine-ordering tasks, while the background computing stream is used to carry out splittable background tasks. The two are kept separate in terms of scheduling. A data ready flag is reserved in the graphics processor's video memory and updated directly by the network adapter. The data ready flag is initialized to a non-ready state and is written to the ready value by the network adapter through direct memory access after the recalled data is written to the predetermined buffer. The network adapter performs write operations in the order of writing recall data, then writing request identifier and version number, and finally writing ready value. Before writing ready value, it completes the write sorting or memory barrier for data buffer and identifier information, so that when the graphics processor reads ready value, it can read complete recall data, request identifier and version number at the same time. The background computing stream reads the data ready flag after each computing subtask is completed. If the data ready flag is not ready, the background computing stream continues to execute subsequent computing subtasks. If the data ready flag has already been written to the ready value, then stop submitting subsequent computational subtasks and write the corresponding instructions of the main computational stream into the high-priority execution queue, so that the main computational stream can resume executing the fine sorting task. Furthermore, the data readiness flag undergoes at least five states in terms of control: unassigned, bound but not ready, data written and pending verification, verification passed and recoverable, and cleared. When the main computing stream enters the waiting state, the data ready flag of the corresponding batch is switched from unassigned to bound but not ready. After the network adapter completes the writing of the recalled data body, it is pushed to the data written and awaiting verification. After the background computing stream reads the ready value at the subtask boundary, it does not directly restore the main computing stream, but first enters the verification branch. When the request identifier and version number are consistent, it switches to the verification passed and can be restored, and writes the main computing stream instruction to the high priority execution queue. When any comparison is inconsistent, it enters the cleared state. If the background computing stream reads the unready state twice in a row, the current batch binding relationship remains unchanged; if a ready value is read but the verification fails, recovery is prohibited and the current batch's occupation of the flag address is released so that subsequent data batches can re-establish the binding relationship. The request identifier is a unique request identifier generated or passed through within the inference node for the current ad inference request, and the version number is the batch identifier for each data batch under the same request, incremented in the order of arrival or update. When the network adapter writes the request identifier and version number, it writes them together with the recall data into the metadata area corresponding to the predetermined data buffer. After the background computing stream reads the ready value, it retrieves the request identifier and version number from the metadata area to perform subsequent verification, thus ensuring a clear reading order between reading the data ready flag, obtaining metadata, and determining recovery. Background tasks are divided and organized into computational subtasks with limited execution time. The upper limit of this time is determined by the window duration formed during the fine-sorting waiting period. The window duration is the time difference between the autonomous computation flow entering the waiting state and the arrival of the recalled data. The window duration is calculated using the waiting start time recorded on the host side and the arrival time of the network adapter write-back. The preset operator duration limit is taken as the minimum value among the median statistical value of the previously recorded window duration, the remaining available time of the current request, and the longest execution time of a single operator allowed by the hardware. When the valid arrival time of the current request has not yet been obtained, a temporary upper limit is first determined using historical statistics and hardware duration limits, and the current window duration is updated after the recall data arrives. When the remaining available time is less than or equal to zero, no new background calculation subtasks will be issued; if the recalled data has not yet arrived, the subsequent subtasks will continue to be executed; if the recalled data has arrived, the issuance of subsequent calculation subtasks will be stopped. Furthermore, the control loop of the background computing flow uses a single computing subtask as the smallest scheduling unit: after a computing subtask is completed, the data ready flag is read first; If the read result is not ready, check if the queue to be sent is empty. If it is not empty, continue to submit the next calculation subtask. If it is empty, continue to wait. If the read result is ready, enter the request identifier and version number verification branch. If the verification passes, stop retrieving the subtask to be sent and trigger the main calculation flow to resume. If the verification fails, clear the data ready flag and return to the waiting state. Candidate execution segments that exceed the preset operator duration limit are split down along the current task boundary; if the remaining execution segments after splitting are less than a minimum scheduling unit, the remaining execution segments are reserved for the next empty window before execution. The waiting start time is recorded on the host side when the main computing stream completes local preprocessing and enters the waiting state. The arrival time is recorded synchronously or written back along with the network adapter when the ready value is written back. The two use a unified clock source or are compared after clock conversion to ensure that the calculation of the idle window duration is consistent. To establish the data ready flag, the system allocates the data batch corresponding to the current fine sorting task to a predetermined data buffer and binds it to a flag address; this flag address usually comes from an array index of multiple flag arrays so that concurrent requests can be isolated from each other; Before the main computing stream enters the waiting state, it records the expected request identifier and expected version number. Before the main computing stream resumes execution, the system reads the request identifier and version number carried by the current batch and compares them with the expected values one by one. If the comparison results are consistent, the main computing stream is allowed to continue execution. If they are inconsistent, the system prohibits the main computing stream from resuming based on the flag address and clears the flag address. The request identifier is used to distinguish different business requests, and the version number is used to distinguish different batches of updates for the same request. If either of them is inconsistent, it is considered that the data is ready and does not meet the conditions for recovery. Furthermore, multiple flag arrays are allocated in a round-robin manner to provide independent slots for concurrent batches. The same slot is not bound to a new waiting batch before the previous batch is cleared. If the slot corresponding to the current round position is still in a bound and not cleared state, the next slot is selected in sequence until an empty slot is found or the search limit for this round is reached. If no available slot is found after reaching the search limit, the main computing stream will remain inactive and the background computing stream will be prevented from injecting new computing subtasks. The upper limit of this round of retrieval is preferably set to the total number of slots in a single flag array or the total number of available slots in multiple flag arrays, to ensure that a single round of search does not extend indefinitely; when an empty slot is found, the array number and array index of that slot are saved as the flag address location information for the current batch, and the unready value or cleared value is written back according to the same location information when clearing it later. If the network adapter has not yet completed writing the reserved data buffer, request identifier, and version number, the flag address remains in an unready state, and the background computing stream continues to execute subsequent computing subtasks; If a request is misaligned, the version is inconsistent, or the data ready flag is overwritten by an older batch, the recovery action will be blocked. The system will retain the current waiting relationship and clear the corresponding data ready flag, in order to wait for the next batch of recalled data to re-establish the matching relationship.
[0019] In one alternative approach, the steps of breaking down the background task into multiple computational subtasks with execution durations not exceeding a preset operator duration limit include: Obtain the candidate task pool and the corresponding computation data for each candidate task from the preset background task set; according to the data type of the candidate task, split the tasks in the candidate task pool into data blocks, word blocks, or sample blocks; Set a task identifier, batch identifier, and version identifier for each computational subtask; control each computational subtask to save its intermediate computational state to the state buffer corresponding to the task identifier and write the execution result to the result buffer corresponding to the batch identifier when it finishes execution; The tasks in the candidate task pool include advertising feature calculation tasks that are divisible, idempotent, and interruptible, and have result caching or reuse indexing mechanisms. Background computation streams can use independent buffers, share read-only parameter areas, or not request specified mutually exclusive resources; Before the step of controlling the background computing stream to execute multiple computing subtasks sequentially, the remaining available time for advertising inference requests is determined based on business latency constraints and the shortest guaranteed execution time of the main computing stream. Obtain the graphics processor's memory bandwidth utilization and L2 cache utilization; allow background computing streams to start when the remaining available time is greater than zero and the memory bandwidth utilization does not exceed the memory bandwidth utilization threshold and the L2 cache utilization does not exceed the L2 cache utilization threshold; otherwise, prohibit the start of background computing streams.
[0020] Retrieve candidate task pools from the preset background task set, and select the splitting method according to the data type of the task; for tasks that can be processed by blocks, split them by data blocks; for tasks that are suitable for word-based processing, split them by word blocks. For tasks suitable for sample-based advancement, they are split into sample blocks. Before splitting, the original computational data range information corresponding to the candidate task is read. Among them, the data block is used to represent a set of feature data within a continuous address interval, the word block is used to represent a segment of word interval in a continuous word sequence, and the sample block is used to represent a continuous sample interval in a batch of samples. The system pre-reads the unit data processing time of various tasks on the corresponding graphics processor and calculates the maximum processing length allowed for the current computational subtask. The calculation formula is as follows: ; in, This represents the maximum allowed processing length for the current computational subtask. To preset the operator duration upper limit, The time consumed per unit of data processing This is the floor symbol; Time consumed per unit of data processing The data was obtained through a combination of offline performance testing and online sliding window scrolling updates: During server initialization, benchmark tests were performed on the target graphics processor for each type of task in the candidate task pool, and the initial average time per unit length of data block, word block, or sample block was recorded as... The initial value; during system operation, it is updated using a weighted average based on the actual execution time of similar subtasks within the most recent 50 statistical windows to ensure the calculated maximum processing length. It can adapt to changes in hardware load; The system will divide the data according to the selected splitting method, ensuring that the splitting length does not exceed the specified limit. Under the constraints, register the start offset, end offset, and processing length for each computational subtask, so that each computational subtask covers only one continuous processing interval; Each computational subtask after splitting has a task identifier, a batch identifier, and a version identifier. After execution, the intermediate computation status is written to the status buffer corresponding to the task identifier, and the execution result is written to the result buffer corresponding to the batch identifier. Furthermore, the intermediate computation states in the state buffer are maintained according to the task identifier, which are categorized into four logical states: not started, in execution, completed and reusable, and expired. When a certain computational subtask starts, the corresponding status changes from not started to running. When the execution ends normally and the version identifiers are consecutive, it changes to completed and reusable. Subsequent batches can directly read this status and continue computation when the task identifiers are consistent and the version identifiers meet the conditions for continuous progress. If the interruption occurs within a computational subtask and has not reached the subtask boundary, the current execution segment will not be written to the reusable state. Instead, it will roll back to the previous completed boundary to reorganize the subsequent computation, so as to ensure that the state buffer only saves the results at the complete subtask boundary. Each computational subtask contains a continuous processing interval, and its input and output boundaries remain in one-to-one correspondence before and after the split. The task identifier is used to uniquely indicate the background task type and its computation instance; the batch identifier is used to indicate the data batch to which the current result belongs; and the version identifier is used to indicate the execution order of the same task or the same batch. When subsequent batches need to reuse existing intermediate calculation states, first compare whether the task identifiers are consistent, then check whether the version identifiers meet the continuous advancement conditions, and write the restored output to the corresponding result buffer according to the batch identifier, so as to form a traceable data chain between state reading, calculation continuation and result attribution. The tasks in the candidate task pool are limited to advertising feature calculation tasks that are divisible, interruptible, and have result caching or reusable indexing mechanisms. The system prioritizes tasks that satisfy idempotency and interruptibility, and controls the background computation flow to run under the conditions of independent buffers, shared read-only parameter areas, or without requesting specified mutual exclusion resources; Furthermore, before a candidate task enters the background task set, it first undergoes a blockability check, an interruptibility check, and a state reusability check in sequence. If the blockability check fails, it is directly excluded. If the blockability check passes but the interruptibility check fails, it is only kept in the ordinary task queue and does not enter the background computation stream. If the first two conditions are met but the state reusability check fails, it is allowed as a candidate for a one-time subtask. However, if the remaining available time is less than the full execution time of the candidate task, it will not be selected. Idempotency is determined by the fact that repeatedly executing the same computational subtask will not change the solidified result. Interruptibility is determined by the fact that stopping at any subtask boundary will not destroy the previously completed state. State reusability is checked by whether the intermediate computational state can be reloaded from the previous completed boundary and the process can continue under the same task identifier. If a task needs to occupy a shared writable buffer that the main computation stream is writing to, or needs to hold a mutex resource that is not released across multiple subtasks, it is determined that the background insertion condition is not met. For tasks with result caching or reuse indexing mechanisms, the reuse index includes at least the task identifier, version identifier, and interval boundary information. The system uses this reuse index to locate the directly readable state buffer location. Before the background computing stream starts, the system will first determine whether there is still available time for the current request; the remaining available time is derived from the business latency constraints and the shortest guaranteed execution time of the main computing stream, and at the same time, the system reads the graphics processor's video memory bandwidth utilization and L2 cache utilization. Background computing streams are only allowed to start when the remaining available time is greater than zero and the video memory bandwidth utilization and L2 cache utilization do not exceed their respective thresholds. If any condition is not met, the background computing stream will be prohibited from starting; the threshold is obtained by correcting the quantile value obtained from previous load statistics with the latency budget of the current request; the remaining available time is calculated as follows: ; in, The total latency budget allowed for the current request. The minimum guaranteed execution time reserved after the main computation flow resumes. This represents the time elapsed since the current request entered the inference node. The video memory bandwidth utilization rate is obtained by the ratio of the video memory bandwidth used in the current statistical window to the rated video memory bandwidth of the graphics processor. The L2 cache utilization rate is obtained by the ratio of the L2 cache capacity used in the current statistical window to the total L2 cache capacity. Both are updated using the same sampling window as the main computing flow scheduling cycle. when When the value is less than or equal to zero, the system directly prevents the background computing stream from starting; when When the execution time is greater than zero but less than the execution time of a minimum scheduling unit, the system will not select a new background task, but will only retain the reused results of the completed state; The memory bandwidth utilization threshold is taken from the preset high quantile value of the bandwidth utilization samples in the most recent statistical windows, and is reduced proportionally when the latency budget of the current request is lower than the business's normal budget. The L2 cache utilization threshold is determined using the same method as the video memory bandwidth utilization threshold; whether to allow background computing streams is determined sequentially: first, check... If the value is greater than zero, then check if the video memory bandwidth utilization rate exceeds the threshold, and then check if the L2 cache utilization rate exceeds the threshold; only when all three conditions are met will the start-up result be output, otherwise the start-up result will be output and the main computing stream will be kept in exclusive waiting interval. When a background task is broken down into multiple computational subtasks, if the resource usage of a subtask exceeds the preset range during execution, subsequent subtasks can be kept in the queue to be dispatched and will continue to proceed when the next window of opportunity arrives. If a candidate task does not meet the requirements of idempotency or does not have the interruptible property, it will not be included in the background task set.
[0021] In one alternative approach, the steps of controlling the background computation stream to execute multiple computational subtasks sequentially include: Obtain the total resources of the graphics processor and the preset background task scheduling ratio, and determine the upper limit of streaming multiprocessor resources, upper limit of video memory bandwidth and upper limit of L2 cache for the background computing stream accordingly. Before each computational subtask is executed, the resource usage information stored in the computational subtask is used to determine whether the streaming multiprocessor usage, video memory bandwidth usage, and L2 cache usage of the computational subtask do not exceed the corresponding streaming multiprocessor resource limit, video memory bandwidth limit, and L2 cache limit, respectively. If none of the above conditions are met, the computational subtasks will be issued; if any one condition is met, the background computation stream will be suspended. The steps for determining the upper limit of the preset operator duration include: determining candidate values for the upper limit of the preset operator duration based on the statistical results of the window duration of previous tasks; The remaining available time for the ad inference request and the maximum execution time of a single operator allowed by the graphics processor to ensure preemptive response are used as the hardware time limit; the minimum value among the candidate value, the remaining available time and the hardware time limit is determined as the preset operator time limit; It also includes: obtaining the 99.9th percentile latency of the fine-ranking task after enabling background computing flow and the 99.9th percentile latency of the fine-ranking task without enabling background computing flow; The tail latency jitter rate is determined by the ratio of the difference between the two to the 99.9th percentile latency of the fine-ranking task when the background computing stream is not enabled. Obtain the duration of the idle window and the execution time of each computational subtask completed during the idle window; The window fill rate is determined by summing the execution times of each computational subtask completed during the window period and the window duration; when the tail latency jitter rate does not exceed the preset jitter threshold and the window fill rate is lower than the target fill rate, the background task scheduling ratio is increased by a preset step size to no more than the upper limit threshold of the ratio. When the tail latency jitter rate does not exceed the preset jitter threshold and the window fill rate is greater than or equal to the target fill rate, the background task scheduling ratio remains unchanged. When the tail latency jitter rate exceeds the preset jitter threshold, the background task scheduling ratio is reduced by a preset step size. Background tasks are obtained from a preset candidate task pool. The method also includes: obtaining the reuse degree, duration matching and resource consumption of multiple candidate tasks in the candidate task pool. Based on the reusability, duration matching, and resource consumption of each candidate task, the scheduling priority score of each candidate task is determined. Candidate tasks are selected as background tasks in descending order of scheduling priority score. In continuous When the tail latency jitter rate exceeds the preset jitter threshold within a statistical window, the background task scheduling ratio will be adjusted to the preset lower limit threshold. It is a preset positive integer.
[0022] After starting the background computing stream, the upper limit of resources that the background computing stream can use is determined based on the total resources of the graphics processor and the preset background task scheduling ratio. Before each computing subtask is sent out, it carries its pre-stored resource usage information. The system then compares each item with the subtask's streaming multiprocessor usage, video memory bandwidth usage, and L2 cache usage to see if they do not exceed the corresponding upper limits. The upper limit of SM resources is obtained by multiplying the total number of available SMs of the graphics processor by the background task scheduling ratio and then rounding down. The upper limit of video memory bandwidth is obtained by multiplying the rated video memory bandwidth by the background task scheduling ratio. The upper limit of L2 cache is obtained by multiplying the allocatable L2 cache capacity by the background task scheduling ratio. The resource usage information of the computation subtask is derived from the execution profile recorded before the task enters the candidate task pool. The execution profile includes at least the average streaming multiprocessor usage, average video memory bandwidth usage, and average L2 cache usage, and can be updated in subsequent windows based on the most recently executed results. Distribution is only allowed if all conditions are met; if any condition exceeds the limit, subsequent distribution will be suspended and the system will wait for the next round of resource assessment. Furthermore, resource checks are performed in a fixed order: first, the usage of streaming multiprocessors is checked, then the usage of video memory bandwidth is checked, and then the usage of L2 cache is checked; if the previous condition is not met, the next condition is not checked, and the computational subtask is directly kept at the head of the queue to be sent. If two consecutive rounds of resource assessment are suspended due to the same resource item exceeding the limit, the parallel execution scale of the computational subtask will be reduced before the next round of assessment, or it will be split into smaller execution segments and re-submitted for assessment; if it still exceeds the limit after splitting, the task will be downgraded to a non-current empty window executable task and will re-enter the candidate sequence after subsequent resource availability. The candidate value for the preset operator duration limit is formed by statistical results of the idle time of historical tasks. The minimum value among the candidate value, the remaining available time of the current request, and the maximum execution time of a single operator allowed by the graphics processor is used as the preset operator duration limit. The previously recorded window duration statistics are used to generate candidate values using the median or quantile values within multiple statistical windows. The remaining available time is then checked again after the request enters the waiting state to prevent the length of the background subtask slice from exceeding the current acceptable range. If the remaining available time for the current request is insufficient, the corresponding upper limit will be reduced accordingly; if the hardware itself allows a short duration for a single operator, then the hardware upper limit will directly become a constraint. Furthermore, the statistical results of the window duration of historical tasks are not directly adopted using a single window value, but rather abnormal windows with deviations exceeding the preset distribution threshold are removed before candidate values are generated; When the business group to which the current request belongs has insufficient samples in the most recent statistical window, the global statistical candidate value is used first, and then the global statistical result is gradually replaced in subsequent windows with the newly added empty window duration of the business group. If a request detects that the remaining available time continues to shrink to below the currently selected candidate value after the waiting state is established, the system will not modify the computational subtask that is already being executed, but will re-divide it according to the new time limit starting from the next subtask to be issued, in order to keep the division boundary stable. Candidate values are preferentially selected from the median value of the empty window duration samples within the most recent statistical windows; when the median value is higher than the hardware duration limit or higher than the remaining available time of the current request, it will not be relaxed further, but will be directly constrained by the smaller of the two values. To ensure the maximum execution time of a single operator allowed by preemptive response, the graphics processor uses the response time parameter pre-recorded on the device side and reloads the corresponding parameter group when switching device models, thus giving different graphics processors a clear source of upper limit. Within the statistics window, the 99.9th percentile latency of the fine-ranking task after enabling background computing flow is calculated and compared with the corresponding latency when background computing flow is not enabled to obtain the tail latency jitter rate; at the same time, the sum of the execution time of each computing subtask completed during the empty window is calculated and compared with the empty window duration to obtain the empty window fill rate. When the tail latency jitter rate does not exceed the preset jitter threshold and the window fill rate is lower than the target fill rate, the background task scheduling ratio is increased by a preset step size, but not exceeding the upper limit threshold; when the tail latency jitter rate exceeds the preset jitter threshold, the background task scheduling ratio is decreased by a preset step size. The preset step size is a fixed proportional adjustment unit, and the upper and lower thresholds of the proportional ratio are used to limit the maximum and minimum values of the scheduling ratio, respectively. Furthermore, the adjustment of the scheduling ratio is executed according to the safety priority cascading judgment: first check whether the tail delay jitter rate exceeds the limit, then check whether the empty window fill rate is lower than the target value, and then execute the ratio increase or decrease; as long as the tail delay jitter rate exceeds the preset jitter threshold, even if the empty window fill rate is still low, the background task scheduling ratio will be reduced first. If the tail delay jitter rate fluctuates around a threshold within multiple consecutive statistical windows, the system will not immediately reverse the adjustment in the current window. Instead, it will require that out-of-bounds or insufficient values in the same direction occur at least a preset number of times before triggering the next proportional change. The tail delay jitter rate is calculated as follows: ; in, This indicates the tail delay jitter rate. To reduce the 99.9th percentile latency of the fine-ranking task after enabling background computation stream, The 99.9th percentile latency for fine-ranking tasks without background computation stream enabled; The window fill rate is obtained by dividing the sum of the execution times of all computational subtasks completed during the window period by the window duration; when When the value is zero or no valid control sample is formed in the current statistical window, the system skips the proportion increase judgment for this window and only retains the downward protection logic. When a background task is selected from the candidate task pool, the system first obtains the reusability, duration matching and resource consumption of the candidate tasks, and then determines the scheduling priority score of each candidate task based on the score, and selects the background task from high to low. Reusability measures the likelihood that the results of a candidate task can be reused in subsequent requests; duration matching measures how well it fits with the duration of the empty window; and resource usage measures the risk of interference to the main computing flow. If the tail latency jitter rate exceeds the preset jitter threshold in multiple consecutive statistical windows, the background task scheduling ratio will be adjusted to the lower limit threshold so that the background injection intensity in subsequent windows returns to the preset lower limit level. If the reuse level of certain candidate tasks is lower than the preset reuse threshold, the duration is mismatched, or the resource consumption exceeds the preset consumption threshold, their scores will be reduced by a preset ratio, and they will not be given priority. Furthermore, the scheduling priority score of each candidate task. The weighted composite calculation is performed using the following mathematical formula: ; in, Scoring is given based on scheduling priority. For the degree of reuse, The ratio of the absolute value of the difference between the expected execution time of the candidate task and the current empty window duration to the current empty window duration represents the normalized duration matching situation. This is a normalized combination value representing resource usage. , , All are preset weight coefficients greater than zero, and resource usage items are calculated by subtracting... It participates in the calculation as a reverse constraint; To prevent extremely small positive numbers with a denominator of zero; This mapping function ensures that candidate tasks with high reusability but resource interference exceeding a preset interference threshold will not occupy the preferred position for a long time. When two candidate tasks have the same score and both meet the current window distribution conditions, the candidate task with a duration matching that is closer to the current window duration is selected first; if the duration matching is still the same, the candidate task with lower resource consumption is selected first to reduce the risk of interference to the recovery of the main computing flow. The degree of reusability is represented by the proportion of the number of times the target candidate task is hit by subsequent requests within the most recent statistical window to its total number of calls; the duration matching is represented by the absolute value of the difference between the expected execution time of the candidate task and the current empty window duration, and the smaller the difference, the higher the degree of matching. Resource usage is represented by a normalized combination of streaming multiprocessor usage, video memory bandwidth usage, and L2 cache usage in the candidate task execution profile. The larger the combination value, the higher the risk of interference. When sorting candidate tasks, tasks that do not meet the current resource limit and remaining available time constraints are first eliminated, and then the scores of the remaining tasks are compared.
Claims
1. A method for dynamic scheduling and allocation of elastic computing power for large-scale advertising model inference tasks, applied to servers equipped with graphics processors and network adapters, characterized in that: include: Receive an ad inference request and trigger the corresponding ad recall task and fine ranking task, wherein the fine ranking task depends on the recall data returned by the ad recall task; determine the background task to be scheduled in parallel with the ad inference request, and establish the main computing flow for the fine ranking task and the background computing flow for the background task; A data ready flag, which is directly updated by the network adapter, is set in the video memory of the graphics processor; The background task is divided into multiple computational subtasks whose execution time does not exceed the preset operator duration limit. The preset operator duration limit is determined based on the idle window duration, which is the time difference between the main computation flow entering the waiting state and the arrival of the recalled data. While the main computing stream completes the local preprocessing of the fine ranking task and waits for the recalled data, the background computing stream is controlled to execute the multiple computing sub-tasks in sequence, and the data ready flag is read after each computing sub-task is completed. If the reading result indicates that the recalled data has not arrived, the background computing stream is controlled to continue executing subsequent computing subtasks; When the reading result indicates that the recalled data has arrived, the submission of subsequent computational subtasks that have not yet been issued to the graphics processor is stopped, and the instruction corresponding to the main computational stream is written into the high-priority execution queue of the graphics processor, triggering the main computational stream to resume execution of the fine-ordering task.
2. The method for dynamic scheduling and allocation of elastic computing power for large-scale advertising model inference tasks according to claim 1, characterized in that, The step of setting a data ready flag that is directly updated by the network adapter in the video memory of the graphics processor includes: Allocate a predetermined data buffer and flag address to the data batch corresponding to the current fine sorting task, and initialize the data ready flag to the not ready state; The network adapter is controlled to write the recall data into the predetermined data buffer via remote direct memory access; after the network adapter completes the remote direct memory access to write the predetermined data buffer, the network adapter is controlled to write the ready value into the flag address via remote direct memory access. When the value read from the flag address is equal to the ready value, it is determined that the recall data has arrived and the main computing stream is allowed to resume execution.
3. The method for dynamic scheduling and allocation of elastic computing power for large-scale advertising model inference tasks according to claim 2, characterized in that, The steps for assigning flag addresses to the data batch corresponding to the current fine sorting task include: Assign array indices to the data batches corresponding to the current fine sorting task in multiple flag arrays; Bind the data batch corresponding to the current fine sorting task to the flag address corresponding to the array index; Record the expected request identifier and expected version number before the main computing stream enters the waiting state; Before the main computing stream resumes execution, the request identifier and version number of the data batch corresponding to the current fine sorting task are obtained, and the request identifier and version number are compared with the expected request identifier and the expected version number, respectively. If the comparison results are consistent, the main computing flow is allowed to resume execution; If the comparison results are inconsistent, the main computing flow recovery is prohibited from being triggered based on the flag address, and the data ready flag value at the flag address is cleared to wait for the next batch of the recalled data.
4. The method for dynamic scheduling and allocation of elastic computing power for large-scale advertising model inference tasks according to claim 1, characterized in that, The steps of splitting the background task into multiple computational subtasks with an execution time not exceeding the preset operator duration limit include: Obtain the candidate task pool and the corresponding computation data for each candidate task from the preset background task set; according to the data type of the candidate task, split the tasks in the candidate task pool into data blocks, word blocks, or sample blocks; Set a task identifier, batch identifier, and version identifier for each computational subtask; Each computational subtask is controlled to save its intermediate computational state to a state buffer corresponding to the task identifier upon completion of its execution, and to write the execution result to a result buffer corresponding to the batch identifier.
5. The method for dynamic scheduling and allocation of elastic computing power for large-scale advertising model inference tasks according to claim 4, characterized in that, The tasks in the candidate task pool include advertising feature calculation tasks that are divisible, idempotent, interruptible, and have result caching or reuse indexing mechanisms. The background computing stream uses an independent buffer, a shared read-only parameter area, or does not request specified mutually exclusive resources.
6. The method for dynamic scheduling and allocation of elastic computing power for large-scale advertising model inference tasks according to claim 1, characterized in that, Before the step of controlling the background computing stream to execute the plurality of computing subtasks sequentially, the method further includes: The remaining available time for the advertising inference request is determined based on business latency constraints and the minimum guaranteed execution time of the main computing flow; the video memory bandwidth utilization and L2 cache utilization of the graphics processor are obtained; The background computing stream is allowed to start when the remaining available time is greater than zero and the video memory bandwidth utilization rate does not exceed the video memory bandwidth utilization rate threshold and the L2 cache utilization rate does not exceed the L2 cache utilization rate threshold; otherwise, the background computing stream is prohibited from starting.
7. The method for dynamic scheduling and allocation of elastic computing power for large-scale advertising model inference tasks according to claim 1, characterized in that, The steps of controlling the background computing stream to execute the plurality of computing subtasks sequentially include: The total resources of the graphics processor and the preset background task scheduling ratio are obtained, and the upper limit of streaming multiprocessor resources, upper limit of video memory bandwidth and upper limit of L2 cache of the background computing stream are determined accordingly. Before each computational subtask is executed, the resource usage information pre-stored in the computational subtask is used to determine whether the streaming multiprocessor usage, video memory bandwidth usage, and L2 cache usage of the computational subtask do not exceed the corresponding streaming multiprocessor resource limit, video memory bandwidth limit, and L2 cache limit, respectively. The computational subtask is issued when none of the above conditions are met; if any one condition is exceeded, the background computation stream is suspended.
8. The method for dynamic scheduling and allocation of elastic computing power for large-scale advertising model inference tasks according to claim 6, characterized in that, The steps for determining the upper limit of the preset operator duration include: Candidate values for the preset operator duration upper limit are determined based on the statistical results of the window duration of previous tasks; The remaining available time for the advertising inference request is obtained, and the maximum execution time of a single operator allowed by the graphics processor to ensure preemptive response is used as the hardware time limit. The minimum value among the candidate value, remaining available time, and hardware duration limit is determined as the preset operator duration limit.
9. The method for dynamic scheduling and allocation of elastic computing power for large-scale advertising model inference tasks according to claim 8, characterized in that, Also includes: Obtain the 99.9th percentile latency of the fine-ranking task after enabling background computing stream and the 99.9th percentile latency of the fine-ranking task without enabling background computing stream; The tail latency jitter rate is determined by the ratio of the difference between the 99.9th percentile latency of the fine-ranking task after enabling background computing flow and the 99.9th percentile latency of the fine-ranking task without enabling background computing flow to the 99.9th percentile latency without enabling background computing flow. Obtain the duration of the idle window and the execution time of each computational subtask completed during the idle window; The window fill rate is determined by summing the execution times of each computational subtask completed during the window period and the window duration. When the tail delay jitter rate does not exceed the preset jitter threshold and the window fill rate is lower than the target fill rate, the background task scheduling ratio is increased by a preset step size to no more than the upper limit threshold of the ratio. When the tail delay jitter rate does not exceed the preset jitter threshold and the window fill rate is greater than or equal to the target fill rate, the background task scheduling ratio remains unchanged. When the tail delay jitter rate exceeds the preset jitter threshold, the background task scheduling ratio is reduced by the preset step size.
10. The method for dynamic scheduling and allocation of elastic computing power for large-scale advertising model inference tasks according to claim 9, characterized in that, The background task is obtained from a preset candidate task pool, and the method further includes: Obtain the reuse rate, duration matching, and resource consumption of multiple candidate tasks in the candidate task pool; Based on the reuse level, duration matching, and resource consumption of each candidate task, the scheduling priority score of each candidate task is determined. Candidate tasks are selected as background tasks in descending order of scheduling priority scores. In continuous When the tail latency jitter rate exceeds the preset jitter threshold within a statistical window, the background task scheduling ratio is adjusted to the preset lower limit threshold. It is a preset positive integer.
