A reinforcement learning multi-agent based cooperative reasoning task scheduling system

Abstract

The application discloses a kind of based on reinforcement learning multi-agent's cooperative reasoning task scheduling system, comprising: stage bounding module, for establishing stage record table;KV cache page metering module, for dividing KV cache page;Coarse charting module, for generating task stage coarse graph edge table;Fine charting module, for generating cache page fine graph edge table;Graphormer bias deduction module, for according to execution deviation and cache page cost deduction attention score and generate candidate scheduling action score table;Action closure module, for deleting unqualified candidate action;Multi-agent scheduling write-back module, for executing highest score action and writing back execution data.The application realizes the efficient scheduling of cooperative reasoning task, improves cache utilization and reasoning stability.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence reasoning task scheduling technology, and in particular to a cooperative reasoning task scheduling system based on reinforcement learning multi-agent systems. Background Technology

[0002] When collaborative inference platforms handle large model inference requests, they often allocate tasks based on inference node load, available GPU memory, queue length, and communication latency, treating the inference request as a whole task and assigning it to a single inference node. Some solutions also use coarse-grained allocation based on pre-filling and decoding stages. These approaches focus on matching computing power at the node level, lacking unified modeling for the dependencies between inference stages, the occupancy status of key-value cache pages, and the continuity of GPU memory pages.

[0003] During large-scale model inference, the key-value (KV) cache grows with the input length and the number of output tokens. Cache retention, cache migration, and cache recalculation have different execution costs, and memory page fragmentation affects cache page writing and subsequent decoding efficiency. Existing scheduling methods mostly treat the KV cache as auxiliary data processing, lacking a processing structure that includes KV cache pages, current memory pages, target memory pages, and migration links in the scheduling graph.

[0004] Existing graph model scheduling methods mostly use inference nodes as graph nodes for scoring, failing to incorporate coarse graph edges for task stages and fine graph edges for cache pages into the Graphormer attention bias, and neglecting to utilize stage execution time skew, transition time skew, cache page fragmentation rate, and cache page read / write counts to correct candidate action scores. This easily leads to scheduling actions resulting in insufficient GPU memory, inconsistent model versions, mismatched cache page formats, and exceeding the estimated completion time limit. Therefore, how to provide a collaborative inference task scheduling system based on reinforcement learning multi-agent systems is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a cooperative inference task scheduling system based on reinforcement learning and multiple agents. This invention utilizes page-based key-value cache management, Graphormer bias deduction, and multi-agent scheduling write-back to achieve fine-grained scheduling of inference tasks, and has the advantages of high cache utilization, high scheduling accuracy, and low inference latency.

[0006] A cooperative reasoning task scheduling system based on reinforcement learning multi-agent according to an embodiment of the present invention includes: The phase delimitation module is used to create a four-line phase record table according to the order of pre-filling, cache page writing, decoding, and return; The KV cache page metering module is used to divide KV cache pages according to fixed cache page rules and establish cache page metering records. The coarse graph tabulation module is used to pair each stage row in the stage record table with the inference nodes that meet the model version and latency threshold, and to register the stage number, node number and stage execution time in the task stage coarse graph side table. The detailed graph table module is used to pair KV cache pages with current video memory pages, target video memory pages, and migration links one by one, and register the cache page number, page location, migration time, recalculation time, and fragmentation rate in the cache page detailed graph side table; The Graphormer bias deduction module is used to establish bias values ​​for coarse and fine graph edges in the Graphormer attention matrix. It includes the stage execution time deviation in the coarse graph edge bias value, and includes the cache page fragmentation rate, cache page read / write count, and migration time deviation in the fine graph edge bias value. It deducts the attention score of the corresponding edge according to the bias value to obtain the candidate scheduling action score table. The action closure module is used to verify the consecutive memory pages, model version, cache page format and expected completion time of each candidate scheduling action, and delete candidate scheduling actions that do not meet the verification conditions. The multi-agent scheduling and write-back module is used to configure an inference agent for each inference node. The inference agent executes the action with the highest remaining score and fills back the actual stage time, actual migration time, decoding throughput and fragmentation rate to the Graphormer bias deduction module.

[0007] Optionally, the operation of creating the four-row stage record table includes: Generate a stage dependency chain according to the execution order of pre-filling, cache page writing, decoding, and postback; In the stage dependency chain, the pre-filled record is pointed to the cache page write record, the cache page write record is pointed to the decode record, and the decode record is pointed to the return record; The input length is associated with the pre-filled record, the output budget is associated with the decoded record, the latency threshold is associated with the stage dependency chain, and the stage dependency chain is sent to the coarse graph tabulation module.

[0008] Optionally, the operation of partitioning KV cache pages includes: Determine the page boundaries of the KV cache according to the fixed cache page rules, and merge the key cache and value cache within the same page boundary into one KV cache page; Establish a cache page metering record for each KV cache page. The cache page metering record is used to mark the page occupancy status of the KV cache page in the inference node. The cached page metering records are sent to the detailed charting module, which then adds migration time, recalculation time, and fragmentation rate based on the cached page metering records.

[0009] Optionally, the operation of creating the coarse chart includes: Using stage records in the stage dependency chain as edge sources and inference nodes as edge ends, inference nodes with inconsistent model versions are removed. For the remaining inference nodes, calculate the stage execution time by recording the corresponding stage resource quantity and the stage processing capacity of the inference node for each stage. Candidate edges whose execution time exceeds the latency threshold during the deletion phase; According to the order of the stage dependency chain, the pre-filled candidate edges, cache page writing candidate edges, decoding candidate edges, and return candidate edges are registered in the task stage coarse graph edge table in sequence, and the preceding edge number and the following edge number are registered between adjacent candidate edges.

[0010] Optionally, the detailed chart creation operation includes: Use the KV cache page as the object of detailed graph tabulation, and locate the current video memory page number group corresponding to the KV cache page in the video memory page occupancy record of the candidate inference node; Scan the free contiguous video memory segments in the candidate inference nodes according to the number of pages required by the KV cache, and delete the candidate inference nodes for which no free contiguous video memory segments were scanned; Register the scanned free contiguous video memory page segments as the target video memory page number group, and select a migration path between the current video memory page number group and the target video memory page number group; The migration time is obtained based on the page size of the KV cache page, the migration link bandwidth, and the migration link queuing time. The recalculation time is obtained based on the fixed token window length, the number of model layers, and the node inference throughput of the target inference node. The fragmentation rate is obtained based on the non-contiguous occupancy ratio of the target memory page number group. Register the current memory page number group, target memory page number group, migration link, migration time, recalculation time and fragmentation rate corresponding to the same KV cache page as cache page candidate edges, and form a cache page detailed graph edge table page by page.

[0011] Optionally, the operation of establishing and applying attention bias includes: Obtain the stage execution time edge by edge of the rough map edge table for each task stage, and retrieve the actual stage time consumed in the previous round of backfilling. When the actual time taken for a stage exceeds the stage execution time, the excess is taken as the stage execution time deviation; when it does not exceed the time taken, the stage execution time deviation is recorded as zero. The stage execution time deviation is converted into a coarse graph edge bias value according to the stage time normalization threshold, and then subtracted from the original attention score of the corresponding coarse graph edge to obtain the biased coarse graph edge attention score. For each edge of the cache page detailed graph edge table, obtain the migration time, fragmentation rate, and cache page read / write count. The cache page read / write count is the sum of the number of reads and writes of the same KV cache page in the candidate scheduling action. Retrieve the actual migration time of the previous round of backfilling. When the actual migration time exceeds the migration time, the excess is taken as the migration time deviation; when it does not exceed the migration time, the migration time deviation is recorded as zero. The migration time deviation, fragmentation rate, and cache page read / write count are converted according to their respective normalization thresholds, multiplied by preset bias weights, and then summed to obtain the fine graph edge bias value. The biased attention score of the corresponding thin edge is obtained by subtracting the bias value of the thin edge from the original attention score.

[0012] Optionally, the operation of obtaining the candidate scheduling action score table includes: Read the edge attention scores of the coarse image after bias and the edge attention scores of the fine image after bias; According to the stage number in the coarse graph edge table of the task stage, the pre-filled candidate edge, the decoding candidate edge, and the return candidate edge are registered as candidate node actions, and the corresponding biased coarse graph edge attention score is registered as the candidate node action score. For cache page write candidate edges, read the cache page candidate edges corresponding to the same KV cache page number. When the current video memory page number group is the same as the target video memory page number group, register it as a cache page reservation action. When the inference node corresponding to the current memory page number group is inconsistent with the inference node corresponding to the target memory page number group, and the migration time does not exceed the recalculation time, it is registered as a cache page migration action. When the inference node corresponding to the current memory page number group is inconsistent with the inference node corresponding to the target memory page number group, and the migration time exceeds the recalculation time, it is registered as a cache page recalculation action. Register the corresponding biased fine-image edge attention score as the cache page processing action score; The candidate node action score under the same inference node number is added to the cache page processing action score under the same KV cache page number, and the result is written into the candidate scheduling action score table.

[0013] Optionally, the operation of deleting candidate scheduling actions that do not meet the verification conditions includes: Read the candidate scheduling action score table and action verification data. The action verification data includes the number of consecutive video memory pages, model version, cache page format, and estimated completion time. When the number of consecutive video memory pages is less than the number of video memory pages required for KV cache pages, the model version is inconsistent with the model version corresponding to the inference task, the cache page format is inconsistent with the format supported by the inference node, or the estimated completion time exceeds the latency threshold, delete the corresponding candidate scheduling action and retain the candidate scheduling actions that have not been deleted in the candidate scheduling action score table.

[0014] Optionally, the operation of performing the final scheduling action includes: Configure an inference agent for each inference node, and extract the candidate scheduling actions that have not been deleted from the candidate scheduling action score table according to the inference node number; Each inference agent selects the highest-scoring action from the candidate scheduling actions of the corresponding inference node and registers it as the final scheduling action; When the final scheduling action is a cache page retention action, the location of the KV cache page in the video memory page remains unchanged; When the final scheduling action is a cache page migration action, the KV cache page migration is performed according to the target memory page number group and migration link; When the final scheduling action is a cache page recalculation action, the corresponding KV cache page is recalculated at the target inference node; After executing the final scheduling action, record the action start time, action end time, actual migration time, number of decoded output tokens, decoding time, and fragmentation rate after execution. Record the difference between the action end time and the action start time as the actual stage time, and record the number of decoded output tokens divided by the decoding time as the decoding throughput.

[0015] Optionally, the operation of performing record write-back includes: The actual stage time is used as the actual stage time of the next round of coarse graph edges, and the actual migration time is used as the actual migration time of the next round of fine graph edges. The decoding throughput and the fragmentation rate after execution are registered to the corresponding KV cache pages. The actual stage time, actual migration time, decoding throughput, and fragmentation rate after execution are used as inputs to the next round of the Graphormer bias deduction module. Reward records are generated based on the actual stage time, actual migration time, and decoding throughput. The action selection parameters of the inference agent are then updated based on these reward records.

[0016] The beneficial effects of this invention are: (1) The present invention divides the inference task into four stages: pre-filling, cache page writing, decoding and backhaul, and establishes a stage dependency chain so that the inference task is no longer allocated as a whole task in a coarse-grained manner. It can match the inference nodes according to different execution stages in the inference process, reducing the scheduling deviation caused by chaotic stage execution order and inaccurate node allocation.

[0017] (2) The present invention divides the KV cache according to the fixed cache page rules, and establishes a cache page detailed graph side table by combining the current video memory page number group, the target video memory page number group and the migration link, so that cache page retention, cache page migration and cache page recalculation can enter the scheduling and scoring process, improve the KV cache utilization rate and reduce the inference overhead caused by invalid migration and repeated recalculation.

[0018] (3) In the Graphormer attention calculation, the present invention introduces coarse graph edge bias value and fine graph edge bias value, writes the stage execution time deviation, migration time deviation, fragmentation rate and cache page read and write count into the candidate scheduling action score, and deletes candidate actions that do not meet the conditions of video memory, version, format and latency through the action closure module, thereby improving the accuracy and execution stability of collaborative reasoning task scheduling. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a schematic diagram of the overall structure of a cooperative reasoning task scheduling system based on reinforcement learning multi-agent proposed in this invention. Figure 2 This is a schematic diagram illustrating the generation of a cache page detail diagram for a cooperative reasoning task scheduling system based on reinforcement learning multi-agent proposed in this invention. Figure 3 This is a schematic diagram of the Graphormer bias deduction for a cooperative reasoning task scheduling system based on reinforcement learning multi-agent proposed in this invention. Detailed Implementation

[0020] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0021] refer to Figures 1-3 A cooperative reasoning task scheduling system based on reinforcement learning multi-agents includes: The stage delimiting module is used to build a four-line stage record table in the order of pre-filling, cache page writing, decoding, and return. After receiving an inference task, the stage delimiting module reads the input token sequence length, output token budget, and latency threshold. It writes the input token sequence length into the pre-filling line, the output token budget into the decoding line, and the latency threshold into the four-line stage record. It also registers the preceding stage marker and the following stage marker between the four-line stage records. The KV cache page metering module is used to divide KV cache pages according to fixed cache page rules and establish cache page metering records. The KV cache page metering module reads the model layer number, attention head number and fixed token window length of the model corresponding to the inference task. First, it divides the KV cache according to the model layer number, then divides the KV cache under the same model layer number according to the attention head number, and then splits the key cache and value cache corresponding to consecutive tokens according to the fixed token window length. The key cache and value cache obtained from each split are registered as a KV cache page. The coarse graph tabulation module is used to pair each row of the stage record table with the inference nodes that meet the model version and latency threshold, and to register the stage number, node number and stage execution time in the task stage coarse graph edge table. The coarse graph tabulation module reads the loaded model version of the inference node, deletes the inference node whose loaded model version is inconsistent with the inference task model version, and then establishes candidate edges with the pre-filled row, cache page write row, decoding row and return row and the remaining inference nodes respectively. The stage execution time is calculated by dividing the stage resource amount by the corresponding stage processing capacity, and candidate edges that exceed the latency threshold are deleted. The detailed graph tabulation module is used to pair KV cache pages with current memory pages, target memory pages, and migration links item by item. It records the cache page number, page location, migration time, recalculation time, and fragmentation rate in the cache page detailed graph side table. The detailed graph tabulation module reads the KV cache page metering records, finds the memory page numbers occupied by the same KV cache page number in the memory page occupancy records, and merges them into the current memory page number group. Then, it scans the continuous free memory page segments in the candidate inference nodes according to the number of memory pages required by the KV cache pages, registers the scanned continuous free memory page segments as the target memory page number group, and selects a migration link between the inference node where the current memory page number group is located and the inference node where the target memory page number group is located. The Graphormer bias deduction module establishes bias values ​​for coarse and fine graph edges in the Graphormer attention matrix. It includes the stage execution time deviation in the coarse graph edge bias value and the cache page fragmentation rate, cache page read / write counts, and migration time deviation in the fine graph edge bias value. It then deducts the attention score from the corresponding edge based on the bias value to obtain a candidate scheduling action score table. The Graphormer bias deduction module reads the stage execution time from the task stage coarse graph edge table and the actual stage time from the previous write cycle, recording the portion exceeding the actual stage time as the stage execution time deviation. It also reads the migration time, fragmentation rate, and cache page read / write count from the cache page fine graph edge table and the actual migration time from the previous write cycle, recording the portion exceeding the actual migration time as the migration time deviation. Finally, it converts the stage execution time deviation, migration time deviation, fragmentation rate, and cache page read / write count into attention deduction values. The action closure module is used to verify the consecutive video memory pages, model version, cache page format, and estimated completion time of each candidate scheduling action, and delete candidate scheduling actions that do not meet the verification conditions. The action closure module reads the target video memory page number group, the model version loaded by the inference node, the KV cache page format, and the estimated completion time corresponding to the candidate scheduling action. When the number of consecutive free video memory pages in the target video memory page number group is less than the number of video memory pages required by the KV cache page, or the model version loaded by the inference node is inconsistent with the model version of the inference task, or the KV cache page format is inconsistent with the format supported by the inference node, or the estimated completion time exceeds the latency threshold, the corresponding candidate scheduling action is deleted from the candidate scheduling action score table. The multi-agent scheduling and write-back module is used to configure an inference agent for each inference node. The inference agent executes the remaining highest-scoring action and fills the actual stage time, actual migration time, decoding throughput, and fragmentation rate back into the Graphormer bias deduction module. The multi-agent scheduling and write-back module divides the candidate scheduling action score table according to the inference node number. Each inference agent only reads the candidate scheduling actions that have not been deleted under the corresponding inference node number and selects the candidate scheduling action with the highest score as the final scheduling action. After execution, it records the action start time, action end time, migration link start time, migration link end time, number of decoded output tokens, and decoding time. It calculates the actual stage time, actual migration time, and decoding throughput, and writes the calculation results into the next round of the Graphormer bias deduction module.

[0022] In this embodiment, the operation of creating a four-line stage record table includes: The stage dependency chain is generated according to the execution order of pre-filling, cache page writing, decoding, and return; the stage delimiting module establishes four stage nodes, and marks the first stage node as a pre-filling record, the second stage node as a cache page writing record, the third stage node as a decoding record, and the fourth stage node as a return record according to the execution order. In the stage dependency chain, the pre-fill record is pointed to the cache page write record, the cache page write record is pointed to the decoding record, and the decoding record is pointed to the return record. The stage delimitation module writes the subsequent stage as the cache page write record in the pre-fill record, writes the preceding stage as the pre-fill record and the subsequent stage as the decoding record in the cache page write record, writes the preceding stage as the cache page write record and the subsequent stage as the return record in the decoding record, and writes the preceding stage as the decoding record in the return record. The input length is associated with the pre-filled record, the output budget is associated with the decoding record, and the latency threshold is associated with the stage dependency chain. The stage dependency chain is then sent to the coarse graph tabulation module. The stage delimitation module reads the number of input tokens for the inference task as the input length and binds the input length to the pre-filled record; it reads the preset upper limit of output tokens as the output budget and binds the output budget to the decoding record; it reads the maximum allowed completion time of the inference task as the latency threshold and binds the latency threshold to the stage dependency chain composed of the four stage nodes.

[0023] In this embodiment, the operation of partitioning KV cache pages includes: The page boundaries of the KV cache are determined according to the fixed cache page rules. Key caches and value caches within the same page boundary are merged and registered as one KV cache page. The KV cache page metering module reads the model layer number list and accesses the KV cache layer by layer according to the model layer number. Within each model layer number, the attention head number list is read and the KV cache is accessed head by head according to the attention head number. Under each attention head number, the key cache and value cache are sequentially truncated from the token start position to the token end position with a fixed token window length as the step size. Each truncated interval corresponds to a page boundary, and the key cache and value cache within the page boundary are jointly registered as one KV cache page. A cache page metering record is created for each KV cache page. The cache page metering record is used to mark the page-based occupancy status of the KV cache page in the inference node. The cache page metering record contains the KV cache page number, model layer number, attention header number, token start position, token end position, page size, inference node number, occupied memory page number group, and KV cache page format. The page size is obtained by counting the number of bytes occupied by the key cache and value cache within the page boundary. The occupied memory page number group is obtained by querying the memory page occupancy record of the inference node. The cache page metering records are sent to the detailed graphing module, which then adds migration time, recalculation time, and fragmentation rate based on these records. The KV cache page metering module does not directly fix the migration time, recalculation time, and fragmentation rate; instead, after selecting the target memory page number group and migration link for the same KV cache page, the detailed graphing module rereads the page size, migration link bandwidth, migration link queuing time, fixed token window length, model layer number, node inference throughput of the target inference node, and the occupancy status of the target memory page number group, and then calculates the migration time, recalculation time, and fragmentation rate.

[0024] In this embodiment, the operation of creating a coarse chart includes: Using stage records in the stage dependency chain as edge sources and inference nodes as edge ends, inference nodes with inconsistent model versions are eliminated; the coarse graph tabulation module reads the loaded model version of each inference node and reads the inference task model version; when the loaded model version of an inference node is inconsistent with the inference task model version, the inference node is marked as a version mismatch node; version mismatch nodes are not used to establish candidate edges with pre-filled records, cache page write records, decoding records, and return records; For the remaining inference nodes, the stage execution time is calculated based on the stage resource quantity and stage processing capacity of the inference node corresponding to the stage record. The coarse graph tabulation module reads the input length for the pre-filled record, the total page size of the KV cache pages to be written for the cache page write record, the output budget for the decode record, and the return data quantity for the return record. The coarse graph tabulation module reads the pre-filled throughput, cache page write bandwidth, decode throughput, and return link bandwidth for the same inference node. The stage execution time of the pre-filled candidate edge is obtained by dividing the input length by the pre-filled throughput, the stage execution time of the cache page write candidate edge is obtained by dividing the total page size of the KV cache pages to be written by the cache page write bandwidth, the stage execution time of the decoded candidate edge is obtained by dividing the output budget by the decode throughput, and the stage execution time of the return candidate edge is obtained by dividing the return data quantity by the return link bandwidth. Delete candidate edges whose stage execution time exceeds the latency threshold; the coarse graph tabulation module compares the stage execution time of each candidate edge with the latency threshold in the stage dependency chain; when the stage execution time is greater than the latency threshold, the candidate edge is marked as a timeout edge and removed from the candidate edge set; Following the sequential order of the stage dependency chain, pre-filled candidate edges, cache page writing candidate edges, decoding candidate edges, and postback candidate edges are registered in the task stage coarse graph edge table. Pre-order and post-order edge numbers are registered between adjacent candidate edges. The coarse graph table module assigns a coarse graph edge index to each retained candidate edge; the coarse graph edge index is bound to the stage number, inference node number, and stage execution time. When the stage records corresponding to two candidate edges are adjacent in the stage dependency chain, the coarse graph edge index of the subsequent candidate edge is registered in the preceding candidate edge, and the coarse graph edge index of the preceding candidate edge is registered in the subsequent candidate edge.

[0025] In this embodiment, the operation of creating detailed charts includes: The KV cache page is used as the object of detailed graph tabulation. The current memory page number group corresponding to the KV cache page is located in the memory page occupancy record of the candidate inference node. The detailed graph tabulation module reads the KV cache page number and the inference node number in the cache page measurement record, and retrieves the memory page with the same cache page number in the memory page occupancy record of the inference node. The retrieved memory pages are arranged in ascending order of number, and memory pages with consecutive numbers are merged into the same current memory page number group. When the same KV cache page corresponds to multiple non-contiguous number segments, multiple current memory page number groups are recorded respectively, and the non-contiguous cases corresponding to multiple number segments are written into the subsequent fragmentation rate calculation process. The system scans for free contiguous video memory segments in candidate inference nodes according to the number of pages required by the KV cache, and deletes candidate inference nodes for which no free contiguous video memory segments are found. The detailed graph tabulation module reads the page size of the KV cache and the capacity of a single video memory page, divides the page size by the capacity of a single video memory page, and rounds up to obtain the number of pages required by the KV cache. In the video memory page occupancy record of the candidate inference nodes, the system sequentially scans the free markers starting from the smallest free video memory page number and counts the number of consecutive free video memory pages. When the number of consecutive free video memory pages is less than the number of pages required by the KV cache, the system continues to scan the next free video memory segment. When all free video memory segments are less than the number of pages required by the KV cache, the corresponding candidate inference node is deleted. The scanned free contiguous video memory page segments are registered as target video memory page number groups, and a migration link is selected between the current video memory page number group and the target video memory page number group. When a video memory page segment with a number of consecutive free video memory pages not less than the number of pages required by the KV cache is scanned, the video memory page numbers in the video memory page segment with the same number of pages required by the KV cache are extracted and registered as target video memory page number groups. The detailed graph table module reads the available links between the inference node where the current video memory page number group is located and the inference node where the target video memory page number group is located, sorts them in ascending order of migration link queuing time, and selects the available link with the first ranking as the migration link. The migration time is obtained based on the page size of the KV cache page, the migration link bandwidth, and the migration link queuing time. The recalculation time is obtained based on the fixed token window length, the number of model layers, and the node inference throughput of the target inference node. The fragmentation rate is obtained based on the proportion of non-contiguous occupancy of the target memory page number group. The detailed graphing module divides the page size of the KV cache page by the migration link bandwidth to obtain the transmission duration, and then adds the transmission duration to the migration link queuing time to obtain the migration time. The detailed graphing module multiplies the fixed token window length by the number of model layers to obtain the recalculation computation amount, and then divides the recalculation computation amount by the node inference throughput of the target inference node to obtain the recalculation time. The detailed graphing module counts the number of occupied memory pages in the inference node where the target memory page number group is located, and counts the number of non-contiguous occupied memory pages in the adjacent area of ​​the target memory page number group. The fragmentation rate is obtained by dividing the number of non-contiguous occupied memory pages by the number of occupied memory pages. The current memory page number group, target memory page number group, migration link, migration time, recalculation time, and fragmentation rate corresponding to the same KV cache page are registered as cache page candidate edges, and a detailed graph edge table of cache pages is formed page by page. The detailed graph table module creates a cache page candidate edge for each KV cache page and each undeleted candidate inference node; the cache page candidate edge is bound to the KV cache page number, current memory page number group, target memory page number group, migration link, migration time, recalculation time, and fragmentation rate; after all KV cache pages are traversed, all cache page candidate edges are sorted by KV cache page number and target inference node number to form a detailed graph edge table of cache pages.

[0026] In this embodiment, the operation of establishing and applying attention bias includes: The task phase coarse graph edge table is processed edge by edge to obtain the phase execution time, and the actual phase time of the previous round of backfilling is retrieved. The Graphormer bias deduction module traverses the task phase coarse graph edge table by coarse graph edge index and reads the phase execution time of each coarse graph edge. Then, in the previous round execution record provided by the multi-agent scheduling backfilling module, the actual phase time is retrieved by the same phase number and the same inference node number. If no previous round execution record is found, the actual phase time is registered according to the phase execution time. When the actual stage time exceeds the stage execution time, the excess is taken as the stage execution time deviation; when it does not exceed the time, the stage execution time deviation is recorded as zero. The Graphormer bias deduction module subtracts the stage execution time from the actual stage time to obtain the stage difference. When the stage difference is greater than zero, the stage difference is recorded as the stage execution time deviation; when the stage difference is not greater than zero, the stage execution time deviation is recorded as zero. The stage execution time deviation is converted into a coarse graph edge bias value according to the stage time normalization threshold, and then subtracted from the original attention score of the corresponding coarse graph edge to obtain the biased coarse graph edge attention score; the Graphormer bias deduction module divides the stage execution time deviation by the stage time normalization threshold to obtain the coarse graph edge bias value; then reads the original attention score output by Graphormer for the coarse graph edge, and subtracts the coarse graph edge bias value from the original attention score; when the difference is less than zero, the biased coarse graph edge attention score is recorded as zero; when the difference is not less than zero, the difference is recorded as the biased coarse graph edge attention score; The migration time, fragmentation rate, and cache page read / write counts are obtained edge by edge from the cache page fine graph edge table. The cache page read / write count is the sum of the number of reads and writes for the same KV cache page in the candidate scheduling actions. The actual migration time of the previous round of backfilling is retrieved. The Graphormer bias deduction module traverses the cache page fine graph edge table by fine graph edge index and reads the migration time and fragmentation rate of each fine graph edge. The cache page read / write count is obtained through the access record table, which records KV cache page read events and KV cache page write events one by one. The Graphormer bias deduction module counts the number of read events and write events corresponding to the same KV cache page number and adds them together as the cache page read / write count. Then, the actual migration time is retrieved from the previous round execution record by the same KV cache page number and the same migration link. If the previous round execution record is not found, the actual migration time is recorded as the migration time. When the actual migration time exceeds the migration time, the excess is taken as the migration time deviation; when it does not exceed the migration time, the migration time deviation is recorded as zero. The Graphormer bias deduction module subtracts the migration time from the actual migration time to obtain the migration difference. When the migration difference is greater than zero, the migration difference is recorded as the migration time deviation; when the migration difference is not greater than zero, the migration time deviation is recorded as zero. The migration time deviation, fragmentation rate, and cache page read / write count are converted according to their respective normalized thresholds, multiplied by preset bias weights, and then summed to obtain the fine graph edge bias value. The Graphormer bias deduction module divides the migration time deviation by the migration time normalization threshold to obtain the migration bias component; divides the fragmentation rate by the fragmentation rate normalization threshold to obtain the fragmentation bias component; divides the cache page read / write count by the read / write count normalization threshold to obtain the read / write bias component; multiplies the migration bias component by the migration time deviation weight, the fragmentation bias component by the fragmentation rate weight, and the read / write bias component by the read / write count weight, and then adds the three products to obtain the fine graph edge bias value. The bias value of the thin graph edge is subtracted from the original attention score of the corresponding thin graph edge to obtain the biased attention score of the thin graph edge. The Graphormer bias deduction module reads the original attention score output by Graphormer for the thin graph edge, subtracts the bias value of the thin graph edge from the original attention score; when the difference is less than zero, the biased attention score of the thin graph edge is recorded as zero; when the difference is not less than zero, the difference is recorded as the biased attention score of the thin graph edge.

[0027] In this embodiment, the operations for obtaining the candidate scheduling action score table include: Read the attention scores of the coarse and fine edges of the graph after biasing; the Graphormer bias deduction module reads the attention scores of the coarse edges of the graph after biasing by the coarse edge index, and reads the attention scores of the fine edges of the graph after biasing by the fine edge index; during the reading process, the stage number, inference node number, KV cache page number, current memory page number group, and target memory page number group are retained; According to the stage number in the coarse graph edge table of the task stage, pre-fill candidate edges, decoding candidate edges, and return candidate edges are registered as candidate node actions, and the corresponding biased coarse graph edge attention scores are registered as candidate node action scores. When the stage number corresponds to a pre-fill record, the candidate edge is registered as a pre-fill node action; when the stage number corresponds to a decoding record, the candidate edge is registered as a decoding node action; when the stage number corresponds to a return record, the candidate edge is registered as a return path action. The candidate node action score directly adopts the biased coarse graph edge attention score of the same candidate edge. For candidate edges for writing cached pages, read the candidate edges of cached pages corresponding to the same KV cached page number. When the current memory page number group is the same as the target memory page number group, register it as a cached page retention action. The Graphormer bias deduction module compares the current memory page number group and the target memory page number group. When the memory page numbers in the two number groups are completely consistent, register the candidate edges of cached pages corresponding to the same KV cached page as cached page retention actions, and register the cached page processing action score as the corresponding biased fine graph edge attention score. When the inference node corresponding to the current memory page number group is inconsistent with the inference node corresponding to the target memory page number group, and the migration time does not exceed the recalculation time, it is registered as a cache page migration action; the Graphormer bias deduction module reads the inference node number of the current memory page number group and the inference node number of the target memory page number group; when the two inference node numbers are different, and the migration time is less than or equal to the recalculation time, the cache page candidate edge corresponding to the same KV cache page is registered as a cache page migration action, and the cache page processing action score is registered as the corresponding biased fine graph edge attention score; When the inference node corresponding to the current memory page number group is inconsistent with the inference node corresponding to the target memory page number group, and the migration time exceeds the recalculation time, it is registered as a cache page recalculation action; when the inference node number of the current memory page number group is different from the inference node number of the target memory page number group, and the migration time is greater than the recalculation time, the cache page candidate edge corresponding to the same KV cache page is registered as a cache page recalculation action, and the cache page processing action score is registered as the corresponding biased fine graph edge attention score; The corresponding biased fine-image edge attention score is registered as the cache page processing action score; the cache page processing action score corresponds one-to-one with the cache page retention action, cache page migration action and cache page recalculation action, and the cache page processing action score is no longer added to the coarse-image edge attention score; The candidate node action score under the same inference node number is added to the cache page processing action score under the same KV cache page number to form the candidate scheduling action score, which is then written into the candidate scheduling action score table. The Graphormer bias deduction module uses the inference node number and KV cache page number as matching keys to retrieve the candidate node action score and the cache page processing action score under the same inference node number and KV cache page number; the two scores are added together and recorded as the candidate scheduling action score; each row in the candidate scheduling action score table records one candidate scheduling action, and registers the inference node number, stage number, KV cache page number, cache page processing action type, and candidate scheduling action score.

[0028] In this embodiment, the operation of deleting candidate scheduling actions that do not meet the verification conditions includes: Read the candidate scheduling action score table and action verification data. The action verification data includes the number of consecutive video memory pages, model version, cache page format, and estimated completion time. When the number of consecutive video memory pages is less than the number of video memory pages required for KV cache pages, the model version is inconsistent with the model version corresponding to the inference task, the cache page format is inconsistent with the format supported by the inference node, or the estimated completion time exceeds the latency threshold, delete the corresponding candidate scheduling action and retain the candidate scheduling actions that have not been deleted in the candidate scheduling action score table. The action closure module reads the candidate scheduling action score table line by line. For each candidate scheduling action, it reads the number of consecutive free memory pages in the target memory page number group and compares it with the number of pages required by the KV cache page. It reads the loaded model version of the inference node corresponding to the candidate scheduling action and compares it with the model version of the inference task. It reads the KV cache page format and compares it with the format supported by the inference node. It reads the stage execution time corresponding to the candidate node action and the processing time corresponding to the cache page processing action. The processing time of the cache page retention action is recorded as zero, the processing time of the cache page migration action is the migration time, and the processing time of the cache page recalculation action is the recalculation time. The stage execution time and the processing time are added together to obtain the expected completion time. If any of the above comparisons are not satisfied, the candidate scheduling action is marked as a deletion row. After all candidate scheduling actions have been verified, the deletion row is removed from the candidate scheduling action score table, and the candidate scheduling actions that are not marked as deletion are retained.

[0029] In this embodiment, the operations for performing the final scheduling action include: Each inference node is configured with an inference agent, which extracts the candidate scheduling actions that have not been deleted from the candidate scheduling action score table according to the inference node number; the multi-agent scheduling write-back module groups the candidate scheduling action score table according to the inference node number, and each group of candidate scheduling actions is assigned to the inference agent corresponding to the same inference node. Each inference agent selects the highest-scoring action from the candidate scheduling actions of the corresponding inference node and registers it as the final scheduling action; the inference agent reads the scores of the candidate scheduling actions in the corresponding group, sorts them from largest to smallest, and selects the candidate scheduling action ranked first; when two candidate scheduling actions have the same score, the candidate scheduling action with the earlier stage in the stage dependency chain is selected; when the stage order is the same, the candidate scheduling action with the shorter expected completion time is selected. When the final scheduling action is a cache page retention action, the location of the KV cache page in the video memory page remains unchanged; the multi-agent scheduling write-back module does not trigger migration link transmission, does not trigger recalculation operation, and registers the current video memory page number group as the video memory page number group after execution; When the final scheduling action is a cache page migration action, the KV cache page migration is performed according to the target memory page number group and the migration link; the multi-agent scheduling write-back module reads the KV cache page from the current memory page number group, transmits it to the target inference node via the migration link, and then writes it into the target memory page number group; after the migration is completed, the current memory page number group is released, and the target memory page number group is registered as the memory page number group after the execution. When the final scheduling action is cache page recalculation, the corresponding KV cache page is recalculated at the target inference node; the target inference node reads the input token and model parameters corresponding to the fixed token window, recalculates the key cache and value cache under the corresponding model layer number and attention head number, and writes the recalculated KV cache page into the target memory page number group; After executing the final scheduling action, record the action start time, action end time, actual migration time, number of decoded output tokens, decoding time, and fragmentation rate after execution. Record the difference between the action end time and action start time as the actual stage time, and record the number of decoded output tokens divided by the decoding time as the decoding throughput. The action start time is recorded when the inference agent issues the final scheduling action, and the action end time is recorded when cache page retention, cache page migration, or cache page recalculation is completed. The actual migration time is started when the migration link begins transmission and stopped when the migration link completes transmission. The fragmentation rate after execution is recalculated based on the non-contiguous occupancy ratio recorded in the memory page occupancy records after the action execution.

[0030] In this embodiment, the operation of writing back the record includes: The actual stage time is used as the actual stage time of the next round of coarse graph edges, and the actual migration time is used as the actual migration time of the next round of fine graph edges. The decoding throughput and fragmentation rate after execution are registered to the corresponding KV cache pages. The multi-agent scheduling and write-back module matches the next round of coarse graph edges and the next round of fine graph edges according to the inference node number, stage number, and KV cache page number. The actual stage time is written into the execution record of the matched coarse graph edge. The actual migration time is written into the migration record of the matched fine graph edge. The decoding throughput and fragmentation rate after execution are written into the measurement record of the matched KV cache page. The actual stage time, actual migration time, decoding throughput, and fragmentation rate after execution are used as inputs to the next round of the Graphormer bias deduction module. The next round of the Graphormer bias deduction module reads the actual stage time and compares it with the stage execution time to obtain a new stage execution time deviation; reads the actual migration time and compares it with the migration time to obtain a new migration time deviation; reads the fragmentation rate after execution and participates in the calculation of the new fine graph edge bias value. The reward record is generated based on the actual stage time, actual migration time, and decoding throughput. The action selection parameters of the inference agent are updated according to the reward record. The multi-agent scheduling write-back module registers the actual stage time as a time penalty item, the actual migration time as a migration penalty item, and the decoding throughput as a throughput gain item. When the throughput gain item increases and the time penalty item and migration penalty item decrease, the action selection parameters of the corresponding final scheduling action are increased. When the throughput gain item decreases or the time penalty item and migration penalty item increase, the action selection parameters of the corresponding final scheduling action are decreased. The updated action selection parameters are used for ranking when the inference agent selects the highest-scoring action in the next round.

[0031] Example 1: To verify the feasibility of this invention in practice, it was applied to a multi-node large-model collaborative inference service scenario. The scenario involves 12 inference nodes with varying GPU memory capacity, remaining cache page space, model loading status, and link bandwidth. Issues also exist between different inference nodes, including cache page migration waiting, insufficient contiguous GPU memory space, and decoding throughput fluctuations. Inference requests originate from three business flows: text generation, retrieval question answering, and multi-turn conversations. The length of a single input ranges from 256 to 2048 tokens, and the output budget ranges from 64 to 512 tokens. Traditional scheduling methods select execution nodes based on node load and GPU memory availability. This can easily allocate long input requests to nodes with high KV cache page fragmentation and migrate decoding requests that require cache page reuse to nodes lacking contiguous GPU memory, resulting in cache page recalculation, cross-node transmission waiting, and queuing accumulation during the decoding phase.

[0032] During application, the stage delimitation module writes each inference request into four types of stage records: pre-filling, cache page writing, decoding, and postback. It also binds the input length, output budget, and latency threshold to the stage dependency chain. In the stage dependency chain, the pre-filling record handles input token processing, the cache page writing record handles KV cache page generation, the decoding record handles output token generation, and the postback record handles the return of the inference result. The KV cache page metering module divides key-value caches and value-value caches according to the model layer number, attention head number, and fixed token window, forming independently registerable KV cache pages. This allows cache pages generated from different layers, attention heads, and token ranges within the same request to participate in subsequent migration, retention, and recalculation decisions. The coarse graph tabulation module establishes candidate edges between stage records and version-matched inference nodes. Stage execution time is determined by stage resource availability and node stage processing capacity. Candidate edges exceeding the latency threshold are removed from the task stage coarse graph edge table.

[0033] The detailed graph table module locates the current memory page number group of the KV cache page in the memory page occupancy record, and then scans the free contiguous memory page segments among the candidate inference nodes according to the number of pages required by the KV cache page. Candidate inference nodes that do not find enough contiguous memory pages are removed from the detailed graph table scope; the scanned free contiguous memory page segments are registered as the target memory page number group. The detailed graph table module selects a migration link between the current memory page number group and the target memory page number group, and obtains the migration time, recalculation time and fragmentation rate based on page size, link bandwidth, link queuing time, fixed token window length, model layer number and node inference throughput, forming a cache page detailed graph edge table. For multi-round session requests, KV cache pages that already exist in the previous round will be preferentially included in the current memory page number group; for long input requests, the detailed graph table module will scan the target memory page number group page by page for multiple KV cache pages to avoid selecting nodes only based on memory availability and ignoring the problem of insufficient contiguous memory pages.

[0034] The Graphormer bias deduction module reads the coarse graph edge table and the fine graph edge table of cache pages for each task stage. Stage execution time deviations are converted into coarse graph edge bias values, while cache page fragmentation rate, cache page read / write counts, and migration time deviations are converted into fine graph edge bias values. After deducting the original attention score of the corresponding edge from the bias value, the system obtains a candidate scheduling action score table. The action closure module verifies the number of consecutive memory pages, model version, KV cache page format, and estimated completion time for each action, deleting unqualified candidate scheduling actions. Each inference node corresponds to one inference agent, which selects the highest-scoring action from the remaining actions and executes it. The execution record is written back to the next round of the Graphormer bias deduction module, forming a feedback update process for cache page migration, recalculation, and decoding throughput. This ensures that edges that experienced migration timeouts, increased fragmentation rates, or decreased decoding throughput in the previous round receive lower attention scores in the next round of scheduling.

[0035] The validation load consisted of 15,000 inference requests, with an average input length of 920 tokens and an average output budget of 256 tokens. The comparison methods were traditional load scheduling, a staged scheduling scheme, and the system of this invention. Traditional load scheduling selects nodes based on node load and available GPU memory; the staged scheduling scheme selects nodes separately for pre-filling and decoding stages; the system of this invention enables KV cache page metering, coarse-to-fine two-layer graphs, Graphormer bias deduction, action closure, and multi-agent scheduling write-back. The comparison data covers first-token latency, end-to-end latency, cache page hits, migration counts, recalculation counts, GPU memory fragmentation rate, GPU utilization, throughput, and timeout rate. In the validation results, short input requests primarily benefit from action closure's removal of faulty version nodes and nodes with insufficient GPU memory; long input requests primarily benefit from the fine-grained cache page graph's scanning of consecutive GPU memory page segments; and multi-turn session requests primarily benefit from Graphormer bias deduction's preservation of cache page reuse paths. During overall operation, the system of this invention maintains a low cache page recalculation ratio even under high load, reduces the number of timeouts for long output requests, decreases the number of cache page migrations between nodes, and exhibits lower decoding throughput fluctuations than the two comparison methods. The comparison results are shown in the table below: Table 1: Scheduling Performance Comparison Table

[0036] The data in the table shows that the system of this invention outperforms the two comparative schemes in terms of inference latency, cache page reuse, memory fragmentation control, and task stability. The average first token latency of the traditional load balancing scheme is 214.6ms, the staged scheduling scheme is 189.3ms, and the system of this invention reduces it to 151.8ms, a reduction of 29.3% compared to the traditional load balancing scheme; the average end-to-end latency is reduced from 1384.5ms to 928.6ms, a reduction of 32.9%; and the P95 end-to-end latency is reduced from 2478.2ms to 1597.3ms, a reduction of 35.5%. This indicates that the present invention reduces the probability of high-latency paths entering the final scheduling action by deducting points from high-time-consuming stage edges through the task stage coarse graph edge table and Graphormer bias deduction.

[0037] Regarding key-value (KV) cache processing, traditional load balancing generates 318 KV cache migrations and 142 KV cache recalculations per 1,000 requests, while the staged scheduling scheme generates 274 and 119 respectively. The system of this invention reduces these to 166 and 61 respectively, representing reductions of 47.8% and 57.0% compared to traditional load balancing. The cache page hit rate increases from 65.4% to 84.6%, an improvement of 19.2 percentage points. This result demonstrates that the fine-grained cache page graph side table can incorporate the current memory page number group, the target memory page number group, and the migration path into the scheduling score, making the selection of cache page retention, migration, and recalculation more closely aligned with the actual execution cost.

[0038] In terms of resource utilization and scheduling stability, the average memory page fragmentation rate decreased from 23.8% to 12.7%, the proportion of non-executable actions entering execution decreased from 6.7% to 0.6%, the average GPU utilization rate increased from 58.9% to 76.8%, the average decoding throughput increased from 1720 tokens per second to 2465 tokens per second, and the task timeout rate decreased from 8.4% to 2.2%. These results demonstrate that this invention improves the continuous stability of collaborative inference scheduling by proactively removing candidate actions with insufficient consecutive memory pages, inconsistent model versions, mismatched KV cache page formats, and exceeding the expected completion time limit through an action closure module, and by using multi-agent scheduling write-back to send the actual stage time, actual migration time, decoding throughput, and fragmentation rate into the next round of bias deduction process.

[0039] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A cooperative reasoning task scheduling system based on reinforcement learning and multiple agents, characterized in that, include: The phase delimitation module is used to create a four-line phase record table according to the order of pre-filling, cache page writing, decoding, and return; The KV cache page metering module is used to divide KV cache pages according to fixed cache page rules and establish cache page metering records. The coarse graph tabulation module is used to pair each stage row in the stage record table with the inference nodes that meet the model version and latency threshold, and to register the stage number, node number and stage execution time in the task stage coarse graph side table. The detailed graph table module is used to pair KV cache pages with current video memory pages, target video memory pages, and migration links one by one, and register the cache page number, page location, migration time, recalculation time, and fragmentation rate in the cache page detailed graph side table; The Graphormer bias deduction module is used to establish bias values ​​for coarse and fine graph edges in the Graphormer attention matrix. It includes the stage execution time deviation in the coarse graph edge bias value, and includes the cache page fragmentation rate, cache page read / write count, and migration time deviation in the fine graph edge bias value. It deducts the attention score of the corresponding edge according to the bias value to obtain the candidate scheduling action score table. The action closure module is used to verify the consecutive memory pages, model version, cache page format and expected completion time of each candidate scheduling action, and delete candidate scheduling actions that do not meet the verification conditions. The multi-agent scheduling and write-back module is used to configure an inference agent for each inference node. The inference agent executes the action with the highest remaining score and fills back the actual stage time, actual migration time, decoding throughput and fragmentation rate to the Graphormer bias deduction module.

2. The cooperative reasoning task scheduling system based on reinforcement learning multi-agent according to claim 1, characterized in that, The operation of creating the four-line stage record table includes: Generate a stage dependency chain according to the execution order of pre-filling, cache page writing, decoding, and postback; In the stage dependency chain, the pre-filled record is pointed to the cache page write record, the cache page write record is pointed to the decode record, and the decode record is pointed to the return record; The input length is associated with the pre-filled record, the output budget is associated with the decoded record, the latency threshold is associated with the stage dependency chain, and the stage dependency chain is sent to the coarse graph tabulation module.

3. The cooperative reasoning task scheduling system based on reinforcement learning multi-agent according to claim 2, characterized in that, The operation of partitioning KV cache pages includes: Determine the page boundaries of the KV cache according to the fixed cache page rules, and merge the key cache and value cache within the same page boundary into one KV cache page; Establish a cache page metering record for each KV cache page. The cache page metering record is used to mark the page occupancy status of the KV cache page in the inference node. The cached page metering records are sent to the detailed charting module, which then adds migration time, recalculation time, and fragmentation rate based on the cached page metering records.

4. A cooperative reasoning task scheduling system based on reinforcement learning multi-agent according to claim 3, characterized in that, The operations for creating the rough chart include: Using stage records in the stage dependency chain as edge sources and inference nodes as edge ends, inference nodes with inconsistent model versions are removed. For the remaining inference nodes, calculate the stage execution time by recording the corresponding stage resource quantity and the stage processing capacity of the inference node for each stage. Candidate edges whose execution time exceeds the latency threshold during the deletion phase; According to the order of the stage dependency chain, the pre-filled candidate edges, cache page writing candidate edges, decoding candidate edges, and return candidate edges are registered in the task stage coarse graph edge table in sequence, and the preceding edge number and the following edge number are registered between adjacent candidate edges.

5. A cooperative reasoning task scheduling system based on reinforcement learning multi-agent according to claim 4, characterized in that, The detailed chart creation operation includes: Use the KV cache page as the object of detailed graph tabulation, and locate the current video memory page number group corresponding to the KV cache page in the video memory page occupancy record of the candidate inference node; Scan the free contiguous video memory segments in the candidate inference nodes according to the number of pages required by the KV cache, and delete the candidate inference nodes for which no free contiguous video memory segments were scanned; Register the scanned free contiguous video memory page segments as the target video memory page number group, and select a migration path between the current video memory page number group and the target video memory page number group; The migration time is obtained based on the page size of the KV cache page, the migration link bandwidth, and the migration link queuing time. The recalculation time is obtained based on the fixed token window length, the number of model layers, and the node inference throughput of the target inference node. The fragmentation rate is obtained based on the non-contiguous occupancy ratio of the target memory page number group. Register the current memory page number group, target memory page number group, migration link, migration time, recalculation time and fragmentation rate corresponding to the same KV cache page as cache page candidate edges, and form a cache page detailed graph edge table page by page.

6. A cooperative reasoning task scheduling system based on reinforcement learning multi-agent according to claim 5, characterized in that, The operation of establishing and applying attention bias includes: Obtain the stage execution time edge by edge of the rough map edge table for each task stage, and retrieve the actual stage time consumed in the previous round of backfilling. When the actual time taken for a stage exceeds the stage execution time, the excess is taken as the stage execution time deviation; when it does not exceed the time taken, the stage execution time deviation is recorded as zero. The stage execution time deviation is converted into a coarse graph edge bias value according to the stage time normalization threshold, and then subtracted from the original attention score of the corresponding coarse graph edge to obtain the biased coarse graph edge attention score. For each edge of the cache page detailed graph edge table, obtain the migration time, fragmentation rate, and cache page read / write count. The cache page read / write count is the sum of the number of reads and writes of the same KV cache page in the candidate scheduling action. Retrieve the actual migration time of the previous round of backfilling. When the actual migration time exceeds the migration time, the excess is taken as the migration time deviation; when it does not exceed the migration time, the migration time deviation is recorded as zero. The migration time deviation, fragmentation rate, and cache page read / write count are converted according to their respective normalization thresholds, multiplied by preset bias weights, and then summed to obtain the fine graph edge bias value. The biased attention score of the corresponding thin edge is obtained by subtracting the bias value of the thin edge from the original attention score.

7. A cooperative reasoning task scheduling system based on reinforcement learning multi-agent according to claim 6, characterized in that, The operation of obtaining the candidate scheduling action score table includes: Read the edge attention scores of the coarse image after bias and the edge attention scores of the fine image after bias; According to the stage number in the coarse graph edge table of the task stage, the pre-filled candidate edge, the decoding candidate edge, and the return candidate edge are registered as candidate node actions, and the corresponding biased coarse graph edge attention score is registered as the candidate node action score. For cache page write candidate edges, read the cache page candidate edges corresponding to the same KV cache page number. When the current video memory page number group is the same as the target video memory page number group, register it as a cache page reservation action. When the inference node corresponding to the current memory page number group is inconsistent with the inference node corresponding to the target memory page number group, and the migration time does not exceed the recalculation time, it is registered as a cache page migration action. When the inference node corresponding to the current memory page number group is inconsistent with the inference node corresponding to the target memory page number group, and the migration time exceeds the recalculation time, it is registered as a cache page recalculation action. Register the corresponding biased fine-image edge attention score as the cache page processing action score; The candidate node action score under the same inference node number is added to the cache page processing action score under the same KV cache page number, and the result is written into the candidate scheduling action score table.

8. A cooperative reasoning task scheduling system based on reinforcement learning multi-agent according to claim 7, characterized in that, The operation of deleting candidate scheduling actions that do not meet the verification conditions includes: Read the candidate scheduling action score table and action verification data. The action verification data includes the number of consecutive video memory pages, model version, cache page format, and estimated completion time. When the number of consecutive video memory pages is less than the number of video memory pages required for KV cache pages, the model version is inconsistent with the model version corresponding to the inference task, the cache page format is inconsistent with the format supported by the inference node, or the estimated completion time exceeds the latency threshold, delete the corresponding candidate scheduling action and retain the candidate scheduling actions that have not been deleted in the candidate scheduling action score table.

9. A cooperative reasoning task scheduling system based on reinforcement learning multi-agent according to claim 8, characterized in that, The operations for performing the final scheduling action include: Configure an inference agent for each inference node, and extract the candidate scheduling actions that have not been deleted from the candidate scheduling action score table according to the inference node number; Each inference agent selects the highest-scoring action from the candidate scheduling actions of the corresponding inference node and registers it as the final scheduling action; When the final scheduling action is a cache page retention action, the location of the KV cache page in the video memory page remains unchanged; When the final scheduling action is a cache page migration action, the KV cache page migration is performed according to the target memory page number group and migration link; When the final scheduling action is a cache page recalculation action, the corresponding KV cache page is recalculated at the target inference node; After executing the final scheduling action, record the action start time, action end time, actual migration time, number of decoded output tokens, decoding time, and fragmentation rate after execution. Record the difference between the action end time and the action start time as the actual stage time, and record the number of decoded output tokens divided by the decoding time as the decoding throughput.

10. A cooperative reasoning task scheduling system based on reinforcement learning multi-agent according to claim 9, characterized in that, The operation of writing back the execution record includes: The actual stage time is used as the actual stage time of the next round of coarse graph edges, and the actual migration time is used as the actual migration time of the next round of fine graph edges. The decoding throughput and the fragmentation rate after execution are registered to the corresponding KV cache pages. The actual stage time, actual migration time, decoding throughput, and fragmentation rate after execution are used as inputs to the next round of the Graphormer bias deduction module. Reward records are generated based on the actual stage time, actual migration time, and decoding throughput. The action selection parameters of the inference agent are then updated based on these reward records.

Images