A token-based user demand matching and scheduling method for multi-source API resources

By deconstructing user requests to generate dynamic mask prefix cache hashes, matching primary API nodes and monitoring token output rates, and reconstructing the Prompt after disconnection to match backup API nodes, the problem of inaccurate backup node capability assessment in existing technologies is solved, and efficient scheduling of complex business instructions is achieved.

CN123115173APending Publication Date: 2026-11-17FUJIAN ZHUGE INTELLIGENT COMPUTER TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611350443.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-02
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing technologies in LLM distributed scheduling rely on hardware and physical load metrics to select backup API nodes, which cannot quantitatively assess their actual ability to parse and follow complex business instructions, resulting in a mismatch between the model's understanding capabilities and business requirements.

Method used

By receiving user requests and deconstructing them into a high-entropy intent kernel and a highly redundant background context, a dynamic mask prefix cache hash is generated to match the primary API node. A streaming channel is established and the token output rate is monitored. After disconnection, the capability gene characteristics of the backup API node are read, the manifold transformation function is called to reconstruct the Prompt, and the token cost is solved by combining Markov decision-making to achieve accurate matching and reconnection scheduling.

Benefits of technology

It enables precise evaluation of the key capabilities of backup API nodes, solves the problem that backup nodes cannot accurately parse complex nested instructions due to capability differences, and reduces the secondary inference overhead and response latency of disconnection retries.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123115173A_ABST
    Figure CN123115173A_ABST
Patent Text Reader

Abstract

This invention relates to the field of distributed scheduling technology for large language models, and particularly to a token user demand matching and scheduling method for multi-source API resources. The method includes: deconstructing user requests into a high-entropy intent kernel and a highly redundant background context; generating a dynamic mask prefix cache hash and probing to match the primary API node; pre-freezing the TPM quota of the primary node in the database and establishing a streaming channel; calculating the discrete gradient signal of the token output rate (TPS) by window; actively disconnecting when the slow output condition is met; unfreezing the remaining quota and converting the received tokens into incremental KV states; reading the capability gene characteristics of the backup API node and reconstructing the Prompt by calling the manifold transformation function; solving the Markov expected token cost of the backup node based on the capability gene characteristics; when the cost is lower than the estimated cost of the primary node, concatenating the incremental KV state and the reconstructed Prompt and distributing it to the backup node for continued scheduling.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of distributed scheduling technology for large language models, and in particular to a token user demand matching scheduling method for multi-source API resources. Background Technology

[0002] With the rapid development of Large Language Model (LLM) technology, calling distributed inference nodes via Application Programming Interfaces (APIs) to process complex business instructions has become the mainstream architecture. To ensure high system availability, when the primary API node experiences network disconnection or service failure, the scheduling system typically needs to degrade the current task and switch it to a backup API node for continued execution. Current technologies, when switching nodes, usually select from a candidate list and switch to a backup API node based on hardware and physical load indicators such as current memory usage, network bandwidth, or response latency. The original prompt and context information are then directly resent to the backup node for inference execution.

[0003] However, the aforementioned existing technologies rely solely on hardware and physical load metrics to screen candidate nodes, neglecting the inherent differences between different API nodes in terms of LLM underlying architecture, training data, and inference capabilities. This results in the scheduling system being unable to quantitatively assess the actual parsing and following capabilities of backup API nodes for complex business instructions, which can easily lead to a mismatch between model understanding capabilities and business requirements. Summary of the Invention

[0004] To overcome the above shortcomings, this invention provides a token user demand matching and scheduling method for multi-source API resources. It aims to improve the scheduling system's inability to quantitatively assess the actual parsing and compliance capabilities of backup API nodes with complex business instructions, which easily leads to a mismatch between model understanding capabilities and business requirements.

[0005] This invention provides the following technical solution: a token user demand matching and scheduling method for multi-source API resources, comprising:

[0006] S1. Receive user requests and deconstruct them into a high-entropy intent core and a high-redundancy background context. Generate a dynamic mask prefix cache hash for the high-redundancy background context based on the high-entropy intent core. Use a probe to match API nodes that have preheated the dynamic mask prefix cache hash as the primary API node.

[0007] S2. Based on the estimated token consumption, pre-freeze the TPM quota of the primary API node in the database, and establish a streaming channel between the scheduling server and the primary API node.

[0008] S3. Calculate the discrete gradient signal of the token output rate TPS in the streaming channel according to the preset window. When the discrete gradient signal meets the preset slow output condition, actively disconnect the connection, send an instruction to the database to unfreeze the remaining TPM quota according to the actual number of received tokens, and convert the received tokens into incremental KV state.

[0009] S4. When the connection is lost, determine the backup API node, read the capability gene characteristics of the backup API node, call the manifold transformation function to reconstruct the original Prompt, and generate a reconstructed Prompt that matches the backup API node.

[0010] S5. Based on the capability gene characteristics, calculate the Markov expected token cost of the backup API node. When the Markov expected token cost is less than the estimated cost of the primary API node, concatenate the incremental KV state with the reconstruction Prompt and distribute it to the backup API node for continued scheduling.

[0011] The present invention has the following beneficial effects:

[0012] 1. In this invention, by retrieving and parsing the capability feature vectors of backup API nodes from the capability feature database, a multi-dimensional capability gene feature including JSONS chema compliance, multi-constraint compliance rate, spatial inference coefficient, and extremely long context decay rate is constructed. This enables precise and quantitative evaluation of the capability performance of backup API nodes in key dimensions such as format compliance, instruction execution, and long text attention. It solves the problem of mismatch between model understanding capability and business requirements caused by the existing technology of selecting backup nodes based on a single indicator such as hardware load or throughput.

[0013] 2. In this invention, by calling the manifold transformation function and combining it with the prompt word mapping matrix, the nested structure in the original Prompt is deconstructed into a single-line chain instruction structure. The context constraint density is adaptively tiled according to the extremely long context decay rate, which realizes the accurate adaptive matching between the Prompt structure and the inference parsing capability of the backup API node. This solves the problem that the backup node cannot accurately parse complex nested instructions due to capability differences in the downgraded scenario, and is prone to format derailment and semantic decay.

[0014] 3. In this invention, by constructing a Markov decision state transition matrix that includes normal generation state, retry correction state, and final state transition path, the Markov expected token cost in closed-form analytical solution is solved. Combined with incremental KV state and reconstructed Prompt splicing distribution, dynamic precise calculation of discontinuous scheduling cost and lossless continuation of historical context calculation state are realized. This solves the problems of large secondary inference overhead of backup nodes, high response latency, and uncontrollable token cost caused by blind degradation in traditional network disconnection retry mechanism. Attached Figure Description

[0015] Figure 1 This is a flowchart of a token user demand matching and scheduling method for multi-source API resources proposed in this invention;

[0016] Figure 2 This is a flowchart of the high-entropy intent deconstruction and primary node probe matching process for a token user demand matching and scheduling method for multi-source API resources proposed in this invention.

[0017] Figure 3 This is a flowchart illustrating the slow release monitoring, dynamic unfreezing, and incremental KV state transformation of a token user demand matching and scheduling method for multi-source API resources proposed in this invention.

[0018] Figure 4 This is a flowchart of the Prompt manifold structure reconstruction process for a token user demand matching and scheduling method for multi-source API resources proposed in this invention.

[0019] Figure 5 This is a flowchart illustrating the Markov expected token cost calculation and continuation decision process for a token user demand matching and scheduling method for multi-source API resources proposed in this invention. Detailed Implementation

[0020] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Example 1:

[0022] In a first embodiment of the present invention, the present invention provides a token user demand matching and scheduling method for multi-source API resources, such as... Figures 1-5 As shown, it includes the following steps:

[0023] S1. Receive user requests and deconstruct them into a high-entropy intent core and a high-redundancy background context. Generate a dynamic mask prefix cache hash for the high-redundancy background context based on the high-entropy intent core. Use a probe to match API nodes that have preheated the dynamic mask prefix cache hash as the primary API node.

[0024] Furthermore, in S1, receiving a user request and deconstructing it into a high-entropy intent kernel and a highly redundant background context includes:

[0025] Extract the token sequence of the current Query and the historical Context from the user request, and construct a cross-sequence attention matrix;

[0026] Slide the extraction window along the token sequence of the current query, calculate the conditional information entropy of the tokens in the window with respect to the historical context, and generate an entropy increment gradient curve;

[0027] Locate the maximum node of the entropy increment gradient curve, and extract the token sequence corresponding to the maximum node as a high-entropy intent kernel;

[0028] Based on the high-entropy intent, reverse semantic masking is performed on the historical Context, and the unmasked context text is aggregated into a highly redundant background context.

[0029] Furthermore, in S1, generating a dynamic mask prefix cache hash based on the high-entropy intent kernel for the high-redundancy background context includes:

[0030] The semantic association position of the high-entropy intent kernel in the high-redundancy background context is mapped to a binary mask matrix;

[0031] The binary mask matrix is ​​used to perform dynamic position masking on the token sequence of the highly redundant background context to generate a masked background sequence;

[0032] Feature dimensionality reduction is performed on the masked background sequence to obtain a fixed-dimensional context feature vector;

[0033] The high-entropy intent kernel is concatenated with the context feature vector, and a dynamic mask prefix cache hash is generated by mapping through a hash function.

[0034] Furthermore, in S1, matching API nodes that have been preheated with the dynamic mask prefix cache hash via probes as primary API nodes includes:

[0035] The scheduling server sends probe request packets containing the dynamic mask prefix cache hash to the cache manager of each API node;

[0036] Parse the probe response packets returned by each API node and extract the video memory block page index table and LRU cache queue of each API node;

[0037] Based on the memory block page index table and the LRU cache queue, calculate the memory hit rate and cache residence time of the dynamic mask prefix cache hash at each API node;

[0038] The API node with the highest memory hit rate and whose cache residency time meets the preset replacement safety period is determined as the primary API node.

[0039] Specifically, in the token-based user demand matching and scheduling process for multi-source API (Application Programming Interface) resources, the scheduling server first receives user requests submitted by clients. These requests include the currently pending query text and the historical dialogue context text. The scheduling server then calls a tokenizer to perform word segmentation and word embedding vectorization on the query text and historical dialogue context text, mapping them to corresponding dimensional data sequences: the current query token sequence and the historical context token sequence. To accurately extract the user's core intent and remove background redundancy, the scheduling server inputs both the current query token sequence and the historical context token sequence into the attention calculation module. By using the current query token sequence as the query matrix and the historical context token sequence as the key and value matrices, cross-sequence dot product attention calculations are performed, thereby constructing a cross-sequence attention matrix.

[0040] Based on this, the scheduling server sets a fixed-length sliding extraction window along the current query tag sequence, for example, a sliding window size of 5 tags and a step size of 1 tag. For each sliding window, the scheduling server extracts the tag vector within the window and, combined with the attention distribution probability at the corresponding position in the cross-sequence attention matrix, calculates the conditional information entropy of the tags within that window to the historical context tag sequence. Conditional information entropy The calculation formula is as follows:

[0041] ;

[0042] in, This indicates the number of markers currently in the sliding window. Indicates the first in the sliding window One tag, Indicates the first The attention weights of each label are normalized to the historical context labels in the cross-sequence attention matrix.

[0043] As the sliding window moves along the current query tag sequence tag by tag, the scheduling server sequentially calculates the conditional information entropy value corresponding to each window, and performs continuous difference operations on these entropy values ​​according to the tag's position index to generate an entropy increment gradient curve describing the trend of entropy value changes. The scheduling server scans the extreme points of this entropy increment gradient curve, locating the extreme nodes in the curve where the gradient change rate changes from positive to negative and the value is greater than a preset critical value. The scheduling server extracts the tags corresponding to these extreme nodes and their adjacent consecutive tags, combining them to form a high-entropy intent kernel.

[0044] After generating the high-entropy intent kernel, the scheduling server uses the high-entropy intent kernel to perform reverse semantic masking processing on the historical context tag sequence. Specifically, it calculates the semantic similarity between each tag in the historical context tag sequence and the tags in the high-entropy intent kernel, and positions tags with semantic similarity greater than a preset association threshold as masked. For the unmasked text tags in the historical context tag sequence, the scheduling server concatenates and aggregates them according to their original physical order to construct a highly redundant background context.

[0045] To achieve efficient prefix cache matching, the scheduling server further generates a dynamic mask prefix cache hash based on the high-entropy intent kernel and the high-redundancy background context. First, the scheduling server scans the contextual association between the high-entropy intent kernel and the high-redundancy background context, converting the semantic association position information corresponding to the high-entropy intent kernel in the high-redundancy background context into bitwise Boolean values ​​and mapping it to generate a binary mask matrix. The dimension of this binary mask matrix is ​​exactly the same as the length of the tag sequence of the high-redundancy background context, with a mask value of 1 at associated positions and a mask value of 0 at non-associated positions. The scheduling server performs a bitwise logical AND operation between this binary mask matrix and the tag sequence of the high-redundancy background context, setting the tag vector corresponding to the position with a mask value of 0 to zero, thereby performing dynamic position masking and outputting the masked background sequence.

[0046] Subsequently, the scheduling server inputs the masked background sequence into the feature extraction module. Using a preset projection matrix, the module performs a linear transformation and principal component analysis (PCA) on the masked background sequence to obtain a fixed-dimensional context feature vector, such as 1024-dimensional vectors. Finally, the scheduling server concatenates the high-entropy intent kernel text data with the extracted context feature vectors at the byte level. The concatenated data stream is then input into a cryptographic hash function, such as the SHA-256 algorithm, for mapping calculation, ultimately generating a dynamic mask prefix cache hash representing the current user request context features.

[0047] After generating the dynamic mask prefix cache hash, the scheduling server uses a probe mechanism to match the best API node as the primary API node. The scheduling server encapsulates the generated dynamic mask prefix cache hash into a lightweight probe request packet and sends this packet concurrently over the network to the cache managers of each candidate API node in the cluster. Upon receiving the probe request packet, each API node's cache manager extracts the dynamic mask prefix cache hash from the packet and queries the block page index table and the least recently used cache queue in the local graphics processor's video memory. Each API node encapsulates the query results into a probe response packet and returns it to the scheduling server. The probe response packet contains information about the pre-warmed prefix cache blocks in the current node's video memory, the video memory block page index table mapping status, and the head and tail pointers of the least recently used cache queue.

[0048] The scheduling server parses the received probe response packets and extracts the video memory block page index table and the least recently used cache queue. Based on the video memory block page index table, the scheduling server calculates the ratio of the number of hit blocks in the corresponding API node's video memory to the total number of blocks, obtaining the video memory hit rate. Simultaneously, the scheduling server combines the location index and enqueue timestamp of the corresponding cache block in the least recently used cache queue to calculate the expected residence time of the dynamic mask prefix cache hash in video memory. The formula for calculating the expected length of stay is as follows:

[0049] ;

[0050] in, This represents the current system's auto-incrementing timestamp. This indicates the initial timestamp when the corresponding cache block entered the LRU queue. This represents the adaptive correction coefficient calculated based on the real-time memory replacement rate of the current node.

[0051] The scheduling server compares the calculated memory hit rate with a preset hit rate threshold, and also considers the cache residency time. The cache dwell time is compared with a preset replacement safety period (e.g., 300ms). The scheduling server selects candidate nodes whose cache dwell time is greater than the preset replacement safety period, and compares the memory hit rate among these candidate nodes. Finally, the API node with the highest memory hit rate and cache dwell time that meets the preset replacement safety period is determined as the primary API node for subsequent streaming channel establishment and request response.

[0052] S2. Based on the estimated token consumption, pre-freeze the TPM quota of the primary API node in the database, and establish a streaming channel between the scheduling server and the primary API node.

[0053] Furthermore, in S2, the pre-freezing of the primary API node's TPM quota in the database based on the estimated token consumption includes:

[0054] Extract the token length of the high-entropy intent kernel and the high-redundancy background context, and calculate the estimated total token consumption by combining the preset maximum generation length;

[0055] The scheduling server sends an atomic deduction instruction containing the primary API node identifier and the estimated total token consumption to the distributed database.

[0056] In the distributed database, a memory lock is temporarily applied to the minute token counter of the primary API node based on the atomic deduction instruction, generating a pre-frozen quota certificate;

[0057] Verify the pre-frozen quota certificate. If the verification is successful, confirm that the TPM quota pre-freezing of the primary API node is complete.

[0058] Specifically, after determining the primary API node, the scheduling server performs quota pre-freezing and streaming channel establishment operations. First, the scheduling server performs tag length statistics on the high-entropy intent core text and the high-redundancy background context text generated in the preceding steps to obtain the number of tags corresponding to the high-entropy intent core. Number of tags corresponding to highly redundant background context The scheduling server reads the maximum tag length that can be generated per request, which is pre-configured by the system. (For example, set to 2048), and calculate the estimated total token consumption for the current request. Estimated total token consumption The calculation formula is as follows:

[0059] ;

[0060] Calculate the estimated total amount of tokens consumed. Then, the scheduling server sends a quota pre-freeze request to the distributed database (such as a Redis cluster). The scheduling server constructs a system containing the unique identifier of the primary API node and the estimated total token consumption. And atomic deduction instructions with the current minute's timestamp window identifier (e.g., rounded down to the nearest 60 seconds according to UNIX timestamps). To prevent competition and overselling of TPM (Tokens Per Minute) quotas for the same API node by different scheduling nodes in high-concurrency scenarios, the distributed database ensures the atomicity of the deduction operation by executing pre-loaded Lua scripts.

[0061] After receiving the atomic deduction instruction, the distributed database locates the minute token counter corresponding to the primary API node in the current minute timestamp window. The Lua script first retrieves the remaining available quota of the token counter for the current minute. And determine the remaining available quota. Is it greater than or equal to the estimated total token consumption? When the deduction condition is met, the distributed database performs a memory lock operation on the counter to temporarily store the data, updating the available value of the counter to [the specified value]. and the deducted The value is written to the pre-frozen memory hash table corresponding to the primary API node. At the same time, the distributed database generates a pre-frozen quota certificate containing a globally unique serial number, the primary API node ID, the pre-frozen quota value, and a temporary expiration time (e.g., 60 seconds), and returns the certificate to the scheduling server.

[0062] After receiving the pre-frozen quota certificate from the distributed database, the scheduling server verifies the certificate's validity. The verification process includes comparing the serial number and digital signature in the certificate, confirming the pre-frozen quota value against the sent data. The verification is complete, and the temporary status of the credentials is checked to ensure they are active. If the credentials are successfully verified, the pre-freezing of the TPM quota on the primary API node is confirmed.

[0063] After the TPM quota pre-freezing is completed, the scheduling server establishes a long-lived streaming channel with the primary API node. Specifically, the scheduling server initiates a handshake request based on HTTP / 2 or Server-Sent Events (SSE) protocol to the network interface address of the primary API node, carrying the serial number of the pre-frozen quota credential and the reconstructed request data in the request header. After the primary API node's network module parses and verifies the validity of the credential, it establishes a bidirectional Transmission Control Protocol (TCP) connection and starts a streaming data transmission channel, preparing to send the generated results to the scheduling server in real time according to the marked blocks.

[0064] S3. Calculate the discrete gradient signal of the token output rate TPS in the streaming channel according to the preset window. When the discrete gradient signal meets the preset slow output condition, actively disconnect the connection, send an instruction to the database to unfreeze the remaining TPM quota according to the actual number of received tokens, and convert the received tokens into incremental KV state.

[0065] Furthermore, in S3, the discrete gradient signal of the token output rate (TPS) in the streaming channel is calculated according to a preset window. Actively disconnecting the connection when the discrete gradient signal meets a preset slow output condition includes:

[0066] Collect the timestamps of Server-SentEvents stream data packets in the streaming channel, count the number of tokens output within a unit window according to a preset sliding time window, and calculate the current TPS sequence;

[0067] Perform first-order and second-order difference operations on the current TPS sequence to generate a discrete gradient signal;

[0068] When the discrete gradient signal remains negative for a preset number of windows and the absolute value of the current TPS sequence is lower than the lower limit inflection point of the reference output rate, it is determined that the preset slow output condition is met.

[0069] The scheduling server sends a TCPFIN handshake packet or an HTTP / 2RST_STREAM frame to the streaming channel to actively disconnect the network connection with the primary API node.

[0070] Furthermore, in S3, converting received tokens into incremental key-value states includes:

[0071] Extract the valid token text received in the streaming channel before the connection was disconnected, and map it to generate the corresponding TokenID sequence;

[0072] Calculate the physical block offset and sequence termination position of the TokenID sequence in the original Prompt concatenation sequence;

[0073] Extract the Key and Value vectors of the TokenID sequence in the last Transformer attention mechanism and combine them to construct an incremental KV state data structure with the physical Block offset;

[0074] The incremental KV state data structure is written into the shared memory pool of the scheduling system to generate the corresponding incremental state handle.

[0075] Specifically, after establishing a long-lived streaming channel with the primary API node, the scheduling server monitors and manages the streaming process in real time. The scheduling server's network monitoring module is mounted on the streaming channel, capturing data frames pushed by the primary API node via the Server-Sent Events (SSE) protocol in real time, and extracting the system's high-precision millisecond-level timestamp for each data frame's arrival. The scheduling server sets a fixed-length sliding time window. (For example, setting the window size to 1000ms and the sliding step size to 200ms), statistics are calculated within the current sliding time window. Total number of legitimate tokens received internally The scheduling server calculates the real-time token output rate within the current window. The calculation formula is as follows:

[0076] ;

[0077] The scheduling server calculates the results from multiple consecutive sliding windows. Arrange the tokens in chronological order to construct the current TPS sequence. To accurately capture the decay trend of the token output rate and eliminate the interference of instantaneous network jitter, the scheduling server performs first-order and second-order difference operations on the current TPS sequence to calculate the discrete gradient signal. First-order difference With second-order difference The calculation formula is as follows:

[0078] ;

[0079] ;

[0080] Discrete gradient signal It consists of a combination of first-order and second-order differences, i.e. The scheduling server will discrete gradient signals. The algorithm is matched against preset slow-release conditions. These conditions include determining if the gradient is continuously negative and determining the rate lower bound inflection point. The scheduling server checks the first-order difference. Whether to continuously within a preset number of windows (For example The value remains negative within the range, while simultaneously determining the current... Is the absolute value lower than the preset benchmark output rate lower limit inflection point? (For example, set to) When the discrete gradient signal is continuous The window is negative and At that time, the scheduling server determines that the current streaming channel meets the preset slow output conditions.

[0081] Once the preset slow-release condition is met, the transport layer control module of the scheduling server immediately triggers the active disconnection mechanism. If the current streaming channel is based on the TCP / HTTP protocol, the scheduling server sends a TCP FIN handshake packet to the network socket of the primary API node; if the current streaming channel is based on the HTTP / 2 protocol, the scheduling server sends an HTTP / 2RST_STREAM frame with a Cancel error code to the corresponding data stream channel, forcibly terminating the network connection with the primary API node and releasing the socket resources.

[0082] After the connection is broken, the scheduling server performs TPM quota settlement and unfreezing of remaining quotas. The scheduling server calculates the total number of valid tokens successfully received in the streaming channel before the connection is broken. The scheduling server extracts the estimated total token consumption corresponding to the pre-frozen quota vouchers from the aforementioned steps. Calculate the unconsumed quota difference. The scheduling server sends a message containing the credential serial number and the quota difference to the distributed database. The unfreezing command, after verifying the serial number in the distributed database, adds back the quota frozen in the current minute counter of the primary API node. Simultaneously unfreeze the memory lock and destroy the pre-frozen quota certificate.

[0083] Meanwhile, the scheduling server converts the tokens received before the disconnection into incremental KV (Key-Value) states for subsequent reconnection scheduling. The scheduling server extracts the valid token text stream received before the disconnection, calls a token segmenter to perform reverse token ID mapping, and generates the corresponding token ID sequence. ,in This indicates the actual number of tokens received. The scheduling server obtains the length of the concatenated sequence of the Prompt (input prompt text) from the original request. ,Will Determine the physical block (Block Memory Division Unit) offset of the current Token ID sequence within the model context. And calculate the sequence termination position. .

[0084] The scheduling server extracts the key vector matrix from the last layer of the Transformer attention mechanism, which was pushed by the primary API node before disconnection via extended response headers or out-of-band cache synchronization channels. With Value vector matrix The scheduling server will use the physical block offset. Sequence termination position Key vector matrix K and Value vector matrix The data is encapsulated and combined to construct an incremental KV state data structure with a physical block offset. Finally, the scheduling server writes this incremental KV state data structure into the shared memory pool of the scheduling system (such as a contiguous memory block based on POSIX shared memory) and registers it in the handle mapping table of the shared memory block, generating a globally unique incremental state handle for subsequent KV cache state loading and seamless continuation of backup API nodes.

[0085] S4. When the connection is lost, determine the backup API node, read the capability gene characteristics of the backup API node, call the manifold transformation function to reconstruct the original Prompt, and generate a reconstructed Prompt that matches the backup API node.

[0086] Furthermore, in S4, reading the capability gene characteristics of the backup API node includes:

[0087] The scheduling server initiates a query request to the capability feature database of the backup API node;

[0088] Retrieve the feature vector corresponding to the backup API node from the capability feature database;

[0089] The feature vector is parsed, and multi-dimensional numerical indicators including JSONSchema compliance, multi-constraint compliance rate, spatial inference coefficient, and extremely long context decay rate are extracted and combined to construct capability gene features.

[0090] Furthermore, in S4, the manifold transformation function is called to restructure the original Prompt, generating a restructured Prompt that matches the backup API node, including:

[0091] Based on the JSONSchema compliance degree and multi-constraint compliance rate in the capability gene characteristics, the prompt word mapping matrix of the backup API node is calculated;

[0092] The original Prompt is input into the manifold transformation function, and the nested structure in the original Prompt is deconstructed into a chained instruction structure through the prompt word mapping matrix;

[0093] According to the format specifications supported by the backup API node, the deconstructed chain instruction structure is subjected to syntax formatting and adaptive tiling of context constraint density to generate a reconstructed Prompt.

[0094] Specifically, after actively disconnecting the network connection with the primary API node, the scheduling server triggers a degradation and continuation mechanism to determine a backup API node and restructure its prompt. First, the scheduling server selects the API node with the highest memory hit rate and no overload from the candidate node list as the backup API node (e.g., node ID Node-B).

[0095] After identifying a backup API node, the scheduling server reads the capability genetic characteristics of that backup API node. Specifically, the scheduling server sends an SQL (Structured Query Language) query request or a Key-Value retrieval instruction containing the unique identifier Node-B of the backup API node to the system's built-in API node capability characteristic database. The capability characteristic database then retrieves and returns a multi-dimensional capability feature vector corresponding to the backup API node. The scheduling server performs field parsing on the feature vector and extracts the following multidimensional numerical indicators:

[0096] JSON Schema (JavaScript Object Notation Schema) compliance : Indicates the probability that a node's output conforms to complex JSON structure constraints;

[0097] Multi-constraint compliance rate : Indicates the success rate of instruction execution for a node under the constraint of multiple concurrent instructions;

[0098] Spatial inference coefficients : A multi-dimensional evaluation score representing the node's ability to process charts, spatial layouts, and complex logical deductions;

[0099] Extremely long context decay rate : This represents the attention decay coefficient of the node attention mechanism as the length of the input context token increases.

[0100] The scheduling server will extract the values and Multidimensional feature combination is performed to construct a capability gene feature vector for backup API nodes. .

[0101] After acquiring the capability gene features, the scheduling server calls the manifold transformation function to restructure the original Prompt, generating a restructured Prompt that highly matches the understanding capabilities of the backup API nodes. First, the scheduling server extracts the JSON Schema compliance from the capability gene features. Compliance rate with multiple constraints Solve the prompt word mapping matrix The formula for solving the mapping matrix is ​​as follows:

[0102] ;

[0103] in, For size The preset linear transformation weight matrix (e.g.) ), For size bias vector (e.g.) ), For the activation function, this embodiment uses the Sigmoid (S-type activation function):

[0104] ;

[0105] Solve Dimensional prompt word mapping matrix Subsequently, the numerical elements in the matrix are used as control parameters to determine the "nesting depth reduction threshold" and "instruction chain splitting granularity" in the subsequent prompt word deconstruction.

[0106] Next, the scheduling server inputs the original Prompt into the manifold transformation function. The manifold transformation function first parses the AST (Abstract Syntax Tree) of the original Prompt, based on the prompt word mapping matrix. The determined control parameters are used to perform low-dimensional manifold projection on the multi-level nested JSON structure, complex conditional branch judgment blocks (such as if-else combinations), and tree-like concurrent instructions contained in the original Prompt. This flattens the multi-level nested relationship into a single-line chain instruction structure arranged in the order of execution time, namely the CoT (Chain-of-Thought) logical command flow.

[0107] Finally, the scheduling server obtains the prompt word format specification specified by the backup API node (e.g., containing specific System tags <|im_start|>system and User tags <|im_start|>user), and performs syntax formatting on the flattened chained instruction structure. Simultaneously, it considers the extremely long context decay rate. Adaptive tiling adjustment is performed on the context constraint density in the instruction: when When the value exceeds the preset attenuation threshold (e.g., 0.10), the scheduling server repeatedly inserts core constraint markers (such as output format emphasis statements) at preset intervals (e.g., every 500 tokens) at the beginning, middle, and end of the chained instructions, flattening the semantic constraint density to offset the attention attenuation under long text, and finally outputting a refactored Prompt that fully matches the reasoning and parsing capabilities of the backup API node.

[0108] S5. Based on the capability gene characteristics, calculate the Markov expected token cost of the backup API node. When the Markov expected token cost is less than the estimated cost of the primary API node, concatenate the incremental KV state with the reconstruction Prompt and distribute it to the backup API node for continued scheduling.

[0109] Furthermore, in S5, the Markov expected token cost of the backup API node, calculated based on the capability gene characteristics, includes:

[0110] Extract the multi-constraint compliance rate and extremely long context decay rate from the capability gene features, and use them to map and solve the single generation failure probability and retry transfer probability of the backup API node;

[0111] By combining the token unit price of the backup API node with the estimated total token consumption, a Markov decision state transition matrix is ​​constructed, which includes the normal generation state, the retry correction state, and the final state transition path.

[0112] Calculate the expected cumulative cost on each state transition path based on the Markov decision state transition matrix;

[0113] The Markov expected token cost is obtained by solving a closed equation for the expected cumulative consumption and the penalty cost generated by the retry correction state.

[0114] Specifically, after generating the reconstruction Prompt, the scheduling server performs cost assessment and continuation distribution operations. First, based on the capability characteristics of the backup API nodes, the scheduling server calculates their Markov expected token cost. .

[0115] The specific solution process is as follows: The scheduling server extracts the multi-constraint compliance rate from the capability gene features. With extremely long context decay rate Mapping to solve for the single generation failure probability of backup API nodes With retry transition probability Probability of failure in a single generation The mapping calculation formula is as follows:

[0116] ;

[0117] Retry transition probability This indicates the probability that the system will trigger automatic correction and retry after a single generation failure. In this embodiment, it is set to... .

[0118] Next, the scheduling server combines the token price of the backup API nodes. Compared with the estimated total token consumption Construct the Markov decision state transition matrix A Markov decision process consists of three core states: the normal generating state and the normal generating state. Retry Correction Status and termination status State transition matrix Represented as:

[0119] ;

[0120] The scheduling server is based on the Markov decision state transition matrix. Calculate the expected cumulative cost along each state transition path. Under normal generation state... Below, the base consumption cost per transaction is In retry correction state Because the introduction of a retry mechanism incurs additional overhead from context retransmission and prompt correction, a pre-defined retry penalty cost is incurred. For example, set to .

[0121] The scheduling server establishes and solves a closed equation relating the expected cumulative cost to the retry penalty cost to obtain the Markov expected token cost. The closed-form equation is expressed as follows:

[0122] ;

[0123] Solve the above closed equation to obtain The analytical solution formula is:

[0124] ;

[0125] Calculate the Markov expected token cost of the backup API node. Then, the scheduling server obtains the estimated cost of the primary API node. Its calculation formula is ,in The token price for the primary API node. The scheduling server determines this. Is it less than .

[0126] when At this point, the scheduling server confirms the economics and feasibility of continuing the scheduling. The scheduling server retrieves the incremental KV state data structure (containing physical block offsets) generated in the previous steps from the shared memory pool. Key vector matrix With Value vector matrix The refactoring Prompt is concatenated with the refactoring Prompt to construct a re-connection request data packet, which is then distributed to the backup API node for re-connection scheduling. The backup API node continues to stream-generate subsequent token text based on the loaded incremental KV state and the refactoring Prompt.

[0127] Example 2:

[0128] In distributed scheduling scenarios of Large Language Models (LLM), when the system loses network connection with the primary API node, existing technologies only select backup nodes based on physical hardware indicators such as memory usage or network latency. This fails to quantitatively assess the actual capabilities of backup nodes in instruction compliance, format constraints, and long text processing, resulting in selected backup nodes that cannot meet the actual parsing requirements of business instructions. To address this issue, this invention provides a token-based user demand matching and scheduling method for multi-source API resources, the steps of which are as follows: Figure 1 As shown. The specific implementation process of this method is as follows:

[0129] The scheduling server receives complex business requests from the client (such as long text analysis tasks requiring the output of JSON in a specific format), performs semantic deconstruction on the input text, and extracts high-entropy intent core text containing core format constraints and key instructions, as well as highly redundant background context text containing background references. The scheduling server retrieves the real-time GPU memory and network bandwidth load of each API node in the cluster and determines the API node with the highest remaining GPU memory and the lowest latency (e.g., node ID Node-A) as the primary API node.

[0130] The scheduling server counts the number of tags in high-entropy intent kernels. (e.g., 150 tokens) and the number of tokens in a highly redundant background context. (e.g., 1800 Tokens), combined with the system's preset maximum generation length (For example, 2048Tokens), calculate the estimated total token consumption:

[0131] ;

[0132] The scheduling server sends a message containing the node ID (Node-A) and... to a distributed database (such as a Redis cluster). The atomic deduction instruction (3998) is executed. The database executes the pre-loaded Lua script, locates the minute counter corresponding to the current minute timestamp window in Node-A, and determines whether the remaining available quota is greater than or equal to... Then, a memory lock is temporarily applied to the quota, the available quota is updated, and a pre-frozen quota credential with a globally unique serial number is returned (the temporary expiration time is set to 60 seconds). After the scheduling server verifies the validity of the credential, it establishes a streaming long connection channel with Node-A based on Server-SentEvents (SSE) through a handshake request based on the HTTP / 2 protocol.

[0133] During streaming, the scheduling server uses a 1000ms sliding window. The system counts the number of tokens received within a unit window in real time and calculates the real-time output rate. Perform first-order differencing continuously on the TPS sequence. With second-order difference The operation generates a discrete gradient signal. When the discrete gradient signal remains negative for five consecutive windows, and the current... The absolute value is lower than the set lower limit inflection point (e.g.) When the preset slow-output condition is met, the scheduling server immediately sends an HTTP / 2 RST_STREAM frame to the streaming channel to actively terminate the connection.

[0134] After the connection is closed, the scheduling server counts the number of valid tokens that were actually successfully received. (For example, 500 Tokens), calculate the unspent quota difference:

[0135] ;

[0136] The scheduling server sends an unfreezing command to the database, adds the quota of 3498 tokens back to Node-A's minute counter, and destroys the tokens. Simultaneously, the scheduling server reverse-maps the 500 token texts received before the disconnection into a token ID sequence using a token segmenter, and calculates their physical block offset within the original Prompt concatenation sequence. and the end position Extract the key vector matrix from the last Transformer attention mechanism. With Value vector matrix The data is combined and encapsulated into an incremental KV state data structure with physical block offsets, written to the shared memory pool, and a globally unique incremental state handle is generated.

[0137] The scheduling server triggers a fallback and connection continuation mechanism, selecting a backup API node (e.g., node ID Node-B). The scheduling server sends a query request to the API node capability feature database, retrieves and parses the feature vector of Node-B, and extracts and constructs a capability gene feature vector. Among them, JSON Schema compliance Multi-constraint compliance rate Spatial inference coefficients Extremely long context decay rate .

[0138] according to and Call the mapping formula:

[0139] ;

[0140] Calculate and solve the prompt word mapping matrix The original Prompt is input into a manifold transformation function, which parses the Abstract Syntax Tree (AST). Based on the mapping matrix, the multi-layered nested JSON structure and complex conditional branches in the original Prompt are deconstructed and flattened into a single-line chained instruction structure (CoT logical command flow). Syntax formatting is performed according to the format specifications supported by Node-B (such as the `<|im_start|>system` tag). Given... The scheduling server inserts core format constraint markers every 500 tokens in the chained instructions to complete the adaptive tiling of context constraint density and generate the refactored Prompt.

[0141] The scheduling server calculates the probability of a single generation failure for Node-B:

[0142] ;

[0143] Set retry transfer probability Combined with Node-B's token price Build a system containing the normal generation state. Retry Correction Status and final state Markov decision state transition matrix Set a baseline consumption cost. Retry penalty cost By solving the closed-form equation:

[0144] ;

[0145] Calculate the Markov expected token cost of Node-B A comparison of the scheduling servers revealed... Less than the estimated cost of the primary API node Node-A (calculated as) ).

[0146] Once the economic viability is confirmed, the scheduling server extracts the incremental key-value (KV) state data structure from the shared memory pool, concatenates it with the reconstructed Prompt to construct a continuation request data packet, and distributes it to the backup API node Node-B. Node-B directly loads the incremental KV state, eliminating the need for repeated calculations of historical context, and seamlessly streams subsequent text based on the reconstructed Prompt, completing highly reliable degraded scheduling.

[0147] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A Token user demand matching and scheduling method for multi-source API resources, characterized in that, include: S1. Receive user requests and deconstruct them into a high-entropy intent core and a high-redundancy background context. Generate a dynamic mask prefix cache hash for the high-redundancy background context based on the high-entropy intent core. Use a probe to match API nodes that have preheated the dynamic mask prefix cache hash as the primary API node. S2. Based on the estimated token consumption, pre-freeze the TPM quota of the primary API node in the database, and establish a streaming channel between the scheduling server and the primary API node. S3. Calculate the discrete gradient signal of the token output rate TPS in the streaming channel according to the preset window. When the discrete gradient signal meets the preset slow output condition, actively disconnect the connection, send an instruction to the database to unfreeze the remaining TPM quota according to the actual number of received tokens, and convert the received tokens into incremental KV state. S4. When the connection is lost, determine the backup API node, read the capability gene characteristics of the backup API node, call the manifold transformation function to reconstruct the original Prompt, and generate a reconstructed Prompt that matches the backup API node. S5. Based on the capability gene characteristics, calculate the Markov expected token cost of the backup API node. When the Markov expected token cost is less than the estimated cost of the primary API node, concatenate the incremental KV state with the reconstruction Prompt and distribute it to the backup API node for continued scheduling. 2.The Token user demand matching and scheduling method for multi-source API resources according to claim 1, characterized in that, In S1, receiving user requests and deconstructing them into a high-entropy intent kernel and a highly redundant background context includes: Extract the token sequence of the current Query and the historical Context from the user request, and construct a cross-sequence attention matrix; Slide the extraction window along the token sequence of the current query, calculate the conditional information entropy of the tokens in the window with respect to the historical context, and generate an entropy increment gradient curve; Locate the maximum node of the entropy increment gradient curve, and extract the token sequence corresponding to the maximum node as a high-entropy intent kernel; Based on the high-entropy intent, reverse semantic masking is performed on the historical Context, and the unmasked context text is aggregated into a highly redundant background context.

3. The Token user demand matching and scheduling method for multi-source API resources according to claim 1, characterized in that, In S1, generating a dynamic mask prefix cache hash based on the high-entropy intent kernel for the high-redundancy background context includes: The semantic association position of the high-entropy intent kernel in the high-redundancy background context is mapped to a binary mask matrix; The binary mask matrix is ​​used to perform dynamic position masking on the token sequence of the highly redundant background context to generate a masked background sequence; Feature dimensionality reduction is performed on the masked background sequence to obtain a fixed-dimensional context feature vector; The high-entropy intent kernel is concatenated with the context feature vector, and a dynamic mask prefix cache hash is generated by mapping through a hash function.

4. The Token user demand matching and scheduling method for multi-source API resources according to claim 1, characterized in that, In S1, matching API nodes that have been preheated with the dynamic mask prefix cache hash via probes as primary API nodes includes: The scheduling server sends probe request packets containing the dynamic mask prefix cache hash to the cache manager of each API node; Parse the probe response packets returned by each API node and extract the video memory block page index table and LRU cache queue of each API node; Based on the memory block page index table and the LRU cache queue, calculate the memory hit rate and cache residence time of the dynamic mask prefix cache hash on each API node; The API node with the highest memory hit rate and whose cache residency time meets the preset replacement safety period is determined as the primary API node.

5. The Token user demand matching and scheduling method for multi-source API resources according to claim 1, characterized in that, In S2, the TPM quota of the primary API node is pre-frozen in the database based on the estimated token consumption, including: Extract the token length of the high-entropy intent kernel and the high-redundancy background context, and calculate the estimated total token consumption by combining the preset maximum generation length; The scheduling server sends an atomic deduction instruction containing the primary API node identifier and the estimated total token consumption to the distributed database. In the distributed database, a memory lock is temporarily applied to the minute token counter of the primary API node based on the atomic deduction instruction, generating a pre-frozen quota certificate; Verify the pre-frozen quota certificate. If the verification is successful, confirm that the TPM quota pre-freezing of the primary API node is complete.

6. The Token user demand matching and scheduling method for multi-source API resources according to claim 1, characterized in that, In S3, the discrete gradient signal of the token output rate (TPS) in the streaming channel is calculated according to a preset window. When the discrete gradient signal meets the preset slow output condition, the connection is actively disconnected, including: Collect the timestamps of Server-SentEvents stream data packets in the streaming channel, count the number of tokens output within a unit window according to a preset sliding time window, and calculate the current TPS sequence; Perform first-order and second-order difference operations on the current TPS sequence to generate a discrete gradient signal; When the discrete gradient signal remains negative for a preset number of windows and the absolute value of the current TPS sequence is lower than the lower limit inflection point of the reference output rate, it is determined that the preset slow output condition is met. The scheduling server sends a TCPFIN handshake packet or an HTTP / 2RST_STREAM frame to the streaming channel to actively disconnect the network connection with the primary API node.

7. The Token user demand matching and scheduling method for multi-source API resources according to claim 1, characterized in that, In S3, converting received tokens into incremental key-value states includes: Extract the valid token text received in the streaming channel before the connection was disconnected, and map it to generate the corresponding TokenID sequence; Calculate the physical block offset and sequence termination position of the TokenID sequence in the original Prompt concatenation sequence; Extract the Key and Value vectors of the TokenID sequence in the last Transformer attention mechanism and combine them to construct an incremental KV state data structure with the physical Block offset; The incremental KV state data structure is written into the shared memory pool of the scheduling system to generate the corresponding incremental state handle.

8. The Token user demand matching and scheduling method for multi-source API resources according to claim 1, characterized in that, In S4, the ability to read the genetic characteristics of the backup API node includes: The scheduling server initiates a query request to the capability feature database of the backup API node; Retrieve the feature vector corresponding to the backup API node from the capability feature database; The feature vector is parsed, and multi-dimensional numerical indicators including JSONSchema compliance, multi-constraint compliance rate, spatial inference coefficient, and extremely long context decay rate are extracted and combined to construct capability gene features.

9. The Token user demand matching and scheduling method for multi-source API resources according to claim 8, characterized in that, In S4, the manifold transformation function is called to restructure the original Prompt, generating a restructured Prompt that matches the backup API node, including: Based on the JSONSchema compliance degree and multi-constraint compliance rate in the capability gene characteristics, the prompt word mapping matrix of the backup API node is calculated; The original Prompt is input into the manifold transformation function, and the nested structure in the original Prompt is deconstructed into a chained instruction structure through the prompt word mapping matrix; According to the format specifications supported by the backup API node, the deconstructed chain instruction structure is subjected to syntax formatting and adaptive tiling of context constraint density to generate a reconstructed Prompt.

10. The Token user demand matching and scheduling method for multi-source API resources according to claim 1, characterized in that, In S5, the Markov expected token cost of the backup API node is calculated based on the capability gene characteristics, including: Extract the multi-constraint compliance rate and extremely long context decay rate from the capability gene features, and use them to map and solve the single generation failure probability and retry transfer probability of the backup API node; By combining the token unit price of the backup API node with the estimated total token consumption, a Markov decision state transition matrix is ​​constructed, which includes the normal generation state, the retry correction state, and the final state transition path. Calculate the expected cumulative cost on each state transition path based on the Markov decision state transition matrix; The Markov expected token cost is obtained by solving a closed equation for the expected cumulative consumption and the penalty cost generated by the retry correction state.