A multi-type large model token convergence scheduling optimization matching method

Abstract

The application discloses a multi-type large model Token convergence scheduling optimization matching method. A reasoning session is established, Token sequences, maximum output boundaries and cache reuse permissions are read; according to Token sequences and effective cache block chains, prefix cache resident Tokens, Tokens to be pre-filled and reserved continuous decoding Tokens are registered; candidate instances are screened according to model versions, cache configurations, tenant isolation marks and instance states; in an observation period, a pre-filling maintenance period and a decoding reservation period, three types of Token resource occupation, resource safety margin, cache resident safety state and batch processing stability state are registered; when pre-filling commitment, decoding reservation, cache resident safety and batch processing stability all pass, the request is converged and issued; during continuous decoding, resource reservation and instance admission state are updated according to newly added decoding Tokens, bound cache block occupation and instance running records.

Description

Technical Field

[0001] This invention relates to the field of large model inference scheduling technology, specifically a method for aggregation, scheduling and optimal matching of tokens in multi-type large models. Background Technology

[0002] In a multi-model inference cluster, the inference gateway sends user requests to different types of large model instances for execution. After a request enters the model, the input content is typically pre-filled to form a key-value cache, and then the output content is generated token by token using a sequential decoding method. For requests with the same system prompts, search context, or tool call templates, some input tokens can reuse existing prefix caches; for requests that do not hit the prefix cache, pre-filling processing needs to be re-executed. Pre-filled tokens, sequential decoding tokens, and prefix cache-resident tokens correspond to different methods of consuming computing resources, GPU memory resources, and bandwidth resources.

[0003] Existing scheduling methods often determine the target model instance based on request token length, current queue length, GPU utilization, instance load, or prefix cache hit status. These methods typically use the resource status at the time of scheduling as the matching criterion, failing to synchronously reflect subsequent resource changes caused by the continuous growth of the key-value cache after a request enters the decoding phase, the continued generation of existing requests, the persistence of hotspots in the prefix cache, and deviations in the actual output length from the estimated length. When multiple requests are converged to the same cache-hit instance, the target instance may experience key-value cache crowding, batch splitting, cache eviction, or request migration during subsequent decoding, leading to increased per-token generation latency and difficulty in maintaining stable resource load among model instances. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method for the aggregation, scheduling, and optimal matching of multi-type large-scale model tokens, thereby resolving the problems mentioned in the background section.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a method for aggregation, scheduling, optimization, and matching of multi-type large-model tokens, comprising: S1. Establish an inference session and read the request type, model version constraints, token sequence, maximum output boundary, cache reuse permission, and instance execution record; S2. Based on the Token sequence and the continuous valid cache block chain, register the prefix cache resident Token, the Token to be prefilled, and the reserved continuous decoding Token; S3. Filter candidate model instances based on model version, tokenizer version, key-value cache configuration, tenant isolation flag, and instance service status; S4. For candidate model instances, register three types of token resource usage, resource security margin, cache residency security status, and batch processing stable status during the current observation period, pre-filling retention period, and decoding reserved period.

[0006] S5. When pre-filling commitment, decoding reservation, cache residency security and batch processing stability are all registered as passed, the request will be aggregated and distributed according to model version, tokenizer version, tenant isolation flag and decoding service constraints; during continuous decoding, the resource reservation and instance acceptance status will be updated according to the newly added continuous decoding token, session binding cache block usage and instance running records.

[0007] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the above-mentioned method for aggregation, scheduling and optimal matching of multi-type large model tokens.

[0008] Furthermore, in S1: The maximum output boundary is registered based on the maximum output length field in the current inference request; If the maximum output length field is missing, it is registered according to the default output boundary corresponding to the request category; When the maximum output length field exceeds the maximum context length of the target model, context verification is performed based on the total number of input tokens and the maximum output length field. If the context verification fails, a restricted output boundary is registered.

[0009] Furthermore, in S2:

[0010] The cache block chain is verified based on the model version, tokenizer version, attention configuration, location encoding configuration, key-value cache precision, key-value cache block size, inference engine cache layout, tenant isolation marker, cache block integrity status, and cache expiration time.

[0011] When the verification is successful and the cached blockchain continuously covers the input token position range, the corresponding position range will be registered as the prefix cache resident token;

[0012] When a token position is missing, the missing position and its subsequent positions are registered as tokens to be pre-filled.

[0013] Furthermore, in S2:

[0014] The number of reserved continuous decoding tokens is registered by deducting the number of generated continuous decoding tokens from the maximum output boundary.

[0015] If the current inference session has not entered the continuous decoding stage, the number of generated continuous decoding tokens will be recorded as zero;

[0016] Upon receiving the end marker, the number of reserved continuous decoding tokens will be recorded as zero.

[0017] Furthermore, in S3:

[0018] Candidate model instances are filtered based on the capability configuration records corresponding to the request category, the total number of input tokens and the context occupancy length formed by the maximum output boundary, and model instances with matching capability configurations, context occupancy lengths not exceeding the maximum context length, online instance service status and complete instance operation records within the current observation period are added to the candidate model instance list.

[0019] Furthermore, in S4:

[0020] Establish a Token state resource ledger with the current observation period as the smallest time unit, and write in the Token resource entries to be pre-filled, the prefix cache resident Token resource entries, the reserved continuous decoding Token resource entries, the existing session resource commitment entries, and the releasable resource entries respectively;

[0021] The token resource entries to be pre-populated cover the current observation period and the pre-population holding period, while the token resource entries reserved for continuous decoding cover the reserved decoding period.

[0022] Furthermore, the resource safety margin is registered based on the allocable key-value cache capacity, the allocated key-value cache capacity, the token resource entries to be prefilled, the prefix cache resident token resource entries, the reserved continuous decoding token resource entries, the existing session resource commitment entries, and the releasable resource entries; when the resource safety margin boundary is retained in the current observation period, the prefill retention period, and the decoding reserved period, the corresponding resource safety margin is registered as passed.

[0023] Furthermore, the cache residency security status is registered based on the prefix cache residency token resource entries, session-bound cache resource entries, newly added cache resource entries for tokens to be pre-filled, and newly added cache resource entries for reserved continuous decoding tokens. When resource usage does not exceed the cache residency capacity boundary and the resource safety margin boundary is reserved, the cache residency security status is registered as passed. When the number of batch splits is less than twice in three adjacent current observation periods, the cache block application waiting time does not exceed the waiting boundary, and the token output interval does not exceed the stable output interval boundary, the batch processing stable status is registered as passed.

[0024] Furthermore, in S5:

[0025] For candidate model instances in the set of admissible instances, the target model instance is determined sequentially based on the continuous coverage range of the prefix cache resident token, the resource safety margin during the decoding reserved period, the token resource entries to be pre-filled, the high retention priority cache blockchain usage, the current number of continuous decoding sessions, and the number of batch splits. During continuous decoding, when the number of continuously generated continuous decoding tokens reaches 80% of the reserved continuous decoding tokens, subsequent observation period resource entries are supplemented according to the continuous decoding token expansion unit. If the supplementation fails, the instance acceptance status of the target model instance is changed to restricted acceptance.

[0026] Compared with existing technologies, this invention provides a method for aggregation, scheduling, and optimal matching of tokens of multiple types in a large model, which has the following beneficial effects: 1. After verifying the input token sequence and the continuous valid cache blockchain, the prefix cache-resident tokens, tokens to be pre-filled, and reserved continuous decoding tokens are processed separately. The resource usage of the three types of tokens during the current observation period, the pre-filling retention period, and the decoding reservation period is included in the same resource ledger. Synchronously verifying subsequent decoding resources before the request enters the pre-filling process can reduce key-value cache crowding, batch processing splitting, and insufficient decoding resources caused by allocating resources only based on request length, current load, or cache hit. 2. Combine model version, tokenizer version, key-value cache configuration and tenant isolation flag to filter executable instances; during continuous decoding, adjust resource reservation and instance acceptance status according to the number of newly added continuous decoding tokens, session binding cache block usage and instance running records, so that prefix cache reuse, cache residency and continuous decoding process adopt consistent resource management. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the process of a method for aggregation, scheduling, optimization, and matching of multi-type large model tokens according to the present invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] Example: Figure 1 A convergence scheduling optimization matching method for multi-type large model tokens is presented, including:

[0030] S1. Establish an inference session and read the request type, model version constraints, token sequence, maximum output boundary, cache reuse permission, and instance runtime record, specifically:

[0031] After the scheduling control terminal receives the current inference request, it establishes an inference session and reads the request type, model version constraints, token sequence, maximum output boundary, cache reuse permission, and instance execution record. The current inference request is submitted by the caller through the inference interface. When the inference interface receives the request, it writes the request access time, connection number, caller identifier, tenant isolation flag, task flag, input text, target capability flag, model version flag, maximum output length flag, stop condition flag, and cache reuse permission flag.

[0032] The scheduling control terminal generates a session number based on the access date, inference interface number, connection number, and request serial number, and establishes an inference session using this session number. The request category is based on the task tag. If the task tag matches the task category entry in the scheduling configuration version, the corresponding request category is registered. If the task tag is empty, the scheduling control terminal reads the inference interface path and the business category registered by the caller; if they match, the corresponding request category is registered. If the inference interface path and the business category registered by the caller do not match, the request category is registered as pending verification, and the subsequent Token status registration process is suspended until the caller supplements a valid task tag. If the task tag, inference interface path, and business category registered by the caller are all missing, it is registered as a general generation request, and the model capability boundary and maximum output boundary corresponding to the general generation request are adopted.

[0033] The scheduling control terminal first checks whether the current inference request carries a model version tag. If the model version tag matches a valid version in the model version registry, the subsequent instance filtering range is limited according to that tag. If no model version tag is carried, the scheduling control terminal reads the model capability configuration record corresponding to the current request category. This configuration record is generated during model deployment and acceptance testing, registering the model version range that can handle this type of request, the Tokenizer version, the maximum context length, the structured output permission status, the tool call permission status, and the restricted content processing boundaries.

[0034] When the current inference request requires structured output, the scheduling control terminal only retains model versions with valid structured output permissions. When the current inference request requires tool invocation, the scheduling control terminal only retains model versions with valid tool invocation permissions. When the current inference request simultaneously requires both long context and fixed output format, the scheduling control terminal retains model versions that simultaneously meet the maximum context length boundary and fixed output format boundary. If no model version matching the current request category exists in the model capability configuration record, the inference session is registered as a pending model version and does not enter the candidate model instance selection process.

[0035] The token sequence is formed by processing the input text in the current inference request using the tokenizer version corresponding to the model version constraint. The scheduling control terminal first reads the tokenizer version from the model version registry, and then calls the corresponding tokenizer version to convert the input text into input tokens arranged consecutively in order of position. After tokenization, the scheduling control terminal writes the total number of input tokens, token position number, first token position, last token position, input token prefix identifier, and tokenizer version into the inference session.

[0036] The input token prefix is ​​formed in sequential order starting from the first token position and is used only for subsequent cache directory verification; the plaintext of the input text is not retained. When the input text is empty, the scheduling control terminal registers the total number of input tokens as zero and restricts the current inference request to requests that do not read the prefix cache. If, after tokenization, there are issues such as broken position numbers, the first token position not being the first position, the last token position not matching the total number of input tokens, the tokenizer version not being returned, or the tokenization service not returning a result within the response period registered in the scheduling configuration version, the scheduling control terminal registers the inference session as pending tokenization verification.

[0037] Tokenization pending requests are reprocessed after the tokenization service resumes; if the total number of input tokens and the token location number obtained from the two processing are completely consistent, the result of the second processing is retained; if there is a difference between the two processing results, the scheduling control terminal registers the current inference request as a tokenization conflict request and stops reading the historical cached blockchain, and subsequently only processes it according to the complete path of the token to be pre-filled.

[0038] The scheduling control unit reads the maximum output length flag from the current inference request. If the maximum output length flag is a positive integer and its total number with the input tokens does not exceed the maximum context length of the target model version, the number of tokens corresponding to this flag is taken as the maximum output boundary. If the current inference request does not carry a maximum output length flag, the scheduling control unit reads the default output boundary corresponding to the request category.

[0039] The default output boundary is derived from the valid output length records of the current request category in the inference logs over the past 30 days and the continuous decoding resource records during the online acceptance phase. In this implementation, the default output boundary for general generation requests is registered as 512 tokens, for short question-and-answer requests as 256 tokens, for structured generation requests as 384 tokens, and for long text analysis requests as 1024 tokens. When the maximum output length marker exceeds the maximum context length of the target model version, the scheduling control terminal does not directly truncate the input token sequence. Instead, it first reads the total number of input tokens. If the sum of the total number of input tokens and the maximum output length marker does not exceed the maximum context length of the target model, the original maximum output boundary is retained. If the sum exceeds the maximum context length of the target model, the current inference request is registered as a context boundary pending verification, and the caller is waited for resubmission of the maximum output length marker. If the caller fails to replenish the marker within the preset waiting period, the scheduling control terminal processes it according to the restricted output boundary of the request category and writes the restricted output marker into the inference session.

[0040] A cache reuse permission is formed by the cache reuse permission flag in the current inference request and the tenant isolation policy. The cache reuse permission is registered as valid when the cache reuse permission flag is valid and the tenant isolation policy allows the current tenant to read historical caches within the same isolation domain. When the cache reuse permission flag is empty, the scheduling control reads the cache policy registered by the caller; if the caller's registered cache policy allows reuse and the tenant isolation policy allows reading, the cache reuse permission is registered as valid; if the caller's registered cache policy prohibits reuse, the cache reuse permission is registered as invalid. When the tenant isolation policy requires an independent cache domain, even if the cache reuse permission flag in the current inference request is valid, the scheduling control will still register the cache reuse permission as restricted and limit subsequent cache directory verification to only be performed within the current tenant isolation domain.

[0041] Instance execution logs are written by each model instance within the current observation period. The current observation period is 200 milliseconds, which is derived from the inference engine decoding round status reporting records, key-value cache block allocation records, and GPU memory status acquisition records. Instance execution logs include the instance number, model version, Tokenizer version, attention configuration, positional encoding configuration, key-value cache precision, key-value cache block size, GPU parallel configuration, inference engine cache layout, instance service status, allocable key-value cache capacity, allocated key-value cache capacity, current pre-filled queue count, current consecutive decoding session count, current decoding batch count, current GPU memory usage, last token output time, and last cache block release time.

[0042] During the current observation period, if an instance's runtime record lacks information on allocable key-value cache capacity, allocated key-value cache capacity, the number of current consecutive decoding sessions, current GPU memory usage, or instance service status, the scheduling control terminal will mark the instance as pending verification and will not include it in the subsequent candidate model instance list. After completing the above verification, the request type, model version constraints, token sequence, maximum output boundary, cache reuse permission, and instance runtime record in the inference session can be used as the basis for subsequent token status registration.

[0043] S2. Based on the token sequence and the continuous valid cache block chain, register the prefix cache resident token, the token to be pre-filled, and the reserved continuous decoding token, specifically:

[0044] The scheduling control unit reads the token sequence, tokenizer version, cache reuse permit, and tenant isolation flag from the inference session, and queries the cache directory record corresponding to the current token sequence. Based on the continuous coverage of the cache blockchain, it registers the prefix cache-resident token, the token to be pre-populated, and the reserved continuous decoding token, respectively.

[0045] The cache directory records are written by the model instance upon completion of pre-population, session completion, cache block release, cache block migration, and model version switching. Each cache directory record includes the cache instance number, cache block chain number, model version, tokenizer version, attention configuration, positional encoding configuration, key-value cache precision, key-value cache block size, GPU parallel configuration, inference engine cache layout, prefix identifier, prefix token start position, prefix token end position, cache generation time, most recent reuse time, cache expiration time, cache block integrity status, cache reusable status, tenant isolation flag, current number of referencing sessions, and number of cache-resident tokens. The scheduling control terminal only reads the cache directory records when a cache reuse permission is valid.

[0046] When the cache reuse permission is invalid or restricted, the scheduling control end will not use any historical cache block chain as the prefix cache source for the current inference request, and will register all input tokens as tokens to be pre-populated.

[0047] The scheduling control unit sequentially performs version verification, isolation verification, continuity verification, and validity verification on each cache directory record. Version verification is based on the model version constraints in the current inference session, the Tokenizer version, and the attention configuration, positional encoding configuration, key-value cache precision, key-value cache block size, GPU parallel configuration, and inference engine cache layout in the model version registry. Version verification is considered passed if all eight configuration items in the cache directory record are completely consistent with the model version corresponding to the current inference session.

[0048] If any configuration is inconsistent, the version verification will fail. Isolation verification is based on the tenant isolation flag in the current inference session and the tenant isolation flag in the cache directory record; if they match, it is registered as passed; otherwise, it is registered as failed. Continuity verification is based on the position of the first and last token in the cache block chain, as well as the connection between adjacent cache block token positions. If the cache block chain continuously covers from the first position to a certain ending position, and there are no uncovered token positions between adjacent cache blocks, the continuity verification is registered as passed.

[0049] The continuity check fails if any part of the cache block chain is missing, the first block position is inconsistent with the first token position of the current inference request, the last block position exceeds the last token position of the current inference request, or the cache block chain order is abnormal. The validity check is based on the cache block integrity status, cache reusability status, cache expiration time, and the number of current referencing sessions. The validity check passes if the cache block integrity status is complete, the cache reusability status is valid, the cache expiration time is later than the current scheduling time, and the number of current referencing sessions has not reached the reuse boundary registered in the scheduling configuration version.

[0050] When a cache directory record simultaneously passes version verification, isolation verification, continuity verification, and validity verification, the scheduling control terminal registers the range of input token positions continuously covered by the cache block chain as prefix cache-resident tokens. The number of prefix cache-resident tokens is determined by the continuous coverage range, not by the number of cache blocks. For example, if the total number of input tokens for the current inference request is 9000, and the cache block chain continuously covers from the first position to the 6400th position, the scheduling control terminal registers the first to the 6400th tokens as prefix cache-resident tokens, and registers the 6401st to the 9000th tokens as tokens to be pre-populated. If there are valid cache block chains for Token 1 to Token 3200 in the cache directory, and valid cache block chains for Token 4000 to Token 6400, but Token 3201 to Token 3999 are not covered by valid cache, the scheduling control terminal will only register Token 1 to Token 3200 as prefix cache resident tokens, and register Token 3201 to Token 9000 as tokens to be pre-filled, without splicing different cache block chains across uncovered positions.

[0051] When multiple consecutive valid cache block chains exist in the cache directory, the scheduling control end retains the cache block chain with the largest termination position. If the termination positions are the same but the cache generation times are different, the cache block chain with the most recent reuse time is retained. If both the termination position and the most recent reuse time are the same, the cache block chain with the smaller cache instance number is retained. This order is fixed and registered by the scheduling configuration version to avoid reading different cache block chains for the same input token sequence in different observation periods.

[0052] The tokens to be pre-populated originate from the consecutive position range in the input token sequence of the current inference request that are not covered by the prefix cache-resident tokens. If the current inference request does not have a cache block chain that has passed all verifications, all input tokens are registered as tokens to be pre-populated. If the current inference request has a cache block chain that has passed all verifications, all input tokens after the termination position of the prefix cache-resident tokens are registered as tokens to be pre-populated.

[0053] When a pre-filled token is registered, its start position, end position, quantity, prefix cache read position, corresponding cache block chain number, and pre-fill status are simultaneously written. The initial pre-fill status is registered as pending admission. If the cache directory records any of the following states during the current observation period: cache block released, cache block chain interrupted, cache block bound to session, cache status pending verification, or cache block version changed, the scheduling control terminal will revoke the prefix cache resident token registration corresponding to the current inference request, change the coverage of the original prefix cache resident token to the pre-filled token, and write the reason why the cache is not reusable.

[0054] When a cached blockchain does not meet the reuse conditions, the scheduling control end writes the reason into the cache non-reusability record. The reasons are categorized as cache released, blockchain interrupted, session bound, cache pending verification, version change, and isolation mismatch. Prefix cache resident tokens are registered only based on cached blockchains that have passed version verification, isolation verification, continuity verification, and validity verification, and are not based on historical hit counts, text similarity, or model names as the basis for cache reuse.

[0055] Continuous decoding tokens are reserved based on the maximum output boundary. When the current inference request has not yet entered the decoding stage, the number of generated continuous decoding tokens is zero, and the number of reserved continuous decoding tokens is the same as the maximum output boundary. When the current inference request resumes decoding from an interrupted session, the scheduling control unit reads the number of generated continuous decoding tokens from the most recent decoding feedback record and subtracts this number from the maximum output boundary to obtain the number of reserved continuous decoding tokens. If the most recent decoding feedback record is missing, the current inference request is registered as pending decoding and is no longer assigned to a new target model instance.

[0056] When the number of continuously generated decoding tokens exceeds the maximum output boundary, the scheduling control terminal writes an output boundary conflict record and no longer reserves continuous decoding resources. Upon receiving an end marker, the number of reserved continuous decoding tokens is updated to zero, and the current inference session enters a completed and pending release state. After the registration of the three types of tokens is completed, the scheduling control terminal writes a token status occupancy record.

[0057] The Token state occupancy record records the quantity of three types of Tokens, the Token location range, the corresponding cache block number, the cache verification status, the pre-fill status, the number of generated continuous decoding Tokens, the number of reserved continuous decoding Tokens, and the registration time. The Token state occupancy record serves as input for filtering candidate model instances in step S3 and as input for establishing the Token state resource ledger in step S4.

[0058] S3. Filter candidate model instances based on model version, Tokenizer version, key-value cache configuration, tenant isolation flag, and instance service status, specifically:

[0059] The scheduling control terminal reads the request type, model version constraints, total number of input tokens, maximum output boundary, tokenizer version, tenant isolation flag, cache reuse permission and token status occupancy record from the inference session, and determines the candidate model instance by combining the model version registration library and instance running record.

[0060] The model version registry is created during model deployment and acceptance testing. It registers the model version, tokenizer version, maximum context length, attention configuration, positional encoding configuration, key-value cache precision, key-value cache block specifications, graphics processor parallel configuration, inference engine cache layout, structured output permission status, tool call permission status, long context permission status, and resource usage reference record number.

[0061] During candidate model instance selection, the scheduling control terminal sequentially checks capability compatibility, context capacity, cache configuration, tenant isolation conditions, instance service status, and instance runtime records. When the current inference request requires structured output, only model instances with valid structured output permissions are retained; when it requires tool invocation, only model instances with valid tool invocation permissions are retained; and when it requires long context, only model instances with valid long context permissions are retained.

[0062] When the current inference request does not have a target capability flag set, the scheduling control terminal reads the model capability configuration record corresponding to the request category and retains instances according to the model version range in the model capability configuration record. If any of the following situations occur during capability compatibility verification: model version is not within the permitted range, structured output permission status is invalid, tool call permission status is invalid, or long context permission status is invalid, the corresponding model instance will not be included in the candidate model instance list.

[0063] The scheduling control unit performs context verification based on the total number of input tokens, the maximum output boundary, and the maximum context length in the model version registry. If the sum of the total number of input tokens and the maximum output boundary does not exceed the maximum context length of the candidate model instance, the current instance is retained in the candidate range; if the sum exceeds the maximum context length, the current instance does not participate in subsequent matching.

[0064] When the current inference request has a prefix cached token, the context verification still uses the total number of complete input tokens, and does not use the number of tokens to be pre-filled to replace the total number of input tokens. When the current inference request uses a restricted output boundary, the inference session retains the restricted output flag, and the scheduling control terminal only retains model instances that satisfy the restricted output boundary.

[0065] Cache compatibility verification is based on the prefix cache-resident token, cache block chain number, and cache configuration of the current model instance in the token state occupancy record. When the current inference request does not have a prefix cache-resident token, the cache compatibility verification is registered as pre-fillable, and the model instance is not excluded due to cache missing. When the current inference request has a prefix cache-resident token, the scheduling control end reads the model version, tokenizer version, attention configuration, position encoding configuration, key-value cache precision, key-value cache block size, graphics processor parallel configuration, and inference engine cache layout of the model instance to which the cache block chain belongs, and then verifies them item by item with the corresponding configurations in the candidate model instances.

[0066] When the model version, tokenizer version, attention configuration, position encoding configuration, key-value cache precision, key-value cache block size, graphics processor parallel configuration, and inference engine cache layout corresponding to the cache block chain are all consistent with the candidate model instance, the cache compatibility status is registered as direct reuse; when there are inconsistent configurations, the cache compatibility status is registered as not directly reused.

[0067] Model instances whose cache compatibility status is not directly reused are still retained in the candidate model instance list. The scheduling control end merges the prefix cache-resident token of the current inference request into the token to be pre-populated, and establishes a resource ledger according to the adjusted number of tokens to be pre-populated, without carrying the hit status of the cache block chain into the current model instance.

[0068] The isolation compatibility verification reads the tenant isolation tag, data residency policy, and candidate model instance resource domain tag of the current inference request. If the tenant isolation tag matches the resource domain tag and the data residency policy allows the current model instance to remain in the candidate model instance list when the current tenant executes in that resource domain; if the tenant isolation tag does not match the resource domain tag and no cross-domain execution permission is registered, the current model instance will not participate in subsequent matching.

[0069] If a cache reuse license is valid but isolation compatibility verification fails, the scheduling control terminal does not retain the direct reuse status resulting from the cache reuse license. Instance service verification is based on the instance service status. Instance service status is registered as online, under maintenance, draining, restarting, and faulty. Online status means that model weights have been loaded, inference engine health checks have passed, key-value cache allocation is effective, graphics processor status is normal, and complete instance operation records have been written for the last three current observation periods.

[0070] The "Maintenance" status indicates that the model instance has received maintenance instructions and will no longer accept new inference sessions. The "Out of Service" status indicates that the model instance has stopped accepting new inference sessions due to resource constraints, model version switching, cache migration, or fault recovery. The "Restarting" status indicates that the model instance is in the process of loading model weights, restoring the inference engine, or rebuilding the cache directory. When the graphics processor status is abnormal, the inference engine is unresponsive, key-value cache allocation is abnormal, cache directory consistency verification fails, or health check fails, the instance service status is registered as "Faulty". The candidate model instance list only retains model instances with an online service status.

[0071] Instance observation verification is based on the completeness of the instance operation record and the collection time. If the allocable key-value cache capacity, allocated key-value cache capacity, current pre-filled queue quantity, current continuous decoding session quantity, current decoding batch quantity, current graphics processor memory usage, last token output time, and instance service status are all present in the instance operation record, and the collection time is within the current observation period, the instance observation verification is registered as valid.

[0072] If any field is missing, the collection time is earlier than the start of the current observation period, or there are conflicting records for the same field, the instance observation verification is registered as pending verification. Model instances whose instance observation verification is pending verification are not included in the candidate model instance list. After completing capability compatibility verification, context compatibility verification, cache compatibility verification, isolation compatibility verification, instance service verification, and instance observation verification, the scheduling control terminal will write all the model instances that have passed verification into the candidate model instance list, and write the model instances whose cache compatibility verification is not directly reused but have passed all other verifications into the candidate model instance list to be pre-populated.

[0073] Both the candidate model instance list and the candidate model instance list to be pre-populated are written with instance number, model version, Tokenizer version, cache compatibility status, instance running record time and current instance service status, which serve as the instance range for establishing the Token status resource ledger in step S4.

[0074] S4. For candidate model instances, register three types of token resource usage, resource security margin, cache residency security status, and batch processing stable status within the current observation period, pre-filling retention period, and decoding reserved period, specifically:

[0075] For candidate model instances, the scheduling control terminal registers three types of token resource usage, resource safety margin, cache retention safety status, and batch processing stable status within the current observation period, pre-filling retention period, and decoding reserved period. The scheduling control terminal first reads the token status usage record, the candidate model instance list, the candidate model instance list to be pre-filled, the instance running record, the resource usage reference record in the model version registration database, and the scheduling configuration version.

[0076] The scheduling configuration version registers a current observation period of 200 milliseconds, a pre-fill retention period of 600 milliseconds, a decoding reservation period of 1600 milliseconds, a resource safety margin boundary of 7% of the allocatable key-value cache capacity of the current model instance, and a cache residency capacity boundary of 65% of the allocatable key-value cache capacity of the current model instance. The current observation period, pre-fill retention period, decoding reservation period, and resource safety margin boundary are all derived from inference engine operation logs, key-value cache block allocation logs, graphics processor memory logs, continuous decoding token output records, and anomaly recovery test records during the model's online acceptance testing period. Different model versions, different graphics processor parallel configurations, different key-value cache precisions, and different inference engine cache layouts are registered with their respective corresponding scheduling configuration versions and do not share resource safety margin boundaries.

[0077] Resource usage reference records are generated during the online acceptance phase based on model version, Tokenizer version, key-value cache precision, parallel configuration of graphics processor, token range to be pre-filled, and continuously decoded token range.

[0078] In the first model version, with four graphics processors configured in parallel, the 1024 tokens to be pre-filled without reusing the prefix cache correspond to 219 megabytes of key-value cache usage. During the continuous decoding phase, each additional 1024 tokens corresponds to 214 megabytes of key-value cache usage, and the 1024 tokens residing in the prefix cache correspond to 214 megabytes of key-value cache usage.

[0079] The second model version uses two graphics processors running in parallel. The 1024 tokens to be pre-filled without reusing the prefix cache occupy 101 megabytes of key-value cache. During the continuous decoding stage, the key-value cache increases by 96 megabytes for every additional 1024 tokens. The 1024 tokens residing as the prefix cache occupy 96 megabytes of key-value cache.

[0080] Resource usage reference records are generated during the model deployment and acceptance phase. Acceptance tasks use fixed input lengths, fixed output lengths, and fixed batch sizes, and are executed when the graphics processor (GPU) is running stably. The scheduling control unit reads the key-value cache usage for each token range from the inference engine cache block allocation log and the GPU memory status log, forming resource usage reference records corresponding to the model version, GPU parallel configuration, and key-value cache precision. Once a current inference request enters a candidate model instance, the scheduling control unit only calls the resource usage reference record consistent with the instance's configuration.

[0081] The scheduling control terminal establishes a Token state resource ledger for each candidate model instance. The Token state resource ledger uses the current observation period as the smallest time unit and records the following items: Token resource entries to be pre-populated, prefix cache resident Token resource entries, reserved continuous decoding Token resource entries, existing session resource commitment entries, releasable resource entries, and resource safety margin entries.

[0082] When the current inference request's cache compatibility status in the candidate model instance is "direct reuse," the pending pre-populated token resource entries in the token state resource ledger are registered according to the number of pending pre-populated tokens in the token state occupancy record, and the prefix cache-resident token resource entries are registered according to the number of prefix cache-resident tokens. When the current inference request's cache compatibility status in the candidate model instance is "not direct reuse," the scheduling control end transfers the number of prefix cache-resident tokens to the pending pre-populated token resource entries, and the prefix cache-resident token resource entries are registered as zero.

[0083] The number of reserved continuous decoding tokens is determined by the maximum output boundary and the number of generated continuous decoding tokens. When the current inference request has not yet entered the decoding stage, the number of generated continuous decoding tokens is recorded as zero, and the number of reserved continuous decoding tokens is registered according to the maximum output boundary. When the current inference request resumes decoding from an interrupted state, the scheduling control terminal reads the number of generated continuous decoding tokens from the most recent decoding feedback record, subtracts this number from the maximum output boundary, and updates the number of reserved continuous decoding tokens.

[0084] The token resource entries to be pre-populated are written to the current observation period and the subsequent observation periods covered by the pre-population retention period. The pre-population retention period is 600 milliseconds, corresponding to three consecutive current observation periods. Within the three observation periods, the scheduling control terminal reads the existing pre-populated resource entries, existing continuous decoding resource entries, prefix cache resident resource entries, existing session resource commitment entries, and releasable resource entries of the candidate model instance, and then writes them to the token resource entries to be pre-populated in the current inference request.

[0085] The reserved continuous decoding token resource entry is written to the subsequent observation period covered by the decoding reserved period, starting from the end of the pre-filled hold period. The decoding reserved period is 1600 milliseconds, corresponding to eight consecutive current observation periods. Within the eight observation periods, the scheduling control terminal reads the existing continuous decoding resource entries, existing session resource commitment entries, prefix cache resident resource entries, releasable resource entries, and resource safety margin entries one by one, and then writes them to the reserved continuous decoding token resource entry of the current inference request.

[0086] Prefix cache resident token resource entries are continuously written from the current observation period to subsequent observation periods before the cache expiration time. When the cache expiration time is missing, the scheduling control terminal does not register the corresponding cache blockchain as a long-term resident resource, but instead registers the prefix cache resident token resource entry as pending verification.

[0087] The resource safety margin is determined based on the allocable key-value cache capacity and the allocated key-value cache capacity of the candidate model instance, and is updated in combination with the token resource entries to be pre-filled, the reserved continuous decoding token resource entries, the prefix cache resident token resource entries, the existing session resource commitment entries, and the releaseable resource entries corresponding to the current inference request.

[0088] During the current observation period, if the allocated key-value cache capacity, the resource usage of each resource entry during the current observation period, and the committed resource amount for the current session still retain a capacity of no less than 7% of the allocatable key-value cache capacity after being written, the resource safety margin for the current observation period is registered as passed. During the pre-fill hold period, if the same condition is met for three consecutive observation periods, the resource safety margin for the pre-fill hold period is registered as passed.

[0089] During the decoding reservation period, if the same condition is met for eight consecutive observation periods, the resource safety margin for the decoding reservation period is registered as passed. If it is less than 7% in any observation period, the corresponding resource safety margin is registered as failed. If the instance operation record is missing, the resource usage reference record is inconsistent with the candidate model instance configuration, the releaseable resource entry has not received cache block release confirmation, or the allocated key-value cache capacity is greater than the allocable key-value cache capacity, the corresponding resource safety margin is registered as pending approval.

[0090] The cache residency safety status is based on the prefix cache residency resource entries of the current candidate model instance, session-bound cache resource entries, newly added cache resource entries formed by tokens to be pre-populated, newly added cache resource entries formed by reserved continuous decoding tokens, and the cache residency capacity boundary. The cache residency safety status is registered as passed if the total number of prefix cache residency resource entries and session-bound cache resource entries does not exceed 65% of the allocatable key-value cache capacity, and if a resource safety margin is still maintained after the current inference request enters.

[0091] If the total number of prefix cache-resident resource entries and session-bound cache resource entries reaches 65%, and the current candidate model instance does not have a cache block chain to be released, the cache residency security status is registered as failed. If the total number of prefix cache-resident resource entries and session-bound cache resource entries is less than 65%, but any of the following conditions are met: cache block release records have not been written back, the cache block chain is currently bound to a session, the cache directory record collection time is earlier than the start of the current observation period, or the cache expiration time is close to the end of the current observation period, the cache residency security status is registered as restricted.

[0092] Batch processing stability is determined based on the current number of decoding batches, the current number of consecutive decoding sessions, the number of batch splits within the three adjacent current observation periods, the wait time for the most recent cache block request, and the interval between the most recent token outputs. A batch processing stability is registered as passed if there are no two batch splits within the three adjacent current observation periods, the wait time for the most recent cache block request does not exceed the waiting boundary registered in the scheduling configuration version, and the interval between the most recent token outputs does not exceed the stable output interval boundary in the model version acceptance record.

[0093] If two batch processing splits occur within three consecutive current observation periods, the cache block request wait time exceeds the wait boundary, or the token output interval exceeds the stable output interval boundary for three consecutive current observation periods, the batch processing stable state is registered as failed. If the instance execution record is not fully returned within the current observation period, the batch processing stable state is registered as pending approval.

[0094] The scheduling control unit writes the tokens to be pre-filled, the prefix cache resident tokens, and the reserved continuous decoding tokens into their respective time ranges, and forms a token status resource ledger for candidate model instances accordingly. Subsequent resource adjustments during aggregation, target model instance matching, execution admission, and continuous decoding phases all read the corresponding records from this resource ledger.

[0095] S5. When pre-filling commitment, decoding reservation, cache residency security, and batch processing stability are all registered as passed, requests will be aggregated and distributed according to model version, Tokenizer version, tenant isolation flag, and decoding service constraints. During continuous decoding, resource reservation and instance acceptance status will be updated based on the newly added continuous decoding token, session-bound cache block usage, and instance runtime records, specifically:

[0096] When the pre-fillable commitment state, decoding reservable state, cache-resident security state, and batch processing stable state are all registered as passed, the scheduling control terminal will aggregate and distribute the current inference request according to the model version, tokenizer version, tenant isolation flag, and decoding service constraints; during continuous decoding, the resource reservation and instance acceptance status will be updated based on the newly added continuous decoding token, session-bound cache block usage, and instance running records.

[0097] The scheduling control terminal first forms a pre-filled committable state based on the Token state resource ledger in step S4. For the current candidate model instance, within the three observation periods covered by the current observation period and the pre-filling retention period, if the resource safety margin after the pre-filled Token resource entry is written is registered as passed, and the current pre-filled queue number has not reached the pre-filled queue boundary registered in the scheduling configuration version, the pre-filled committable state is registered as passed.

[0098] Prefilling can be registered as failing when the current prefill queue count reaches the prefill queue boundary, the resource safety margin for any prefilling maintenance period is registered as failing, or the instance service status changes from online to maintenance, draining, restarting, or faulty. Prefilling can also be registered as pending when the instance runtime record is incomplete, prefilled resource entries are pending approval, or the cache read status is pending approval.

[0099] The code for the code to be reserved is based on the eight current observation periods covered by the code reservation period. If, within the eight current observation periods, the resource safety margin after the reserved continuous code token resource entries are written is registered as passed, and the number of sequences that the code batch can accommodate for each current observation period has not reached the boundary, the code to be reserved is registered as passed.

[0100] If any of the following conditions are met: the resource safety margin in any current observation period is registered as failing; the current candidate model instance has a continuous decoding session in a reserved extension state and has not completed resource replenishment; the current decoding batch number reaches the decoding batch boundary; or the current continuous decoding session number reaches the sequence number boundary, the decoding reservation state is registered as failing.

[0101] When decoding feedback records are missing, releaseable resource entries have not received cache block release confirmation, or the batch processing stable state of the current candidate model instance is pending, the decoding reservable state is registered as pending. The scheduling control terminal writes the candidate model instance whose pre-filled committable state, decoding reservable state, cache residency safe state, and batch processing stable state are all registered as passed into the set of admissible instances.

[0102] Inference sessions accessed within the same current observation period are first aggregated according to model version constraints, Tokenizer version, tenant isolation flag, request type, maximum context length requirement, and cache reuse permission to form a basic compatibility set. Within the basic compatibility set, the scheduling control terminal further aggregates candidate records based on the Token range to be pre-filled, the Token range resident in the prefix cache, the reserved continuous decoding Token range, the first Token service constraint, and the continuous decoding Token output constraint.

[0103] Inference sessions with no more than 1024 tokens to be pre-filled are classified as short pre-filling ranges; those with more than 1024 but no more than 4096 tokens are classified as medium pre-filling ranges; those with more than 4096 but no more than 12288 tokens are classified as long pre-filling ranges; and those with more than 12288 tokens are classified as ultra-long pre-filling ranges. These ranges are determined based on the pre-filling execution period acceptance records obtained for the corresponding model version under the target graphics processor configuration.

[0104] Inference sessions with no more than 256 consecutively decoded tokens are recorded in the short decoding range; those with more than 256 but no more than 768 tokens are recorded in the medium decoding range; those with more than 768 but no more than 1536 tokens are recorded in the long decoding range; and those with more than 1536 tokens are recorded in the ultra-long decoding range. These ranges are determined based on the consecutive decoding token growth record and key-value cache block expansion record of the current model version.

[0105] Inference sessions with different pre-filled token ranges, different reserved continuous decoding token ranges, different tenant isolation markers, different tokenizer versions, and different first token service constraints are not written to the same aggregation candidate record.

[0106] After the candidate records are aggregated, the scheduling control terminal performs a preferred matching for the set of admissible instances. The preferred matching adopts a fixed step-by-step verification order. The first verification order reads the continuous coverage range of the prefix cache-resident token, the cache-resident security status, and the resource safety margin for the decoding reserved period. Candidate model instances with a longer continuous coverage range of the prefix cache-resident token, a passed cache-resident security status, and resource safety margins higher than the resource safety margin boundary in all eight current observation periods are given priority for retention.

[0107] If the first verification order cannot be distinguished, the scheduling control terminal enters the second verification order, reads the token resource entries to be prefilled and the number of prefilled queues during the prefilling retention period, and prioritizes the retention of candidate model instances with fewer token resource entries to be prefilled and fewer prefilled queues.

[0108] If the second verification order cannot be distinguished, the scheduling control terminal enters the third verification order, reads the high retention priority cache block chain occupancy, the amount of releaseable cache resources, and the cache block release confirmation status of the current candidate model instance. Candidate model instances with low high retention priority cache block chain occupancy, large amount of releaseable cache resources, and valid cache block release confirmation status are given priority for retention.

[0109] If the third verification sequence cannot distinguish between them, the scheduling control terminal proceeds to the fourth verification sequence. It reads the number of current continuous decoding sessions, the number of batch splits within the three adjacent current observation periods, and the interval of the most recent token output. Candidate model instances with fewer continuous decoding sessions, fewer batch splits, and token output intervals not exceeding the stable output interval boundary are prioritized for retention. If the four verification sequences still cannot distinguish between them, the candidate model instance with the smaller instance number is selected. The instance number is fixed and assigned during model deployment and registration, and does not change with the running status.

[0110] After the optimal matching is completed, the scheduling control terminal writes an execution admission record to the target model instance. The execution admission record includes the aggregation number, inference session number set, target model instance number, model version, tokenizer version, prefix cache block chain number, the range of token positions to be pre-populated, pre-populated resource commitment number, decoding resource reservation number, pre-population retention period start point, decoding reservation period start point, cache residency protection number, and backup model instance number.

[0111] After receiving the admission record, the target model instance first performs a local read verification on the prefix cache block chain. The local read verification is based on the cache block chain number, model version, tokenizer version, attention configuration, position encoding configuration, key-value cache precision, key-value cache block size, graphics processor parallel configuration, inference engine cache layout, tenant isolation flag, and prefix token position range.

[0112] When all cache blocks are consistent and the cache block chain is continuous and complete, the target model instance reads the prefix cache block chain and processes the tokens to be pre-populated. If any of the following conditions are met: the cache block chain has been released, the cache block chain is interrupted, the cache block chain is bound to a session, the version is inconsistent, the isolation is inconsistent, or the cache status is pending, the target model instance writes back the cache read failure record to the scheduling control terminal.

[0113] Upon receiving a cache read failure record, the scheduling control terminal converts all prefix cache-resident tokens in the current inference session into tokens to be pre-filled, and re-establishes the token resource entries to be pre-filled and the reserved continuous decoding token resource entries for the target model instance. If the pre-filling commitment state and the decoding reservation state are still passed after re-verification, the current inference session executes in the original target model instance according to the complete token path to be pre-filled. If any state fails after re-verification, the scheduling control terminal cancels the pre-filling resource commitment number and the decoding resource reservation number that have not yet been started, and returns the current inference session to the set of admissible instances for re-matching.

[0114] After the target model instance completes pre-population, a pre-population feedback record is written to the scheduling control terminal. The pre-population feedback record includes the inference session number, pre-population batch number, actual number of pre-populated tokens, actual number of prefix cache tokens read, new cache block chain number, number of new cache blocks, pre-population completion time, actual key-value cache usage, and pre-population exception status.

[0115] The scheduling control unit verifies the actual number of pre-filled tokens against the range of token positions to be pre-filled in the execution admission record, verifies the actual number of prefix cache tokens read against the cache read verification record, and verifies the newly added cache block number against the cache directory record. If the verifications match, the current inference session enters continuous decoding mode. If the actual number of pre-filled tokens is lower than the resource commitment tokens, the scheduling control unit registers the difference as a releaseable pre-filled resource amount and releases it after the pre-fill retention period ends.

[0116] When the actual number of pre-filled tokens exceeds the number of resource-committed tokens, the scheduling control terminal registers the current inference session as a token boundary deviation and re-verifies the resource safety margin within the decoding reservation period. If the resource safety margin remains sufficient within the decoding reservation period, the decoding resource reservation for the current inference session is retained. If the resource safety margin fails to meet the requirement in any current observation period, the target model instance's instance acceptance status is changed to restricted acceptance. In the restricted acceptance state, new pre-filled requests and new continuous decoding requests are stopped. Inference sessions already in the continuous decoding state continue to use existing decoding resource reservations.

[0117] During continuous decoding, the target model instance is written to the decoding feedback record before the end of each current observation period. The decoding feedback record includes the inference session number, the current decoding batch number, the number of continuously decoded tokens generated, the number of newly added continuously decoded tokens in this current observation period, the number of cache blocks bound to the current session, the cache usage of the current session, the time of the last token output, the end flag status, the reason for stopping, and the output length status.

[0118] After reading the decoding feedback record, the scheduling control terminal writes the number of newly added continuous decoding tokens in the current observation period to the token status resource ledger, changes the reserved continuous decoding token resource entry for the corresponding future observation period to the executed continuous decoding token resource entry, and updates the cached resource entry bound to the current session. When the output length status is sufficient, the scheduling control terminal retains the original decoding resource reservation.

[0119] When the output length status enters the near-reservation phase, the scheduling control terminal reads the caller's output length permission record. If the permission record allows continued generation, decoding resources are appended at 128 tokens for the first model version, 192 tokens for the second model version, and 256 tokens for the third model version. Each extension unit registers its progress based on the key-value cache block specifications and continuous decoding stability acceptance records for its corresponding model version.

[0120] The scheduling control terminal will write the newly added decoding resources into the subsequent four current observation periods and verify the resource safety margin for each period. If the verification results for all four current observation periods pass, the number of reserved continuous decoding tokens for the current inference session will be increased; if the verification fails in any of the observation periods, the instance acceptance status of the target model instance will be adjusted to restricted acceptance, and new aggregation candidate records will be stopped from being issued to that model instance.

[0121] When the current inference session reaches its maximum output limit and the caller has not submitted an extension permission, the scheduling control unit updates the output length status to "output complete" and releases unused decoding resource reservations. When the current inference session receives an end marker, cancellation instruction, or stop condition trigger record, the scheduling control unit transfers it to the "complete and await release" state, and then processes the cache blocks bound to the current session based on cache reuse permission, tenant isolation marker, cache residency security status, and cache residency capacity limits.

[0122] When the cache reuse permit is valid, the tenant isolation flag is consistent, the cache residency security state is passed, and the prefix cache residency capacity does not exceed 65%, the current session's bound cache block chain is changed to the prefix cache residency token. If any condition is not met, the current session's bound cache block chain enters the release path. In the release path, the model instance first writes back the cache block release confirmation record, and then the scheduling control terminal removes the corresponding session-bound cache resource entry and the reserved continuous decoding token resource entry from the token state resource ledger.

[0123] Through a continuous process of continuous decoding feedback, resource reservation updates, instance acceptance status changes, cache retention, and cache release, the current inference request maintains the same set of token status criteria in the pre-filling stage, continuous decoding stage, and cache residency stage.

[0124] This embodiment also provides a computer-readable storage medium storing a computer program. When the computer program is loaded and executed by the processor of the scheduling control terminal, it completes the following tasks: establishing an inference session, processing a token sequence, registering tokens residing in the prefix cache, registering tokens to be pre-filled, registering tokens reserved for continuous decoding, filtering candidate model instances, establishing a token state resource ledger, verifying resource security margins, verifying cache residing security status, verifying batch processing stability, converging and issuing requests, updating continuous decoding resource reservations, and adjusting instance acceptance status.

[0125] The above content is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for aggregation, scheduling, optimization, and matching of tokens in a large multi-type model, characterized in that, include: S1. Establish an inference session and read the request type, model version constraints, token sequence, maximum output boundary, cache reuse permission, and instance execution record; S2. Based on the Token sequence and the continuous valid cache block chain, register the prefix cache resident Token, the Token to be prefilled, and the reserved continuous decoding Token; S3. Filter candidate model instances based on model version, tokenizer version, key-value cache configuration, tenant isolation flag, and instance service status; S4. For candidate model instances, register three types of token resource usage, resource security margin, cache residency security status, and batch processing stable status during the current observation period, pre-filling retention period, and decoding reserved period. S5. When pre-filling commitment, decoding reservation, cache residency security and batch processing stability are all registered as passed, the request will be aggregated and distributed according to model version, tokenizer version, tenant isolation flag and decoding service constraints; during continuous decoding, the resource reservation and instance acceptance status will be updated according to the newly added continuous decoding token, session binding cache block usage and instance running records.

2. The aggregation, scheduling, optimization, and matching method for multi-type large-scale model tokens according to claim 1, characterized in that, In S1: The maximum output boundary is registered based on the maximum output length field in the current inference request; If the maximum output length field is missing, it is registered according to the default output boundary corresponding to the request category; When the maximum output length field exceeds the maximum context length of the target model, context verification is performed based on the total number of input tokens and the maximum output length field. If the context verification fails, a restricted output boundary is registered.

3. The aggregation, scheduling, and optimal matching method for multi-type large-scale model tokens according to claim 1, characterized in that, In S2: The cache block chain is verified based on the model version, tokenizer version, attention configuration, location encoding configuration, key-value cache precision, key-value cache block size, inference engine cache layout, tenant isolation marker, cache block integrity status, and cache expiration time. When the verification is successful and the cached blockchain continuously covers the input token position range, the corresponding position range will be registered as the prefix cache resident token; When a token position is missing, the missing position and its subsequent positions are registered as tokens to be pre-filled.

4. The aggregation, scheduling, optimization, and matching method for multi-type large-scale model tokens according to claim 1, characterized in that, In S2: The number of reserved continuous decoding tokens is registered by deducting the number of generated continuous decoding tokens from the maximum output boundary. If the current inference session has not entered the continuous decoding stage, the number of generated continuous decoding tokens will be recorded as zero; Upon receiving the end marker, the number of reserved continuous decoding tokens will be recorded as zero.

5. The aggregation, scheduling, and optimal matching method for multi-type large-scale model tokens according to claim 1, characterized in that, In S3: Candidate model instances are filtered based on the capability configuration records corresponding to the request category, the total number of input tokens and the context occupancy length formed by the maximum output boundary, and model instances with matching capability configurations, context occupancy lengths not exceeding the maximum context length, online instance service status and complete instance operation records within the current observation period are added to the candidate model instance list.

6. The aggregation, scheduling, optimization, and matching method for multi-type large-scale model tokens according to claim 1, characterized in that, In S4: Establish a Token state resource ledger with the current observation period as the smallest time unit, and write in the Token resource entries to be pre-filled, the prefix cache resident Token resource entries, the reserved continuous decoding Token resource entries, the existing session resource commitment entries, and the releasable resource entries respectively; The token resource entries to be pre-populated cover the current observation period and the pre-population holding period, while the token resource entries reserved for continuous decoding cover the reserved decoding period.

7. The aggregation, scheduling, and optimal matching method for multi-type large-scale model tokens according to claim 6, characterized in that, Resource safety margin is registered based on the allocable key-value cache capacity, allocated key-value cache capacity, token resource entries to be prefilled, prefix cache resident token resource entries, reserved continuous decoding token resource entries, existing session resource commitment entries, and releasable resource entries. When resource safety margin boundaries are maintained in the current observation period, prefill retention period, and decoding reserved period, the corresponding resource safety margin is registered as passed.

8. The aggregation, scheduling, and optimal matching method for multi-type large-scale model tokens according to claim 6, characterized in that, The cache-resident security status is registered based on prefix cache-resident Token resource entries, session-bound cache resource entries, newly added cache resource entries for tokens to be pre-filled, and newly added cache resource entries for reserved continuous decoding Tokens. When resource usage does not exceed the cache residency capacity boundary and the resource safety margin boundary is reserved, the cache residency safety status is registered as passed; when the number of batch splits is less than twice in three adjacent current observation periods, the cache block application waiting time does not exceed the waiting boundary, and the token output interval does not exceed the stable output interval boundary, the batch processing stable status is registered as passed.

9. The aggregation, scheduling, and optimal matching method for multi-type large-scale model tokens according to claim 1, characterized in that, In S5: For candidate model instances in the set of admissible instances, the target model instance is determined sequentially based on the continuous coverage range of the prefix cache resident token, the resource safety margin during the decoding reserved period, the token resource entries to be pre-filled, the high retention priority cache blockchain usage, the current number of continuous decoding sessions, and the number of batch splits. During continuous decoding, when the number of continuously generated continuous decoding tokens reaches 80% of the reserved continuous decoding tokens, subsequent observation period resource entries are supplemented according to the continuous decoding token expansion unit. If the supplementation fails, the instance acceptance status of the target model instance is changed to restricted acceptance.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the aggregation, scheduling, optimization, and matching method for multi-type large model tokens as described in claim 1.

Images