A large model streaming usage charging and token quota synchronous management method

Abstract

The application discloses a large model streaming usage charging and token quota synchronous management method, relates to the technical field of cloud computing resource charging, and is characterized in that: when a session is accessed, a quota lease with a version number and a valid period is granted according to an available quota of an account, and a session credit window is established in the quota lease; in a streaming segment return process, the session credit window is deducted according to a first-arrival condition of a token threshold and a time threshold, intra-lease renewal is performed, and a cancellation frame carrying cumulative usage, a metering bit, a control bit, a lease version number and a pricing category is generated; a central quota service maintains a central maximum water level according to a session identifier and a pricing category, and performs a maximum combination on repeated, out-of-order or late cancellation frames; a terminal state frame or a reconstruction frame is generated when a session ends or is abnormally disconnected, and an initial credit window, a cancellation window and a cutting-grant parameter are written back according to a settlement residual error. The method can reduce the overhead of token-by-token remote verification, reduce quota virtual occupation, and improve the consistency level of streaming usage settlement.

Description

Technical Field

[0001] This invention relates to the field of cloud computing resource billing technology, specifically a method for large-scale streaming usage billing and token quota synchronization management. Background Technology

[0002] As large-scale models are introduced into scenarios such as office assistants, customer service, code generation, and contract review via API services, callers configure a limit for an account, and then multiple business systems concurrently initiate streaming generation requests. Streaming output only retrieves the prompt text, model type, and maximum generation parameters upon access; the final number of tokens depends on the context, stopping conditions, and user disconnection. Currently, the common process involves authentication, rate limiting, quota judgment, and call forwarding at the gateway layer. After the response is completed, an accounting record is generated based on actual usage. Rate control often uses requests per minute, tokens per minute, fixed window counters, or token buckets to control the frequency. Related materials also summarize the pain points in this field as follows: when the generation length is unpredictable and multiple concurrent sessions share the same account limit, there is a coupling relationship between the limit verification overhead, limit usage deviation, and settlement consistency after termination.

[0003] International publication WO2015120460A1 discloses a system and method for API rate limiting and distribution. The system consists of an application programming interface (API) proxy system that receives client requests, constructs proxy requests based on tenant internal authentication information, and sends the proxy requests to the interface endpoint corresponding to the store or tenant resource. After the endpoint accesses or modifies its local database, it returns a response to the proxy system, which then returns the response and time information to the client. This proxy system includes an authentication service, authentication cache, API proxy engine, and API rate limiting engine. It can run in server clusters and load balancing environments and can calculate the allowed request quota for each client within the current time window based on the target resource's configuration limits, the number of active clients in the current time window, and an adjustable scaling factor. When the number of client requests exceeds the limit, it returns a rate limiting response or a waiting time to the proxy system; otherwise, the proxy system prepares to forward the proxy requests to the interface endpoint. The document also discloses that the quota can vary with the time window, the number of active clients, and the scaling factor, and can be implemented using an exponential distribution function to address the significant resource pressure under sudden surges in access.

[0004] The aforementioned existing technologies can handle access rate limiting for interface requests, but they still have some limitations in large-scale streaming generation billing scenarios. Firstly, traditional rate limiting uses "requests" as the control object, and the quota within the time window can be determined when the request arrives. However, the usage of a streaming generation session is continuously generated during the response process, and the prompt text, maximum generation parameters, and actual completed token count are not equivalent. Existing cloud-based large-scale model services also reflect this difference; for example, the tokens allocated per minute during deployment are mapped to request rate limiting, but the estimated maximum number of processing tokens when the request is received does not equal the billing amount after processing. Secondly, traditional proxy rate limiting focuses more on whether to allow access, whether to wait, and whether to return a rate limiting response, lacking unified handling of segmented output, connection interruptions, backend exceptions, duplicate reporting, and out-of-order arrival within the streaming response. If each token is synchronously verified, the number of remote verifications increases with the generation length; if the quota is pre-allocated based on the maximum generation parameters, short replies and user interruptions will result in temporary occupancy. If billing is only completed after the response is finished, usage confirmation delays are likely to occur when abnormal disconnections occur.

[0005] Therefore, the technical problem that needs to be solved in this field is: how to achieve unified management of low-latency access control, process usage constraints and billing consistency after termination under the conditions of uncertain generation length, multiple sessions sharing account quota and complex session termination methods in the scenario of large model streaming output. Summary of the Invention

[0006] (a) Technical problems to be solved

[0007] To address the shortcomings of existing technologies, this invention provides a method for synchronizing streaming usage billing and token quota management in a large-scale model. During the streaming segment return process, the session credit window is deducted according to the priority conditions of token thresholds and time thresholds, lease renewal is performed, and a revocation frame carrying cumulative usage, metering bits, control bits, lease version number, and pricing category is generated. The central quota service maintains the highest central water level according to session identifier and pricing category, and performs a maximum merging of duplicate, out-of-order, or late revocation frames. When a session ends or abnormal connection is lost, a final state frame or a reconstruction frame is generated, and the initial credit window, revocation window, and revocation parameters are written back based on the settlement residual. This method can reduce the overhead of per-token remote verification, reduce quota occupancy, and improve the consistency level of streaming usage billing; it solves the technical problems described in the background art.

[0008] (II) Technical Solution

[0009] To achieve the above objectives, the present invention provides the following technical solution:

[0010] A method for managing streaming usage billing and token limit synchronization in a large-scale model, executed by a limit management system, includes: receiving streaming generation requests, granting a limit lease with a lease version number and validity period according to the account's available limit, and establishing a session credit window containing a session identifier and a pricing category within the limit lease;

[0011] During the fragment return period, the session credit window is deducted according to the priority conditions of the token threshold and the time threshold. If the balance is lower than the token threshold, the credit lease is renewed. If the balance is not lower than the token threshold, the credit lease is not renewed. If the balance is insufficient, the boundary termination code is output and a revocation frame containing the cumulative usage, final state bit, meter bit, control bit and lease version number is generated.

[0012] Upon receiving a cancellation frame, maintain the center's highest water level according to the session identifier and pricing category. When the cumulative usage exceeds the center's highest water level, update the confirmed usage based on the difference and cancel the unclosed lease amount. If it is not higher, do not update.

[0013] A final state frame is generated when the session ends. If no final state frame is received after the orphan determination time, a reconstruction frame is generated. All frames are merged according to the highest water level in the center.

[0014] Based on the merge result, a write-back message carrying the parameter set version number and lease epoch number is generated, and the initial credit window, revocation window and grant parameters for subsequent sessions are adjusted.

[0015] Furthermore, the credit limit lease includes account identifier, processing unit identifier, lease version number, validity period, lease balance, and word segmenter version; the session credit window includes session identifier, window number, pricing category, input cumulative amount, output cumulative amount, and billable cumulative amount;

[0016] The quota management system maintains the highest water level in the center according to the session identifier and the pricing category, so that input tokens, output tokens and cache hit tokens are collected separately.

[0017] Furthermore, the quota management system receives the engine increment token count returned by the inference engine along with the streaming fragments; when the engine increment token count is complete and valid, it writes the engine increment token count into the output cumulative amount;

[0018] When the engine incremental token count is missing, the shadow segmenter with the same version as the segmenter is invoked, and the supplementary incremental token count is obtained by combining the segmentation tail cache formed at the end of the previous streaming segment. The supplementary incremental token count and the caliber tag are written into the verification frame.

[0019] Furthermore, when the quota management system processes the cancellation frame, it first locates the central session status according to the session identifier, pricing category, and lease version number; when the cumulative usage is higher than the central maximum level, it updates the confirmed usage according to the difference between the two, and deducts the corresponding lease unclosed amount when the control bit is valid;

[0020] When the cumulative usage is not higher than the highest water level in the center, the confirmed usage will not be updated and the amount of leases not closed will not be deducted.

[0021] Furthermore, when the central credit service is unavailable and the remaining amount of the local valid credit lease covers the session credit window, the credit management system continues to open the window within the credit lease and writes an autonomous time period marker in the session context;

[0022] When the available local credit limit lease balance is insufficient, the credit limit management system rejects new streaming generation requests.

[0023] When a temporary trial lease is pre-issued by the central credit limit service, the credit limit management system will only establish a minimum session credit window within the upper limit and validity period of the temporary trial lease.

[0024] Furthermore, when the write-ahead log is successfully written, the quota management system will add the revocation frame to the sending batch;

[0025] When the write-ahead log fails to write, the credit management system stops batch packet assembly and synchronously sends the current reimbursement frame to the central credit service; when the sending channel is congested, the credit management system writes a hysteresis flag in the session context and reduces the re-granting speed of the session credit window; when the sending channel recovers, the credit management system continues to deliver according to the window sequence number in the write-ahead log.

[0026] Furthermore, when no final state frame is received and no new reimbursement frame is received within the orphan determination time, the quota management system reads the highest water level of the pre-written log and the highest water level of the engine session log, respectively.

[0027] When the window sequence number, lease version number, and fragment boundary are all continuous, the higher water level is selected to generate the reconstruction frame; when the window sequence number, lease version number, and fragment boundary are not continuous, the lower water level is selected to generate the reconstruction frame; when log corruption prevents the water level from being read, the session is frozen at the highest recorded center water level.

[0028] Furthermore, the quota management system configures the processing unit that holds quota leases and generates reimbursement frames as at least one of the following: gateway node, inference engine instance, companion process, and stand-alone worker process;

[0029] The processing unit stores the session identifier, lease version number, and pricing category in the same session context; the reversal frames generated by different processing units all enter the same central highest water level merging path.

[0030] Furthermore, the credit management system will send the revocation frame to the central credit service via at least one of the following: message bus, batch remote call interface, and heartbeat piggyback message;

[0031] The central quota service does not use the sending channel type and arrival order as the conditions for billing; when the same reimbursement frame arrives repeatedly, the central quota service will not update based on the highest central quota level; when a reimbursement frame with a higher cumulative usage arrives later, the central quota service will update based on the difference based on the highest central quota level.

[0032] Furthermore, the credit limit management system configures an input token processing strategy table for the account, which includes an immediate cancellation flag and a final state aggregation flag.

[0033] When the immediate verification flag is valid, the input token quantity obtained by the server-side tokenizer is written into the first verification frame when the session is accessed; when the final state aggregation flag is valid, the input token quantity is temporarily stored in the session context and carried separately from the output cumulative quantity in the final state frame.

[0034] Furthermore, the lease grant message generated by the credit management system includes account identifier, processing unit identifier, lease version number, validity period, and lease balance; the write-off frame includes session identifier, window sequence number, cumulative usage, final state bit, meter bit, control bit, lease version number, and pricing category; the write-back message includes parameter set version number, lease epoch number, initial credit window, write-off window, grant parameters, effective time, and rollback threshold.

[0035] Furthermore, the credit management system outputs a session status flag in at least one location in the interface response, management interface, and file log; the session status flag includes in progress, settlement in progress, settled, orphan session, and rebuilt; when the credit management system terminates the output due to insufficient session credit window, it outputs the credit limit termination code; when the credit management system completes the freeze based on the rebuilt frame, it outputs the rebuilt flag and the corresponding session identifier.

[0036] Furthermore, the credit management system outputs the account identifier, session identifier, pricing category, cumulative usage, confirmed usage, center maximum water level, lease version number, and session status flag in the accounting receipt;

[0037] When the session status is marked as settled, the billing receipt carries the final state frame identifier; when the session status is marked as rebuilt, the billing receipt carries the rebuilt frame identifier; when the session status is marked as an orphan session, the billing receipt carries the trigger record corresponding to the orphan determination duration.

[0038] (III) Beneficial Effects

[0039] This invention provides a method for synchronizing large-scale streaming usage billing and token limit management, which has the following beneficial effects:

[0040] By setting up a two-tiered credit limit structure of credit limit lease and session credit window, the account credit limit is no longer reserved all at once according to the maximum generation length, but is gradually occupied on the processing unit side according to the actual progress of the session; the credit limit lease carries the account-level balance boundary and the session credit window carries the single session output boundary. The two work together to reduce the situation of credit limit being occupied in vain after a short session ends, and also avoids the central credit limit verification being triggered by each output token.

[0041] By generating local cancellation at the first window, credit window deduction, lease renewal, and cancellation frames, streaming segments are merged into window-level metering events during the generation process. The cancellation frame carries cumulative usage, final state bit, metering bit, control bit, lease version number, and pricing category, ensuring that metering in the stream, lease cancellation, and subsequent settlement are transmitted along the same data carrier, avoiding duplicate processing caused by out-of-order incremental call detail records.

[0042] By employing a central highest water level merging mechanism, duplicate, out-of-order, and delayed cancellation frames are all processed according to the same cumulative water level rule. When the cumulative usage exceeds the central highest water level, only the difference is updated for confirmed usage. When the cumulative usage is not higher than the central highest water level, duplicate entries are not made, ensuring that account usage and lease closure amounts remain consistent at least once in the delivery channel. Through isomorphic merging of final state frames, orphan session determination, and reconstruction frames, normal completion, connection disconnection, processing unit failure, inference engine failure, and timeout termination all enter the same settlement path.

[0043] The reconstructed frame adopts the cumulative water level semantics of the write-off frame, avoiding the need to set up a separate compensation branch for abnormal recovery and reducing the risk of duplicate or missed settlements due to final state loss. Pre-occupancy error, truncation event, lease lifetime, synchronization lag, reconstruction deviation, and peak in-transit exposure are transformed into operational statistics, which are then written back to the initial credit window, write-off window, and grant parameters by the water level management chain. This enables the aforementioned static credit limit control to adjust according to changes in account water level and generation length distribution. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the overall architecture of the large-scale streaming usage billing and token limit synchronization management method of the present invention;

[0045] Figure 2 This is a flowchart illustrating the process of credit limit lease granting and session credit window opening in this invention.

[0046] Figure 3 This is a flowchart illustrating the process of stream segment window cancellation and cancellation frame generation in this invention.

[0047] Figure 4 This is a schematic diagram of the asynchronous reporting of cancellation frames and the process of merging the highest central water level in this invention;

[0048] Figure 5 This is a schematic diagram of the state transition process in the session finality settlement and orphan session reconstruction process of this invention;

[0049] Figure 6 This is a closed-loop diagram illustrating the water level management and parameter write-back process based on operational statistics in this invention.

[0050] Figure 7 This is a schematic diagram illustrating the field transmission relationships between the quota lease, session context, revocation frame, and write-back message in this invention. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] Please see Figures 1-7 This invention provides a method for managing streaming usage billing and token quota synchronization for large models, comprising: Step 1, before the final length of the large model streaming generation is determined, the account quota is allocated to a processing unit close to the streaming output path, forming a session credit window that is reproducible, reclaimable, and can be called by subsequent windows for resale without blocking the first output segment. The account quota is split into two parts: a quota lease and a session credit window. The central quota service only handles coarser-grained quota leases; the gateway processing unit only opens windows for a single session within the local quota lease; and the inference engine accepts the already approved context.

[0053] When multiple streaming queries are initiated simultaneously from a shared account, the number of output tokens for a single session is uncertain at the time of access. If each session queries the central quota service for quota token by token, the initial response is subject to remote call control; if the quota is pre-deducted in full according to the maximum output length, a large amount of temporarily unavailable quota will be left after the short session ends.

[0054] Through a two-tiered structure of credit limit leases and session credit windows, the central-side fund consistency and local-side low-latency release are separated: the central credit limit service confirms the available credit limit of an account and grants it a credit limit lease. The gateway processing unit then allocates a smaller initial credit window within this credit limit lease. Then, the session credit window is occupied according to window consumption rather than according to worst-case length.

[0055] The following processing is performed collaboratively by the gateway processing unit, the central credit limit service, and the inference engine. After the client sends a streaming generation request, the gateway processing unit first parses the account identifier, model identifier, generation parameters, prompt text, and tokenizer version, and then reads the local credit limit lease table. If the local lease balance is sufficient to cover the initial credit window... The gateway processing unit directly generates the session context and hands the request over to the inference engine; if the local lease availability is insufficient, the gateway processing unit carries the number of active flows for this unit. Recent consumption rate and local lease balance Request a transfer from the central credit limit service. The central credit limit service does not care how many tokens are ultimately generated in this session; it only verifies the credit limit boundary that the account can still be transferred and returns a credit limit lease with a version identifier and validity period.

[0056] To avoid discrepancies between permission checks and measurement standards across different components, the session context is generated with an account identifier, session identifier, lease version number, and tokenizer version. Subsequent frames do not reinterpret account ownership; they simply continue accumulating along this context. Step one does not yield a semantic container for clearing order funds, but rather funds that can be continuously reconciled in subsequent steps.

[0057] When the gateway processing unit initiates a handover request, it will include the account identifier, node identifier, request volume, and number of active flows. Recent consumption rate Tokenizer version and local lease balance The message is encapsulated as a sharing authorization message. After receiving the message, the central quota service first refreshes the available quota at the account level. Then, the amount of quota sliced ​​is determined according to the demand weight. This slice size is not equivalent to a single session credit limit, but rather a pre-authorized balance shared by multiple active sessions within the same processing unit. Therefore, its granularity is larger than the session credit window but smaller than the total account credit limit.

[0058]

[0059] Where: Lease open amount This represents the outstanding balance of credit service leases issued by the central authority but not yet closed due to write-off, final rollback, or expiration. The value is not less than 0, and its purpose is to limit the credit limit that the processing unit can continue to autonomously consume. (Total number of leases issued) This indicates the total amount of quota leases granted to each processing unit in the central lease register; the amount of leases that have been closed. This indicates the amount of credit leases that have been released after step three (center merging) and step four (final state rollback or lease expiration recovery).

[0060]

[0061] Where: Account available credit limit This indicates the remaining balance of the central quota service that can still be allocated, with a value not less than 0. It is used to constrain the upper limit of this allocation and serves as the input for the account water level ratio in step five; total account quota. This indicates the total token limit that the account has purchased or configured, derived from the account limit ledger.

[0062] Account usage confirmed This represents the amount of chargeable amount after confirmation of the highest water level at the center in step three or four; maximum function. This indicates that the larger value is selected from the candidate values ​​to avoid negative values ​​in the available account balance.

[0063]

[0064] Where: Original cutting quantity This represents the number of candidate leases calculated based on the account's available credit, allocation ratio, and active traffic; allocation ratio This indicates the percentage of the available credit limit that can be granted to a single processing unit, preferably between 0.5% and 20%; available credit limit. Using the aforementioned definition; demand amplification factor This represents a short-term compensation for active flow counts, preferably 1 to 2; active flow counts This indicates the number of sessions being generated or awaiting cancellation for the same account in the current processing unit, with a value not less than 1; maximum function. Used to provide a non-zero denominator when the active stream count has not yet been refreshed.

[0065]

[0066] Quota slice volume This indicates that the credit limit lease amount granted to the processing unit this time will neither exceed the account's available credit limit nor be lower than the lower limit required to meet the short-term opening window of the waiting queue; cancellation window This indicates the local revocation token threshold for step two; number of pending sessions. This represents the number of sessions waiting to open a window or renew the license, derived from the processing unit's wait queue; minimum function. With the maximum function Together, they form an execution order that prioritizes candidates and then is constrained by account balance, preventing the lower limit rule from exceeding the actual available balance in scenarios with small balances.

[0067] For example, when a corporate account simultaneously requests long text generation in multiple Q&A windows, the first gateway processing unit already has several unfinished sessions. The central quota service assigns a leased version number to the gateway processing unit based on the number of active flows reported and its local speed. The quota lease. In this case, the gateway processing unit only updates the local lease table and does not wait for the inference engine to send back any output tokens.

[0068] In cases of low balances, if existing sessions with open windows are consecutive, the central credit limit service will prioritize newly added session credit limits. If the account has available credit limits... If only a small number of windows can be covered, then the quota slicing is based on the number of active flows. Compressed quota slice volume This prevents a single processing unit from monopolizing the remaining credit limit. Consequently, the central credit limit service handles the allocation less frequently. This ensures that the gateway processing unit has sufficient credit to meet the balance generated in a short period, while the account level still relies on the available credit limit. The restrictions do not extend indefinitely due to simultaneous requests from multiple nodes.

[0069] When the lease balance in the local credit lease schedule covers the initial credit window At that time, the gateway processing unit allocates session credit windows from the remaining lease amount. And generate a globally unique session identifier. Window number Input cumulative amount Output cumulative amount Billable cumulative amount Lease version number and pricing categories Among them, window number Initialize to 0, input cumulative amount The source text is the prompt text processed by the server-side word segmenter according to the word segmenter version. The input token count is obtained, and the output cumulative count is obtained. With a session starting point of 0, the billable cumulative amount Increment after conversion according to pricing category, pricing category Distinguish between three categories: input, output, and buffer hit.

[0070] The gateway processing unit first completes the input accumulation in the first frame. Local verification is then performed, followed by sending a message carrying the session credit window to the inference engine. In this context, the inference engine does not participate in determining the account limit.

[0071]

[0072] In the formula: Conversation Credit Window This indicates that the credit limit window is available for the first time in this session for output-side streaming generation; initial credit window. This indicates the base window size after system configuration or write-back in step five; maximum generation parameter. Indicates the output limit declared in the request; local lease slack. Indicates the outstanding balance of credit leases that have not yet been allocated to the current processing unit; input the cumulative amount. This represents the number of input tokens that the prompt text has been calculated by the server-side tokenizer and verified by the first frame; the maximum function. This formula is used to prevent negative values ​​in the output-side credit window when the cumulative input amount approaches the local lease balance. It ensures that input occupancy, request cap, and lease balance are uniformly constrained from the start of the session.

[0073] For example, when a user submits a contract summary request, the gateway processing unit first uses the server-side tokenizer to calculate the input token count of the prompt text, and then writes the tokenizer version into the same session context. If the local credit limit is sufficient, the gateway processing unit allocates a session credit window. The request is then immediately handed over to the inference engine, and the caller receives the first stream of output; if the input token is already close to the initial credit window, the request is then processed. The gateway processing unit still uses the session credit window. The upper bound is allowed, and subsequent windowing is handled by step two. In actual deployment, the gateway processing unit runs on the access layer service, and the tokenizer and inference engine use the same vocabulary file; when the two versions are inconsistent, the session context retains the tokenizer version. Subsequent settlements will be aggregated according to this version's standards, without temporarily changing the standards during the process.

[0074] Furthermore, the usage on the prompt side is included in the same cumulative caliber at the beginning of the session, while the usage on the output side is handled by the session credit window, so that the input and output do not form two separate credit paths.

[0075] When the central credit line service is unavailable, but the remaining amount of the local valid credit lease still covers the initial credit window. When the gateway processing unit enters the autonomous window within the lease, it only allows the session within the existing credit limit lease and marks the autonomous period in the session context; when the remaining local effective credit limit lease amount is less than the preset minimum credit window When the gateway processing unit returns an insufficient credit code, it will not generate a session credit window for the new session; when a temporary trial lease issued by the central credit service already exists in the system, the gateway processing unit will only draw the minimum credit window within the upper limit and validity period of the temporary trial lease. Minimum credit window It does not represent an overdraft limit without a lease, but rather the minimum session credit window drawn from the valid lease or temporary trial lease.

[0076] This boundary processing ensures that granted quota leases remain usable under weak network conditions, while preventing the processing unit from allowing new streaming sessions when there is absolutely no authorized balance. When a grant request times out, the gateway processing unit exponentially backs up the grant request for retrying, with the waiting time determined by the initial waiting time, the growth factor, the maximum waiting time, and the disturbance time.

[0077]

[0078] In the formula: the first Waiting time for each retry Indicates the first Waiting time before making another request after a timeout; maximum waiting time. This indicates the maximum retry waiting time, preferably no more than one-third of the lease term; initial waiting time. Indicates the waiting time before the first retry; growth coefficient The retreat growth multiple is indicated, preferably between 1.5 and 2; the disturbance duration is also considered. This represents a non-negative perturbation obtained by hash mapping of node identifier and retry sequence number. Its purpose is to reduce the sharing congestion caused by multiple processing units retrying at the same time.

[0079] In parallel embodiments, the holder of the credit lease is not limited to the gateway processing unit. For systems where the inference engine and counter are deployed in the same process, the inference engine instance directly holds the credit lease, and the gateway processing unit only transmits the session context. For systems where multiple model services share a gateway, the accompanying process holds the credit lease and distributes session credit windows to multiple inference engines. For private single-machine deployments, the credit lease degenerates into a sharded balance within the single-machine worker process, and the central credit service is handled by the local credit management process. The same terminology remains unchanged: the credit lease is always a pre-authorized credit with a version identifier, validity period, and reclaimable balance, and the session credit window is always a consumable window allocated from the credit lease to a single streaming session.

[0080] For example, when the external billing link is disconnected from the enterprise intranet but the local credit limit management process still has unexpired credit limit leases, the accompanying process continues to open small session credit windows for authenticated accounts and sets the session identifier for each session. Lease version number and word segmenter version Write to the local context. After the link is restored, subsequent verification frames can still enter the central merging path according to the same version.

[0081] Furthermore, the data plane under weak network conditions will not completely stop due to the short-term unavailability of the center, and all autonomous windows are limited by the remaining quota lease, so they will not be separated from the subsequent large-scale merging and final settlement system.

[0082] Step 2: During the continuous output of streaming fragments by the inference engine, the generated output tokens are converted into local verification results and asynchronously reported verification frames according to the first-to-reach window. This prevents the session credit window formed in Step 1 from being slowed down by the successive remote verification of individual tokens, and provides a water level input with cumulative semantics for the maximum merging in Step 3.

[0083] In the large-scale streaming output service, the caller sees segments of text arriving in sequence, while account credit consumption occurs on the token sequence behind these segments. If the processing unit reports to the central credit service every time it receives a token, the number of communications increases along with the output length. If settlement is only done once after the session ends, long sessions will continuously occupy the session credit window without mid-way convergence. Step two takes the session context, session credit window, lease version number, and tokenizer version output from step one. Without changing the inference engine's generation rhythm, it aggregates several output tokens into a local reimbursement action and encapsulates this action into subsequent repeatable, out-of-order, and replayable reimbursement frames.

[0084] This step is executed collaboratively by the processing unit and the inference engine. For each streaming segment generated, the inference engine returns the segment text, the engine incremental token count, the segment end marker, and the current session identifier to the processing unit. The processing unit reads the window sequence number, input cumulative amount, output cumulative amount, billable cumulative amount, session credit window balance, and lease version number from the session context. It first determines whether the current segment meets the redemption trigger condition, and then performs credit window deduction, lease renewal, and redemption frame generation. This processing logic does not require the central credit limit service to participate in the hot path; the central credit limit service only updates the account level after receiving the batch redemption frames in step three.

[0085] In the preferred deployment, the processing unit is located at the access layer or next to the inference node. The inference engine sends streaming segments through in-process queues, shared memory queues, or local loopback channels; message queues with the same delivery function can also be used. The processing unit uses a monotonic clock instead of the system wall clock to calculate the write-off interval. This avoids clock rollback affecting window judgment. Therefore, step two is located between the generation path and the settlement path, which is close to the token generation end to obtain the accurate fragment usage, and postpones the synchronization action to the window boundary to avoid each token carrying the fund status.

[0086] The processing unit maintains the number of tokens to be revoked, the last revocation time, and the caliber tag in the session context. When the inference engine returns the engine incremental token count, the processing unit uses this value as the main count for this segment. If the inference engine fails to return the engine incremental token count due to abnormal restart, segment splicing, or cross-model proxy, the processing unit calls the shadow segmenter corresponding to the same segmenter version as in step one to perform supplementary segmentation on the segment text and writes the fallback source in the caliber tag. The triggering condition adopts a first-come-first-served logic of token threshold and time threshold, so that long segments will not delay revocation for a long time, and short segments will not amplify the synchronization overhead due to frequent messages.

[0087]

[0088] Where: number of fragment increment tokens It is in the The number of output tokens added within each write-off judgment period; engine incremental token count The count comes from the inference engine's return of streaming segments; the number of incremental tokens from shadow segmentation. Used by the processing unit and the word segmenter version The same shadow segmenter was used for calculation; engine count valid flag. This indicates whether the engine incremental token count is complete and valid; if it is complete and valid, it is 1; otherwise, it is 0.

[0089]

[0090] In the formula: word segmentation mapping Indicates based on the word segmenter version Tokenize the text sequence; buffer the token segmentation tail. This represents the character or byte sequence retained at the end of the previous streaming segment; its length is determined by the maximum span after merging by the server-side tokenizer. Current streaming segment. This represents the text fragment output by the inference engine; concatenation symbol. This indicates that the token end cache is concatenated with the beginning and end of the current streaming segment. The processing unit removes the segmentation result portion that has already been confirmed by the previous verification frame, and only adds the incremental token count of the newly added portion to the segment. This is to avoid double counting or omission at the boundary of a segment.

[0091]

[0092] Where: trigger flag Indicates the first The next segment to be judged determines whether a verification frame is generated, with a value of 0 or 1. Its function is to unify token triggering and time triggering into the same gating result; indicator function. The OR operator represents a value of 1 if the condition within the parentheses is true and 0 if it is false. Its function is to convert conditional statements into programmable gating variables; it is a logical OR symbol. This means that it can be triggered as long as any condition is met, and its function is to make the token boundary and the time boundary share the same trigger entry point;

[0093] Fragment Increment Token Count This represents the cumulative number of output tokens since the last reconciliation but which have not yet formed a reconciliation frame. The value is not less than 0, and its function is to receive incremental tokens from the engine or supplementary counts from the shadow segmenter; Reconciliation window This represents the token threshold required to complete a local write-off, preferably ranging from 8 to 2048, and more preferably from 64 to 256, used to create an adjustable boundary between write-off sensitivity and message count.

[0094] Current moment The processing unit receives the first The local monotonic clock reading for each segment increments as the segment arrives, its purpose being to prevent low-speed output from failing to clear for extended periods; the last clearing time... This represents the local monotonic clock reading before the previous write-off frame was written, and its function is to correlate it with the current segment time. Forming a time interval.

[0095] Verification interval This indicates the longest allowed verification interval, preferably 50 milliseconds to 1 second, and more preferably 100 milliseconds to 300 milliseconds, to cover short responses, low-speed models, and network jitter scenarios.

[0096] For example, the caller requests the generation of a contract terms summary, and the inference engine returns several short streaming fragments in succession. The output tokens of the first two short streaming fragments have not yet reached the verification window. The processing unit only accumulates the fragment increment token count. Continue allowing passage; although there is still not much new text when the third segment arrives, the current segment time... Interval with the last write-off The write-off interval has been reached. The processing unit immediately proceeds to local verification.

[0097] This enables fast-generated sessions to be revoked based on token boundaries, while slow-generated sessions are revoked based on time boundaries, both triggered by the same flag. Next, proceed to the next step.

[0098] As a preferred implementation, the processing unit retains both the engine incremental token count and the shadow segmenter result in the caliber tag of the session context, but only incorporates the selected count into the fragment incremental token count. When the two are inconsistent, subsequent frames continue to accumulate along the same caliber, without switching the segmentation caliber midway through the same session. This approach ensures that the session accumulation output in step two has a continuous baseline and provides a recalcible source for the final state settlement in step four.

[0099] When the flag is triggered When the value is 1, the processing unit increments the token count by the fragment increment. The remaining balance in the session credit window will be deducted, and the accumulated session balance will be increased simultaneously. If the remaining balance in the session credit window after deduction is lower than the redemption window... The processing unit does not access the central quota service, but allocates the renewal quota within the quota lease already granted in step one. If the local lease balance is insufficient to form the renewal quota, the processing unit issues the remaining budget and boundary policy to the inference engine based on the session credit window balance, causing the inference engine to stop outputting at word boundaries, sentence boundaries, or paragraph boundaries, and writes the quota boundary termination code in the session context. The subsequent budgeting process is handled by step five.

[0100]

[0101] In the formula: Continuous supply quantity Indicates the first The token limit transferred from the credit lease to the session credit window during the next local renewal must have a value no less than 0; cancellation window Following the aforementioned definition, its function is to set the basic metric for renewing grants; local lease surplus. Indicates the first The remaining credit lease balance not yet allocated to the Session Credit Window prior to the next renewal; maximum generation parameters. Use the output limit declared in step one's request; output cumulative amount This indicates that the current session ends on [date]. Cumulative output token count after each verification; Maximum function This formula is used to reset the renewal amount to zero when the cumulative output amount reaches the maximum generation parameter. The renewal is simultaneously subject to the reimbursement window, local quota lease, and request output boundary constraints.

[0102] For example, a corporate account is running multiple copy generation sessions simultaneously, one of which continuously outputs long paragraphs. After several reconciliations, the processing unit finds that the session's credit window balance has fallen below the reconciliation window. Therefore, from the local lease surplus Continued grant quotas drawn from the middle The renewal amount is written back to the session context; another session under the same account is close to the limit boundary, and the processing unit cannot get enough renewal amount after deduction, so it sends a stop flag to the inference engine at the end of the current segment. The caller receives the text that has been segmented instead of the segment that has been truncated within the word.

[0103] This fixes the credit limit boundary processing within the fragment boundary and the session credit window, neither increasing the pressure on the central credit limit service nor causing the already generated fragments to lose their measurement basis. For ease of implementation, the processing unit synchronously saves the renewal source, the remaining balance before renewal, the remaining balance after renewal, and the credit limit boundary termination code within the session context. The renewal source points to the lease version number, the remaining balance before and after renewal is used to offset the lease in transit in step three, and the credit limit boundary termination code is used by the caller to distinguish between normal completion and the end of the credit limit boundary. This combination of fields does not change the generated content; it only limits how the generated content is reclaimed and how further interaction continues.

[0104] After credit window deduction is completed, the processing unit generates a reversal frame. The reversal frame includes at least the session identifier, window number, input cumulative amount, output cumulative amount, billable cumulative amount, final state bit, meter bit, control bit, lease version number, pricing category, and caliber flag. Specifically, the window number monotonically increases with the session, the input cumulative amount remains constant along the pricing category after the first frame, the output cumulative amount increases with each streaming segment, the billable cumulative amount is formed according to the pricing discount factor, the final state bit remains 0 in this step, the meter bit indicates that the frame carries billable usage, and the control bit indicates that the frame also participates in lease reversal.

[0105] The processing unit first appends the revocation frame to the write-ahead log and then puts it into the sending batch. The write-ahead log preferably uses an append file, embedded key-value storage, or sequential log segments. The group commit trigger condition is a fixed time slice or a fixed number of frames arriving first.

[0106]

[0107] Where: cumulative session count Indicates the first The cumulative output token count after the next verification, starting from the beginning of the session, has a value no less than 0. Its purpose is to be used in step three to calculate the increment by taking the largest value; the cumulative amount from the previous session... Indicates the first The cumulative number of output tokens after each reconciliation serves to provide a continuous baseline for this cumulative process.

[0108] Fragment Increment Token Count Using the aforementioned definition, its function is to incorporate newly added outputs within the current window into the cumulative output. When it is necessary to aggregate items by pricing category, the processing unit further maintains the cumulative amount of each session category, calculated as follows:

[0109]

[0110] Where: Cumulative amount of conversation categories Representing a session In pricing category Next Cumulative amount per unit after the last reimbursement; cumulative amount per category in the previous session. Indicates the previous write-off water level; pricing conversion factor. Indicates pricing category The corresponding credit limit conversion ratio is derived from the account pricing configuration; category incremental token count This represents the amount of new tokens added within the same session and pricing category. This mapping ensures that inputs, outputs, and cache hits maintain a central highest watermark in step three, preventing different pricing categories from sharing a single watermark field.

[0111] For example, the processing unit increments the window number of a session from 5 to 6, and outputs the cumulative amount and the billable cumulative amount. A write-off frame is created, and the lease version number and pricing category are permanently saved. Subsequently, a brief network congestion occurs, and the batch of data sent is temporarily not delivered to the central quota service. After the processing unit restarts, it first reads the pre-written log, retrieves the unconfirmed write-off frames, and continues delivery. This is because the write-off frame carries the cumulative credit limit. Rather than simply incremental, even if there are repeated deliveries later, they can be absorbed by merging and taking the largest value in step three.

[0112] In parallel implementation, the construction of the write-off frame can be executed by the gateway processing unit, or by the inference engine instance or accompanying process; in low-computing-power edge deployments, the write-ahead log can be replaced by a circular memory buffer and periodic snapshots, but the cumulative water level is still retained. The system includes three fields: window sequence number, lease version number, and other relevant fields. For batch reporting channels, the processing unit preferably adds the revocation frame to the sending batch after writing it to the write-ahead log. The sending batch can be flushed according to frame number or time slice arrival. If the sending batch writing fails, the processing unit reconstructs the sending batch along the write-ahead log without rereading the inference engine output text. This path decouples output generation, window revocation, and channel delivery, ensuring that short-term channel congestion does not force the inference engine to stop generating already acquired quota coverage.

[0113] Triggered by the first-to-issue window, output tokens are merged into window-level reimbursement actions, and metering costs no longer increase linearly with individual tokens. Through lease renewal and fragment boundary truncation, the session credit window can continue within the granted limit, and the settlementable fragment boundary is still preserved when the limit is insufficient.

[0114] Step 3: Send the merged frame to the central quota service asynchronously, and the central quota service will perform the maximum merge according to the cumulative water level of the same session and the same pricing category, so that the metering entry and lease write-off are closed in the same frame.

[0115] In scenarios where multiple gateways and inference nodes serve the same account, step two compresses continuous output into reconciliation frames carrying session identifiers, window sequence numbers, input cumulative amounts, output cumulative amounts, billable cumulative amounts, final state bits, metering bits, control bits, lease version numbers, and pricing categories. If the central quota service still receives messages strictly in order, network jitter at any node will cause subsequent sessions to wait; if received using ordinary incremental call detail records, duplicate deliveries and out-of-order arrivals will result in duplicate billing or missed reconciliations. Step three does not base channel reliability on ordered transmission, but rather moves reliability into frame semantics and central merging rules: processing units allow multiple deliveries of the same batch, and the central quota service only recognizes higher cumulative water levels, using the water level difference as the incremental amount. In this way, local reconciliation in step two is not blocked by central congestion, and final state settlement in step four also freezes session water levels along the same path.

[0116] The processing unit reads unconfirmed write-off frames from the write-ahead log, forms a sending batch based on the batch number or batch delay priority condition, and sends it to the central credit service via message bus, batch call interface, or heartbeat message. Upon receiving the sending batch, the central credit service locates the central session status by account identifier and session identifier, and then reads the category center's highest credit limit, lease version number, and pricing category. For write-off frames higher than the category center's highest credit limit, the central credit service increases the account's confirmed usage by the difference, and simultaneously reduces the lease's unclosed amount based on the control bit and lease version number; for write-off frames not higher than the category center's highest credit limit, the central credit service does not record them twice, nor does it repeatedly write off leases. Node lease replenishment requests share the same channel as the sending batch. The central credit service determines whether to grant a new credit lease after completing the merging of this batch, ensuring that the replenishment basis falls on the updated account view.

[0117] The processing unit does not wait for the session to end before reporting the cancellation frame; instead, it continues batch grouping after the write-ahead log in step two. The triggering of a sending batch is determined by both the number of frames within the batch and the batch waiting time; once either reaches its boundary, it enters the delivery queue. The delivery channel employs at least-once semantics, allowing the processing unit to retransmit the same sending batch if it has not received confirmation from the center. The sending batch stores the batch identifier, first frame window number, last frame window number, and lease version number in the memory queue. The write-ahead log stores the complete cancellation frame, and the confirmation message only deletes the received batch index without deleting the session context.

[0118]

[0119] Where: Batch trigger flag Indicates the first Whether a candidate batch of data enters the delivery queue, with a value of 0 or 1, serves to merge frame number boundaries and time boundaries into a single trigger result; indicator function This indicates that the value is 1 if the condition within the parentheses is true and 0 if it is false. Specifically, it is a mapping from Boolean conditions to integer gating values, and its purpose is to facilitate the judgment by the batch assembly thread.

[0120] Number of frames per batch Indicates the first The number of aggregated cancellation frames in each candidate sending batch, with a value not less than 0, serves to limit the payload size; batch frame number threshold. This indicates the number of frames required to trigger a batch delivery, preferably between 16 and 256, which serves to establish a boundary between the number of network packets and the cost of single-batch recovery; the current batch time. This represents the monotonic clock reading at the time the candidate batch was last written to the write frame, and its purpose is to prevent low-speed session delays; the last batch time. This represents the monotonic clock reading when the previous batch entered the delivery queue, and its function is to correlate it with the current batch's time. Calculate the waiting time;

[0121] Batch latency threshold This indicates the maximum allowed waiting time for sending a batch, preferably between 20 milliseconds and 500 milliseconds. Its purpose is to provide reporting boundaries for short sessions and low-speed sessions.

[0122] For example, a processing unit simultaneously generates sessions for its service digest and code interpretation sessions, using a batch latency threshold. The first user to arrive enters the sending batch; and according to the code explanation, the number of frames within the batch... Reaching the batch frame threshold They are then added to the sending batch. Both carry the accumulated amount of the original session in the delivery channel, and there is no requirement for the central quota to arrive in a specific order, i.e., they arrive without regard to the order. This ensures that the sending batch does not delay low-speed sessions due to waiting for the batch to be full, nor does it send high-speed sessions frame by frame; after the processing unit restarts, it continues to deliver along the original cancellation frame.

[0123] The central quota service maintains a category-wide maximum water level for each billing category in each session. When a reimbursement frame arrives, the central quota service does not use the window sequence number as the basis for billing, but rather whether the cumulative amount of the frame category exceeds the category-wide maximum water level; the window sequence number is used to detect rollbacks. Since the cumulative amount of a session category increases incrementally from the start of the session, duplicate frames, old frames, and out-of-order frames will all fall below or equal to the category-wide maximum water level, and therefore will not be added to the account's confirmed usage again.

[0124]

[0125] Where: Incremental amount credited for session categories Indicates sending batch Chinese dialogue In pricing category The actual newly confirmed quota unit, with a value not less than 0; cumulative amount of frame category. This indicates the cumulative water level of the same session and the same pricing category carried by the cancellation frame; the highest water level at the center of the category. This indicates the highest water level previously confirmed for the same session and pricing category within the central quota service; maximum function. Used to absorb duplicate frames, old frames, and out-of-order frames; session This refers to the session object generated in step one and continuously carried over in step two; pricing category. Indicates the specific category of input, output, or buffer hit; batch number sent. This indicates the monotonic number of the batch sent by the processing unit.

[0126] For example, the processing unit first delivers a cancellation frame with window sequence number 5 and a cumulative frame count of 320, and then, due to network retransmission, delivers an older cancellation frame with window sequence number 4 and a cumulative frame count of 256. The central quota service has already set the central maximum water level after processing the previous frame. Pushed to 320, the session category increment is recorded when the next frame arrives. If the value is 0, the confirmed usage and unclosed lease amount in the account remain unchanged. If a cancellation frame with window number 6 and cumulative frame amount of 384 subsequently arrives, the central quota service will only use 64 as the new usage and cancel the unclosed lease amount under the corresponding lease version number according to the control bit.

[0127] This reduces the deduplication status from message level to session level; the metering bits and control bits are separated within the same frame, allowing billing flow and lease cancellation to share the same water level difference.

[0128] After merging and consolidating data, the central credit service refreshes the account view and reads the lease replenishment requests piggybacked by the processing units' sending batches. The replenishment requests include node identifier, account identifier, local lease balance, active flow count, recent consumption rate, and lease version number. The central credit service first obtains the centrally calculated credit limit based on confirmed usage and cancelled leases, and then returns new credit leases following the allocation strategy in step one. If a processing unit has no sending batches arriving for an extended period, the central credit service places its leases in a observation period; any uncancelled balances that have not been renewed after the expiration date are reclaimed.

[0129]

[0130] Where: Lease open amount This indicates the outstanding balance of credit line leases that have been issued but not yet closed by the central credit line service; the total number of leases issued. This indicates the total number of granted leases recorded in the center's lease registration form; the number of leases that have been closed. This indicates the amount of credit leases that have been released through control bit write-off, final state rollback, or expiry date recovery.

[0131]

[0132] Where: Central accounting quota This indicates the account limit used to determine whether to continue allocating funds after the completion of this batch of mergers; total account limit. This indicates the total token limit that the account has configured or purchased; the account's confirmed usage. This indicates the amount of chargeable fees that have been confirmed and credited for the center's credit service; and the amount of leases that have not yet been closed. Following the previous definition; maximum function This is used to avoid negative values ​​in the central accounting limit. This ledger only deducts open leases, and no longer simultaneously deducts and releases the balance of the same lease. When the central limit service processes the control bit, it updates the open lease balance by accounting increment for the same session category:

[0133]

[0134] Where: the amount of unclosed leases after the update Indicates the unclosed lease balance after processing the write-off frame; control bit This indicates the lease revocation permission for the revocation frame; the value is 1 when valid and 0 when invalid; session category accounting increment. Following the aforementioned definition, the old epoch frames recalled in step five can retain their metrological semantics without repeatedly releasing the old lease balance;

[0135] Furthermore, to limit the maximum exposure during abnormal periods, the central quota service also maintains an upper bound on the exposure, which is used to output the peak time series of the exposure in transit to step five.

[0136]

[0137] Where: synchronous lag term It is the sum of the unmerged differences between the highest watermark in the write-ahead log of the processing unit and the highest watermark in the center; the highest watermark in the write-ahead log. Indicates the session in the local write-ahead log of the processing unit. The highest cumulative water level; the highest water level at the center This indicates the highest confirmed session water level for the central quota service, using the summation symbol. Summate the set that is not yet closed in the central view.

[0138]

[0139] In the formula: the upper limit of the opening This indicates that the central credit limit service estimates the maximum open credit limit of the account for the current period; open lease amount. This indicates the outstanding balance of credit limit leases that have been issued but not yet closed; number of active sessions. This represents the number of sessions that are still in the generation, awaiting resale, or awaiting finality stage. Other steps are as shown in Step 1, in the resale window. Similarly defined, synchronous lag items are included during write-off. Repeat the definition above.

[0140] For example, after the central quota service processes a batch of data sent by a certain node, it discovers that the node's local lease balance continues to decrease with each frame crediting, while supplementary requests indicate that the node still has multiple active sessions. The central quota service first calculates the quota based on the central accounting standard. Verify account boundaries and then return to the supplementary lease; if channel backlog causes synchronization lag... The central quota service requests the processing unit to temporarily increase the reimbursement window during the return packet process. The batch grouping interval is reduced and the re-granting speed is decreased, allowing the channel to process historical sending batches first. In parallel implementation, the sending channel can use a dedicated message bus, direct batch call interface, or heartbeat messages; the central merging status can be stored in an in-memory table and synchronously written to a persistent log, or stored in a sharded key-value store; for private single-machine deployment, the central quota service is handled by the local quota management process, and the sending batch degenerates into an in-process queue, still executing the highest watermark merging.

[0141] This causes the lease supplement to occur after the account view is refreshed, and the outstanding balance is not increased based on the old balance; upper limit of exposure. Leases, active sessions, and synchronization lags are grouped into the same dimension.

[0142] By sending batches to achieve grouping and at least one channel, the processing unit merges and delivers multiple frames for cancellation. Even if a channel fails, it will still replay along the write-ahead log, without changing the streaming output rhythm of the inference engine. By merging the largest value at the center water level, duplicate frames, old frames, and out-of-order frames are all absorbed by the same comparison rule. The confirmed usage of the account and the amount of leases not closed only change with the addition of water levels.

[0143] Step 4: Convert the window verification state in the streaming generation process into the session final settlement state, and convert the unclosed state into a single-chain data that can be reconstructed, rolled back, and fed back.

[0144] In large-scale streaming billing scenarios, session termination is not always triggered by the inference engine outputting a completion flag. Caller-initiated disconnection, gateway processing unit restart, inference engine instance migration, network channel backlog, and timeout shutdown can all leave unfinished segments from step two, unmerged batches from step three, and the local session credit window in an intermediate state simultaneously. If a separate recovery branch is established for final-state settlement, normal frames, compensation frames, and manually corrected frames will coexist within the same session, requiring the central quota service to deduplicat each frame, thus disrupting the maximum merging semantics established in step three. Therefore, step four does not redefine the settlement path but retains the reconciliation frame structure, only changing the frame meaning through the final state bit, reconstruction flag, and caliber flag, allowing normal termination and abnormal reconstruction to share the same central highest water level comparison rule.

[0145] The client, processing unit, inference engine, and central credit service collaborate to execute this step. After the client closes the streaming connection or the inference engine returns a generation end flag, the processing unit reads the session identifier, lease version number, pricing category, tokenizer version, session credit window remaining capacity, and highest sequence number of the write-ahead log from the session context. The inference engine outputs the server-side authority cumulative count aggregated by pricing category, which the processing unit writes to the final state frame and sets the final state bit to 1. After receiving the final state frame, the central credit service still merges according to the highest central water level from step three, without creating new accounting rules due to the final state bit. The final state bit only puts the session into a freeze process; after freezing, it is no longer allowed to renew credit limits from the credit lease to the session's credit window.

[0146] If the orphanage determination period has expired If the final state frame is still not received, the central quota service queries the source processing unit and the inference engine for the highest accumulated water level, and constructs a reconstructed frame based on the available water level. Therefore, the action sequence of step four is as follows: final state identification, isomorphic crediting, balance rollback, orphan reconstruction, and residual suspension. Each subsequent action takes the water level and state output by the previous action as input.

[0147] The final state frame is not a new billing object, but rather the termination form of the reimbursement frame in step two. When generating the final state frame, the processing unit retains the session identifier, window sequence number, input cumulative amount, output cumulative amount, billable cumulative amount, lease version number, pricing category, metering bit, and control bit, and sets the final state bit to 1. The metering bit continues to indicate that the frame can generate an additional billing amount, and the control bit continues to indicate that the frame can reimburse the corresponding credit lease. After receiving the final state frame, the central credit service first obtains the difference between the final state frame's cumulative amount and the highest water level in the category center, then collects input tokens, output tokens, and cache hit tokens according to the pricing category, and finally rolls back the unused balance of the session credit window to the local credit lease, and reserves the lease balance not occupied by the final state session to other sessions in the same processing unit or reclaims it according to the validity period.

[0148]

[0149] Where: Final state increment This represents the actual number of newly confirmed tokens added to the final state frame. The value is not less than 0, and its purpose is to prevent duplicate final state frames from being recorded again; the cumulative number of final state frames. This indicates the server-side authoritative cumulative count written to the final state frame by the processing unit at the end of the session. The value is not less than the cumulative amount at the start of the session, and its function is to provide a final comparison level for the central quota service.

[0150] The highest water level in the center This represents the cumulative session water level confirmed by the central quota service before processing the final frame. Its value is not less than 0, and its function is to absorb out-of-order old cancellation frames; maximum function. This means taking the larger of the two candidate values. Specifically, it means taking 0 when the difference is less than 0. Its function is to suppress backflush frames, duplicate frames, and late frames to a uniform zero increment.

[0151] For example, the caller requests to compose a product description. After the inference engine outputs the last streaming fragment, it returns a generation end flag. The processing unit reads the last window sequence number from the write-ahead log and writes the server-side authoritative cumulative count, categorized by pricing type, into the final state frame. If the central quota service detects that the cumulative amount in the final state frame is higher than the central maximum water level, it only uses the difference between the two as the final state credit increment. The session state is then transitioned from in progress to settlement, and then to settled. If the same final state frame arrives again due to a batch retransmission, the final state increment is recorded. If the value is 0, the central quota service will remain frozen as it is.

[0152] This ensures that the final settlement is still completed along the largest merging path, avoiding the need to add a separate deduplication table at the end of the session; at the same time, the pricing category enters the itemized settlement with the final frame, so that inputs, outputs and buffer hits will not be mixed up in the final stage.

[0153] When the central credit limit service continuously receives reimbursement frames from other sessions of a certain account, but the orphanage determination period is reached... If no new revocation frame or termination frame is received for the specified session, the session is transferred to an orphan state. Orphan determination is not based solely on the client connection state, but rather on a combination of batch channel latency, session silence time, and the queryability of the write-ahead logs from step two, to prevent low-speed generated sessions from being prematurely terminated. The central quota service sends a reconstruction query message to the source processing unit, with query fields including session identifier, lease version number, last window sequence number, and query type; the source processing unit returns the highest cumulative write-ahead log amount, and the inference engine returns the highest cumulative engine session log amount.

[0154] After obtaining the available water level, the central quota service constructs a reconstruction frame with reconstruction markers. The reconstruction frame still carries the session identifier, window sequence number, input cumulative amount, output cumulative amount, billable cumulative amount, final state bit, meter bit, control bit, lease version number and pricing category, and merges the largest value along step three.

[0155]

[0156] In the formula: orphan determination time This represents the shortest decision boundary for the central quota service to wait for the final frame or a new cancellation frame. Its value is greater than the batch latency threshold, and its function is to distinguish between short-term channel delays and genuine connection loss. (Batch latency threshold) The waiting time when sending batches in step three is used, preferably between 20 milliseconds and 500 milliseconds. The purpose is to include the normal queuing time of the reporting channel in the orphan determination.

[0157] Conversation silence duration This indicates the time the inference engine is allowed to maintain the session when no new fragments are generated. It is preferably configured according to the model streaming output heartbeat interval and business timeout strategy. Its purpose is to protect low-speed generation and long inference sessions.

[0158] When the write-ahead log maximum watermark is inconsistent with the engine session log maximum watermark, the central quota service first generates a trusted reconstruction watermark, and then constructs a reconstruction frame:

[0159]

[0160] Where: credible reconstructed water level Indicates the cumulative water level used to generate the reconstructed frame; the highest water level in the write-ahead log. This indicates the highest cumulative amount of data for the same session in the source processing unit's write-ahead log; the highest watermark in the engine session log. Indicates the highest cumulative amount for the same session in the inference engine session log; continuity flag. The value is 1 if the window number, lease version number, and fragment boundary are all consecutive, and 0 otherwise; maximum function With the minimum function They are used for two types of scenarios: continuously verifiable and continuously unverifiable.

[0161] For example, a processing unit restarts during the generation of long text, the client does not receive a complete end signal, but the central quota service does not receive the final state frame. When the orphan determination time has expired... Then, the processing unit will query its write-ahead log and read the last session cumulative amount from the appended log segment. At the same time, the inference engine will return the cumulative amount corresponding to the segments it has already generated from the session log. If the two are consistent, a reconstruction frame can be generated according to the water level; otherwise, a trusted reconstruction water level will be used. Generate reconstructed frames and record these unrecorded discrepancies in the reconstructed bias distribution.

[0162] The recovery path does not leave the verification frame structure; normal frames, final frames, and reconstructed frames all originate from the highest water level in the center. Absorption; even if the write-ahead log and the engine session log are restored one after the other, the higher water level that is reached later can still be made up by taking the larger one and merging them, and the lower water level will not be reverted to the confirmed usage.

[0163] After the session enters the settled state, the processing unit and the central credit service jointly generate operational statistics. These operational statistics are not supplementary descriptions of the billing flow, but rather input objects for step five, water level management; their fields are fixed and generated in step four. The processing unit provides the initial credit window, total renewal amount, truncation position, session credit window exhaustion point, and segmenter version; the central credit service provides actual consumption, lease lifetime, validity recovery delay, synchronization lag, reconstruction bias, and peak in-transit exposure. These fields are aggregated by account, model, and time period to form the pre-occupancy error sequence, truncation event stream, lease lifetime histogram, synchronization lag tail distribution, reconstruction event distribution, and peak in-transit exposure time series.

[0164]

[0165] Where: Pre-occupancy error This indicates the direction and magnitude of the deviation between the pre-allocated credit limit and the actual credit consumption for the session. The value can be negative, zero, or positive. Its purpose is to help step five determine whether the initial credit window is consistently too small or too large. Initial Credit Window Using the initial window opening amount from step one, the value is not less than 0, and its function is to describe the pre-allocated quota at the start of the session.

[0166] Total amount of renewed grants This represents the cumulative renewed credit limit transferred from the credit lease to the session credit window in step two of the stream. The value is not less than 0, and its purpose is to describe the credit limit additionally used during the session. Actual consumption. This represents the data value indicating the final usage of the session confirmed by the central quota service after the final frame or reconstructed frame is merged. The value is not less than 0 and serves as a pre-occupancy error. Settlement benchmark. Maximum function. As defined before, its function is in actual consumption. Provide a non-zero denominator when the value is 0 to avoid the expression becoming uncomputable due to an empty reply session.

[0167] For example, after a set of short Q&A sessions for a corporate account ends, the central credit service will record the actual consumption of each session. Initial credit window returned by the processing unit Total amount of renewal grants Merging, resulting in pre-occupancy error If a session terminates due to a quota boundary, the processing unit simultaneously writes the truncation location, quota snapshot, and session context to the truncation event stream. If a session is settled by a reconstructed frame, the central quota service writes the difference between the highest quota level in the pre-written log and the highest quota level in the engine session log to the reconstructed event distribution. In parallel implementations, runtime statistics can be written to sequential logs, sharded key-value stores, or time-series databases. In private single-machine deployments, runtime statistics can be output in hourly batches by the local quota management process.

[0168] In step five, without re-parsed the original generated text, epoch recall, quantile ladder pre-occupancy, and frequency division write-back can be performed based on the six types of residuals solidified in step four; pre-occupancy error It shares the same data chain as the session credit window, credit lease, and central maximum water level. Through final-state frame sealing, normal completion and active disconnection settlement follow the same maximum merging rule, while the final-state credit increment... It is determined solely by the cumulative amount of the final frame and the highest water level at the center.

[0169] Step 5: Based on the pre-occupancy error sequence, truncated event stream, lease lifetime histogram, synchronous lag tail distribution, reconstructed event distribution, and peak time series of in-transit exposure output in Step 4, the static quota lease system maintains the same funding state chain as when the account level drops sharply, the generation length distribution drifts, and the dual-source counting error occurs.

[0170] Although steps one through four have enabled the streaming generation session to have local windowing, window cancellation, maximum merging, and isomorphic reconstruction capabilities, it is still based on static initial credit windows, static cancellation windows, and static lease validity periods. When the same account initiates the generation of long texts in a concentrated period of time, or when the output length distribution of a certain model version shifts, static parameters cause the pre-allocated amount in transit to be higher than the actual consumption for a long time, or to frequently trigger hard truncation at the limit threshold.

[0171] Step five does not re-establish the billing path, but instead reads the six types of operational statistics solidified in step four, determines the account level, and issues epoch recall, segmented windowing, frequency division write-back, and budgeting closing instructions to the processing unit holding the quota lease, so that the aforementioned static link is transformed into a closed-loop link constrained by the settlement result.

[0172] The administrator subscribes to the six types of operational statistics output in step four and merges them by account identifier, model identifier, and lease epoch number. The central quota service provides the administrator with the account's available quota, the amount of unclosed leases, the number of active sessions, and synchronization lag items; the processing unit provides the administrator with the local lease balance, the current write-off window, truncation events, and dual-source count differences. The administrator first calculates the account water level ratio and then determines whether it has entered the sufficiency, warning, or critical range. When it is in the critical range, the administrator first performs an epoch recall to shrink unclosed leases, and then generates a parameter set version number through the write-back arbitrator, distributing the initial credit window, write-off window, grant ratio, demand amplification factor, and safety margin according to the same version system. The write-back message contains the target account, target model, parameter effective time, grayscale ratio, rollback threshold, and signature. After receiving it, the processing unit only applies it to new sessions or new renewal windows. Write-off frames that have already been written to the write-ahead log continue to enter step three according to the original lease epoch.

[0173] The administrator uses the account balance ratio as the recall entry point, but does not directly shut down all sessions. The account balance ratio is jointly determined by the account's available credit and the amount in transit. The amount in transit places unrecovered leases, active session window occupancy, and synchronization lag on the same scale, thus reflecting the credit limits that the center has not yet closed. In the ample range, the administrator only records the balance status; in the alert range, it restricts the granting of new leases; and in the critical range, it sends a recall message with the new lease epoch number to each processing unit. The processing unit freezes the renewal actions under the old lease epoch, seals the old lease balance, and returns a receipt.

[0174]

[0175] Where: Account water level ratio This indicates the relative relationship between the account's available credit limit and the amount of credit in transit. The value is not less than 0 and is used to classify accounts into three tiers: ample, warning, and critical. Available Credit Limit Using the available account limit from step one, with a value no less than 0, this represents the balance that the center can still allocate. Pre-allocated quota. This represents the amount of leases that have not yet been closed by a final state or reconstruction. Its value is not less than 0 and it is used to trigger an epoch recall. (Unclosed lease amount) This indicates the outstanding credit lease balance that has not been written off or recovered for each processing unit, serving to describe edge-side occupancy. Active sessions. This indicates the number of sessions still in the generation, pending reimbursement, or pending final state. The value is not less than 0 and is used to estimate the remaining quota within the window. Reimbursement Window Using the same verification window as in step two, preferably between 8 and 2048, we can measure the window space of active sessions that have not yet been reported.

[0176] Synchronous lag term The synchronization lag term from step three is used, with a value not less than 0, to compensate for missed frames during batch queuing and processing unit restarts. Maximum function This indicates that the larger value is selected from the candidate values, and its effect is to determine the pre-allocation amount in transit. The expression remains computeable when the value is 0.

[0177]

[0178] Where: Recall deadline This indicates the waiting time for the processing unit to complete the sealing and receipt of the old lease; minimum recall time limit. This indicates the lower limit for avoiding premature failure; the maximum recall period. Indicates the maximum period not exceeding the lease term; critical water level threshold. This indicates the account water level relative to the boundary when entering the critical zone; the pre-occupied amount in transit. Using the previous definition; global consumption rate This represents the token consumption rate of the account, calculated as a cumulative water level increment across multiple processing units; maximum function. and minimum function Establish defined time-limited clamping rules.

[0179] For example, when a corporate account runs contract generation sessions simultaneously on multiple processing units, the central credit service discovers the account level ratio. Entering critical phase. The administrator issues a new lease epoch, recall mode, and recall deadline. And the recall message for the credit window. Upon receiving the message, a certain processing unit stops renewing credit limits from the old lease to the session credit window, writes the remaining balance of the old lease to the archive record, and sends back the old epoch number and the highest window number in transit. Before receiving the acknowledgment, the central credit service still executes the metering and control bits on the old lease epoch frame; after receiving the acknowledgment, late old lease epoch frames only retain the metering bits, and the control bits are no longer used to cancel the lease.

[0180] This ensures that old frames do not lose measurement and new leases are not repeatedly consumed by old control bits, while the keep-alive credit window supports the current segment in completing semantic termination.

[0181] When the water level has not reached the critical threshold but the pre-occupancy error has been biased for a long time, the administrator does not recall existing leases, but adjusts the initial credit window for the next batch of sessions. The administrator uses the actual consumption from step four as input and maintains a decay bucketing histogram by account and model. The histogram uses logarithmic bucketing, with the lowest bucket covering short responses and the highest bucket covering long document generation. After each final state or reconstruction, the actual consumption is placed into the corresponding bucket, and periodic decay is performed on the weights of the old buckets. The quantile value is obtained from the position where the cumulative bucket weight first exceeds the target proportion, which belongs to the finite state calculations that can be performed in stream processing tasks.

[0182]

[0183] Where: quantile consumption Indicates target quantile The corresponding upper bound of the consumption bucket; bucket upper bound Indicates the first The right boundary of a logarithmic bucket; bucket weights Indicates the first The sample weights of each bucket after periodic decay; the total number of buckets. Indicates the number of buckets in the histogram; target quantile. This indicates the percentile selected based on the account's water level tier.

[0184]

[0185] Where: Initial credit window after update This represents the new window size opened in step one of the write-back process, and its value falls between the minimum and maximum credit windows. Minimum Credit Window This indicates the lower limit for allowing new session windows to open, preferably no less than the window required for a short reply, to avoid frequent probing. Maximum Credit Window This indicates the upper limit of the initial window opening, preferably not exceeding the maximum requested generation parameters and lease balance, to avoid pre-deducting the entire amount. (Percentage consumption) This indicates the attenuation bin histogram at the target quantile. The corresponding actual consumption, target quantile Use lower values ​​for the ample range, middle values ​​for the warning range, and higher values ​​for the critical recovery period.

[0186] Pre-occupancy correction factor This represents the windowing margin adjusted by the slow loop based on the pre-occupancy error, preferably ranging from -0.3 to 0.5, used to compensate for the length bias of the model and tenants. Minimum function and maximum function These are all boundary clamping functions used to ensure the initial credit window after the update. Do not cross the guardrail area.

[0187] For example, if an account's recent customer service summary conversations are mostly short outputs, step four continuously provides positive pre-occupancy error. The manager observes an increase in the weight of low-scoring buckets in the histogram and then adjusts the updated initial credit window accordingly. The value is sent to Step 1; when a new session connects, the processing unit opens a window according to this value, and the re-granting mechanism of Step 2 is still retained. Long sessions will consume more resources when crossing the threshold. Then, the remaining budget is replenished from the credit lease. If the session fails to renew the credit in step two, the processing unit sends the remaining budget and security margin to the inference engine. The inference engine terminates at a period, paragraph end, or list item end and returns a budget termination code and context replay credentials.

[0188] In budgeted semantic closing, the processing unit determines the closing budget using the following formula:

[0189]

[0190] Where: Final budget This indicates the number of tokens the inference engine can continue to generate and search for semantic boundaries; session credit window margin. This indicates the credit window limit that has not yet been consumed in the current session; keep-alive credit window. Represents a non-recallable window reserved for a segment that has already started; maximum function This is used to ensure that the final budget is not negative.

[0191] In parallel implementations, decaying bucket histograms can be replaced by online quantile algorithms or lightweight regression models trained on requested features; budgeted termination can be performed by the inference engine or by the accompanying process issuing stop flags at output segment boundaries. If a lightweight regression model is chosen, the requested feature vectors... This includes account anonymization features, model identifiers, and cumulative input volume. Maximum generated parameters Pricing Category Based on the historical quantile bucket numbers, the model obtains the predicted quantile consumption. The actual sample consumption comes from the merged results of the final frame or reconstructed frame in step four; the training objective uses quantile loss. The model parameters Indicates trainable parameters, actual sample consumption Indicates the first The final usage of each historical session, quantile residual function This is the quantile regression loss function.

[0192] This reduces the amount of unused credit in short sessions, while long sessions are extended through renewing, and insufficient credit is no longer simply truncated within word segments.

[0193] The controller employs a fast, medium, and slow loop frequency division for parameter write-back to prevent different statistics from simultaneously driving the same parameter. The fast loop reads the truncated event stream and the synchronization lag tail distribution, and writes back the write-back window. With water level threshold boundary; read lease lifetime histogram and fragmentation rate in the central ring, and write back the cut ratio. With demand amplification factor Slow loop reads the pre-occupancy error sequence and writes back the pre-occupancy correction coefficient. All write-backs first enter the write-back arbitrator. The write-back arbitrator compares the parameter set version numbers and only allows the newer version to enter the processing unit after its effective time has arrived. If the truncation event or the peak of the en route exposure exceeds the rollback threshold during the observation period, the previous parameter set version number is restored.

[0194] The regulator uses a stability index to identify output length distribution drift, and the calculation in the main embodiment is as follows:

[0195]

[0196] Where: stability index This indicates the deviation of the current consumption distribution from the baseline consumption distribution; the current barrel percentage. Indicates the current period number The sample percentage of each bucket; the percentage of the benchmark bucket. Indicates the historical baseline period Sample percentage of each bucket; smoothing constant This indicates preventing small positive numbers with a logarithmic denominator of 0; total number of bins. Following the aforementioned definition, when both the engine incremental count and the shadow segmenter count exceed the tolerance limit consecutively, the administrator adjusts the trust level based on the dual-source difference rate and the consecutive tolerance count:

[0197]

[0198] Where: Two-source difference rate Indicates the first The relative difference between the two types of counts within a single write-off period; Engine incremental token count Incremental token count for shadow segmentation Using the definition from step two; continuous out-of-tolerance count Indicates the number of times a session or account exceeds the tolerance within the observation window; tolerance threshold. Derived from account trust level and historical bias; observation window length This indicates the number of most recently revoked frames used for trust determination.

[0199]

[0200] Where: the updated pre-occupancy correction coefficient This represents the correction factor for the next version of the slow loop output, and calculates the initial credit window after the update. Pre-occupancy correction factor The same definition as above expresses the windowing margin for the current parameter set. Correction step size. Adjust the speed for each slow loop, preferably by decreasing the parameter set version number to mitigate oscillations caused by continuous write-backs.

[0201] Smoothing pre-occupancy error This represents the account-level deviation obtained by applying an exponential weighting to the pre-occupancy error series. Its value can be negative, zero, or positive, and its purpose is to describe the long-term pre-occupancy direction. Target pre-occupancy error. This indicates the permissible pre-emptive bias target, preferably close to zero and configured according to business risk. (Modified lower bound) and the upper limit of correction These represent the guardrail boundaries for the pre-occupancy correction factor, preferably -0.3 and 0.5. Minimum function and maximum function Using the aforementioned definition, its function is to adjust the pre-occupancy correction factor after the update. Located within the guardrail area.

[0202] For example, after a model upgrade, both the engine incremental count and the shadow segmenter count continuously exceed the acceptable range. The processing unit writes the dual-source difference into the reconciliation frame caliber marker, and step four transmits the dual-source deviation distribution to the administrator. The administrator first enters the settlement path with the lower available count, while simultaneously lowering the trust level of the account; the write-back arbitrator issues a smaller reconciliation window with the next parameter set version number. A smaller initial credit window and an earlier entry into the warning zone allow this account to generate more write-off frames and reduce exposure to old leases before the differences fade.

[0203] In parallel implementation, the feedback law can employ exponential weighting, proportional integral, or Bayesian posterior update; drift detection can utilize the population stability index, relative entropy, or cumulative sum detection; and the recall strategy can employ full recall, recall sorted by lease utilization, or full node proportional reduction. In parallel implementation, the central quota service also performs joint optimization on multiple processing units, solving for the node allocation ratio as the variable. and node verification window The objective function is as follows:

[0204]

[0205] Where: Node cutting ratio Indicates the first The allocation ratio of each processing unit; node cancellation window. Indicates the first The write-off window for each processing unit; vacancy estimate. This represents the amount of unused quota estimated based on historical pre-occupancy error and active flow count; exposure estimate. This represents the open account balance estimated based on lease balance, active sessions, and synchronization lag; weighting factor. and This represents the relative weights of the vacancy penalty and the exposure penalty. This joint optimization can be implemented using coordinate descent, linear search, or a constrained solver. This approach places the source of discrepancies, parameter write-back, and lease epochs within the same version chain, ensuring that the data surface continues to operate according to the previous parameter set even if the governance surface fails.

[0206] Through account water level ratio Epoch recall and old frame rulings proactively close unclosed leases on critical accounts, and old lease epoch frames can still be measured without double write-off. This is achieved through the updated initial credit window. With budgeted semantics as the final step, static pre-allocation is driven by the actual consumption distribution, and responses at the quota boundary end with fragment or paragraph boundaries.

[0207] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for synchronizing large-scale streaming usage billing and token limit management, executed by a limit management system, characterized in that: include: Receive streaming generation requests, grant credit leases with lease version numbers and validity periods according to the account's available credit, and create a session credit window containing session identifiers and pricing categories within the credit lease; During the fragment return period, the session credit window is deducted according to the priority conditions of the token threshold and the time threshold. If the balance is lower than the token threshold, the credit lease is renewed. If the balance is not lower than the token threshold, the credit lease is not renewed. If the balance is insufficient, the boundary termination code is output and a revocation frame containing the cumulative usage, final state bit, meter bit, control bit and lease version number is generated. The cumulative usage increments from the start of the session. The credit management system sends the revocation frame to the central credit service via at least one of the following: message bus, batch remote call interface, and heartbeat piggyback message; Upon receiving a cancellation frame, first locate the central session status according to the session identifier, pricing category, and lease version number; maintain the central maximum water level according to the session identifier and pricing category; when the cumulative usage is higher than the central maximum water level, update the confirmed usage according to the difference between the two, and deduct the corresponding lease unclosed amount when the control bit is valid; when the cumulative usage is not higher than the central maximum water level, do not update the confirmed usage and do not deduct the lease unclosed amount. The central quota service does not use the sending channel type and arrival order as the conditions for billing; when the same reimbursement frame arrives repeatedly, the central quota service will not update based on the highest central quota level; when a reimbursement frame with a higher cumulative usage arrives later, the central quota service will update based on the difference based on the highest central quota level. A final state frame is generated when the session ends. If no final state frame is received after the orphan determination time, a reconstruction frame is generated. All frames are merged according to the highest water level in the center. Based on the merge result, a write-back message carrying the parameter set version number and lease epoch number is generated, and the initial credit window, revocation window and grant parameters for subsequent sessions are adjusted.

2. The method for simultaneous management of usage-based billing and token limits according to claim 1, characterized in that: The credit limit lease includes account identifier, processing unit identifier, lease version number, validity period, lease balance, and word segmenter version; the session credit window includes session identifier, window number, pricing category, input cumulative amount, output cumulative amount, and billable cumulative amount; The quota management system maintains the highest water level in the center according to the session identifier and the pricing category, so that input tokens, output tokens and cache hit tokens are collected separately.

3. The method for simultaneous management of usage-based billing and token limits according to claim 2, characterized in that: The quota management system receives the engine increment token count returned by the inference engine along with the streaming fragments; when the engine increment token count is complete and valid, it writes the engine increment token count to the output cumulative amount; When the engine incremental token count is missing, the shadow segmenter with the same version as the segmenter is invoked, and the supplementary incremental token count is obtained by combining the segmentation tail cache formed at the end of the previous streaming segment. The supplementary incremental token count and the caliber tag are written into the verification frame.

4. The method for simultaneous management of usage-based billing and token limits according to claim 1, characterized in that: When the central credit service is unavailable and the remaining amount of the local valid credit lease covers the session credit window, the credit management system continues to open the window within the credit lease and writes an autonomous time period marker in the session context; When the available local credit limit lease balance is insufficient, the credit limit management system rejects new streaming generation requests. When a temporary trial lease is pre-issued by the central credit limit service, the credit limit management system will only establish a minimum session credit window within the upper limit and validity period of the temporary trial lease.

5. The method for simultaneous management of usage-based billing and token limits according to claim 4, characterized in that: When the write-ahead log is successfully written, the quota management system will put the cancellation frame into the sending batch; When the write-ahead log fails to write, the credit management system stops batch packet assembly and synchronously sends the current reimbursement frame to the central credit service; when the sending channel is congested, the credit management system writes a hysteresis flag in the session context and reduces the re-granting speed of the session credit window; when the sending channel recovers, the credit management system continues to deliver according to the window sequence number in the write-ahead log.

6. The method for simultaneous management of usage-based billing and token limits according to claim 5, characterized in that: When no final state frame is received and no new reimbursement frame is received after the orphan determination time has expired, the quota management system reads the highest water level of the pre-written log and the highest water level of the engine session log respectively. When the window sequence number, lease version number, and fragment boundary are all continuous, the higher water level is selected to generate the reconstruction frame; when the window sequence number, lease version number, and fragment boundary are not continuous, the lower water level is selected to generate the reconstruction frame; when log corruption prevents the water level from being read, the session is frozen at the highest recorded center water level.

7. The method for simultaneous management of usage-based billing and token limits according to claim 6, characterized in that: The quota management system configures the processing unit that holds quota leases and generates revocation frames as at least one of the following: gateway node, inference engine instance, companion process, and stand-alone worker process; The processing unit stores the session identifier, lease version number, and pricing category in the same session context; the reversal frames generated by different processing units all enter the same central highest water level merging path.

8. The method for simultaneous management of usage-based billing and token limits according to claim 7, characterized in that: The credit limit management system configures an input token processing strategy table for accounts. The input token processing strategy table includes an immediate cancellation flag and a final state aggregation flag. When the instant verification mark is valid, the input token quantity obtained by the server-side tokenizer is written into the first verification frame when the session is accessed. When the final state aggregation flag is valid, the input token quantity is temporarily stored in the session context and carried separately from the output cumulative quantity in the final state frame.

9. The method for simultaneous management of usage-based billing and token limits according to claim 1, characterized in that: The lease grant message generated by the credit limit management system includes account identifier, processing unit identifier, lease version number, validity period, and lease balance; The write-off frame includes session identifier, window sequence number, cumulative usage, final state bit, meter bit, control bit, lease version number, and pricing category; the write-back message includes parameter set version number, lease epoch number, initial credit window, write-off window, grant parameters, effective time, and rollback threshold.

10. The method for simultaneous management of usage-based billing and token limits according to claim 9, characterized in that: The quota management system outputs a session status flag in at least one location: in the interface response, management interface, and file log. Session status markers include in progress, settlement in progress, settled, orphan session, and rebuilt; when the credit management system terminates output due to insufficient session credit window, it outputs the credit limit termination code; when the credit management system completes the freeze based on the rebuilt frame, it outputs the rebuilt marker and the corresponding session identifier.

11. The method for simultaneous management of usage-based billing and token limits according to claim 10, characterized in that: The credit management system outputs the account identifier, session identifier, pricing category, cumulative usage, confirmed usage, center maximum credit level, lease version number, and session status flag in the accounting receipt; When the session status is marked as settled, the billing receipt carries the final state frame identifier; when the session status is marked as rebuilt, the billing receipt carries the rebuilt frame identifier; when the session status is marked as an orphan session, the billing receipt carries the trigger record corresponding to the orphan determination duration.

Images