Border-aware large model collaborative inference and rollback method

Abstract

The application discloses a boundary-aware large model collaborative inference and rollback method. On the basis of pre-deploying or caching model weights, a base station generates a candidate collaborative inference strategy, and determines a target collaborative inference strategy based on boundaries. In the autoregressive generation stage, the base station periodically re-evaluates whether the current strategy still satisfies the boundary condition, and triggers a collaborative rollback when the boundary condition is not satisfied. The method can avoid continuing to execute a negative yield collaborative strategy after link deterioration or output length growth, can identify memory risk in advance, avoid memory overflow or strategy interruption in the subsequent word generation process caused by only relying on the initial memory state access, and reduce communication overhead, reduce memory pressure and improve the stability of long output large model inference.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to a method for collaborative reasoning and backoff of large models based on boundary awareness. Background Technology

[0002] With the increasing application of large language models in mobile terminals, in-vehicle devices, robots, AR / VR devices, and IoT terminals, a single terminal often struggles to simultaneously meet constraints on model parameter (weights and configuration) storage, key-value cache memory, real-time inference latency, and energy consumption. Therefore, edge-cloud collaborative inference has become an important technical strategy, with user terminals, base station-side MEC servers, and cloud GPU servers jointly undertaking large model inference tasks.

[0003] Existing solutions typically employ static strategy selection, binary offloading, or fixed-sharding deployment. Binary offloading usually executes the complete inference task locally or remotely, failing to leverage the layered collaboration capabilities between endpoints, edge computing, and the cloud. Fixed-sharding deployment typically determines model shards and physical execution nodes before inference begins, eliminating the need to adjust strategies based on changes in links, queues, and GPU memory during runtime.

[0004] For generative inference in autoregressive large models, a complete inference cycle typically includes a Prefill phase and a Decode phase. The input to the Prefill phase is the sequence of input tokens from the user request, such as the prompt sequence. Instead of directly outputting the final response token, the Prefill phase outputs intermediate states for the Decode phase, primarily including the key-value caches generated at each layer and the activation states passed between layers. The Decode phase, based on the key-value caches and related activation states formed in the Prefill phase, generates output tokens token by token in an autoregressive manner. For each new token generated, the system correspondingly increases the key-value cache for that token.

[0005] Therefore, the Prefill and Decode phases have different resource characteristics. The Prefill phase mainly processes the complete input token sequence, and its computational and activation transmission volume typically increases with the length of the input tokens. The Decode phase generates outputs token by token, with strict autoregressive dependencies between tokens. It proceeds sequentially according to the token order, and the KV Cache continuously grows with the number of output tokens. If the link rate decreases, there is queue backlog at the edge or cloud, the number of output tokens given by the business is too long, or the KV Cache usage exceeds the safety threshold during the Decode process, the originally beneficial collaborative inference strategy may turn into a negative strategy.

[0006] Existing collaborative reasoning solutions mainly suffer from the following technical shortcomings:

[0007] (1) Collaboration strategies are usually determined once at the beginning of inference and lack a runtime re-evaluation mechanism.

[0008] Existing solutions typically determine inference strategies based on the device state, link state, and load state at the time of request arrival. However, the following situations may occur during actual inference: fluctuations in the wireless link from the terminal to the edge; a decrease in backhaul bandwidth from the edge to the cloud; queue backlog caused by other user requests at the edge or cloud; the actual number of tokens generated by the large model exceeding the initial prediction; and the KV cache continuously growing during the Decode phase.

[0009] Therefore, a strategy that satisfies the cooperation conditions at the start of inference may become a negative-return strategy during operation. Existing solutions lack a dynamic re-evaluation mechanism for such changes.

[0010] (2) Lack of boundary judgment on whether the "cooperation benefits are still positive"

[0011] Cross-device collaborative inference incurs overhead for transmitting intermediate activation features. If link bandwidth is insufficient or input / output token lengths are large, the communication overhead may outweigh the latency gains from multi-device computation.

[0012] Existing solutions typically only optimize static total latency or average latency, lacking runtime judgment on the following boundary relationship: "remaining computational benefit > "remaining communication cost".

[0013] Therefore, the system may continue to execute collaboration strategies that are no longer profitable.

[0014] (3) Lack of advance control over the risk to video memory caused by the dynamic growth of KV Cache

[0015] In large model inference, model weights can usually reside on nodes and be shared by multiple requests, while KV Cache is a request-level dynamic state that grows with the input length, generation length, and number of concurrent requests.

[0016] Existing collaborative inference solutions often check whether the GPU memory meets the initial requirements at the start of inference, but do not fully consider the continuous growth of KV Cache during the Decode process. This may lead to: edge server GPU memory approaching the threshold; cloud server GPU memory being filled by multiple requests; GPU memory overflow (OOM, Out of Memory) occurring during subsequent token generation; and the ongoing collaborative strategy being forced to stop.

[0017] (4) Lack of a linkage mechanism between entry and exit

[0018] Existing solutions typically focus on "how to choose the optimal strategy," but lack a complete control loop: whether to allow entry into collaborative reasoning when a request arrives; whether the benefit and resource boundaries are still satisfied during collaborative execution; how to back off when the boundaries are no longer satisfied; and how to prevent ineffective collaboration from continuing to consume bandwidth, computing power, and GPU memory.

[0019] Therefore, existing solutions lack robustness in scenarios involving multiple users, dynamic links, and long-context, large-model inference. Summary of the Invention

[0020] The following is an overview of the topics described in detail in this article.

[0021] The purpose of this application is to at least partially solve one of the technical problems existing in the related technologies. The embodiments of this application provide a method for collaborative reasoning and rollback of large models based on boundary awareness.

[0022] An embodiment of this application provides a boundary-aware collaborative reasoning and backoff method for large models, comprising:

[0023] During the pre-filling phase, node collaborative reasoning requests and resource status information are obtained;

[0024] Candidate collaborative reasoning strategies are generated based on the model sharding pre-deployment status, node collaborative reasoning requests, and resource status information.

[0025] The target collaborative reasoning strategy is selected from the candidate collaborative reasoning strategies based on the number of users and boundary constraints.

[0026] During the autoregressive generation phase, in the process of generating tokens autoregressively, the remaining token length, remaining computational benefits, remaining communication costs, and remaining key-value cache growth are reassessed based on the current resource status information. The current strategy is then reassessed to determine whether it meets the boundary constraints. If the collaboration benefit boundary, memory security boundary, or link state boundary is exceeded, the current strategy is rolled back.

[0027] According to certain embodiments of this application, the candidate collaborative inference strategy includes: a set of model splitting points in the pre-filling stage and the autoregressive generation stage, a mapping of model shards to execution nodes, a set of communication links used between shards, serial or parallel execution relationships, and a set of key-value cache resident nodes corresponding to each model shard.

[0028] According to certain embodiments of this application, after generating candidate collaborative reasoning strategies, the method further includes:

[0029] When a node has pre-deployed or cached model shards, the model shards are allocated to the node for execution based on the node's collaborative inference request;

[0030] When a candidate strategy requires a node to execute an undeployed model shard, the candidate strategy is marked as unavailable or requires cold start loading and additional loading latency is accounted for.

[0031] According to certain embodiments of this application, a target collaborative reasoning strategy is obtained by filtering from the candidate collaborative reasoning strategies based on the number of users and boundary constraints, including:

[0032] For single-user scenarios, determine whether the candidate collaborative reasoning strategies corresponding to a single node's collaborative reasoning request satisfy the boundary constraints to determine the target collaborative reasoning strategy;

[0033] For multi-user scenarios, resource competition is conducted within the effective candidate strategy set of each user to determine the target collaborative reasoning strategy.

[0034] According to certain embodiments of this application, the boundary constraints corresponding to a single-user scenario include one or more of the following: deployment constraints, computational benefit constraints, communication benefit constraints, token length constraints, memory security constraints, and latency constraints.

[0035] According to certain embodiments of this application, the deployment constraint is used to constrain the allocation of the model shard to the node for execution based on the node's collaborative inference request, provided that the node has already pre-deployed or cached the model shard.

[0036] The computational benefit constraint is used to ensure that the computational benefit of the candidate strategy relative to the single-node benchmark is greater than 0.

[0037] The communication benefit constraint is used to ensure that the computational latency savings brought about by cross-node collaboration are greater than the communication latency generated by cross-node transmission of intermediate activation features or related intermediate states.

[0038] The token length constraint is used to ensure that, under the conditions of the current input length, the given output length of the service, and the effective transmission rate of the link, the computational benefit of the candidate strategy can cover the communication cost.

[0039] The memory safety constraint is used to constrain the sum of the input length threshold and the output length threshold to be less than or equal to the maximum supported context length. The maximum supported context length represents the maximum total length of input and output tokens that the strategy can support, provided that the memory safety threshold of each node is met.

[0040] The delay constraint is used to constrain the end-to-end delay corresponding to the policy to be less than or equal to the maximum tolerable inference delay.

[0041] According to certain embodiments of this application, the boundary constraints corresponding to multi-user scenarios include: wireless bandwidth constraints, backhaul bandwidth constraints from the edge server to the cloud server, and multi-user video memory constraints; the resource contention selection includes:

[0042] Given the constraints of wireless bandwidth, backhaul bandwidth from edge server to cloud server, and multi-user video memory, select the strategy combination that minimizes the total system cost.

[0043] The wireless bandwidth constraint is used to constrain the sum of the minimum wireless bandwidth required by each user under the corresponding valid candidate strategy to be less than or equal to the total wireless bandwidth that the system can allocate.

[0044] The backhaul bandwidth constraint from the edge server to the cloud server is used to constrain the minimum backhaul bandwidth from the edge server to the cloud server required by each user under the corresponding valid candidate strategy to be less than or equal to the total backhaul bandwidth from the edge server to the cloud server that the system can allocate.

[0045] The multi-user memory constraint is used to constrain the sum of the weight memory usage of model shards and the key-value cache required by each user under the corresponding effective candidate strategy to be less than or equal to the safe memory threshold of the node.

[0046] According to certain embodiments of this application, during the autoregressive generation stage, a re-evaluation of whether the current strategy satisfies the boundary constraints is triggered when any of the following multiple evaluation trigger conditions are met:

[0047] The number of tokens generated has reached a preset threshold.

[0048] The current effective transmission rate of the link has decreased;

[0049] The current key-value cache usage exceeds the warning threshold;

[0050] The current queue load exceeds the preset queue load threshold;

[0051] The actual number of tokens generated exceeded the predicted value;

[0052] A new node collaborative reasoning request has arrived, and the priority of the new node collaborative reasoning request is higher than that of the current node collaborative reasoning request.

[0053] According to certain embodiments of this application, when the remaining computational benefit of continued collaboration is not greater than the remaining communication cost, it is determined that the collaboration benefit boundary has been crossed.

[0054] When the sum of the key-value cache currently occupied by a node and the key-value cache expected to be added to the node by the remaining tokens exceeds the safe memory threshold, it is judged as a violation of the safe memory boundary.

[0055] When the remaining token length, effective transmission rate, or latency constraints cause the current policy to no longer meet the single-user boundary conditions, it is judged as a link state boundary out of bounds.

[0056] According to certain embodiments of this application, the rollback action includes one or more of the following actions:

[0057] Stop cross-device transmission of the intermediate activation feature (activation) corresponding to subsequent tokens;

[0058] Ensure that subsequent autoregressive generation phases are executed within the current key-value cache resident node or the current executable sub-policy.

[0059] The terminal-base station-cloud collaboration is reverted to base station-cloud collaboration, terminal-base station collaboration, or single-node execution;

[0060] Reduce the concurrency of low-priority requests;

[0061] Postpone new node collaborative reasoning requests that do not meet the admission criteria.

[0062] The above-mentioned solution has at least the following beneficial effects:

[0063] By proposing a boundary determination mechanism for the remaining collaboration benefits in the Decode stage, the collaboration strategy is no longer statically selected before inference begins. Instead, during the generation process, the mechanism determines whether continuing collaboration still yields positive benefits based on the remaining tokens, real-time link rate, and activation transmission cost. This avoids continuing to execute negative-benefit collaboration strategies after link deterioration or output length increase.

[0064] By incorporating the residence location and dynamic growth of the KV Cache into the collaboration boundary judgment, and distinguishing between cross-node transmission activation and local continuously growing KV Cache, it is possible to identify potential memory risks in the Decode stage in advance, and avoid memory overflow or policy interruption in the subsequent token generation process due to admission based solely on the initial memory state.

[0065] By performing a cooperative rollback that matches the KV Cache resident location when the boundary is not satisfied, unnecessary cross-node activation transfers can be stopped without migrating or migrating large-scale KV Caches, and the cooperative strategy can be rolled back to a node combination that is still executable, thereby reducing communication overhead, reducing memory pressure, and improving the stability of long-output large model inference. Attached Figure Description

[0066] The accompanying drawings are used to provide a further understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0067] Figure 1 This is a structural diagram of the system running the boundary-aware large model collaborative reasoning and backoff method of this embodiment;

[0068] Figure 2 This is a flowchart illustrating the steps of a boundary-aware collaborative reasoning and backoff method for large models.

[0069] Figure 3 This is a diagram illustrating the serial inference, inter-layer activation transmission, and local KVCache residency relationships during the Prefill and Decode phases.

[0070] Figure 4 This is a signaling flowchart for collaborative reasoning in a large-scale model with boundary awareness. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0072] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, or the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0073] The embodiments of this application provide a boundary-aware collaborative inference and rollback method for large models. In the Prefill stage, based on pre-deployed or cached model weights, the base station generates candidate collaborative inference strategies and determines the target collaborative inference strategy based on effective transmission rate boundaries, input / output token length boundaries, KV cache memory boundaries, and latency boundaries. In the autoregressive Decode stage, the base station periodically re-evaluates whether the current strategy still satisfies the boundary conditions and triggers collaborative rollback when the boundary conditions are not met.

[0074] The embodiments of this application will be further described below with reference to the accompanying drawings.

[0075] Reference Figure 2 A boundary-aware collaborative reasoning and backoff method for large models includes the following steps:

[0076] Step S100: In the pre-filling stage, obtain node collaborative reasoning requests and resource status information;

[0077] Step S200: Generate candidate collaborative reasoning strategies based on the model sharding pre-deployment status, node collaborative reasoning requests, and resource status information;

[0078] Step 300: Based on the number of users and according to boundary constraints, select the target collaborative reasoning strategy from the candidate collaborative reasoning strategies;

[0079] Step 400: In the autoregressive generation stage, during the autoregressive generation of tokens, the remaining token length, remaining computational revenue, remaining communication cost, and remaining key-value cache growth are re-evaluated based on the current resource status information. The current strategy is re-evaluated to determine whether it meets the boundary constraints. If the collaboration revenue boundary, memory security boundary, or link state boundary is exceeded, the current strategy is rolled back.

[0080] Reference Figure 1 It should be noted that the system running the boundary-aware large model collaborative reasoning and rollback method of this embodiment includes terminal devices, base stations (edge ​​servers) and cloud servers.

[0081] Terminal devices generate large model inference requests; base stations are responsible for request state aggregation, candidate collaborative inference strategy generation, boundary judgment, target strategy selection, runtime re-evaluation, and rollback control; cloud servers provide cloud model sharding execution capabilities.

[0082] Large models can be divided into multiple model layers or model fragments. Model weights are pre-deployed or cached on terminals, base stations or cloud servers on a slow time scale. Large-scale model weights are not transmitted temporarily by default during the online inference phase.

[0083] Reference Figure 3 Large model inference includes a Prefill phase and a Decode phase.

[0084] The input for the Prefill phase is the node collaborative reasoning request. The input token sequence in the sequence is denoted as: .

[0085] The Prefill phase processes the complete input token sequence in the order of the model layers and outputs the intermediate states required for the Decode phase, including at least:

[0086] Each model layer generates a key-value cache (KV Cache) for the input token sequence, denoted as 𝐾. ;

[0087] The intermediate activation features that need to be transmitted between fragments are denoted as... ;

[0088] The equivalent intermediate representation used to initiate the Decode phase is denoted as... .

[0089] Therefore, the Prefill phase can be abstractly represented as:

[0090] .

[0091] The input to the Decode phase is the intermediate state output from the Prefill phase and the generated token sequence. For the nth output token, the Decode phase can be represented as:

[0092] .

[0093] in, Indicates the first Generate a token. Indicates the generation of the first The KVCache is updated after each token.

[0094] During the Decode phase, each time a token is generated, a new KV Cache corresponding to that token is added, and activations that need to be transmitted may be generated at cross-node split points.

[0095] For step S100, when the terminal device initiates a node collaborative inference request for a large model, the base station receives the request and parses the request parameters:

[0096] ;

[0097] in, This represents user A's node collaboration inference request; This indicates the length of the input token for request 𝑢, which is the length of the input context during the Prefill phase; This indicates the predicted output token length for request 𝑢, i.e., the number of tokens expected to be generated during the Decode phase; Indicates the maximum tolerable inference delay for request 𝑢; This indicates the priority of request 𝑢.

[0098] The base station further aggregates resource status information from the edge and cloud, including:

[0099] Terminal-side status: available computing power, available memory or video memory, deployed model fragments on the terminal, and link status from the terminal to the base station;

[0100] Base station status: Base station deployed model fragment set, current computation queue length, current video memory usage, current KV cache usage, and current available computing resources;

[0101] Cloud-side status: cloud-deployed model shard set, cloud-current queue load, cloud-based video memory status, and link status from base station to cloud server.

[0102] For step S200, first determine the model sharding pre-deployment status.

[0103] The target large model is divided into multiple model layers or model slices: ;

[0104] Define model sharded deployment state variables: ;

[0105] in, Indicates the first Have the model layers or model shards been deployed on the node? ; Number the model layer or model segment; The execution node can be a terminal, base station, or cloud server. If the node... Already capable of executing the first The parameters of each model segment, then If the node If the conditions for directly executing the model's sharding are not met, then... .

[0106] Then, the base station generates candidate collaborative reasoning strategies based on the model's pre-deployment status, node collaborative reasoning requests, and resource status information.

[0107] Collaborative reasoning strategies can include one or more links. A link refers to a communication connection between two nodes, such as a terminal-base station link, a base station-cloud link, or a terminal-cloud link.

[0108] Specifically, each candidate strategy includes the execution node sequence, the model splitting point set, the model shards that each node is responsible for, the transmission direction of activation at the splitting point, and the expected resident nodes of the KV Cache.

[0109] A candidate collaborative reasoning strategy can be represented as:

[0110] .

[0111] in, Indicates candidate collaborative reasoning strategies. This represents the set of model split points in the Prefill phase. This indicates the mapping from model shards to execution nodes during the Prefill phase. This represents the mapping from model shards to execution nodes during the Decode phase. This represents the set of communication links used between fragments during the Prefill phase. This represents the set of communication links used between fragments during the Decode phase. This represents the set of KV Cache resident nodes corresponding to each model shard. This indicates whether the Prefill phase is executed sequentially or in parallel. This indicates whether the Decode phase is executed serially or in parallel.

[0112] It's important to note that the split points for the Prefill and Decode phases can be the same or different. For example, the Prefill phase can be divided into three parts, executed on the terminal, base station, and cloud respectively, while the Decode phase can be divided into two parts, executed only on the base station and cloud. Although the split points for the two phases can be determined independently, end-to-end performance needs to be jointly optimized because the KV Cache and activation states output by the Prefill phase directly affect the input, memory usage, and communication overhead of the Decode phase.

[0113] For the Decode phase, it's necessary to further distinguish between the serializability between tokens and the parallelizability of computation within a single token. Due to the autoregressive generation mechanism, the 𝑔 The generation of an oken depends on the output of the nth token and the updated KV Cache, therefore different output tokens must be executed sequentially. However, for a single forward inference of a single token, different model slices, expert branches, or parallelizable subtasks can be executed in parallel or pipelined provided that the model dependencies are satisfied.

[0114] In some embodiments, after generating candidate collaborative inference strategies, a strategy for filtering out undeployed model shards is executed:

[0115] When a node has pre-deployed or cached model shards, the model shards are allocated to the node for execution based on the node's collaborative inference request;

[0116] When a candidate strategy requires a node to execute an undeployed model shard, the candidate strategy is marked as unavailable or requires cold start loading and additional loading latency is accounted for.

[0117] Specifically, define the online execution selection variables: ;

[0118] in, The request indicates that the nth model shard of Request A is executed on node t during the Prefill phase; The request indicates that the nth model shard of request 𝑢 is executed on node 𝑛 during the Decode phase.

[0119] If the nth model shard of request 𝑢 is to be executed on node 𝑛, then the following must be satisfied:

[0120] ;

[0121] This expression means that request A can only allocate a shard to node A for execution if node A has already pre-deployed or cached the nth model shard. If a candidate strategy requires a node to execute an undeployed model shard, then the candidate strategy is marked as unavailable, or marked as requiring a cold start and incurring additional loading latency.

[0122] For step S300, the base station determines the target cooperative reasoning strategy, which includes two scenarios: a single-user scenario and a multi-user scenario.

[0123] The target collaborative reasoning strategy is selected from the candidate collaborative reasoning strategies based on the number of users and boundary constraints, including the following steps:

[0124] Step S310: For a single-user scenario, determine whether the candidate collaborative reasoning strategy corresponding to a single node collaborative reasoning request satisfies the boundary constraints to determine the target collaborative reasoning strategy.

[0125] Step S320: For multi-user scenarios, resource competition is conducted within the effective candidate strategy set of each user to determine the target collaborative reasoning strategy.

[0126] Boundary constraints for single-user scenarios include one or more of the following: deployment constraints, computational benefit constraints, communication benefit constraints, token length constraints, memory security constraints, and latency constraints.

[0127] The deployment constraint is used to constrain the allocation of the model shard to the node for execution based on the node's collaborative inference request, provided that the model shard has been pre-deployed or cached on the node.

[0128] The computational benefit constraint is used to ensure that the computational benefit of the candidate strategy relative to the single-node benchmark is greater than 0.

[0129] The communication benefit constraint is used to ensure that the computational latency savings brought about by cross-node collaboration are greater than the communication latency generated by cross-node transmission of intermediate activation features or related intermediate states.

[0130] The token length constraint is used to ensure that, under the conditions of the current input length, the given output length of the service, and the effective transmission rate of the link, the computational benefit of the candidate strategy can cover the communication cost.

[0131] The memory safety constraint is used to constrain the sum of the input length threshold and the output length threshold to be less than or equal to the maximum supported context length. The maximum supported context length represents the maximum total length of input and output tokens that the strategy can support, provided that the memory safety threshold of each node is met.

[0132] The delay constraint is used to constrain the end-to-end delay corresponding to the policy to be less than or equal to the maximum tolerable inference delay.

[0133] Boundary constraints for multi-user scenarios include: wireless bandwidth constraints, backhaul bandwidth constraints from edge servers to cloud servers, and multi-user video memory constraints; the resource contention selection includes:

[0134] Given the constraints of wireless bandwidth, backhaul bandwidth from edge server to cloud server, and multi-user video memory, select the strategy combination that minimizes the total system cost.

[0135] The wireless bandwidth constraint is used to constrain the sum of the minimum wireless bandwidth required by each user under the corresponding valid candidate strategy to be less than or equal to the total wireless bandwidth that the system can allocate.

[0136] The backhaul bandwidth constraint from the edge server to the cloud server is used to constrain the minimum backhaul bandwidth from the edge server to the cloud server required by each user under the corresponding valid candidate strategy to be less than or equal to the total backhaul bandwidth from the edge server to the cloud server that the system can allocate.

[0137] The multi-user memory constraint is used to constrain the sum of the weight memory usage of model shards and the key-value cache required by each user under the corresponding effective candidate strategy to be less than or equal to the safe memory threshold of the node.

[0138] For example, for a single link l, the communication delay is calculated using the actual amount of data to be transmitted and the link's transmission rate: ;

[0139] in, This indicates the amount of data that needs to be transmitted on link l. This indicates the transmission rate of link l.

[0140] If link l is a wireless link, the transmission rate can be determined by the allocated bandwidth and channel quality:

[0141] ;

[0142] in, This indicates the bandwidth allocated to the wireless link. This indicates the signal-to-interference-plus-noise ratio (SIR) of the wireless link.

[0143] For the multiple links used serially in strategy 𝛺, the total communication delay is:

[0144]

[0145] in, This represents a candidate collaborative reasoning strategy. Representation Strategy The set of communication links used between the various model fragments

[0146] If we define the equivalent effective transmission rate of this serial link set... Its physical meaning is: using a virtual link to transmit the total amount of data. The rate at which the same communication delay is generated, i.e.:

[0147] For parallel communication branches, if the data volume and rate of the nth parallel branch are respectively... and Then the delay of the parallel communication stage is: ;

[0148] If we define the equivalent effective transmission rate of a parallel communication phase, its physical meaning is: the rate at which a single virtual link transmits the total amount of data in the parallel branches while producing the same delay as the parallel phase, that is: ;

[0149] The equivalent transmission rate mentioned above is only used to simplify the expression of policy-level communication capabilities. In actual admission judgment, the actual transmission delay of each link and the serial / parallel relationship are still the basis.

[0150] The single-user scenario is as follows:

[0151] 1. Determine the baseline latency and calculate the benefits of candidate strategies.

[0152] The base station first calculates the baseline latency for request 𝑢 when executed by a single node in the fastest possible way:

[0153] ;

[0154] in, This indicates that the length of the nth model slice processed on node 𝑛 is... Prefill computation delay when inputting the token sequence. This represents the Decode computation latency when the nth model shard generates a single output token on node 𝑛.

[0155] For a candidate collaborative inference strategy 𝛺, its end-to-end latency consists of the Prefill stage latency and the Decode stage latency: ;

[0156] in, This indicates the time delay required for the Prefill stage to process the complete input sequence. This indicates the average or estimated latency for generating a single token during the Decode phase. This indicates a request for the given output token length or the maximum number of tokens to be generated.

[0157] The computational latency during the Prefill phase is: ;

[0158] The computation latency for a single token during the Decode phase is: ;

[0159] If all computation and communication tasks in the strategy are serial, then the delay in the Prefill phase can be expressed as: ;

[0160] The latency of a single token during the Decode phase can be expressed as:

[0161] ;

[0162] in, This represents the amount of activation or related intermediate state data transmitted on link l when a single token is generated during the Decode phase.

[0163] If the strategy contains parallel branches, the calculation follows the rule of "serial summation, parallel maximum calculation". For the parallel phase, if the computation and communication latency of the nth parallel branch is:

[0164] ;

[0165] The latency of this parallel phase is: ;

[0166] If there are convergence calculations or subsequent serial calculations after the parallel phase, the total latency is:

[0167] ;

[0168] in, This indicates the serial processing latency before the parallel phase. This represents the computation or transmission delay for converging the results of parallel branches. This indicates the delay in subsequent serial processing.

[0169] To uniformly describe the relationship between serial and parallel execution, the execution process of candidate strategy 𝛺 can be abstracted as a directed acyclic graph. In the graph, nodes represent computational or communication tasks, and edges represent dependencies. The policy latency is then equal to the critical path length of the execution graph. ;

[0170] in, This represents all dependency strategies in the execution graph. This represents the computation or communication latency of task 𝑣 in terms of strategy. This expression naturally includes two cases: serial summation and parallel maximum maximization.

[0171] in, Nodes represent computational or communication tasks, and edges represent dependencies between tasks. Let be all directed strategies from the starting point to the ending point in the graph, and denote a single strategy as . , This represents the computation or communication latency of task v in the strategy; the strategy latency is taken as the length of the key strategy with the largest sum of latency among all strategies, thus uniformly covering both serial summation and parallel maximum taking.

[0172] The computational benefit of the candidate strategy relative to the single-node benchmark is defined as:

[0173] ;

[0174] If If the candidate strategy does not yield any benefit on the computation side, the base station will filter it out; otherwise, the base station will further determine whether the computational benefit can cover the communication cost generated by the cross-node intermediate state transmission.

[0175] 2. Calculate the total delay of candidate strategies based on the serial / parallel relationship.

[0176] For each link l in the candidate strategy 𝛺, the amount of data that needs to be transmitted in the Prefill phase can be expressed as: ;

[0177] The amount of data that needs to be transmitted to generate a single output token during the Decode phase can be represented as:

[0178] ;

[0179] in, This represents the amount of Prefill intermediate state data transmitted for each input token on link l. This represents the amount of intermediate Decode state transmitted for each output token on link l.

[0180] In the case of serial communication, the communication cost of the candidate strategy is:

[0181] ;

[0182] In the case of parallel communication, 𝑆 is first calculated separately for each parallel communication branch. Then, take the maximum value of the parallel branch and sum it with the values ​​of the preceding and following serial stages.

[0183] The basic conditions for the admission of collaborative strategies in the communication dimension are: ;

[0184] The physical meaning of this condition is that the computational latency savings from cross-node collaboration must outweigh the communication latency incurred by transmitting activations or related intermediate states across nodes. If this condition is not met, it means that although the collaboration strategy may be faster on the computational side, the communication cost has already offset or exceeded its computational gains, and therefore it should not be admitted.

[0185] For a single critical link l, if other links and calculation terms are fixed, the critical effective transmission rate of that link can be derived from the above inequality. Let the communication delay excluding link l be... The total amount of data transmitted on link l is: ;

[0186] Then link l needs to satisfy: ;

[0187] Therefore, the critical effective transmission rate of link l is obtained: ;

[0188] Only when the actual effective transmission rate meets the following: Only the communication boundary corresponding to this link is satisfied.

[0189] 3. Derive the input / output token length boundaries.

[0190] The calculated payoff of candidate strategy 𝛺 can be approximated as: ;

[0191] in, This represents the computational latency saved by each input token during the Prefill phase relative to the single-node baseline. This represents the computational latency saved by each output token during the Decode phase relative to the single-node baseline.

[0192] The communication cost of candidate policies can be expanded as follows:

[0193] ;

[0194] Based on the collaborative benefit conditions 𝛥 We can obtain:

[0195]

[0196] in, ; ;

[0197] This represents the net benefit per unit of input token during the Prefill phase, which is equal to the savings in calculating per unit of input token minus the cost of transmitting intermediate states of per unit of input token. This represents the net benefit per unit output token during the Decode phase, which is equal to the savings in calculating the unit output token minus the cost of transmitting the intermediate state of the unit output token.

[0198] when This demonstrates that, given the current input length, the given output length of the service, and the effective transmission rate of the link, the computational benefits of the candidate strategy can cover the communication costs.

[0199] If the output length is fixed ,and Then the input length threshold value is:

[0200] ;

[0201] when At that time, strategy Admission is possible along the input length dimension.

[0202] If the input length is fixed ,and The output length threshold value is:

[0203] ;

[0204] when At that time, policy 𝛺 is admissible in the output length dimension.

[0205] 4. Derive the single-user video memory boundary.

[0206] For candidate strategy 𝛺, the model weights carried on node 𝑛 are stored as follows:

[0207] ;

[0208] in, This represents the set of model layers assigned to node in strategy 𝛺. This represents the memory usage of the i-th model segment.

[0209] The request is for the KV cache memory generated by 𝑢 on node 𝑛 to be:

[0210] ;

[0211] in, This represents the KV Cache memory coefficient corresponding to the nth model shard unit token. This represents the batch size of request 𝑢, i.e., the number of sequences processed concurrently in one inference operation, which is 1 in a single request scenario.

[0212] Candidate strategies must satisfy the following memory boundary conditions:

[0213] in, Represents a node The memory limit reserved for this type of computation can be safely used to support model weights and KV cache, thus avoiding decoding.

[0214] This yields the maximum supporting context length for strategy 𝛺:

[0215]

[0216] Among them, the maximum supported context length Under the premise of meeting the safe memory threshold of each node, the strategy The maximum total length of input and output tokens that can be supported.

[0217] The candidate strategy satisfies the memory boundary condition as follows: ;

[0218] 5. Form a set of effective candidate strategies for a single user.

[0219] For request 𝑢, only candidate strategies that meet the following conditions will be admitted:

[0220] The model partitioning deployment constraints are satisfied, i.e. and ;

[0221] There is a positive return on the computational side, i.e., 𝛥 ;

[0222] The communication revenue boundary is satisfied, i.e., 𝛥 ;

[0223] The input / output token length reward boundary is satisfied, i.e. ;

[0224] The memory boundary is satisfied, that is ;

[0225] End-to-end inference latency satisfies user latency constraints, i.e. .

[0226] The set of strategies that satisfy the above conditions is denoted as: ;

[0227] Base stations can The candidate strategy with the minimum end-to-end latency, the minimum resource consumption, or the minimum overall cost is selected as the target collaborative reasoning strategy for request 𝑢.

[0228] Multi-user scenarios are as follows:

[0229] When there are multiple user requests in the system, the base station competes for resources within the set of valid candidate strategies for each user.

[0230] Define strategy selection variables: ;

[0231] in, The user is prompted to select a candidate strategy. This indicates that the strategy is not selected.

[0232] Each user selects an effective strategy: .

[0233] The wireless bandwidth constraint is:

[0234] Among them, minimum wireless bandwidth Indicates user In candidate strategies Minimum wireless bandwidth required, total wireless bandwidth This indicates the total wireless bandwidth that the system can allocate.

[0235] The edge-to-cloud backhaul bandwidth constraint is:

[0236] in, Indicates user In candidate strategies Minimum edge-cloud backhaul bandwidth required. This represents the total edge-cloud backhaul bandwidth that the system can allocate.

[0237] Multi-user video memory constraints are:

[0238] in, Indicates the first The memory usage of the weights in each model slice This represents the safe memory threshold for node n.

[0239] Under the premise of satisfying the above resource constraints, the base station selects the strategy combination that minimizes the total system cost:

[0240] Among them, the decision vector Indicates the selection of variables by all strategies. The decision vector is composed of the entire multi-user strategy selection scheme.

[0241] Priority, existing risk, or time-delay default risk can also be added to form a weighted cost function.

[0242] For step S400, in the autoregressive generation stage, during the autoregressive generation of the token, the remaining token length, remaining computational revenue, remaining communication cost, and remaining key-value cache growth are reassessed based on the current resource status information.

[0243] Let the current number of generated tokens be... The updated predicted total output token count is The remaining number of tokens is:

[0244] Among them, the number of tokens generated This indicates the number of output tokens currently generated, and updates the predicted total output. This represents the total number of predicted output tokens that are re-estimated during the Decode process. This indicates the number of tokens that still need to be generated.

[0245] The base station reacquires the current effective transmission rate, current queue load, current KV cache usage, current remaining video memory, and the remaining computational benefits and remaining communication costs of the current cooperation strategy.

[0246] A re-evaluation of whether the current policy satisfies the boundary constraints is triggered when any of the following evaluation trigger conditions are met:

[0247] The number of tokens generated has reached a preset threshold.

[0248] The current effective transmission rate of the link has decreased;

[0249] The current key-value cache usage exceeds the warning threshold;

[0250] The current queue load exceeds the preset queue load threshold;

[0251] The actual number of tokens generated exceeded the predicted value;

[0252] A new node collaborative reasoning request has arrived, and the priority of the new node collaborative reasoning request is higher than that of the current node collaborative reasoning request.

[0253] Reassess whether the current strategy satisfies the boundary constraints.

[0254] Specifically, if the remaining computational benefits of continued collaboration are not greater than the remaining communication costs, it is judged that the collaboration benefit boundary has been crossed.

[0255] When the sum of the key-value cache currently occupied by a node and the key-value cache expected to be added to the node by the remaining tokens exceeds the safe memory threshold, it is judged as a violation of the safe memory boundary.

[0256] When the remaining token length, effective transmission rate, or latency constraints cause the current policy to no longer meet the single-user boundary conditions, it is judged as a link state boundary out of bounds.

[0257] Specifically, if the remaining computational gains from continued collaboration do not exceed the remaining communication costs:

[0258]

[0259] This will trigger a payout.

[0260] in, This represents the computational latency savings and remaining communication costs that can be achieved by continuing to execute the current collaboration strategy based on the current number of remaining tokens, compared to single-node execution. This indicates the communication latency required for the remaining tokens to transmit activation at the cross-node split point; when the remaining computational benefit is not greater than the remaining communication cost, it is determined that the current strategy has crossed the collaboration benefit boundary.

[0261] If the KV cache memory does not meet the requirements during the remaining decoding process:

[0262]

[0263] This will trigger a rollback due to memory risk.

[0264] Among them, the current KV Cache occupies Represents a node Currently used KV Cache memory, predicted remaining growth This indicates that the remaining tokens will be stored on the node. The expected increase in KV Cache video memory, and the safe video memory threshold. Represents a node The safe memory threshold; when the sum of the two exceeds the safe memory threshold, a memory risk rollback is triggered.

[0265] If the remaining token length, effective transmission rate, or latency constraints cause the current policy to no longer meet the single-user boundary conditions, then policy rollback is triggered.

[0266] When the collaborative benefit boundary, memory security boundary, or link state boundary is exceeded, the rollback action of the current policy is executed.

[0267] The rollback action includes one or more of the following actions:

[0268] Stop cross-device transmission of the intermediate activation feature (activation) corresponding to subsequent tokens;

[0269] Ensure that subsequent autoregressive generation phases are executed within the current key-value cache resident node or the current executable sub-policy.

[0270] The terminal-base station-cloud collaboration is reverted to base station-cloud collaboration, terminal-base station collaboration, or single-node execution;

[0271] Reduce the concurrency of low-priority requests;

[0272] Postpone new node collaborative reasoning requests that do not meet the admission criteria.

[0273] After the request is completed, the system releases the KV Cache corresponding to the request, retains the model weights pre-deployed on the slow time scale, and records the actual number of output tokens, actual latency, reason for rollback, peak KV Cache usage, and link status changes.

[0274] Reference Figure 4 For example, the signaling flow for boundary-aware large-model collaborative reasoning is as follows.

[0275] Phase 1: Request Initiation and Global State Convergence;

[0276] ① Inference Request: The terminal device (UE) initiates a large model inference request to the base station / MEC, reporting core constraint parameters (such as model ID, prompt word length, maximum tolerable latency, etc.).

[0277] ② Status reporting: The collaborating device synchronizes its current resource base status (including model sharding deployment status, computing queue load, remaining capacity of video memory / KV cache, and network link status) to the base station / MEC, which is the control center.

[0278] Phase Two: Multidimensional Boundary Assessment and Strategy Distribution;

[0279] ③ Admission Boundary Determination: Based on the collected global information, the base station / MEC performs strict admission calculations locally. Three core considerations are defined here: revenue boundary (whether the calculated revenue exceeds the communication cost), KV memory boundary (whether the predicted dynamic memory is safe), and latency boundary (whether it meets user needs).

[0280] ④ Strategy distribution: After the boundary judgment is passed, the base station / MEC generates the optimal "execution node sequence" and distributes the specific cooperation strategy (such as where the split point is and who is responsible for which layers) to the terminal equipment and the cooperating equipment respectively.

[0281] Phase 3: Model Collaborative Execution;

[0282] ⑤ Collaborative Execution and Data Transfer: Each node begins pipeline operations according to the issued strategy. Activation flows between collaborating devices, forming a dynamic distributed computing link.

[0283] Phase 4: Dynamic monitoring and rollback loop in the Decode phase

[0284] ⑥ Decode Phase Reassessment: During the long text's token-by-token autoregressive generation process, the base station / MEC continuously monitors the system status. The core assessment points are: as generation progresses, whether the remaining collaborative benefits are still positive, whether the communication link deteriorates, and whether the ever-expanding KV memory reaches a dangerous level.

[0285] ⑦ Fallback mechanism trigger: Once the reassessment finds that any boundary no longer meets the conditions, the base station / MEC will immediately broadcast a fallback signaling to the terminal equipment and cooperating equipment, forcibly "stopping subsequent cross-node activation" transmission, and downgrading the policy (e.g. disconnecting cooperating equipment with poor communication quality, falling back to pure edge or cloud execution), thereby ensuring system stability and avoiding negative-return collaboration.

[0286] In this embodiment, during the large-model collaborative inference execution, based on the number of generated tokens, the predicted number of remaining tokens, the current effective link rate, the activation transmission volume of each model fragment, and the computational latency of each node, the remaining computational benefits and remaining communication costs for continuing to execute the current collaborative strategy are calculated. When the remaining computational benefits are not greater than the remaining communication costs, it is determined that the current collaborative strategy has crossed the collaborative benefit boundary, and a re-evaluation or rollback of the collaborative strategy is triggered. Instead of statically selecting a collaborative strategy before inference begins, the system determines whether continuing collaboration still has positive benefits during the generation process based on the remaining tokens, real-time link rate, and activation transmission costs. This avoids continuing to execute a negative-benefit collaborative strategy after link deterioration or output length increase.

[0287] The KV Cache residing node for each model shard is determined based on its execution location on the endpoint, edge, and cloud nodes. The KV Cache growth of each node in the subsequent Decode stage is predicted based on the input token length, the number of generated tokens, the predicted remaining token count, and the unit memory coefficient of the model shard KV Cache. When the sum of the current KV Cache usage and the predicted growth of any node exceeds the safe memory threshold, a memory risk rollback is triggered. By incorporating the KV Cache's residing location and dynamic growth into the collaboration boundary judgment, and distinguishing between cross-node transmission activations and locally continuously growing KV Caches, potential memory risks during the Decode stage can be identified in advance, avoiding memory overflow or policy interruption during subsequent token generation due to admission based solely on the initial memory state.

[0288] When the Decode phase detects that the collaboration benefit boundary, KV Cache memory boundary, or latency boundary is not met, subsequent token activation transmission at the cross-node split point is stopped. Based on the current KV Cache resident node, deployed model sharding parameters, and executable sub-policies, the original edge-cloud collaboration strategy is rolled back to edge-cloud collaboration, edge-edge collaboration, or single-node execution, and activation transmission is no longer continued. By executing a collaboration rollback that matches the KV Cache resident location when the boundary is not met, unnecessary cross-node activation transmission can be stopped without migrating or with minimal migration of large-scale KV Caches. The collaboration strategy is rolled back to a still executable node combination, thereby reducing communication overhead, reducing memory pressure, and improving the stability of long-output, large-model inference.

[0289] The above is a detailed description of the preferred embodiments of this application, but this application is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A boundary-aware collaborative reasoning and backoff method for large models, characterized in that, include: During the pre-filling phase, node collaborative reasoning requests and resource status information are obtained; Candidate collaborative reasoning strategies are generated based on the model sharding pre-deployment status, node collaborative reasoning requests, and resource status information. The target collaborative reasoning strategy is selected from the candidate collaborative reasoning strategies based on the number of users and boundary constraints. During the autoregressive generation phase, in the process of generating tokens autoregressively, the remaining token length, remaining computational benefits, remaining communication costs, and remaining key-value cache growth are reassessed based on the current resource status information. The current strategy is then reassessed to determine whether it meets the boundary constraints. If the collaboration benefit boundary, memory security boundary, or link state boundary is exceeded, the current strategy is rolled back.

2. The boundary-aware large-model collaborative reasoning and backoff method according to claim 1, characterized in that, The candidate collaborative inference strategy includes: the model splitting point set in the pre-filling stage and the autoregressive generation stage, the mapping from model shards to execution nodes, the set of communication links used between shards, the serial or parallel execution relationship, and the set of key-value cache resident nodes corresponding to each model shard.

3. The boundary-aware large-model collaborative reasoning and backoff method according to claim 1, characterized in that, After generating candidate collaborative reasoning strategies, the method further includes: When a node has pre-deployed or cached model shards, the model shards are allocated to the node for execution based on the node's collaborative inference request; When a candidate strategy requires a node to execute an undeployed model shard, the candidate strategy is marked as unavailable or requires cold start loading and additional loading latency is accounted for.

4. The boundary-aware large-model collaborative reasoning and backoff method according to claim 1, characterized in that, The target collaborative reasoning strategy is selected from the candidate collaborative reasoning strategies based on the number of users and boundary constraints, including: For single-user scenarios, determine whether the candidate collaborative reasoning strategies corresponding to a single node's collaborative reasoning request satisfy the boundary constraints to determine the target collaborative reasoning strategy; For multi-user scenarios, resource competition is conducted within the effective candidate strategy set of each user to determine the target collaborative reasoning strategy.

5. The boundary-aware large-model collaborative reasoning and backoff method according to claim 4, characterized in that, Boundary constraints for single-user scenarios include one or more of the following: deployment constraints, computational benefit constraints, communication benefit constraints, token length constraints, memory security constraints, and latency constraints.

6. The boundary-aware large-model collaborative reasoning and backoff method according to claim 5, characterized in that, The deployment constraint is used to constrain the allocation of the model shard to the node for execution based on the node's collaborative inference request, provided that the node has already pre-deployed or cached the model shard. The computational benefit constraint is used to ensure that the computational benefit of the candidate strategy relative to the single-node benchmark is greater than 0. The communication benefit constraint is used to ensure that the computational latency savings brought about by cross-node collaboration are greater than the communication latency generated by cross-node transmission of intermediate activation features or related intermediate states. The token length constraint is used to ensure that, under the conditions of the current input length, the given output length of the service, and the effective transmission rate of the link, the computational benefit of the candidate strategy can cover the communication cost. The memory safety constraint is used to constrain the sum of the input length threshold and the output length threshold to be less than or equal to the maximum supported context length. The maximum supported context length represents the maximum total length of input and output tokens that the strategy can support, provided that the memory safety threshold of each node is met. The delay constraint is used to constrain the end-to-end delay corresponding to the policy to be less than or equal to the maximum tolerable inference delay.

7. The boundary-aware large-model collaborative reasoning and backoff method according to claim 4, characterized in that, Boundary constraints for multi-user scenarios include: wireless bandwidth constraints, backhaul bandwidth constraints from edge servers to cloud servers, and multi-user video memory constraints; the resource contention selection includes: Given the constraints of wireless bandwidth, backhaul bandwidth from edge server to cloud server, and multi-user video memory, select the strategy combination that minimizes the total system cost. The wireless bandwidth constraint is used to constrain the sum of the minimum wireless bandwidth required by each user under the corresponding valid candidate strategy to be less than or equal to the total wireless bandwidth that the system can allocate. The backhaul bandwidth constraint from the edge server to the cloud server is used to constrain the minimum backhaul bandwidth from the edge server to the cloud server required by each user under the corresponding valid candidate strategy to be less than or equal to the total backhaul bandwidth from the edge server to the cloud server that the system can allocate. The multi-user memory constraint is used to constrain the sum of the weight memory usage of model shards and the key-value cache required by each user under the corresponding effective candidate strategy to be less than or equal to the safe memory threshold of the node.

8. The boundary-aware large-model collaborative reasoning and backoff method according to claim 1, characterized in that, During the autoregressive generation phase, a re-evaluation of whether the current strategy satisfies the boundary constraints is triggered when any of the following evaluation trigger conditions are met: The number of tokens generated has reached a preset threshold. The current effective transmission rate of the link has decreased; The current key-value cache usage exceeds the warning threshold; The current queue load exceeds the preset queue load threshold; The actual number of tokens generated exceeded the predicted value; A new node collaborative reasoning request has arrived, and the priority of the new node collaborative reasoning request is higher than that of the current node collaborative reasoning request.

9. The boundary-aware large-model collaborative reasoning and backoff method according to claim 1, characterized in that, If the remaining computational benefit of continued collaboration is not greater than the remaining communication cost, it is judged that the collaboration benefit boundary has been exceeded. When the sum of the key-value cache currently occupied by a node and the key-value cache expected to be added to the node by the remaining tokens exceeds the safe memory threshold, it is judged as a violation of the safe memory boundary. When the remaining token length, effective transmission rate, or latency constraints cause the current policy to no longer meet the single-user boundary conditions, it is judged as a link state boundary out of bounds.

10. The boundary-aware large-model collaborative reasoning and backoff method according to claim 1, characterized in that, The rollback action includes one or more of the following actions: Stop cross-device transmission of the intermediate activation feature (activation) corresponding to subsequent tokens; Ensure that subsequent autoregressive generation phases are executed within the current key-value cache resident node or the current executable sub-policy. The terminal-base station-cloud collaboration is reverted to base station-cloud collaboration, terminal-base station collaboration, or single-node execution. Reduce the concurrency of low-priority requests; Postpone new node collaborative reasoning requests that do not meet the admission criteria.

Images