A blockchain incentive-based consumer-grade GPU cluster distributed large language model inference system

Abstract

The application belongs to the technical field of distributed computing, and specifically relates to a consumer-grade GPU cluster distributed large language model inference system based on a blockchain incentive. The system adopts six technical measures: a low video memory quota perception fragmentation placement mechanism, a hierarchical pipeline parallel inference protocol, a two-stage optimistic fraud proof protocol, an inference response and economic settlement asynchronous separation mechanism, a pledge perception seven-dimensional scheduling score function, a node eight-state state machine and a hot standby fault-tolerant mechanism; the system is a three-layer architecture, and the technical measures cooperate to realize a complete closed loop of low video memory node participating in large model inference in an untrusted P2P network; the application can decouple logical service quota and physical GPU video memory, so that real-time performance and economic security can be considered; replayable segment-level verification is realized, and the correctness proof problem of untrusted nodes is solved; the influence of blockchain delay on user experience is eliminated; and verifiable and incentivized distributed AI inference services are realized in a heterogeneous untrusted P2P network.

Description

Technical Field

[0001] This invention belongs to the field of distributed computing technology, specifically relating to a consumer-grade GPU cluster distributed large language model inference system. Background Technology

[0002] As the parameter size of large language models (LLMs) continues to increase, such as DeepSeek-V3 (671B parameters) and LLaMA-3.1 (70B parameters), their inference services rely on high-end data center GPUs (such as NVIDIA H100 80GB), which presents the following problems:

[0003] The high memory barrier for large model inference. Existing inference frameworks require that the GPU memory of a single node or homogeneous cluster must be able to accommodate the complete model weights. A 671B parameter model still requires approximately 340GB of memory after INT4 quantization, equivalent to at least 8 H100 GPUs (640GB of memory), making it impossible for ordinary consumers to participate in inference services, creating a hardware barrier of up to several million yuan.

[0004] There is a lack of decentralized inference incentive mechanisms. Currently, cloud inference services are monopolized by a few companies, and hundreds of millions of consumer-grade GPUs (GTX 1050, RTX 3060, etc.) worldwide are almost completely idle in the LLM inference field, lacking quantifiable, verifiable, and tradable economic incentives to encourage them to join the inference service network.

[0005] Verifying the correctness of distributed reasoning is difficult. The forward computation results of neural networks are continuous floating-point numbers, making it impossible to verify whether nodes have performed the computation accurately in a simple way, unlike hash computation (Proof of Work). Existing technologies (TrueBit's interactive verification game, Verde's replayable ML dispute, and zkLLM's zero-knowledge proof) are either too cumbersome (zkLLM's proof time does not yet support synchronous settlement) or not specifically designed for low-memory, heterogeneous P2P environments. Using "activation value hashes" as raw PoW allows untrusted nodes to submit incorrect computation results in the correct hash format, creating a security vulnerability.

[0006] Inference scheduling in heterogeneous networks is complex. Consumer-grade nodes have highly heterogeneous computing power, memory, network bandwidth, and geographical location. Although existing distributed inference frameworks such as Petals have achieved hierarchical pipelined inference across Internet nodes, their node selection strategies lack a staking weight dimension and cannot be linked with decentralized economic incentive mechanisms. Frameworks such as DistServe assume a homogeneous high-speed interconnected environment, which is not suitable for consumer-grade P2P networks.

[0007] The timing conflict between inference response and economic settlement. Existing designs often place on-chain settlement in the critical path of inference completion, but there is a fundamental contradiction between the blockchain confirmation delay (seconds to minutes) and the user's expectation of real-time streaming response (milliseconds), making it difficult to balance user experience and economic security. Summary of the Invention

[0008] The purpose of this invention is to provide a consumer-grade GPU cluster distributed large language model inference system based on blockchain incentives, in order to solve the problems of difficulty in verifying the correctness of distributed inference, complexity of inference scheduling under heterogeneous networks, and time-series conflicts between inference response and economic settlement.

[0009] This invention provides a blockchain-incentivized distributed large language model inference system for consumer-grade GPU clusters. This system organizes globally distributed consumer-grade GPU devices into a distributed system capable of collaboratively running ultra-large-scale language model inference tasks. Through a task scheduling mechanism based on a staking-aware scoring function, a two-phase optimistic fraud proof protocol, and a smart contract automatic settlement mechanism, this system achieves verifiable and incentivized distributed AI inference services within a heterogeneous, untrusted P2P network. The core technical measures proposed in this invention include the following six aspects:

[0010] (a) Constructing a low-memory quota-aware fragmentation placement mechanism;

[0011] (ii) Design a hierarchical pipelined parallel inference protocol;

[0012] (iii) Design a two-phase optimistic fraud proof protocol;

[0013] (iv) Constructing an asynchronous separation mechanism between reasoning response and economic settlement;

[0014] (v) Design a seven-dimensional scheduling and scoring function for pledge perception;

[0015] (vi) Construct an eight-state machine for nodes and a hot standby fault-tolerant mechanism;

[0016] Specifically, the large language model reasoning system of this invention is designed with a three-layer architecture (see appendix). Figure 1 The above six core technical measures are coordinated to achieve a complete closed loop for low-memory nodes to participate in large-scale model inference in an untrusted P2P network; the three-layer architecture consists of, from top to bottom, the node capability layer, the scheduling and execution layer, the verification layer, and the settlement layer; wherein:

[0017] The node capability layer includes low memory quota-aware fragmentation placement and node octa-state machine and hot standby fault tolerance capabilities. The former solves the static management capability of nodes, where each node holds and executes only one segment of the entire model (Dense architecture) or a group of experts (MoE architecture) based on the low memory quota-aware fragmentation placement mechanism; the latter solves the dynamic lifecycle management problem of nodes; together, they constitute the complete technical support system for working nodes.

[0018] The task scheduling layer is the nerve center of the system. Based on the seven-dimensional scoring function of staking perception, it greedily selects the optimal set of pipeline nodes for each request from the distributed hash table. It adopts a hierarchical pipeline parallel inference protocol as the main inference protocol of the Dense model, manages the micro-batch request queue, and continuously fills the pipeline with requests from different users to eliminate pipeline bubbles. The task scheduling layer and the node capability layer maintain bidirectional real-time synchronization: the scheduling layer feeds back the failure rate history information to the capability layer, and the capability layer provides the scheduling layer with node capability (Tier) and staking information.

[0019] The verification and settlement layer involves a two-phase optimistic fraud proof protocol and an asynchronous separation mechanism between inference response and economic settlement. Specifically, all key economic operations are implemented through smart contracts (programs running on the blockchain), including: automatic allocation of tokens (digital assets on the blockchain) according to layer ratios, staking lock-up and release, staking slash penalty, and sharded storage proof. Storage (PoSt) periodic verification and node blacklist writing; the token economic system is designed around the three principles of "allocation according to contribution ratio, protection according to staking ratio, and punishment according to behavior results", and adopts a two-stage optimistic fraud proof protocol, that is, a dual-track mechanism of "optimistic execution + post-challenge": under normal circumstances, if no one challenges the phase commitment submitted by the node, it will directly enter the settlement; for tasks that are challenged or randomly sampled, the verification layer triggers a replayable fragment verification process to reduce the dispute to the smallest computational fragment that can be independently adjudicated; in the settlement stage, an asynchronous separation mechanism between reasoning response and economic settlement is adopted, that is, by explicitly defining "user-visible reasoning completion" and "on-chain economic settlement finality" as two independent time-series events, the settlement finality and user-side reasoning completion are decoupled from each other, so that real-time performance and economic security can be balanced.

[0020] The following section provides a more detailed introduction to the six core technical measures:

[0021] (a) Regarding the low memory quota-aware fragmentation placement mechanism;

[0022] By abstracting the large model uniform format weight (GGUF) sharding list, logical memory quotas are decoupled from physical GPU memory, allowing nodes with a 1GB logical quota to run on any consumer-grade GPU with ≥1GB of memory; specifically including:

[0023] (1) Decoupling of logical quotas and physical memory

[0024] This invention introduces a usable_vram_quota (abbreviated as VRAM_quota). The concept of ) decouples the service capabilities that a node promises to the system from its physical GPU memory capacity. This is the maximum amount of video memory voluntarily declared by the node, which can be equal to or less than the actual video memory of the physical GPU. The system... Calculate the number of model layers that this node can support, deducting physical memory. The remaining capacity can be used for the operating system, CUDA runtime, or other local applications, and does not compete for inference services.

[0025] Maximum number of layers a node can support Determined by the following memory allocation formula:

[0026]

[0027] in: This indicates the available logical quota of video memory for a node; The weight of a single layer is indicated by the fragmentation list; This indicates the amount of KV cache memory used by a single concurrent request; This indicates the maximum number of concurrent requests per node, and is configured by system parameters. This indicates that CUDA runtime reserves video memory with the system.

[0028] when When it is insufficient to support an entire floor, that is Below When this happens, the system will automatically trigger CPU overflow mode: the GPU prioritizes the computation of the attention layer, and the FFN weights overflow into the CPU memory to maximize GPU utilization efficiency.

[0029] (2) GGUF fragment list

[0030] This invention introduces the GGUF Shard Manifest as the authoritative description file for model sharding. Each shard record in the manifest contains inferences. All required metadata. Core fields include: model unique identifier and version number; shard identifier and the range of layers it covers; shard filename and SHA256 integrity hash; file size in bytes; quantization type; list of tensor names contained in the shard; KV cache memory usage per concurrent request; minimum required inference engine version; and execution configuration identifier for replayable verification. See Appendix 1 for the JSON format of the shard list.

[0031] When a node starts, it determines its own... Substitute into the memory allocation formula to calculate Then, the vectorized inference engine receives the layer number parameter to precisely control the number of layers the GPU can handle; exceeding... Quota-based tiers automatically overflow into CPU memory for execution. See Appendix 2 for specific command-line invocation instructions.

[0032] (3) Service Classification Definition

[0033] based on Regarding the range of values, this invention defines three standard service tiers:

[0034] (a) Tier-S1: (e.g., 1.5GB), suitable for entry-level consumer GPUs, capable of supporting several layers at the bottom or top of a model;

[0035] (b) Tier-S2: (e.g., 3.5GB), which can support more consecutive layers, making it suitable for intermediate layer providers;

[0036] (c) Tier-S3: It can support larger layers and is suitable for high-performance consumer-grade GPUs.

[0037] Grading threshold , Stored in an on-chain parameter contract, the layer size can be adjusted through community governance to suit different models. When assembling the pipeline, the scheduler prioritizes nodes of the same level to ensure that the computation time between nodes is similar, reducing pipeline bubbles.

[0038] (ii) Regarding the hierarchical pipelined parallel inference protocol;

[0039] This system implements multi-node parallel inference based on hierarchical pipeline parallelism, two-stage inference, and micro-batch scheduling technologies, and supports the MoE expert parallel routing extension protocol; specifically including:

[0040] (1) Overall structure of the inference protocol

[0041] This invention employs layer-wise pipeline parallelism as the primary inference protocol for the Dense model (see appendix). Figure 2 , Figure 3 Given the total number of layers in the model. Maximum number of layers per node Number of nodes required for a single production line All in the system The number of concurrent pipelines that can be formed from each available node. The calculation formula is as follows.

[0042]

[0043] Let the single token generation throughput of each pipeline be... (token / s), then the total system throughput is .in It is determined by the time of the slowest stage of the production line. Under steady-state full load, Each node works concurrently, and at each time step, each node processes activation values ​​from different requests. The system throughput is affected by the computation time of a single bottleneck node. With transmission delay Common constraints:

[0044]

[0045] in, The computation time for a single node to process one Decode token step in its assigned layer, rather than the entire time. The sum of the times of each node. When the network latency between nodes... much smaller In the same gigabit network (within the same region), the throughput bottleneck is determined by the GPU's computing power. Cross-regional scenarios Enlarging becomes a bottleneck; at this point, higher elevations should be prioritized. Tier-S3 nodes to reduce pipeline depth This reduces the cumulative network latency in the end-to-end latency of a single request.

[0046] (2) Activation value data packet format

[0047] Nodes transmit activation tensors via the gRPC protocol, with each packet carrying inference state and commitment fields. The packet contains the following core fields: a request unique identifier; an index of currently completed layers; a compressed activation tensor; KV cache metadata for receiving nodes to update their local cache; an execution configuration identifier for replayable validation constraints; and an input activation Merkle commitment for receiving nodes to verify activation continuity. The format definition is given in Appendix 3.

[0048] The size of a single token in the activation tensor is determined by the hidden layer dimension of the model. Decide( (Bytes, bf16 format), which can be reduced in size by zstd compression.

[0049] (3) Two-stage reasoning

[0050] Pre-filling phase (executed in one go): After the first node receives the embedding tensor of the input token, it performs forward computation of all layers sequentially along the pipeline. Each node fills the KV cache of its responsible layer in turn, and the last node outputs the logits of the first token. The time taken in the pre-filling phase is mainly determined by the length of the input sequence and the network transmission.

[0051] Decode phase (executed in an autoregressive loop, see appendix) Figure 2 Each time a new token is generated, the activation tensor is passed around the pipeline. Each node queries its local KV cache to complete the attention calculation, updates and passes the activation value to downstream nodes. Each Decode step generates a new stage commitment for on-chain verification.

[0052] (4) Micro-batch scheduling: Eliminating bubbles in the pipeline

[0053] The scheduler maintains a global request queue, continuously feeding inference requests from different users into the pipeline (see appendix). Figure 3 During the start-up bubble phase (the transition time from an empty to a full pipeline, approximately equal to the pipeline depth)... (At each time step), downstream nodes are in a waiting state, and the GPU is partially idling; after entering the steady-state full load period, each time step in the pipeline has exactly one... The requests are located on different nodes, and each node's GPU is always performing computational tasks, with utilization approaching 100%.

[0054] The percentage of time spent in the bubble phase is related to the pipeline depth and the request arrival rate, and its calculation formula is as follows.

[0055]

[0056] in This represents the number of micro-batches processed simultaneously in parallel. The larger the batch size, the lower the bubble ratio. The scheduler strikes a balance between latency constraints (user wait time) and bubble rate by dynamically adjusting the micro-batch size.

[0057] (5) MoE Expert Parallel Routing Extension Protocol

[0058] For large language models in a hybrid expert (MoE) architecture, a parallel routing extension protocol is designed for the Expert-FFN layer, enabling the Top-K experts of each MoE layer to execute concurrently on an expert execution network distributed across different nodes (see Appendix). Figure 4 ).

[0059] MoE routing packets are generated by the Router Network after computation and broadcast concurrently to the Top-K expert nodes. The packets contain the following core fields: request identifier; a list of sequence positions of each token in the current batch; a list of Top-K expert IDs selected by the router; a list of corresponding normalized gating weights; current capacity metadata for each expert; an overflow handling policy identifier; and a compressed token hidden state, the format of which is defined in Appendix 4.

[0060] After all expert nodes have completed their concurrent computations, they will return the output activation tensor to the weighted aggregation node for execution. Then it enters the dense attention layer of the next Transformer module. The end-to-end latency of the entire expert execution network is approximately equal to the single-expert computation latency, which is the core throughput advantage of the MoE architecture over the Dense architecture in distributed inference.

[0061] The expert capacity overflow three-level strategy is to first route to the same-region replica, then use CPU-MoE as a backup, and finally report capacity events and dynamically expand replicas to ensure the service availability of expert nodes under high load conditions.

[0062] (iii) Regarding the two-stage optimistic fraud proof agreement;

[0063] The raw activation hash is upgraded to a phased commitment that includes input / output activation Merkle commitments and execution configuration identifiers, and disputes are narrowed down to independently verifiable computational fragments through a challenge-replay protocol; specifically including:

[0064] (1) Security vulnerabilities and solution design of raw hashing

[0065] The current design uses the activation value hash as on-chain proof of a node's work, which contains a fundamental security vulnerability: a dishonest node can submit a "correctly formatted" commitment using any vector through a hash function, while the activation value corresponding to that hash is completely different from the honest computation result. The hash proves that "the data exists," but it does not prove that "the computation is correct."

[0066] This invention proposes a two-stage optimistic fraud proof, upgrading the semantics of commitment from "I submitted a certain activation value" to "My input activation is consistent with the output of the preceding node, and produces a deterministic output activation under a given execution configuration." Specifically, it is divided into a commitment phase and a challenge-replay phase.

[0067] (2) First stage: Stage Commitment

[0068] After each node completes forward inference, it generates the following stage commitment structure, signs it, and broadcasts it on the chain. The core principle of the commitment field design is that all factors affecting the calculation result are fixed by the commitment, making "replay under the same input and configuration" the only verification path. This commitment structure contains the following fields: a unique request identifier; a model version hash; a SHA256 hash of the shard list, used to fix the weight content; the layer range undertaken by the node; the Merkle root activated by the input, which can be compared with the output commitment of the upstream node to detect data tampering; the Merkle root activated by the output, which is the target of the dispute; the token range to be processed; a random seed allocated by the scheduler, used to fix the randomness of sampling; an execution configuration identifier, which constrains the quantization type and precision mode; a nanosecond timestamp; and an Ed25519 signature of the node's private key. The complete type definition of the structure is shown in Appendix 5.

[0069] The promised Merkle root constructor: will Activation tensor of each token step Calculate the hash of each leaf node one by one Then, a binary Merkle tree is constructed, and the root node hash is used as input to activate a Merkle commitment or as output to activate a Merkle commitment. The validator only needs to provide the path length during the challenge. The Merkle proof can prove the existence and correctness of any single-step activation value without retransmitting the complete activation tensor.

[0070] Continuity verification: Node Received node When processing the activation value data packet, verify its input activation Merkle commitment field against the node. The output activation Merkle commitment in the StageCommitment is consistent with the value of the Merkle commitment, forming a commitment chain across nodes to ensure that the activation value is not tampered with during the flow between nodes.

[0071] (3) Second stage: dispute narrowing and replayable verification

[0072] Any network participant can argue that a node's output activating a Merkle commitment contradicts honest computation, and after initiating a challenge, the system proceeds as follows: The two-way split agreement narrows the scope of the dispute:

[0073] The Bisection Protocol executes as follows: The midpoint of the current disputed interval is selected, and the challenged node is required to provide a Merkle proof of the activation value at that position. If the proof does not match the hash value of the corresponding intermediate node in the commitment tree, the disputed range is shrunk to the left half of the interval; if they match, it is shrunk to the right half. This process is repeated, halving the disputed range each round, until the disputed range is compressed into a single-level computational fragment of a single token step.

[0074] The dispute is ultimately narrowed down to a single step (single layer, single token), where a referee node performs a deterministic replay under the same execution configuration identifier constraint. The execution logic for replay verification is as follows: First, the inference engine is initialized with the quantization type, precision mode, and random seed specified by the execution configuration identifier, and the corresponding shard weights are loaded; then, the disputed input activation is used as input to perform a deterministic forward computation to obtain the replay output; finally, the Merkle root of the replay output is calculated and compared with the output activation Merkle commitment in the phase commitment—if they match, honesty is determined; if they do not match, cheating is determined, triggering a staking forfeiture penalty.

[0075] The execution configuration identifier locks the quantization type, precision mode, and random seed, providing a deterministic basis for replay verification. Considering the legitimate implementation-level differences in GPU floating-point operations across different hardware architectures or driver versions, this invention employs the following mechanism for determining replay consistency:

[0076] Deterministic path: When the inference path uses INT4 integer lookup table operations (quantized matrix multiplication), the output results have bit-exact consistency across hardware bits under the same weights and inputs. In this case, replay_fragment directly compares the Merkle roots.

[0077] Tolerance Path: When the inference path contains bf16 floating-point operations, an error tolerance threshold is introduced. The judgment criteria have been relaxed from equivalence comparison to:

[0078]

[0079] in, The quantization precision is determined by the quantization precision configuration in the execution configuration identifier (e.g., under bf16 precision). (corresponding to the least significant bit of bf16).

[0080] Referee node constraint: The referee node must declare the same GPU architecture as the audited node. This field is written into the stage commitment as a required subfield of the execution configuration identifier to avoid cross-generation GPU architecture differences exceeding the tolerance threshold.

[0081] If the replay result exceeds the tolerance range, the smart contract will automatically trigger a slash penalty; if the result is consistent, the challenge fails and the challenger's stake is deducted to prevent malicious challenges and harassment.

[0082] (4) Economic security analysis of the verification mechanism

[0083] Let the reward for a single successful cheating by a node be... The penalty amount for confiscation of pledged shares after inspection is... The overall detection rate is (Based on the joint probability of random sampling and valid challenges), the expected payoff for cheating is:

[0084]

[0085] make The sufficient condition is Overall detection rate Based on random sampling rate Challenger monitoring coverage and conditional detection rate The three parties jointly decided:

[0086]

[0087] set up , (Deterministic judgments that can be replayed and verified) to ensure that in the most unfavorable scenario ( The game equilibrium remains based on honest reasoning, and the system is set up accordingly. (That is, the slash amount is no less than 6 times the single reward). In actual deployment, Set as the total staked amount for the node (usually much higher than) Furthermore, with the improvement of the challenger incentive mechanism (driven by whistleblower rewards), (Enhancement), security margin increases with network activity.

[0088] (iv) On the asynchronous separation mechanism of reasoning response and economic settlement

[0089] User streaming begins immediately after the minimum offline acceptance conditions are met, and the on-chain economic endgame is determined after the challenge window closes; the two events do not block each other. Specifically, this includes:

[0090] Users expect a first-token latency (TTFT) and streaming output in milliseconds to seconds for inference response timing, while the blockchain's settlement finality (from the commit phase to the closing of the challenge window) takes several minutes. If on-chain settlement is placed on the critical path of the response, users will experience unacceptable waiting delays. This invention completely decouples the above contradiction by explicitly defining "user-visible inference completion" and "on-chain economic settlement finality" as two independent timing events.

[0091] (1) Formal definition of two time-series events

[0092] event (Inference Response Complete): Triggered when the API gateway determines that the Offline Acceptance Condition (OAC) is met, defined as:

[0093]

[0094] in, For the first Phase commitments for each pipeline node, This represents the pipeline depth. Once OAC is satisfied, the API gateway immediately transmits the inference results to the user via Server-Sent Events (SSE), regardless of whether the challenge window is closed.

[0095] event (Economic Settlement Endgame): The Window of Opportunity (Default 5 minutes) If the dispute is closed and there is no valid challenge, or if the dispute adjudicator contract completes its ruling, the smart contract will execute the token distribution.

[0096] The temporal relationship between the two events is as follows: ,Right now It occurred much earlier than Users begin receiving streaming output within seconds of completing inference, which runs completely in parallel with the on-chain settlement process.

[0097] (2) Parallel execution on the user side and the chain side

[0098]

[0099] exist and During this period, the system is in an "optimistic acceptance" state: the user has received a response, the node's token rewards are locked and awaiting release, and the challenge window remains open. If the challenge is successful, the system will execute a staking forfeiture penalty and rollback based on the existing on-chain records. The rollback only affects token allocation and does not affect the reasoning results already delivered to the user.

[0100] The OAC mechanism ensures economic security, not the deterministic correctness of the reasoning result. The correctness guarantee provided by this invention is probabilistic: provided that less than 1 / 3 of the pipeline nodes are malicious and the effective challenge mechanism is functioning normally, the probability of an incorrect reasoning result being accepted by the user is limited. Constraints. For application scenarios where correctness requirements exceed this upper bound of probability, it is recommended to wait until the challenge window closes before using the inference results, or to additionally require the output commitment of the last node and the input commitment of the first node to complete on-chain lightweight causality verification in the OAC condition. The main applicable scenarios of this invention are general dialogue, code generation, and other applications where probabilistic correctness is acceptable.

[0101] (v) Regarding the seven-dimensional scheduling scoring function for pledge perception

[0102] Node staking weight, historical failure rate, and shard continuity are incorporated into the scheduling score to link the scheduling strategy with the decentralized incentive mechanism; specifically including:

[0103] (1) Definition of scoring function

[0104] This invention proposes a seven-dimensional scoring function to select the optimal set of nodes from a distributed hash table (DHT) during the pipeline assembly phase. The scoring function integrates real-time performance metrics (latency, throughput, queue depth), historical reliability metrics (failure rate), economic incentive metrics (staking weight), and topology optimization metrics (geographic affinity, sharding continuity).

[0105]

[0106] Normalization functions for each dimension The definitions are as follows, totaling 7 items:

[0107] (1) Measured RTT from node to downstream node: Default attenuation coefficient When RTT=3ms When RTT=100ms .

[0108] (2) Historical average throughput: The historical average tokens / s of the nodes are normalized to the global maximum value.

[0109] (3) Recent task failure rate: The higher the failure rate, the lower the score. The score is updated based on the exponentially weighted moving average over the past 30 days.

[0110] (4) Current task queue depth: , This represents the number of tasks currently pending, used for load balancing.

[0111] (5) Pledge weight: The proportion of tokens staked by a node to the total amount staked across the entire network.

[0112] (6) Geographical affinity: The value is set in three tiers: 1.0 within the same Autonomous System (ASN), 0.6 within the same country, and 0.3 across different geographical continents, to reduce cross-regional latency.

[0113] (7) Continuity of segments: If adjacent node segments are continuous, otherwise take 0.5 to reduce data retransmission.

[0114] Default weight vector: It is stored in an on-chain parameter contract and supports adjustments for community governance.

[0115] (2) Greedy assembly algorithm for pipeline

[0116] The scheduler uses a greedy strategy, sequentially determining the position of each stage of the pipeline (total...). Select the optimal node (i). The specific steps are as follows: For the first node... For each layer segment, a candidate node set is selected from the DHT node set that meets the service level requirements, covers the target layer segment, and is currently registered. Based on this, the geographic location of previously selected nodes is updated. and Dimension, calculate the comprehensive score of each candidate node in the current context; select the node with the highest score as the first candidate node. The segment is assigned to a specific recipient and removed from the candidate set to avoid duplicate assignment; this process is repeated until all segments are assigned. Each layer completes node selection. The pseudocode representation of the algorithm is shown in Appendix 6.

[0117] Staking weight dimension in the scoring function This directly links node staking behavior to task allocation priority, creating a positive cycle of "higher staking, more allocation, and higher returns," while a negative penalty mechanism of "higher failure rate, lower score, and fewer allocations" drives nodes to maintain high availability.

[0118] (3) Dynamic updates of scoring

[0119] The scheduler maintains sliding window statistics for each node: each time an inference task is completed (or times out / fails), the exponentially weighted moving average of that node is updated. ) and moving average ( When node staking changes, Real-time recalculation; when DHT receives a node online / offline notification, it triggers a refresh of the hot standby candidate pool for the relevant pipeline.

[0120] (vi) On the 8-state machine of nodes and hot standby fault tolerance mechanism

[0121] Define an eight-state node lifecycle state machine, and in conjunction with client-side recovery cache, implement a highly available inference service with rapid failover, specifically including:

[0122] (1) Node lifecycle eight-state machine

[0123] This invention defines a complete node lifecycle state machine (see appendix). Figure 5 This covers the entire process from registration to settlement. The hot standby switching logic driving the scheduler sequentially enters the following states: registered, assigned, warm, active, degraded, replaced, disputed, and settled. The semantics of each state and the typical triggering conditions for switching are as follows:

[0124] (1) Registered: The node has completed storage proof and staking and is waiting for task allocation. After the scheduler scores and selects it, it will be converted to allocated.

[0125] (2) Allocated: The node has received the task instruction and is loading the KV Cache context checkpoint. After the activation value arrives for the first time, it will be marked as preheated.

[0126] (3) Preheated: The KV Cache has been pre-filled and can immediately participate in the Decode computation. After the first forward computation is performed, it becomes active.

[0127] (4) Active: During normal inference, the node continuously receives and transmits activation values. After more than 15 seconds of continuous heartbeats, it is downgraded; when the phase commitment is challenged, it is replaced.

[0128] (5) Degradation: Heartbeat timeout or error rate increase triggers hot standby startup process. After the standby node completes the switchover, the current node becomes replaced; if the node recovers its heartbeat, it becomes active again.

[0129] (6) Replaced: The current node has been replaced by a hot standby node, and the inference pipeline continues normally. After the node is re-registered, it will be marked as registered.

[0130] (7) In Dispute: The stage commitment is challenged and awaits the decision of the referee node. After the final determination is made, regardless of whether it is honesty or cheating, it will be converted to settlement.

[0131] (8) Settled: The economic settlement for the current reasoning task is complete. The node enters the next task cycle and is re-registered or assigned.

[0132] (2) Hot standby switching and client-side cache recovery

[0133] The scheduler maintains 1–2 pre-warmed hot standby nodes for each stage in the pipeline. The hot standby nodes have completed shard loading and KV cache initialization. During switching, only the activation value from the last step needs to be passed in to immediately resume computation without reloading the weights.

[0134] Client-side recovery cache: API gateway cache most recently (Default 32) Intermediate activation tensors for decoding steps. When a transition from an active state to a degraded state is detected, the gateway retrieves the last activation value from the cache and sends it directly to the hot standby node that is already in a preheating state. The hot standby node can continue forward computation from this activation value without having to re-execute from the pre-filling stage.

[0135] When a hot standby node takes over, in addition to the activation value, a complete KV cache must be rebuilt (the self-attention mechanism requires holding the key-value history from the beginning of the sequence to the current position). This invention defines a three-level KV cache migration strategy, which the scheduler selects based on pipeline health and bandwidth budget:

[0136] Strategy 1 (Real-time Synchronization, Applicable to Critical Pipelines): During the pre-warm-up phase, the hot standby node establishes an incremental KV cache synchronization channel with the primary node. After each Decode step is completed, the newly added KV block is pushed to the hot standby node. During failover, the hot standby node's KV cache is already synchronized, eliminating the need for rebuilding and minimizing failover overhead.

[0137] Strategy 2 (Periodic checkpoints, suitable for general pipelines): The scheduler checks every... (Default 16) Decode steps write the complete KV cache of the master node to the highly available distributed storage. After failover, the hot standby node loads the most recent checkpoint and uses the client to restore the cached activation value for replay. Within a few steps of history, the total cost of reconstruction shall not exceed The calculation time for each step.

[0138] Strategy 3 (Lightweight Rebuild, suitable for low-bandwidth environments): Hot standby nodes do not undergo pre-synchronization. After failover, the hot standby node, in conjunction with the client, restores the cache (most recent). The initial KV state (activated value) is forcibly stored in the distributed storage when the pre-filling phase is completed, and the complete KV cache from the end of the pre-filling phase to the switching occurs is rebuilt, introducing a reconstruction time proportional to the length of the historical sequence.

[0139] Pipeline checkpoint mechanism: The scheduler checks every [time / time]. (Default 8) A checkpoint node is set up at layer 8 to write the currently active intermediate tensor to a highly available distributed storage. When a serious pipeline failure occurs, recovery starts from the most recent checkpoint, with a replay cost of at most [missing information]. The computational cost per layer, rather than the entire pre-filling stage.

[0140] (3) Node registration and sharded storage proof

[0141] To join the network, a node must complete five steps in sequence: reporting hardware capabilities and service level declaration, staking tokens on-chain, downloading GGUF weighted shards via P2P, submitting shard storage proof, and broadcasting (NodeAnnouncement). The shard storage proof prevents fraudulent node registration; the on-chain contract randomly selects several tensor names and requires nodes to provide corresponding data segments to verify that the node actually holds the shard files. The storage proof is re-verified every 7 days.

[0142] The core structure of the node registration record includes the following fields: unique node identifier, identifier of the model it supports, range of the support layer, available video memory logical quota, service level enumeration, uplink bandwidth, geographic region code, autonomous system number, number of staked tokens, hash of the shard list held, and shard storage proof. The complete type definition of the structure can be found in Appendix 7.

[0143] The main technical features and advantages of this invention are as follows:

[0144] (1) The low memory quota-aware fragment placement mechanism driven by GGUF fragment list decouples the logical service quota from the physical GPU memory and defines a Tier-S1 / S2 / S3 three-level service layer.

[0145] (2) Two-phase optimistic fraud proof protocol, which replaces the raw activation hash with a phase commitment containing input / output activation Merkle commitment, and achieves replayable fragment-level verification by disputed binary search, thus solving the problem of proving the correctness of untrusted nodes;

[0146] (3) The reasoning response and on-chain economic settlement are asynchronously separated to eliminate the impact of blockchain latency on user experience;

[0147] (4) The seven-dimensional scoring function of staking perception (combining RTT, throughput, failure rate, queue depth, staking weight, geographical affinity, and sharding continuity) drives task scheduling and is linked with decentralized economic incentives.

[0148] (5) An eight-state node state machine and a client-side recovery cache enable low-latency and high-availability inference during failover. Attached Figure Description

[0149] Figure 1This is a schematic diagram of the architecture of a blockchain-incentivized consumer-grade GPU cluster distributed large language model inference system.

[0150] Figure 2 This is a complete flowchart of the pipelined parallel inference part of the hierarchical pipelined parallel inference protocol.

[0151] Figure 3 This is a micro-batch pipeline scheduling diagram for the hierarchical pipelined parallel inference protocol.

[0152] Figure 4 This is a comparison diagram of the MoE expert parallel protocol and the Dense model for the hierarchical pipelined parallel inference protocol.

[0153] Figure 5 This is a flowchart illustrating the proof-of-work process for node registration and inference in the octagonal state machine and hot standby fault-tolerant mechanism. Detailed Implementation

[0154] The method of the present invention will be further described below with reference to specific embodiments. It should be understood that the following embodiments are used to illustrate the present invention and not to limit the scope of the present invention. In different embodiments, the details of the technical solution of the present invention may vary, but its spirit and principle remain consistent.

[0155] Example 1: Joint inference of large-scale nodes in the same region (Dense architecture model, high concurrency scenario)

[0156] 1. Deployment parameters and fragment placement calculations

[0157] This embodiment employs 10,000 consumer-grade PC nodes distributed across the same geographical region (same ASN), with an RTT (Real-Time To Transaction) between nodes not exceeding 10ms. Each node is equipped with a consumer-grade GPU with 4GB of physical video memory, and the usable_vram_quota is configured to 1,024MB (Tier-S1), with the remaining physical video memory reserved for the operating system and other applications. Each node has 16GB of system memory, supporting CPU overflow computation. The inference object is a Dense Transformer model (INT4 quantization) with approximately 8B parameters, consisting of 32 layers, with each layer having approximately 140MB of weights.

[0158] Layer allocation calculation:

[0159]

[0160] Actual conservative configuration (Reserving more KV cache space for batch processing buffering), at this time, the weight of each node occupies MB, KV Cache reserves 80MB, system reserves 200MB, total usage is 840MB, less than MB, with the remainder available for context checkpoint caching.

[0161] Pipeline concurrency capability:

[0162] Total number of layers in the model Each node :

[0163]

[0164] Single-node Decode computation time Taking a GTX 1060 (192 GB / s memory bandwidth) running Q4 quantization with 4 layers (4096 hidden dimensions) as an example, with each layer's weight approximately 36MB, the memory read and computation for the single token decoding step is approximately... ,Pick Inter-node transmission delay (Same autonomous system RTT).

[0165] Apply the modified throughput formula (M01):

[0166]

[0167] Total system throughput:

[0168]

[0169] Bubble ratio in production line ( Micro-batch parallelism): After reaching steady state, it drops to In actual scheduling, the bubble period is compressed into the startup phase by continuously filling in different user requests. – (See attached) Figure 3 ).

[0170] 2. Node Registration Process

[0171] Taking Node-A as an example, the complete registration process is as follows:

[0172] Step 1: Report hardware capabilities. Node-A sends a registration request to the DHT, reporting usable_vram_quota=1024MB, bandwidth=100Mbps, region=CN-BJ, and asn=4134. The system classifies it into Tier-S1 according to the service layer definition. node_id=SHA256(pubkey_A).

[0173] Step 2: On-chain staking. Node-A calls the smart contract `stake(100 TOKEN)`, locking the tokens to the contract. Staking weight. Write the score to the scheduler's score cache.

[0174] Step 3: P2P Download of Fragments. Node-A pulls the fragment list file from IPFS, according to... Select the bearer layer range [0, 3], download the corresponding 4 GGUF fragment files (each about 140MB, total 560MB), and the SHA256 verification passes.

[0175] Step 4: Submit storage proof. Node-A constructs a PostProof { node_id, layer_range=[0,3],shard_hash, timestamp} and broadcasts it on-chain. The contract randomly selects three tensor names (e.g., blk.0.attn_q.weight, blk.2.ffn_down.weight, blk.3.attn_v.weight), and Node-A provides the corresponding data segment and Merkle proof. The contract verification is successful.

[0176] Step 5: Broadcast NodeAnnouncement. Node-A broadcasts { node_id, layer_range=[0,3],tier=S1, bandwidth=100Mbps, region=CN-BJ, asn=4134, stake=100, shard_manifest_hash= <hash>The status changes to registered, and the inference service enters the ready state.

[0177] 3. The step-by-step execution process of a complete reasoning process

[0178] User Request: The user submits a reasoning request with an input of 512 tokens and a desired output of 256 tokens. The API gateway pre-locks tokens from the user-hosted contract. Tokens (For a single token price), the request is pushed into the global task queue.

[0179] Scheduler scoring for node selection (applying a staking-aware seven-dimensional scheduling scoring function): The scheduler filters candidate nodes from the DHT according to Tier-S1, selecting the highest-scoring node from each of the eight pipeline positions (layers [0–3], [4–7], …, [28–31]). Taking candidate node Node-A in layer [0–3] as an example, the scoring is calculated as follows (using default settings). ):

[0180] RTT dimensions ( ):

[0181] Throughput dimension ( ):

[0182] Reliability dimension ( ):

[0183] Queue Dimension ( ):

[0184] Staking Dimensions ( ):

[0185] Geographical dimension ( ):

[0186] Continuity dimension ( ):

[0187] total:

[0188] Select the 8 nodes with the highest scores to form an ordered pipeline: Node-A[0–3], Node-B[4–7], …, Node-H[28–31], and change the state of each node to assigned.

[0189] Node preheating (already assigned to preheated): The scheduler sends task instructions to 8 nodes respectively. Each node loads the KV Cache context checkpoint according to exec_profile_id=cuda_bf16_deterministic. After it is ready, the status changes to preheated.

[0190] Pre-filling phase execution: Node-A receives an embedding tensor of 512 tokens (approximately [size missing]). The pre-filling phase (MB) performs forward computation on layers 0–3, fills the local KV cache, generates an activation tensor, and packages it into an activation value data packet (ActivationPacket) containing input_commit=MerkleRoot({h_0,…,h_{511}}), which is then sent to Node-B via gRPC. Node-A then becomes active. Each node executes the process sequentially, with the final node, Node-H, outputting the logits of the first token. The total pre-filling phase takes approximately 2–5 seconds (including 8 network transmissions, each with approximately 10ms × 8 = 80ms network overhead, and computation taking approximately 2–4 seconds).

[0191] Decode autoregressive loop (executes 256 times): In each loop step, Node-A receives the single token activation from the previous step (approximately...). (Approximately 4.8KB after zstd compression) Query the local KV Cache to perform attention calculations, generate new activation values, update input_commit, and send it to Node-B... and so on. Node-H samples to obtain the new token and streams it to the user through the API gateway. Each loop takes approximately... ms, which is approximately 15 tokens per second. It would take about 17 seconds to complete the process with 256 tokens.

[0192] Phase Commitment Commitment (in parallel with Decode): After completing all Decode steps for this inference, each node constructs a phase commitment using fields such as request identifier, shard list hash, layer range, input activation Merkle commitment, output activation Merkle commitment, token processing range, random seed, and execution configuration identifier. This commitment is then signed with the node's private key and broadcast to the chain. Taking Node-A as an example, the values ​​for each field in its commitment are: request identifier "req-20260524-001"; layer range from layer 0 to 3; input activation Merkle commitment constructed from the input activation tensor sequence of 256 Decode steps; output activation Merkle commitment constructed from the corresponding output activation sequence; token processing range from 0 to 255; random seed a fixed value pre-allocated by the scheduler; execution configuration identifier "cuda_bf16_deterministic"; and signature generated using the Ed25519 algorithm with the node's private key.

[0193] Event reasoning response completed and triggered The API gateway detected that 6 out of 8 nodes were already active. The phase commitment is visible on the chain, satisfies OAC, and begins to stream inference results to users via server push events (SSE), with users perceiving the token-by-token streaming output.

[0194] 5-minute challenge window: In this example, no validator issues a challenge. After the window closes, Triggered, smart contract according to Distribute token rewards to each node and unfreeze locked staked tokens.

[0195] 4. Fault switching demonstration (application of state machine and hot standby mechanism)

[0196] Suppose that when the reasoning process reaches the 150th token, Node-E (layers 16–19, active state) suddenly disconnects due to a local network failure:

[0197] (1) Heartbeat timeout detection: If Node-D does not receive a heartbeat packet from Node-E for 15 consecutive seconds, it sends TIMEOUT_EVENT { node_id: "Node-E", position: 5, last_act_cached: true} to the scheduler. The scheduler marks Node-E's status as downgraded.

[0198] (2) Hot standby node activation: The scheduler retrieves the hot standby candidate pool for the corresponding layer segment [16–19] of Node-E, selects Node-E2 with the highest score (which is in a preheated state and has preloaded the weights of layer 16–19), and changes its state to active.

[0199] (3) Restore cache and retransmit: The API gateway retrieves the activation value (approximately 4.8KB) output by Node-D in step 149 from the client-side restore cache and sends it directly to Node-E2. Node-E2 does not need to repeat the pre-filling stage and continues to calculate forward from the activation value in step 150.

[0200] (4) Inference recovery: After Node-E2 completes the calculation, it sends the 150th step activation tensor to Node-F, and the entire pipeline returns to normal.

[0201] (5) On-chain records: Node-E disconnection events are recorded on-chain and scored by the scheduler. Due to this disconnection, the credit score will be adjusted. If the number of disconnections exceeds the threshold within 30 days, a 20% staking slash will be triggered; this is an isolated event and only the credit score will be updated.

[0202] Example 2: Cross-regional public P2P network deployment (Internet heterogeneous node scenario)

[0203] 1. Deployment parameters and scheduling strategies

[0204] This embodiment simulates nodes distributed across three regions: Asia (40%), Europe (35%), and North America (25%). Cross-regional RTT varies due to network differences: approximately 100–150 ms between Asia and Europe, 150–200 ms between Asia and America, and 80–120 ms between Europe and America; RTT within the same region is 10–30 ms. Node service levels are mixed, with Tier-S1 accounting for 60%, Tier-S2 for 30%, and Tier-S3 for 10%. Geographic affinity parameters are configured according to the following rules: nodes within the same Autonomous System (ASN)... The value is set to 1.0, 0.6 for nodes within the same country, and 0.3 for nodes across continents. The scheduler prioritizes building pipelines within the same region with high geographical affinity.

[0205] 2. Setup of the scheduler's geographic sensing pipeline

[0206] When the scheduler receives an inference request, it uses the seven-dimensional scheduling scoring function based on the pledge perception. Based on the (geographic affinity) dimension, priority will be given to building pipelines within the same region. For example, consider a request from an Asian user:

[0207] If there are enough Tier-S1 nodes in the Asian region, the scoring function selects all 8 nodes located in Asia. ), Optimal, establish production lines in the same region. tokens / s;

[0208] If there is a shortage of nodes in the Asian region (high queue depth) (Decline), the scheduler retreats to allow the introduction of 1–2 European nodes ( In the pipeline, the RTT for intercontinental jumps is approximately 120ms, and after stacking 8 nodes... MS became the bottleneck. The system automatically suppresses the number of such mixed pipelines by using tokens / s and scheduling scores.

[0209] 3. Performance comparison in cross-regional scenarios

[0210] In a purely local scenario (RTT ≤ 10ms), a single pipeline throughput is approximately 15 tokens / s, with a first token latency of 2–5 seconds and a failover time of no more than 200ms. The pipeline bubble ratio approaches zero in steady state. However, in a hybrid pipeline with cross-regional nodes (representative RTT approximately 120ms), the throughput of a single pipeline drops to 4–8 tokens / s, the first token latency extends to 5–15 seconds, and the cross-continental hot standby failover has an upper limit of approximately 600ms due to the additional RTT time. The startup bubble ratio follows the same pattern in both scenarios. (Approximately 47% in micro-batch production) all approach zero in steady state.

[0211] The scheduler continuously monitors the scores of nodes in each region, overflowing requests from high-load regions to adjacent regions, and processes this information in the scoring function. (Queue depth) and The combined penalty of (geographic affinity) automatically suppresses the proportion of cross-regional pipelines, maintaining the overall system throughput within an acceptable range.

[0212] 4. Economic security verification

[0213] This example simulates a malicious node Node-X submitting incorrect calculation results during the inference process:

[0214] Phase commitment commit: After completing the computation of layer [8–11], Node-X deliberately commits an incorrect output activation Merkle commitment, corresponding to a randomly forged activation tensor.

[0215] Continuity check trigger: When downstream node Node-Y receives the activation value packet, it compares the output activation Merkle commitment of Node-X with the input activation Merkle commitment of the received packet. If an inconsistency is found, it sends an alarm to the scheduler.

[0216] Challenge Initiation: Node-Y or an independent monitoring node can act as a validator to initiate an on-chain challenge to Node-X's phase commitments, staking a certain amount of tokens as challenge deposit.

[0217] The binary search protocol is executed as follows: In the first round, the contract requires Node-X to provide a Merkle proof of the activation value of the midpoint (token 128) in the token range [0–255]. If the Merkle proof provided by Node-X contradicts the root of the commitment tree, the scope of the dispute is narrowed to [0–128]. In the second round, the scope of the dispute is narrowed to [0–63], in the third round to [0–31], and so on. The round is reduced to a single step (token 0, the first Decode step); the referee node replays the forward computation of the layer [8–11] corresponding to token 0 under the constraint exec_profile_id=cuda_bf16_deterministic. The output activation Merkle root does not match the output activation Merkle commitment (output_commit) in the Node-X commitment, and cheating is judged.

[0218] Slash Execution: The smart contract confiscates all 100 tokens staked in Node-X, of which 5% (5 tokens) is transferred to the reporting validator, and the remainder is destroyed or added to the protocol vault; Node-X is permanently blacklisted, and subsequent NodeAnnouncement broadcasts will be rejected by the DHT.

[0219] Detection rate statistics: In the 3-month simulation operation, in the replay verification triggered by 5% random sampling, the replay consistency rate of honest nodes in deterministic execution mode was 99.97% (a very small number of inconsistencies were caused by hardware ECC errors, which were handled by the outlier removal mechanism); the detection rate of active cheating nodes was 100% (any node that submits an erroneous output to activate the Merkle commitment will fail the replay verification).

[0220] Example 3: Expert Parallel Asynchronous Inference for Ultra-Large-Scale Models Based on the MoE Architecture

[0221] 1. Model parameters and node allocation

[0222] This embodiment uses a 671B parameter MoE architecture large language model (referencing the DeepSeek-V3 architecture, 61 Transformer blocks, 58 of which contain MoE layers, each layer has 256 experts, each time activating Top-8, and each expert is approximately 22MB after INT4 quantization) as an example to illustrate the complete execution process of the expert parallel routing protocol.

[0223] This embodiment deploys 500 Tier-S1 nodes ( GB). According to the video memory allocation formula, after deducting the system reserved (200MB) and KV Cache reserved (80MB) per node, the available storage space is... One expert. The total number of experts across the entire network is... The minimum number of nodes required for a single replica to be covered is: In this embodiment, the total number of slots exceeds the minimum requirement by approximately 11%, and the global average replicability is approximately 1.11 times. For hotspot experts using high-frequency routing, replicas can be dynamically expanded through a three-level overflow strategy, achieving a local replicability of 3 to 5 times. MLA compressed KV Cache (32x compression ratio) allows each 1GB node to support approximately 8,192 token contexts.

[0224] The responsibilities of the 500 nodes are divided as follows: approximately 80 nodes are responsible for the main pipeline role of the dense Attention layer (each node is responsible for the Attention layer of approximately one Transformer block, approximately 200MB), and the remaining GPU memory can be used for storage. One expert serves as an expert node; the remaining approximately 420 nodes are pure expert nodes, each storing 33 experts (fully configured). The total number of valid expert slots across the network is [number missing]. It can achieve full model expert single copy coverage and leave about 6% redundancy.

[0225] 2. Execution process of the MoE layer during a single inference iteration

[0226] Taking the execution of the 12th Transformer block of the model (including the MoE Expert-FFN layer) as an example, the input sequence length is 1,024 tokens (the first Decode step of one request):

[0227] (1) Dense Attention Layer Calculation: The mainstream waterline node Node-P12 performs the calculation of the 12th Attention layer (MLA compressed attention, KV occupies about 10MB), generating the hidden state vector hidden (size approximately (KB, bf16 format), transmitted to the Router Network.

[0228] (2) Router Network: The router (running locally on Node-P12) calculates the gating score for the hidden state and selects 8 experts from 256 experts, numbered 7, 23, 89, 134, 178, 201, 234, and 251, with corresponding normalized gating weight values ​​of approximately 0.18, 0.15, 0.14, 0.13, 0.12, 0.11, 0.10, and 0.07, respectively. Capacity metadata shows that expert #89 has a small remaining capacity in the current batch (approximately 2 tokens), and its replica nodes include Node-X3 and Node-X9; the overflow handling strategy is set to prioritize replicas in the same region (replica_first). The hidden state is compressed in bf16 format by zstd and sent concurrently to the above 8 expert nodes.

[0229] (3) Expert capacity detection and overflow handling: Expert #89 (capacity_tokens=2, i.e., only 2 tokens remain in the current batch) is detected as being close to its capacity limit. Since only 1 token is being processed at present, the capacity is sufficient and there is no need for overflow. If the batch contains more than 2 tokens routed to Expert #89, it is handled according to a three-level overflow strategy: Level 1 (same-region replica routing): Node-X9 (higher score, more empty queue) is selected from replica_node_ids=[X3,X9], and the overflow token is routed to the Expert #89 replica on Node-X9, with a latency increment of about 3ms (same as ASN RTT); Level 2 (CPU-MoE alternative): If both X3 and X9 are saturated, the scheduler specifies a node with ≥16GB of memory to load Expert #89 weight for execution through the --cpu-moe mode, with a latency increment of about 200ms, which is suitable for asynchronous task scenarios. Level 3 (Capacity Event Reporting): Regardless of the path taken, expert_id=89, overflow_count+=1 is recorded to the scheduler, triggering the automatic replication logic. That is, in the next scheduling cycle, the weight of Expert #89 is preloaded on high-scoring idle nodes.

[0230] (4) 8-Expert Concurrent Computation: The routed data packet is concurrently broadcast to 8 expert nodes (Node-X1 to Node-X8). Each node performs Expert FFN forward computation (forward inference with approximately 22MB weights, taking approximately 100-200ms) on its own GPU, outputs an Expert activation vector (approximately 14KB) and returns it to Node-P12.

[0231] (5) Weighted aggregation: After receiving the outputs of 8 experts, Node-P12 performs a weighted summation:

[0232]

[0233] The aggregation result is used as the output of this Transformer block and fed into the dense Attention layer of the next block for computation. The end-to-end latency of 8-expert concurrent computation is approximately equal to the latency of single-expert computation (approximately 150–250 ms), compared to the serial computation time. ms, concurrency brings about 8 times the latency reduction.

[0234] 3. Overall performance metrics of MoE inference

[0235] Taking the deployment scale of this embodiment (500 nodes, input 1,024 tokens) as an example, the time consumption of each stage is as follows:

[0236] Pre-filling stage (executed all at once): 61 Transformer blocks are processed serially, each block containing Attention computation (approximately 100ms) and concurrent Expert-FFN computation (approximately 200ms), with pure computation time approximately... Including the overhead of data transmission and scheduling coordination between nodes (approximately 1.7–6.7 s within the same region and approximately 5.7–11.7 s across regions), the total pre-filling phase time is approximately 20–25 s within the same region and approximately 24–30 s across regions.

[0237] Decode latency per step: With 8 experts executing concurrently, the latency is approximately 200ms, which is the same as the single expert's computation time. Adding routing distribution, result aggregation, and network transmission, which take about 50 to 200ms, the total latency per step is about 250 to 400ms.

[0238] The total time to output 256 tokens is approximately 65 to 110 seconds, which is suitable for asynchronous task scenarios with less stringent response time requirements, such as code generation and documentation writing.

[0239] Challenge window handling: Phase commitment submission and Decode execution run in parallel, without occupying the inference thread and having no impact on user-side output.

[0240] In the MoE scenario, the total inference time is approximately 65 to 110 seconds, which is much shorter than the 5-minute on-chain challenge window. Triggered approximately 100 seconds later (Decoding complete, OAC satisfied), the user begins receiving the result; 5 minutes later... When triggered, 500 nodes receive token rewards based on the proportion of experts / layers they undertake; asynchronous settlement has no impact on user experience.

[0241] appendix:

[0242] Appendix 1:

[0243] { "model_id": <model-hash>", "version": "1.0", "shards": [ { "shard_id": "shard_layers_s_e", "layer_range": [s, e], "file": " <shard-filename>.gguf", "sha256": " <integrity-hash>", "size_bytes": " <total-bytes>", "quant_type": "Q4_K_M", "tensor_names": ["blk.s.attn_q.weight", "..."], "kv_footprint_per_req_kb": "<B_kv in KB> ", "required_runtime": "llama.cpp>=b3800", "exec_profile_id": "cuda_bf16_deterministic" } ] }

[0244] Appendix 2:

[0245] llama-server \ --model<shard_manifest> .json --shard<shard_id> \ --n-gpu-layers<L_max> \ --ctx-size<context_window> \ --vram-quota<Q_in_MB>

[0246] Appendix 3:

[0247] message ActivationPacket { / / Activation value data packet definition string request_id = 1; / / Unique identifier for the request int32 layer_idx = 2; / / The current layer index is complete. bytes hidden = 3; / / bf16 activation tensor (zstd compressed, approximately d_model × 2 bytes) bytes kv_meta = 4; / / KV Cache metadata (updated by the receiving node's local cache) string exec_profile = 5; / / Execution configuration identifier used for replayable verification bytes input_commit = 6; / / This data packet's input activates the Merkle commitment (for receiving nodes to verify continuity)}

[0248] Appendix 4:

[0249] message MoERoutingPacket { string request_id = 1; repeated int32 token_positions = 2; / / Sequence position of each token in the batch repeated int32 expert_ids = 3; / / List of Top-K expert IDs repeated float gate_weights = 4; / / Normalized gate weights (summing to 1) ExpertCapacityMeta capacity = 5; / / Current capacity metadata for each expert string overflow_policy = 6; / / Overflow handling policy identifier bytes hidden_state = 7; / / Token hidden state (bf16, zstd compression) }

[0250] Appendix 5:

[0251] struct StageCommitment { request_id: String, / / Unique identifier for the request model_version: String, / / Model version hash (including architecture and quantization method) shard_manifest_hash: [u8; 32], / / Shard manifest SHA256 (fixed weight content) layer_range: [u32; 2], / / Range of the layer [start, end] input_commit: [u8; 32], / / Input activates the Merkle root (comparable to the upstream output commitment) output_commit: [u8; 32], / / Output the activated Merkle root (disputed target) token_interval: [u64; 2], / / The token interval to be processed [start, end] random_seed: u64, / / Random seed assigned by the scheduler (fixed sampling randomness) exec_profile_id: String, / / Execution configuration identifier (quantization type, precision mode) timestamp: u64, / / Nanosecond timestamp node_signature: [u8; 64], / / Signature of node private key Ed25519 }

[0252] Appendix 6:

[0253] Input: DHT node set V, pipeline depth P, each segment [s_i, e_i] Output: an ordered list of nodes [n_1, n_2, ..., n_P] for i in 1..P: candidates = {v ∈ V : tier(v) satisfies the hierarchical requirements of segment i} AND layer_range(v) covers [s_i, e_i] AND status(v) == registered} / / Considering the geographical locations of the previously selected nodes, update f_6 and f_7 n_i = argmax_{v ∈ candidates} Score(v | n_{1..i-1}) V = V \ {n_i} / / Selected nodes do not participate in subsequent level competition

[0254] Appendix 7:

[0255] struct NodeAnnouncement { node_id: [u8; 32], / / SHA256(pubkey) model_id: String, / / Identifier of the model layer_range: [u32; 2], / / Range of the carrying layer, including start and end layer numbers [start, end] usable_vram_quota: u32, / / Logical video memory quota (MB) tier: ServiceTier, / / Service tier enumeration {S1, S2, S3} bandwidth_mbps: u16, / / Uplink bandwidth (Mbps) region: String, / / Geographic region (ISO 3166-1 standard) asn: u32, / / Autonomous System Number, used for geographic affinity calculations stake_amount: u256, / / Number of staked tokens shard_manifest_hash: [u8; 32], / / Hash of the shard manifest list post_proof: PostProof, / / Fragmented storage proof } < / hash>

Claims

1. A consumer-grade GPU cluster distributed large language model inference system based on blockchain incentives, characterized in that, This system organizes globally distributed consumer-grade GPUs into a distributed system capable of collaboratively running ultra-large-scale language model inference tasks. The proposed technical measures include: a low-memory quota-aware fragmentation mechanism; a hierarchical pipelined parallel inference protocol; a two-phase optimistic fraud proof protocol; an asynchronous separation mechanism between inference response and economic settlement; a staking-aware seven-dimensional scheduling and scoring function; and an eight-state node state machine and hot-standby fault-tolerance mechanism. The system employs a three-layer architecture, with the above six technical measures working in tandem to achieve a complete closed loop for low-memory nodes participating in large-scale model inference within an untrusted P2P network. The three-layer architecture, from top to bottom, consists of a node capability layer, a scheduling and execution layer, a verification layer, and a settlement layer. The node capability layer includes a low memory quota-aware fragmentation placement mechanism and a node octa-state machine and hot standby fault tolerance mechanism. The former solves the static management capability of nodes, with each node holding and executing only one segment of the entire model or a group of expert execution networks according to the low memory quota-aware fragmentation placement mechanism; the latter solves the dynamic lifecycle management problem of nodes; together, they constitute the complete technical support system for working nodes. The task scheduling layer is the nerve center of the system. Based on the seven-dimensional scoring function of staking perception, it greedily selects the optimal set of pipeline nodes for each request from the distributed hash table. It adopts a hierarchical pipeline parallel inference protocol as the main inference protocol of the Dense model, manages the micro-batch request queue, and continuously fills the pipeline with requests from different users to eliminate pipeline bubbles. The task scheduling layer and the node capability layer maintain bidirectional real-time synchronization: the scheduling layer feeds back historical failure rate information to the capability layer, and the capability layer provides node capability and staking information to the scheduling layer. The verification and settlement layer involves a two-phase optimistic fraud proof protocol and an asynchronous separation mechanism between inference response and economic settlement. Specifically, all key economic operations are implemented through smart contract programs running on the blockchain, including: using tokens as digital assets on the blockchain, automatic allocation according to the layer ratio, staking locking and release, staking confiscation, sharded storage of proof cycle verification, and writing to the node blacklist. The revenue-generating economic system is designed around the three principles of "allocation according to contribution ratio, protection according to staking ratio, and punishment according to behavior result," and adopts a two-phase optimistic fraud proof protocol, namely, a dual-track mechanism of "optimistic execution + post-challenge": under normal circumstances, if no one challenges the phase commitment submitted by the node, it directly enters the settlement; for tasks that are challenged or randomly sampled, the verification layer triggers a replayable fragment verification process to reduce the dispute to the smallest computational fragment that can be independently adjudicated; in the settlement phase, an asynchronous separation mechanism between inference response and economic settlement is adopted, that is, by explicitly defining "user-visible inference completion" and "on-chain economic settlement finality" as two independent time-series events, the settlement finality and user-side inference completion are decoupled from each other, so that real-time performance and economic security can be balanced.

2. The large language model reasoning system according to claim 1, characterized in that, The low memory quota-aware fragmentation placement mechanism, through the abstraction of a unified format weighted fragmentation list in a large model, decouples logical memory quotas from physical GPU memory, allowing nodes with a 1GB logical quota to run on any consumer-grade GPU with ≥1GB of memory; specifically including: (1) Decoupling of logical quotas and physical memory Introducing available video memory logical quotas ( The concept of ) decouples the service capabilities that a node promises to the system from its physical GPU memory capacity; It is the maximum amount of video memory voluntarily declared by the node, equal to or less than the actual video memory of the physical GPU; according to Calculate the number of model layers that this node can support, deducting physical memory. The remaining space is used for the operating system, CUDA runtime, or other local applications, and does not compete for inference services. Maximum number of layers a node can support Determined by the following memory allocation formula: in: This indicates the available logical quota of video memory for a node; The weight of a single layer is indicated by the fragmentation list; This indicates the KV Cache memory used by a single concurrent request, specified in the shard list; This indicates the maximum number of concurrent requests per node, and is configured by system parameters. This indicates that CUDA reserves video memory with the system during runtime, and is a fixed value; when When it is insufficient to support an entire floor, that is Below This will automatically trigger the CPU overflow mode: the GPU prioritizes the computation of the attention layer, and the FFN weights overflow to CPU memory to maximize GPU utilization efficiency. (2) List of weight partitions for large models A unified format weight sharding list for large models is introduced as the authoritative description file for model sharding. Each sharding record in the list contains inferences. All required metadata; When a node starts up, it depends on its own... Substitute into the above formula to calculate The number of GPU layers can be precisely specified by setting startup parameters; any excess layers will be automatically executed in system memory through the CPU overflow mechanism. (3) Service Classification Definition based on The range of values ​​is defined, and three standard service levels are defined: (a) Tier-S1: Suitable for entry-level consumer GPUs, capable of supporting several layers at the bottom or top of a model; (b) Tier-S2: It can support more continuous layers and is suitable for intermediate layer providers; (c) Tier-S3: It can support larger layers and is suitable for high-performance consumer-grade GPUs; Grading threshold , The parameters are stored in the on-chain contract and can be adjusted through community governance to adapt to the layer size of different models. When the scheduler builds the pipeline, it prioritizes nodes of the same level to make the computation time of each node similar and reduce pipeline bubbles.

3. The large language model reasoning system according to claim 1, characterized in that, The hierarchical pipelined parallel inference protocol implements multi-node parallel inference based on hierarchical pipelined parallelism, two-stage inference, and micro-batch scheduling technologies, and supports the MoE expert parallel routing extension protocol; specifically including: (1) Overall structure of the inference protocol; Hierarchical pipelined parallelism is used as the primary inference protocol for the Dense model; given the total number of layers in the model. Maximum number of layers per node Number of nodes required for a single production line ;all Number of concurrent pipelines formed by available nodes The calculation formula is as follows: Let the single token generation throughput of each pipeline be... The total system throughput is ;in Determined by the time of the slowest stage of the production line; under steady-state full load, Each node works concurrently, and at each time step, each node processes activation values ​​from different requests. The throughput is affected by the computation time of a single bottleneck node. With transmission delay Common constraints: in, The computation time for a single node to process one Decode token step in its assigned layer; when there is network latency between nodes. much smaller At that time, the throughput bottleneck is determined by the GPU's computing power. Cross-regional scenarios Enlarging becomes a bottleneck; in this case, prioritize higher dimensions. Tier-S3 nodes to reduce pipeline depth This reduces the cumulative network latency in the end-to-end latency of a single request; (2) Activation value data packet format Nodes use gRPC to transmit activation tensors, and each data packet contains inference state and commitment fields; The size of a single token in the activation tensor is determined by the hidden layer dimension of the model. It was determined that compression with zstd could reduce the amount of data transmitted. (3) Two-stage reasoning In the pre-filling phase, the process is executed all at once: after the first node receives the embedding tensor of the input token, it performs forward computation of all layers sequentially along the pipeline. Each node fills the KV cache of its responsible layer in turn, and the last node outputs the logarithmic probability of the first token. The time consumption of the pre-filling phase is mainly determined by the length of the input sequence and the network transmission. During the Decode phase, an autoregressive loop is executed: for each new token generated, the activation tensor is passed around the pipeline; each node queries its local KV Cache to complete attention calculation, updates and passes the activation value to downstream nodes; each Decode step generates a new stage commitment for on-chain verification. (4) Micro-batch scheduling: Eliminating bubbles in the pipeline The scheduler maintains a global request queue, continuously feeding inference requests from different users into the pipeline; during the initial bubble phase, downstream nodes are in a waiting state, and the GPUs partially idle; after entering the steady-state full load phase, each time step in the pipeline has exactly [number missing] requests. The requests are located on different nodes, and each node's GPU is always performing computational tasks, with utilization approaching 100%. The percentage of time spent in the bubble phase is related to the pipeline depth and the request arrival rate, and its calculation formula is as follows: in, This refers to the number of micro-batches processed simultaneously in parallel. The larger the batch size, the lower the bubble ratio; the scheduler achieves a balance between delay constraints and bubble rate by dynamically adjusting the micro-batch size. (5) MoE Expert Parallel Routing Extension Protocol For large language models with hybrid expert architecture, a parallel routing extension protocol is designed for the Expert-FFN layer, enabling the Top-K experts of each MoE layer to execute concurrently on the expert execution network distributed across different nodes; Routing packet format: After the routing network calculation is completed, a routing packet containing relevant fields is generated and broadcast to K expert nodes; After all expert nodes have completed their concurrent computations, they will return the output activation tensor to the weighted aggregation node for execution. Then it enters the dense attention layer of the next Transformer module; the end-to-end latency of the entire expert execution network is approximately equal to the single expert computation latency, which is the core throughput advantage of the MoE architecture over the Dense architecture in distributed inference; The expert capacity overflow three-level strategy is as follows: first, route to the same region replica; then, use CPU-MoE as a backup; and finally, report capacity events and dynamically expand replicas to ensure the service availability of expert nodes under high load conditions.

4. The large language model reasoning system according to claim 1, characterized in that, The two-phase optimistic fraud proof protocol upgrades the raw activation hash to a phased commitment that includes input / output activation Merkle commitments and execution configuration identifiers, and narrows down disputes to independently verifiable computational fragments through a challenge-replay protocol; specifically including: (1) Security vulnerabilities and solutions for raw hashing That is, the two-phase optimistic fraud proof, which upgrades the semantics of commitment from "I submitted a certain activation value" to "My input activation is consistent with the output of the predecessor node and produces a deterministic output activation under a given execution configuration"; specifically, it is divided into a commitment phase and a challenge-replay phase; (2) Phase 1: Phase Commitment; After each node completes forward reasoning, it generates the following stage commitment structure, signs it, and broadcasts it to the chain. The core principle of the commitment field design is that all factors affecting the calculation result are committed to be fixed, making "replay under the same input and configuration" the only verification path. The promised Merkle root constructor: will Activation tensor of each token step Calculate the hash of each leaf node one by one Then construct a binary Merkle tree, using the root node hash as input to activate the Merkle commitment or as output to activate the Merkle commitment; the validator only needs to provide the path length during the challenge. The Merkle proof can prove the existence and correctness of any single-step activation value without retransmitting the complete activation tensor. Continuity verification: Node Received node When processing the activation value data packet, verify its input activation Merkle commitment field against the node. The stage commitment output activates the Merkle commitment to ensure consistency, forming a commitment chain across nodes to ensure that the activation value is not tampered with during the flow between nodes; (3) Second stage: dispute reduction and replayability verification If a verifier believes that a node's output activating a Merkle commitment is inconsistent with honest computation, and initiates a challenge, the system will proceed as follows: The two-way split agreement narrows the scope of the dispute: Ultimately, the dispute was narrowed down to a single step, with the referee node performing a deterministic replay under the same execution configuration identifier constraint; The following mechanism is used to determine replay consistency: Deterministic path: When the inference path uses INT4 integer lookup table operations, the output results have precise consistency across hardware bits under the same weights and inputs. In this case, the replay stage directly compares the Merkle roots. Tolerance Path: When the inference path contains bf16 floating-point operations, an error tolerance threshold is introduced. The judgment criteria have been relaxed from equivalence comparison to: in, Determined by the quantization precision configuration in the execution configuration identifier; Referee node constraint: The referee node must declare the same GPU architecture as the audited node. This field is written into the phase commitment as a required subfield of the execution configuration identifier to avoid cross-generation GPU architecture differences exceeding the tolerance threshold. If the replay result exceeds the tolerance range, the smart contract will automatically trigger a staking penalty; if the result is consistent, the challenge fails and the challenger's stake is deducted. (4) Economic security analysis of the verification mechanism Let the reward for a single successful cheating by a node be... The penalty amount for confiscation of pledged shares after inspection is... The overall detection rate is The expected benefit of cheating is: make The sufficient condition is Overall detection rate Based on random sampling rate Challenger monitoring coverage and conditional detection rate The three parties jointly decided: set up , To ensure that in the worst-case scenario... The next game equilibrium is still based on honest reasoning, setting... That is, the penalty amount for confiscation of pledged shares shall not be less than 6 times the reward for a single instance; in actual deployment, The total pledged amount of the node is set to be much higher than Furthermore, as the challenger incentive mechanism improves, the security margin increases with network activity.

5. The large language model reasoning system according to claim 1, characterized in that, The asynchronous separation mechanism between inference response and economic settlement ensures that user streaming output begins immediately after the minimum offline acceptance conditions are met, while the on-chain economic endgame is determined after the challenge window closes, with the two events not blocking each other; specifically including: Users expect inference response timing with millisecond to second delays for the first token and streaming output, while the blockchain's settlement endgame, from the commitment during the submission phase to the closing of the challenge window, takes several minutes. If on-chain settlement is placed on the critical path of the response, users will experience unacceptable waiting delays. By explicitly defining "user-visible inference completion" and "on-chain economic settlement endgame" as two independent timing events, the above contradictions can be completely decoupled. (1) Formal definition of two time-series events event Inference response completion: Triggered when the API gateway determines that the offline minimum acceptance condition (OAC) is met, defined as: in, For the first Phase commitments for each pipeline node, This represents the pipeline depth; once OAC is satisfied, the API gateway immediately transmits the inference results to the user via server push events, regardless of whether the challenge window is closed. event The Final Chapter of Economic Settlement: A Challenging Window If the dispute is closed and there is no valid challenge, or if the dispute adjudicator contract completes its ruling, the smart contract will execute the token distribution. The temporal relationship between the two events is as follows: ,Right now It occurred much earlier than Users begin receiving streaming output within seconds of completing the inference process, which runs completely in parallel with the on-chain settlement process; (2) Parallel execution on the user side and the chain side: exist and In between, the system is in an "optimistic acceptance" state: the user has received the answer, the node's token reward is locked and awaiting release, and the challenge window remains open; if the challenge is successful, the system will execute a staking confiscation penalty and rollback based on the existing on-chain records, where the rollback only affects token allocation and does not affect the reasoning results already delivered to the user.

6. The large language model reasoning system according to claim 1, characterized in that, The staking-aware seven-dimensional scheduling scoring function incorporates dimensions such as node staking weight, historical failure rate, and shard continuity into the scheduling score, linking the scheduling strategy with the decentralized incentive mechanism; specifically including: (1) Definition of scoring function A seven-dimensional scoring function is used to select the optimal set of nodes from the DHT during the pipeline assembly phase. The scoring function integrates: real-time performance metrics, including latency, throughput, and queue depth; historical reliability metrics, i.e., failure rate; economic incentive metrics, i.e., staking weight; and topology optimization metrics, including geographical affinity and sharding continuity. Normalization functions for each dimension The definitions are as follows, totaling 7 items: (1) Measured RTT from node to downstream node: ; (2) Historical average throughput: Normalize the historical average tokens / s of nodes to the global maximum value; (3) Recent task failure rate: The higher the failure rate, the lower the score. The score is updated based on the index-weighted moving average over the past 30 days. (4) Current task queue depth: , This represents the current number of tasks pending processing, used for load balancing. (5) Pledge weight: The proportion of tokens staked by a node to the total amount staked across the entire network; (6) Geographical affinity: The system is categorized into three levels: 1.0 for the same autonomous system, 0.6 for the same country, and 0.3 for different geographical continents, to reduce cross-regional latency. (7) Continuity of segments: If adjacent node segments are continuous, otherwise take 0.5 to reduce data retransmission; Default weight vector: It is stored in an on-chain parameter contract and supports adjustments by community governance; (2) Greedy assembly algorithm for pipeline The scheduler uses a greedy strategy, sequentially determining the position of each stage of the pipeline (total...). Select the optimal node (i). The specific steps are as follows: For the i) For each layer segment, a candidate node set is selected from the DHT node set that meets the service level requirements, covers the target layer segment, and is currently registered. Based on this, the geographic location of previously selected nodes is updated. and Dimension, calculate the comprehensive score of each candidate node in the current context; select the node with the highest score as the first candidate node. The segment is assigned to a specific recipient and removed from the candidate set to avoid duplicate assignment; this process is repeated until all segments are assigned to a specific recipient. Node selection has been completed for each layer segment; Staking weight dimension in the scoring function This directly links node staking behavior to task allocation priority, forming a positive cycle of "the higher the staking, the more allocation, and the higher the reward," while a negative penalty mechanism of "the higher the failure rate, the lower the score, and the less allocation" drives nodes to maintain high availability. (3) Dynamic updates of scoring The scheduler maintains sliding window statistics for each node: Upon completion of each inference task (i.e., timeout or failure), the node's exponentially weighted moving average and sliding mean are updated; when a node's staking changes, Real-time recalculation; when DHT receives a node online / offline notification, it triggers a refresh of the hot standby candidate pool for the relevant pipeline.

7. The large language model reasoning system according to claim 1, characterized in that, The node octet state machine and hot standby fault tolerance mechanism, namely, defining an octet lifecycle state machine and cooperating with the client-side recovery cache, enables a highly available inference service with rapid fault switching, specifically including: (1) Node lifecycle eight-state machine A complete node lifecycle state machine is defined, covering the entire process from registration to settlement. The hot standby switching logic driving the scheduler sequentially enters the registered, allocated, preheated, active, downgraded, replaced, disputed, and settled states. The semantics and typical triggering switching conditions for each state are as follows: (1) Registered: The node has completed the sharded storage proof and staking, and is waiting for task allocation; after the scheduler scores and selects it, it will be converted to allocated; (2) Allocated: The node has received the task instruction and is loading the KV Cache context checkpoint; after the activation value arrives for the first time, it will be marked as preheated; (3) Preheated: The KV Cache has been pre-filled and can immediately participate in the Decode computation; after the first forward computation is performed, it becomes active; (4) Active: During normal inference, the node continuously receives and transmits activation values; after 15 seconds of continuous heartbeats, it is downgraded; when the phase commitment is challenged, it is replaced. (5) Degradation: Heartbeat timeout or error rate increase triggers hot standby startup process; after the standby node completes the switch, the current node becomes replaced; if the node recovers its heartbeat, it becomes active again; (6) Replaced: The current node has been replaced by a hot standby node, and the inference pipeline continues normally; after the node is re-registered, it becomes registered; (7) In dispute: the phase commitment is challenged and awaits the decision of the referee node; after the final determination is made regardless of whether it is honesty or cheating, it is converted to settlement; (8) Settled: The economic settlement of the current reasoning task has been completed; the node enters the next task cycle and is re-registered or assigned; (2) Hot standby switching and client-side cache recovery The scheduler maintains 1-2 hot standby nodes in a preheated state for each stage of the pipeline. The hot standby nodes have completed the shard loading and KV cache initialization. When switching, only the activation value of the last step needs to be passed in to immediately resume the calculation without reloading the weights. Client-side cache recovery: API gateway cache most recently The intermediate activation tensor of each decoding step; when an active to degrade transition is detected, the gateway retrieves the last activation value from the cache and sends it directly to the hot standby node that is already in a preheated state. The hot standby node continues forward calculation from this activation value without having to re-execute from the pre-filling stage. When a hot standby node takes over, in addition to the activation value, the complete KV cache must be rebuilt. This means the self-attention mechanism requires holding the key-value history from the beginning of the sequence to the current position. A three-level KV cache migration strategy is defined, and the scheduler selects the appropriate strategy based on pipeline health and bandwidth budget. Strategy 1, real-time synchronization, suitable for critical pipelines: During the preheating period, the hot standby node establishes an incremental synchronization channel with the master node for KVCache. After each Decode step is completed, the newly added KV block is pushed to the hot standby node. During the switchover, the hot standby node's KVCache is already synchronized, so there is no need to rebuild it, minimizing the switchover overhead. Strategy 2, periodic checkpoints, suitable for general pipelines: the scheduler checks every [time / period]. Each Decode step writes the complete KVCache of the master node to the highly available distributed storage; after failover, the hot standby node loads the most recent checkpoint and uses the client to restore the activation values ​​already saved in the cache for replay. Within a few steps of history, the total cost of reconstruction shall not exceed The calculation time for each step; Strategy 3, lightweight reconstruction, suitable for low-bandwidth environments: hot standby nodes do not perform pre-synchronization; after switching, the hot standby node combines the client recovery cache with the initial KV state of the distributed storage when the pre-filling phase is completed, and rebuilds the complete KV cache from the end of pre-filling to the switching, introducing a reconstruction time proportional to the length of the historical sequence. Pipeline checkpoint mechanism: The scheduler checks every [time / time]. The layer sets up a checkpoint node to write the currently active intermediate tensor to a highly available distributed storage; when a serious pipeline failure occurs, recovery starts from the most recent checkpoint, with a replay cost of at most [value missing]. The computational cost per layer, not the entire pre-filling stage; (3) Node registration and sharded storage proof To join the network, a node must complete five steps in sequence: reporting hardware capabilities and service level declaration, staking tokens on-chain, downloading GGUF weighted shards via P2P, submitting shard storage proof, and broadcasting. Among these steps, the shard storage proof prevents nodes from registering fraudulently. The on-chain contract randomly selects several tensor names and requires nodes to provide corresponding data segments to verify that the node actually holds the shard files. The storage proof is re-verified every 7 days.

Images