A distributed large language model inference method and system based on request self-carrying routing

Abstract

The application relates to a distributed large language model inference method and system based on request self-carrying routing, and belongs to the technical field of artificial intelligence. The method comprises the following steps: S1. A central dispatcher maintains a full cluster view; S2. The central dispatcher synchronizes the full cluster view to a plurality of gateways through a push protocol; S3. When a gateway receives an inference request, a complete request path is constructed based on the locally cached full cluster view; S4. The gateway inserts the path in the form of a binary frame into the request header and sends it to the path first-hop node through a QUIC protocol; S5. The nodes directly forward the inference intermediate state according to the path in the frame, and the central dispatcher is not on the critical path of the request execution; S6. When any node senses that the downstream node is unreachable, the node locally selects a replacement node from the backup node list in the frame to continue forwarding, without the need to query the central dispatcher. The application enables a large language model to complete an inference task in a wide area network environment by a plurality of consumer-grade GPU nodes.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a distributed large language model inference method and system based on request-carrying routing. Background Technology

[0002] With the rapid growth in the parameter size of Large Language Models (LLMs), a single GPU server can no longer handle the inference tasks of mainstream large models. For example, DeepSeek-V3 has 671 billion total parameters, and Qwen3-235B-A22B has 235 billion total parameters. Even top-tier consumer-grade GPUs (32GB VRAM) such as the RTX 5090 cannot load the complete model.

[0003] The industry commonly adopts the following distributed deployment solutions to handle large model inference:

[0004] (1) Tensor Parallelism (TP): The weights of a single layer are distributed across multiple GPUs, with each GPU responsible for a portion of the computation. This requires high-speed interconnection between GPUs (NVLink or InfiniBand) and is only applicable to single-machine multi-GPU data center scenarios.

[0005] (2) Pipeline Parallelism (PP): The model is split into layers and distributed across multiple machines, with each machine responsible for several consecutive layers. It can be deployed across machines, but in traditional solutions, the layer allocation of each node is fixed at startup and cannot be dynamically adjusted during runtime.

[0006] (3) Expert Parallelism (EP): The experts in the MoE model are distributed across multiple machines. High-speed interconnection between nodes is also required. Summary of the Invention

[0007] The purpose of this invention is to address the problems existing in the background technology by proposing a distributed large language model inference method and system based on request-carrying routing, which enables a large language model to complete inference tasks collaboratively by several consumer-grade GPU nodes in a wide area network environment.

[0008] The technical solution of this invention: A distributed large language model inference method and system based on request-carried-routes, comprising the following steps:

[0009] S1. The central scheduler maintains a full cluster view, which includes several GPU nodes. Each node loads several layers of the large language model after it has been sharded according to the transformer block hierarchy. Each layer is loaded on at least two nodes.

[0010] S2. The central scheduler synchronizes the full cluster view to several Gateways via a push protocol. This synchronization employs a dual-channel approach: event-triggered and periodically fallback.

[0011] When S3.Gateway receives an inference request, it constructs a complete request path based on the local cached full cluster view. The path includes the node identifier for each hop, the node loading layer range, and a list of backup nodes.

[0012] S4.Gateway inserts the path into the request header in binary frame form and sends it to the first-hop node of the path via the QUIC protocol;

[0013] S5. Nodes directly forward intermediate states for inference along the intra-frame path, and the central scheduler is not on the critical path for the execution of the request;

[0014] S6. When any node detects that a downstream node is unreachable, it locally selects a replacement node from the intra-frame backup node list to continue forwarding without querying the central scheduler.

[0015] Preferably, the perception of downstream node unreachability is implemented based on kernel TCP / QUIC connection status signals (RST, EOF, ICMPunreachable), with a latency on the order of 1ms.

[0016] Preferably, the event trigger is executed when a node joins / leaves / the loading layer changes / the quality changes; the cycle fallback is executed unconditionally once every 30 seconds.

[0017] Preferably, the construction of the complete request path is based on multi-dimensional scoring, and the dimensions include at least latency, load, computing power, quality, and available video memory.

[0018] Preferably, the intermediate inference state is a hidden state in the middle jump and a token after the sample in the last jump.

[0019] Preferably, the last hop node simultaneously sends the token back to both the Gateway and the first hop node, with the first hop node used as input for the next decoding step.

[0020] Preferably, after a node has completed local recovery, it asynchronously reports a PeerDead event to the central scheduler, which then updates the overall cluster view accordingly.

[0021] A distributed large language model inference system based on request-carrying routing includes a central cluster scheduling module, a cluster view synchronization module, a gateway path orchestration module, a request frame encapsulation and transmission module, a node distributed forwarding module, a node fault self-healing and replacement module, and a status feedback and reporting module. The central cluster scheduling module constructs and maintains a global GPU node cluster view. The cluster view synchronization module completes the view distribution through a dual mechanism of event triggering and periodic fallback. The gateway path orchestration module generates a complete forwarding path including backup nodes based on multi-dimensional indicators. The request frame encapsulation and transmission module completes the embedding of path information and transmits it based on the QUIC protocol. The node distributed forwarding module directly transmits intermediate inference data without central scheduling. The node fault self-healing module quickly selects a backup node to continue inference based on network status signals. The status feedback and reporting module completes the feedback of inference results and asynchronous reporting of node abnormal events, realizing decentralized and efficient distributed inference.

[0022] Compared with the prior art, the present invention has the following beneficial technical effects:

[0023] This invention decouples long-term route planning from single-request execution, enabling complete bypassing of the central scheduler for critical request paths. This innovation addresses the technical problem that in traditional distributed inference systems, each cross-node hop requires a query to the central scheduler, introducing tens of milliseconds of additional latency in WAN scenarios and making the central scheduler a performance bottleneck and single point of failure. The solution is as follows: the central scheduler generates a long-term route once when a node changes and pushes it to all Gateway caches; when a request arrives, the Gateway constructs a complete real-time route based on the cache and inserts the route into the request header in binary frame form; nodes communicate directly based on intra-frame routes, and the central scheduler is no longer involved in the critical request path.

[0024] The large language model is decomposed into independent deployment units at the transformer block level, and each layer is required to have redundant replicas on multiple nodes. This is fundamentally different from the traditional pipelined parallel stage-level deployment.

[0025] It utilizes signals such as RST, EOF, and ICMPunreachable from the kernel TCP / QUIC protocol stack as the fastest and most accurate signal source for fault detection, with a latency on the order of 1ms, which is significantly faster than the application layer heartbeat timeout (on the order of seconds) that is commonly relied upon in the industry.

[0026] The InstantRoute of this invention carries a backup node list in advance in the request frame, and can immediately switch locally after fault detection without querying the central scheduler, achieving end-to-end P99 jitter of less than 1 second. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a diagram of the overall system architecture in this invention;

[0029] Figure 2 This is a data structure diagram in this invention. Detailed Implementation

[0030] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0031] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0032] Example 1

[0033] like Figure 1-2 As shown, the present invention proposes a distributed large language model inference method and system based on request-carrying routing, which includes the following steps:

[0034] S1. The central scheduler maintains a full cluster view, which includes several GPU nodes. Each node loads several layers of the large language model after it has been sharded according to the transformer block hierarchy. Each layer is loaded on at least two nodes.

[0035] S2. The central scheduler synchronizes the full cluster view to several Gateways via a push protocol. This synchronization employs a dual-channel approach: event-triggered and periodically fallback.

[0036] When S3.Gateway receives an inference request, it constructs a complete request path based on the local cached full cluster view. The path includes the node identifier for each hop, the node loading layer range, and a list of backup nodes.

[0037] S4.Gateway inserts the path into the request header in binary frame form and sends it to the first-hop node of the path via the QUIC protocol;

[0038] S5. Nodes directly forward intermediate states for inference along the intra-frame path, and the central scheduler is not on the critical path for the execution of the request;

[0039] S6. When any node detects that a downstream node is unreachable, it locally selects a replacement node from the intra-frame backup node list to continue forwarding without querying the central scheduler.

[0040] The perception of downstream node unreachability is achieved based on kernel TCP / QUIC connection status signals (RST, EOF, ICMPunreachable), with a latency on the order of 1ms.

[0041] The event is triggered when a node joins / leaves, the loading layer changes, or the quality changes; the cycle is executed unconditionally every 30 seconds as a fallback.

[0042] The construction of the complete request path is based on a multi-dimensional score, which includes at least latency, load, computing power, quality, and available video memory.

[0043] The intermediate state of the inference is a hidden state in the middle jump and a token after the sample in the last jump.

[0044] The last hop node simultaneously sends the token back to both the Gateway and the first hop node, with the first hop node serving as the input for the next decoding step.

[0045] After a node completes local recovery, it asynchronously reports a PeerDead event to the central scheduler, which then updates the overall cluster view accordingly.

[0046] A distributed large language model inference system based on request-carrying routing includes a central cluster scheduling module, a cluster view synchronization module, a gateway path orchestration module, a request frame encapsulation and transmission module, a node distributed forwarding module, a node fault self-healing and replacement module, and a status feedback and reporting module. The central cluster scheduling module constructs and maintains a global GPU node cluster view. The cluster view synchronization module completes the view distribution through a dual mechanism of event triggering and periodic fallback. The gateway path orchestration module generates a complete forwarding path including backup nodes based on multi-dimensional indicators. The request frame encapsulation and transmission module completes the embedding of path information and transmits it based on the QUIC protocol. The node distributed forwarding module directly transmits intermediate inference data without central scheduling. The node fault self-healing module quickly selects a backup node to continue inference based on network status signals. The status feedback and reporting module completes the feedback of inference results and asynchronous reporting of node abnormal events, realizing decentralized and efficient distributed inference.

[0047] Layer-level granular model deployment

[0048] The large language model consists of several transformer block layers, each containing self-attention and feed-forward sub-layers. This invention treats each layer as an independent deployment unit with the following properties:

[0049] Layer identifier (layer_id): globally unique, such as Qwen3-8B::layer-12; weight shard file: split by layer from formats such as safetensors; KVcache slot: indexed at runtime by (request_id, layer_id); dependency relationship: the output of layer N is the input of layer N+1;

[0050] Each node loads a set of continuous or discrete layers based on its local VRAM, computing power, and network conditions. Node A may load [layer0, layer1, ..., layer11], node B may load [layer8, layer9, ..., layer19] (partially overlapping with A as redundancy), and node C may load [layer20, ..., layer35].

[0051] Key constraints of layered granular deployment:

[0052] (1) Redundancy constraint: Each layer has at least N replicas on N nodes (default N=2) to provide fault tolerance;

[0053] (2) VRAM constraint: The total layer weight loaded on each node plus KVcache shall not exceed 0.85 times the local VRAM;

[0054] (3) Continuity constraint (optional): The loading of each node in the dense model is a continuous interval, which is convenient for pipelines; the MoE model can be a discrete set of experts.

[0055] LongTermRoute (long-term route)

[0056] LongTermRoute (hereinafter referred to as LTR) is a full cluster view maintained by the central scheduler, containing the following information: LongTermRoute {generation: int64, / / Monotonically increasing version number model_id: string, / / Model identifier layer_count: int, / / Total number of model layers nodes: [{node_id: string, peer_addr: string, / / QUIC data plane address loaded_layers: [int], / / Loaded layer range quality_score: float, / / Quality score 0-1 compute_capacity: float, / / Computing power vram_free_mb: int,}, ...],

[0057] layer_to_nodes: { / / Reverse index: list of nodes for each level 0: [node-A, node-B], 1: [node-A, node-B], ...}}

[0058] LTR update trigger conditions:

[0059] When a node joins / leaves the cluster, the node's loading layer changes (result of dynamic scheduling). Significant changes in the node's quality_score are observed. Heartbeat reports show substantial changes in the vram_free_mb of some nodes. The LTR (Local Time Response) pushes this information to all Gateways via a dual-channel approach: event-driven and periodic fallback.

[0060] (1) Event triggering: When any of the above conditions are met, the Scheduler immediately broadcasts the new LTR version, and the Gateway updates its local cache upon receiving it. The delay of event triggering is on the order of 100ms.

[0061] (2) Periodic fallback: The Scheduler broadcasts a complete LTR unconditionally every 30 seconds to prevent the Gateway from holding an expired view due to event loss.

[0062] (3) Subscriber ack: Gateway sends an ack after receiving LTR. Scheduler counts the ack ratio and triggers a full re-push when it falls below the threshold.

[0063] InstantRoute instant route construction

[0064] When the Gateway receives an inference request, it constructs an InstantRoute (hereinafter referred to as IR) based on the locally cached LTR. The IR is the complete execution path for a single request, containing: InstantRoute {request_id: string, ltr_generation: int64, / / Which version of LTR is used to construct hops: [{node_id: string, / / Primary node peer_addr: string, layer_range: [int, int], / / The layer range processed by this hop backup_nodes: [ / / List of backup nodes for this hop {node_id, peer_addr}, ...]}, ... / / Complete coverage [0, layer_count)]}

[0065] IR construction algorithm (pseudocode):

[0066] functionbuild_instant_route(request,ltr):

[0067] hops=[]

[0068] current_layer=0

[0069] while current_layer <ltr.layer_count:

[0070] # Select the main node based on score among the nodes covering the current_layer

[0071] candidates=ltr.layer_to_nodes[current_layer]

[0072] primary=score_and_pick(candidates, request)

[0073] #Select the maximum coverage of the master node's load layer range

[0074] primary_layers=ltr.nodes[primary].loaded_layers

[0075] layer_range_end=compute_max_continuous(

[0076] primary_layers, current_layer)

[0077] #Select backup_nodes: Other nodes that also cover this layer's scope.

[0078] backups = find_backup_nodes(

[0079] ltr, current_layer, layer_range_end, exclude=primary)

[0080] hops.append({

[0081] node_id: primary,

[0082] peer_addr:ltr.nodes[primary].peer_addr,

[0083] layer_range: [current_layer, layer_range_end],

[0084] backup_nodes: backups

[0085] })

[0086] current_layer=layer_range_end

[0087] returnInstantRoute(

[0088] request_id,ltr.generation,hops)

[0089] Path scoring (score_and_pick) considers multiple dimensions:

[0090] (1) Current node delay (RTT to Gateway);

[0091] (2) Current number of active requests for the node (avoiding hotspots);

[0092] (3) Node quality_score (prioritize higher quality);

[0093] (4) Node compute_capacity relative to model requirements;

[0094] (5) Check if the node's vram_free_mb is sufficient for this request;

[0095] The weight distribution is configurable, and the typical weights are latency 0.45 / load 0.20 / compute 0.15 / quality 0.10 / vram 0.10.

[0096] The request frame carries its own routing

[0097] After constructing the IR, the Gateway serializes the IR into binary (msgpack encoded) and inserts it into the header of the request frame. The format of the request frame:

[0098] The Gateway sends the first frame to the IR.hops[0].peer_addr node through a QUIC unidirectional stream.

[0099] Forwarding and execution between nodes

[0100] After node N (the primary node in IR.hops[k]) receives the request frame:

[0101] (1) Parse the frame header + payload to obtain the InstantRoute and input_tokens;

[0102] (2) Verify whether IR.ltr_generation is compatible with the locally cached LTR version (if severely outdated, return the STALE_ROUTE error);

[0103] (3) Verify whether IR.hops[k].layer_range is within the loading layer range of this node;

[0104] (4) Send input_tokens and layer_range to the local Python inference engine through UnixSocket;

[0105] (5) The inference engine executes forward and returns the hiddenstate or the final token;

[0106] (6) If this node is the last hop (k == len(hops)-1): - Directly stream the token back to the Gateway (return_addr) through QUIC;

[0107] (7) - At the same time, send the token back to IR.hops[0] through QUIC (as the input for the next decode);

[0108] (8) If this node is an intermediate hop (k < len(hops)-1): - Construct the next-hop request frame (replace input_tokens with hiddenstate in the payload);

[0109] (9) - Send to IR.hops[k+1].peer_addr via QUIC;

[0110] Localized fault recovery

[0111] Node N may encounter the following failure when performing the forwarding in step 5.5 (sending to IR.hops[k+1]):

[0112] (1) Kernel-level RST: The target node process has died, the kernel immediately returns to RST, and the local node's perception delay is <1ms;

[0113] (2) Kernel-level ICMPunreachable: The target node is unreachable, and the detection time is less than 1ms;

[0114] (3) QUICEOF: Millisecond-level detection of abnormal connection closure;

[0115] (4) Application-level timeout: The target node receives the data but does not respond for an extended period of time (default 5 seconds);

[0116] Upon detection of any of the above faults, node N immediately executes the local recovery process:

[0117] (1) Query the list of IR.hops[k+1].backup_nodes in the request frame;

[0118] (2) Try to back up nodes in the order of the list until the forwarding is successful or the backup list is exhausted;

[0119] (3) If successful: Keep the original InstantRoute, but replace hops[k+1] with the backup node;

[0120] (4) If all backups fail: return an error to the upstream (IR.hops[k-1]) and the upstream will continue to try its backup node; if all upstreams also fail, the Gateway will request a new InstantRoute from the Scheduler after receiving the error.

[0121] (5) After local recovery is completed, the PeerDeadEvent is asynchronously reported to the Scheduler, and the Scheduler updates the LTR and broadcasts the new version accordingly;

[0122] Key properties of local recovery:

[0123] Non-blocking of critical path requests: Node N continues forwarding immediately after completing local failover, and PeerDeadEvent is sent asynchronously;

[0124] Reachable without relying on the Scheduler: Even if the Scheduler crashes, the local recovery mechanism still works;

[0125] Generated tokens are not lost: If a fault occurs and the last hop has already returned part of the token, the Gateway does not require retransmission, but only switches the subsequent hop.

[0126] This invention decouples long-term routing planning from single-request execution, enabling complete bypassing of the central scheduler on critical request paths. This innovation addresses a key technical problem: in traditional distributed inference systems, each cross-node hop requires a query to the central scheduler, introducing tens of milliseconds of additional latency in WAN scenarios and making the central scheduler a performance bottleneck and single point of failure. The solution is as follows: the central scheduler generates a long-term route once when a node changes and pushes it to all Gateway caches; when a request arrives, the Gateway constructs a complete real-time route based on the cache and inserts the route into the request header as a binary frame; nodes communicate directly based on intra-frame routes, and the central scheduler is no longer involved in the critical request path.

[0127] The large language model is decomposed into independent deployment units at the transformer block level, and each layer is required to have redundant replicas on multiple nodes. This is fundamentally different from the traditional pipelined parallel stage-level deployment.

[0128] It utilizes signals such as RST, EOF, and ICMPunreachable from the kernel TCP / QUIC protocol stack as the fastest and most accurate signal source for fault detection, with a latency on the order of 1ms, which is significantly faster than the application layer heartbeat timeout (on the order of seconds) that is commonly relied upon in the industry.

[0129] The InstantRoute of this invention carries a backup node list in advance in the request frame, and can immediately switch locally after fault detection without querying the central scheduler, achieving end-to-end P99 jitter of less than 1 second.

[0130] ongTermRoute achieves view synchronization between the Gateway and nodes through the monotonic generation version number: the ltr_generation carried in the request frame is compared with the node's local view. If it is severely outdated, execution is refused (returning STALE_ROUTE). When the node's local view expires, it actively pulls the latest version from the Scheduler, supporting gradual upgrades where new and old nodes coexist.

[0131] Scenario: Qwen3-8B model, 36 layers, deployed on three cross-city GPU nodes.

[0132] node Location GPU Loading layer range Node A Beijing RTX 5090 (32GB) 0-11 Node B Shanghai RTX4090 (24GB) 12-23 Node C Guangzhou RTX4090 (24GB) 24-35 Node D Beijing RTX 5090 (32GB) 0-17 (Redundancy) Node E Shanghai RTX4090 (24GB) 18-35 (Redundancy)

[0133] The client initiates a Qwen3-8B inference request, as follows:

[0134] (1) Request to reach Beijing Gateway. Gateway IR is constructed as follows: hops = [node A0-11, node B12-23, node C24-35], and backup_nodes for each hop are [node D], [], and [node E], respectively.

[0135] (2) Gateway writes IR into the request frame header and sends it to node A via QUIC;

[0136] (3) Node A calls the local Python inference engine to execute layer0-11 and obtains the hidden state;

[0137] (4) Node A retrieves the next-hop information from the IR and sends the hidden state to Node B via QUIC;

[0138] (5) If node B is offline (sensing <1ms), node A immediately queries IR.hops[1].backup_nodes (empty in this example) and reports an error to Gateway to trigger rerouting;

[0139] (6) If node B is online, node B executes layer 12-23 and continues to forward to node C;

[0140] (7) Node C executes layer24-35+sample to obtain the token;

[0141] (8) Node C directly sends the token back to Beijing Gateway via QUIC;

[0142] (9) Node C simultaneously sends the token back to node A via QUIC as input for the next decoding step;

[0143] In this embodiment, the end-to-end single-token latency is approximately 870ms, mainly composed of the forward time of each node and the cross-city RTT. The Scheduler is not queried during the entire request execution process.

[0144] Example 2: Local recovery of node B after it goes offline

[0145] Based on Example 1, assume that node B suddenly loses power during the 50th step of decoding:

[0146] (1) When node A forwards the 50th step hidden state to node B, QUIC receives RST and is aware of it in <1ms.

[0147] (2) Node A queries IR.hops[1].backup_nodes and finds it to be empty (Node B has no direct backup);

[0148] (3) Node A returns PEER_DEAD_NEED_REROUTE upstream (Gateway in this example);

[0149] (4) After receiving the error, the Gateway reconstructs the IR based on the latest LTR (at this time, node E takes over layers 12-23) and continues inference;

[0150] (5) The total delay from fault detection to new path establishment is approximately 50ms;

[0151] (6) Node A asynchronously reports PeerDeadEvent to the Scheduler;

[0152] (7) The Scheduler updates the LTR (marks node B as offline) and broadcasts the new version.

[0153] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited thereto. Various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention.

Claims

1. A distributed large language model inference method based on request self-carried routing, characterized in that: Includes the following steps: S1. The central scheduler maintains a full cluster view, which includes several GPU nodes. Each node loads several layers of the large language model after it has been sharded according to the transformer block hierarchy. Each layer is loaded on at least two nodes. S2. The central scheduler synchronizes the full cluster view to several Gateways via a push protocol. This synchronization employs a dual-channel approach: event-triggered and periodically fallback. When S3.Gateway receives an inference request, it constructs a complete request path based on the local cached full cluster view. The path includes the node identifier for each hop, the node loading layer range, and a list of backup nodes. S4.Gateway inserts the path into the request header in binary frame form and sends it to the first-hop node of the path via the QUIC protocol; S5. Nodes directly forward intermediate states for inference along the intra-frame path, and the central scheduler is not on the critical path for the execution of the request; S6. When any node detects that a downstream node is unreachable, it locally selects a replacement node from the intra-frame backup node list to continue forwarding without querying the central scheduler.

2. The distributed large language model inference method based on request-carrying routing as described in claim 1, characterized in that, The downstream node unreachability detection is achieved based on kernel TCP / QUIC connection status signals (RST, EOF, ICMPunreachable), with a latency on the order of 1ms.

3. The distributed large language model inference method based on request-carrying routing as described in claim 2, characterized in that, The event is triggered when a node joins / leaves, the loading layer changes, or the quality changes; the cycle is executed unconditionally every 30 seconds as a fallback.

4. The distributed large language model inference method based on request-carrying routing as described in claim 3, characterized in that, The construction of the complete request path is based on a multi-dimensional score, which includes at least latency, load, computing power, quality, and available video memory.

5. The distributed large language model inference method based on request-carrying routing as described in claim 4, characterized in that, The intermediate inference state is a hidden state in the middle jump and a token after the sample in the final jump.

6. The distributed large language model inference method based on request-carrying routing as described in claim 5, characterized in that, The last hop node simultaneously sends the token back to both the Gateway and the first hop node, with the first hop node serving as the input for the next decoding step.

7. The distributed large language model inference method based on request-carrying routing as described in claim 6, characterized in that, After a node completes local recovery, it asynchronously reports a PeerDead event to the central scheduler, which then updates the overall cluster view accordingly.

8. A distributed large language model inference system based on request-carrying routing, based on the distributed large language model inference method based on request-carrying routing according to claim 7, characterized in that, The system includes a central cluster scheduling module, a cluster view synchronization module, a gateway path orchestration module, a request frame encapsulation and transmission module, a node distributed forwarding module, a node fault self-healing and replacement module, and a status feedback and reporting module. The central cluster scheduling module constructs and maintains a global GPU node cluster view. The cluster view synchronization module completes the view distribution through a dual mechanism of event triggering and periodic fallback. The gateway path orchestration module generates a complete forwarding path including backup nodes based on multi-dimensional indicators. The request frame encapsulation and transmission module completes the embedding of path information and transmits it based on the QUIC protocol. The node distributed forwarding module directly transmits intermediate inference data without central scheduling. The node fault self-healing module quickly selects a backup node to continue inference based on network status signals. The status feedback and reporting module completes the feedback of inference results and asynchronous reporting of node abnormal events, realizing decentralized and efficient distributed inference.

Images