A stateless routing method for distributed caching

Abstract

The application discloses a stateless routing method of distributed cache, and relates to the field of distributed computing and artificial intelligence reasoning. In the configuration stage, a calibration bias value is obtained according to preset weight values of each candidate node, and is published in the form of a configuration snapshot; in the request stage, a multi-round dialogue reasoning request is parsed to confirm session identification information, joint hashing is performed on the session identification and node identification, a pseudo-random variable is obtained by normalization, a sum of the pseudo-random variable and the calibration bias value is taken as a routing index, a node with the maximum routing index is selected as a target storage node and the request is forwarded, and the node reuses a local stored historical KV Cache. The method moves the probability calibration to the configuration stage, and only needs hash, addition and comparison operations in the request stage; under an ideal independent continuous uniform hash model, the probability of each node being selected after calibration is equal to the weight proportion, and actual implementation is verified according to an error threshold.

Description

Technical Field

[0001] This invention relates to the fields of distributed computing and artificial intelligence reasoning, specifically to a stateless routing method for distributed caching in a distributed cluster of multi-turn dialogue services for large language models. Background Technology

[0002] In multi-turn dialogue services driven by Large Language Models (LLM), key-value caching (KV Cache) technology is commonly used to maintain dialogue context and improve inference efficiency. This technology caches intermediate key-value pairs generated by the attention mechanism during inference, allowing the encoding calculation of historical text to be directly reused in subsequent dialogue turns, thereby avoiding redundant calculations, significantly reducing single-response latency and improving the overall system throughput.

[0003] As the user base expands and dialogue complexity increases, the storage requirements of the KV Cache grow linearly or even exponentially, making single-machine storage insufficient and driving the system to evolve towards a distributed cluster architecture. Especially under sudden traffic fluctuations, dynamic scaling of servers is necessary. How to efficiently and intelligently manage and allocate the KV Cache becomes a key challenge.

[0004] Existing distributed cache routing can employ consistent hashing based on virtual nodes or name-based HRW mapping. Different implementations vary in metadata cost, load distribution, and the scope of impact from node changes; large-scale migration or server overload is not an inevitable consequence of all consistent hashing schemes. This method is designed for multi-turn dialogue services, using stable session identifiers and node weights to calculate the target storage node.

[0005] Therefore, without maintaining per-session routing mapping, it is necessary to convert the expected request share of each node into a pre-configurable scoring parameter so that deterministic routing can be achieved in the request phase using only addition and comparison operations. Summary of the Invention

[0006] To overcome the shortcomings of the prior art, this invention provides a method for dynamically scaling distributed environments that does not maintain per-session routing mapping, but instead calibrates the additive bias based on node weights during the configuration phase, performs joint hashing on the session identifier and node identifier during the request phase, adds the normalized hash value to the corresponding bias, and selects the node with the highest score as the target storage node.

[0007] This invention achieves the above objective through the following scheme: a stateless routing method with distributed caching, comprising:

[0008] (1) Perform request parsing to confirm session identification information;

[0009] (2) Traverse the candidate nodes and obtain the pseudo-random variables of the candidate nodes;

[0010] (3) Obtain the preset weight value and calibration bias value of the candidate node. The calibration bias value is obtained by calibration based on the preset weight value of all candidate nodes during the configuration phase. Calculate the routing index of the candidate node. The routing index is the sum of the pseudo-random variable of the candidate node and the calibration bias value to obtain the routing index set.

[0011] (4) Compare the routing metrics in the routing metric set, select the optimal node as the target storage node and perform request forwarding.

[0012] Preferably, the session identification information is a data field used to uniquely identify a continuous multi-turn dialogue session.

[0013] Preferably, the session identification information is a request feature for routing decisions and serves as one of the input factors for subsequent joint hash calculations, so that requests initiated by the same session at any time can generate a deterministic numerical sequence bound to itself when participating in routing calculations.

[0014] Preferably, step (2) includes:

[0015] (2.1) Obtain the list of available nodes and node identifiers;

[0016] (2.2) Concatenate the request feature with the node identifier to construct a union string;

[0017] (2.3) Perform a deterministic hash operation on the concatenated string and verify the output bit width;

[0018] (2.4) Normalize the verified integer hash value to generate pseudo-random variables for candidate nodes.

[0019] Preferably, each node in the node list in step (2.1) has a globally unique and stable node identifier, which is used to uniquely distinguish each node at the network level, and the node identifier is not changed when the node address is changed.

[0020] Preferably, the preset weight value of the node can be preset according to the node's storage capacity, bandwidth and / or load rate; and can reflect the node's carrying capacity in a true and linear way. The larger the weight value, the more user sessions and the amount of accumulated KV Cache data the node is expected to handle.

[0021] Preferably, the calibration bias value is determined during the configuration phase based on the preset weight values ​​of all candidate nodes, such that under the ideal independent continuous uniform hash model, the probability of any request being routed to that node is equal to the proportion of the preset weight value of that node to the sum of the preset weight values ​​of all nodes; in actual implementation, the calibration results are verified according to preset absolute error thresholds and relative error thresholds.

[0022] Preferably, step (4) includes:

[0023] (4.1) Perform a maximum value search on the set of routing metrics, select the candidate node corresponding to the routing metric with the largest value, and determine it as the target storage node for this request;

[0024] (4.2) If multiple nodes have the same maximum routing index, the system selects the node according to the fixed lexicographical order of the node identifiers;

[0025] (4.3) After determining the target storage node, the system forwards the current inference request to the node. After receiving the request, if the target storage node has a historical KV Cache previously generated by the session in its local storage system, it will reuse it directly, only perform inference calculation on the dialogue content added in the current request, and continue to store the newly generated KV Cache data in its local storage.

[0026] As a priority, the calibration bias value is determined in the configuration phase by solving a calibration problem: in the configuration phase, the difference between the probability of each node being selected and the target allocation share of each node under the ideal independent continuous uniform hash model is used as the residual. Through iterative solution, the probability that any request is routed to the node under the ideal independent continuous uniform hash model is equal to the proportion of the preset weight value of the node to the sum of the preset weight values ​​of all nodes.

[0027] As a priority, the calibration bias value, node set, preset weight value, hash specification, and winning rule are published together in the form of a configuration snapshot; the same version of the configuration snapshot is read during the request phase; when the node set changes or the preset weight value is adjusted, the configuration snapshot is recalibrated and a new configuration snapshot is published.

[0028] The beneficial effects of this invention are as follows: 1. It supports session affinity under a fixed configuration, ensuring that all requests in the same session are stably routed to the same node, enabling local reuse of existing KV cache, reducing cross-machine access, and lowering network overhead and latency; 2. In the ideal independent continuous uniform hash model, the node selection probability after accurate calibration is equal to the weight ratio, achieving precise load allocation based on node capabilities; the actual implementation is verified through error thresholds, avoiding allocation bias introduced by discrete approximation schemes; 3. The probability calibration is moved to the configuration stage, and only hash, addition, and comparison operations need to be performed on each candidate node during the request stage, resulting in extremely low routing computation overhead, suitable for high-concurrency inference request scenarios; 4. It eliminates the need to maintain per-session routing mapping, only maintaining configuration metadata such as node and calibration bias values, avoiding synchronization and lock contention issues in the per-session mapping table, and allowing the gateway to be horizontally scalable. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the steps of the method of the present invention;

[0030] Figure 2 This is a schematic diagram of the specific process of step (2) of the method of the present invention;

[0031] Figure 3 This is a schematic diagram of the specific process of step (3) of the method of the present invention. Detailed Implementation

[0032] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but the scope of protection of the present invention is not limited thereto:

[0033] Example: In multi-turn dialogue services driven by large language models, as the user base and dialogue complexity continue to grow, the storage space required for the generated KV cache expands dramatically, and the storage capacity of a single server can no longer meet the needs of massive concurrent dialogues. This necessitates the evolution of the system architecture from a single machine to a distributed cluster with multi-machine collaboration. During this process, if the KV cache for the same session is not assigned to the same machine, due to the difficulty in achieving cross-machine KV cache storage access, that machine will be unable to read the KV cache stored on other servers, leading to an increase in the overall system computation and wasted computing resources.

[0034] This invention provides a stateless routing method for distributed caching. This method is based on probability calibration during the configuration phase and additive scoring during the request phase. It eliminates the need to maintain per-session routing mappings and determines the target storage node each time an inference request arrives. For example... Figure 1 As shown, the specific steps include:

[0035] (1) Perform request parsing to confirm session identification information.

[0036] When a client initiates a multi-turn dialogue inference request, the microservice gateway or routing layer first parses the request to extract the session identification information carried within.

[0037] The session identifier is a data field used to uniquely identify an ongoing multi-turn dialogue session. In specific implementations, this session identifier is preferably a unique session number assigned by the system to the dialogue during the session establishment phase; a single user account or device identifier is insufficient to distinguish sessions when supporting multiple parallel sessions and should be used in combination with the session number. The session identifier must meet the following technical characteristics: remain constant throughout the entire lifecycle of the same session, and have unique distinguishing capabilities between different sessions.

[0038] The session identifier is determined as the request characteristic for this routing decision and serves as one of the input factors for subsequent joint hash calculations. This ensures that requests initiated by the same session at any given time will generate a deterministic numerical sequence bound to itself when participating in the routing calculation. Only when the routing decision is bound to a characteristic that uniquely identifies the session can it be ensured that all inference requests generated by that session are directed to the same storage node. This guarantees the complete local reuse of historical KV cache and avoids cross-node network latency and data migration overhead.

[0039] In the actual request parsing process, the session identification information can be located in a custom field in the HTTP request header, the request path parameter, a JSON field in the request body, or a session cookie. The specific location is determined by the system implementer according to the interface specification. After the gateway extracts this information, it temporarily stores it in the context variable of this request processing for use in subsequent steps.

[0040] (2) Traverse candidate nodes Obtain candidate nodes pseudorandom variables .

[0041] In practice, the candidate node set is read from the configuration snapshot used in this request, and for each candidate node... This involves performing joint hash calculations and normalization processes to generate pseudo-random variables that are jointly determined by the node and the current requesting session. Specifically, as follows... Figure 2 As shown, the implementation includes the following steps:

[0042] (2.1) Obtain the list of available nodes and node identifiers.

[0043] The system retrieves a complete list of currently healthy KV Cache storage nodes from the service registry or cluster management component. Each node in this list has a globally unique node identifier. This node identifier is used to uniquely distinguish each storage node at the network layer. In practice, the node identifier can preferably be a combination of the node's Internet Protocol address and communication port, or it can be the node's hostname, service name, or a unique number assigned by the system. As long as the node identifier is unique and stable within the cluster, the implementation requirements of this method are met; stability means that the node identifier does not change with changes in the node address.

[0044] (2.2) Concatenate the request feature and the node identifier to construct a union string.

[0045] For any candidate node in the node list, the system concatenates the session identifier information determined in step (1) with the node identifier of that candidate node at the string level to generate a union string. This concatenation operation follows a predefined format specification. For example, the session identifier information can be placed first, followed by the node identifier, and connected in the middle with a specific delimiter to form a union string similar to "session identifier information#node identifier". For example, delimiter escaping rules can be unified or the range of field characters can be limited to avoid different field combinations from generating the same input. The uniformity of this format specification ensures that for the same pair of "session-node", the generated union string is completely consistent at the binary level each time.

[0046] (2.3) Perform a hash operation on the combined string and verify the output bit width.

[0047] Perform a deterministic hash operation on the combined string generated in step (2.2) to generate a fixed-width integer hash value. The hash operation can be implemented using cryptographic or non-cryptographic hash functions commonly used in the field, such as SHA-256, MurmurHash3, or xxHash. This operation is deterministic: for identical input strings, the output hash value will be the same regardless of the time or the routing node on which the operation is performed. The actual distribution and collision characteristics are determined by the selected function and the input; the avalanche effect alone cannot prove that the outputs of different inputs are strictly independent, and hashing is not a mathematically collision-free unique mapping.

[0048] The hash operation outputs a b-bit unsigned integer hash value. Let M = 2^b, and verify. Zero values ​​are valid and are not incremented or replaced. All terminals use the same input encoding, hash algorithm, truncation bit, and stable seed.

[0049] (2.4) Normalize the verified integer hash value to generate pseudo-random variables for candidate nodes.

[0050] The verified integer hash value is converted to a floating-point representation and divided by M to obtain a discrete value in the interval [0,1). This discrete value is then used as a pseudo-random variable for the current request session and the current candidate node. ,Right now The configuration version is used to select a complete parameter snapshot, without including it in the joint hash input, to avoid rehashing all nodes simply because the version number has changed.

[0051] The pseudo-random variable These are deterministic discrete pseudo-random values. Subsequent probability derivations are based on an ideal independent continuous uniform hash model, where the pseudo-random variables of each candidate node are independent and all follow a continuous uniform distribution on the interval [0,1). The avalanche effect itself cannot prove independence and uniformity; finite bit width and the true key distribution may introduce bias.

[0052] (3) Obtain the preset weight value and calibration bias value of the node and calculate the routing index of the node to obtain the routing index set.

[0053] The preset weight values ​​are used to determine the target allocation share for each node during the configuration phase; during the request phase, the calibrated calibration bias values ​​are read for additive scoring. The calibration bias values ​​depend on all candidate nodes and their preset weight values, and cannot be represented as a function dependent solely on the preset weight value of a single node. Figure 3 As shown, the steps include the following.

[0054] (3.1) For candidate nodes, read the preset weight values ​​from the current configuration snapshot. With calibration bias value .

[0055] The preset weight value The preset weight value is determined by the cluster administrator based on the physical resource configuration of the candidate node. In a real-world distributed KV cache storage scenario, this preset weight value is preferably positively correlated with the node's available capacity for KV cache storage. The configuration end rejects invalid weights such as negative numbers, infinity, or non-numerical values; nodes with zero weight do not participate in the competition. Target allocation shares are set for the remaining N positive weight nodes. When N=0, no route can be selected; when N=1, a unique node is selected directly without calibration.

[0056] (3.2) Obtain pseudo-random variables and verify configuration consistency.

[0057] Read the pseudo-random variable obtained in step (2) Confirm that it is located in the [0,1) interval, and confirm that the candidate node set, calibration bias value and winning rule come from the same configuration snapshot. New weights must not be mixed with old biases.

[0058] (3.3) Calculate the routing index of the candidate node, wherein the routing index is the sum of the pseudo-random variable and the calibration bias value.

[0059] For each candidate node, the pseudo-random variable is added to the calibration bias value to obtain the routing metric for that candidate node. No logarithmic, exponential, or target probability integrals are performed during the request phase.

[0060]

[0061] k is the session identifier; This represents the pseudo-random variable obtained by session k with respect to candidate node i through joint hashing and normalization. This indicates the routing metric for candidate node i in this session; This indicates the target storage node ultimately selected by session k; This is the calibration bias value obtained for the entire set of candidate nodes. Adding the same constant to all calibration bias values ​​only shifts the routing indices of each node as a whole, without changing their relative magnitudes or the final routing result. Therefore, this formula compares the routing indices of each candidate node... Select the node with the largest value.

[0062] The mathematical basis and configuration generation process for bias calibration are given below. Assume N≥2, and the allocation share for each target is... Furthermore, the sum is 1, and the pseudo-random variables of each candidate node are independent and all follow a continuous uniform distribution on the interval [0,1). The following derivation is performed under the ideal independent continuous uniform hash model; this ideal independent continuous uniform hash model does not refer to re-random sampling when a fixed session makes repeated requests.

[0063] Step 1: Establish the winning probability of each node. Using pseudo-random variables... For node i to win, all other nodes must satisfy the condition that... Define the truncation function:

[0064]

[0065] Based on independence, multiplying the probabilities of each conditional event and then integrating u over [0,1], we obtain:

[0066]

[0067] In a continuous model, the probability of a tie for the maximum value is zero, and the sum of the probabilities of all nodes winning is 1. The calibration objective is:

[0068]

[0069] The above integrals are at 0, 1, and within (0,1). , The process involves segmentation; the integrand of each segment is a polynomial of degree N-1, which can be integrated piecewise or calculated using Gaussian integrals of sufficient order.

[0070] Step 2: Establish the convex calibration problem and demonstrate the existence of a solution. Define the objective function for the configuration phase:

[0071] 1

[0072] For a fixed pseudo-random variable vector (U1,…,U... N ), It is a pointwise maximum of several affine functions with respect to the bias vector θ, and therefore a convex function; taking its expectation also makes it a convex function. Under a continuous distribution, the probability of tandem maxima is zero; except for tandem points, about The partial derivative is 1 when node i wins, and 0 otherwise. Therefore, taking the expectation, we have:

[0073]

[0074] Since adding the same constant to all calibration biases does not change the relative magnitude of the scores at each node, the bias vector has a common translational degree of freedom. This degree of freedom is eliminated by fixing the Nth calibration bias to 0. Let... The minimum value among the shares allocated to each objective is:

[0075]

[0076] Since the Nth calibration bias is fixed at 0, when the norm of the vector composed of the remaining free biases approaches infinity... It also tends towards infinity. Combining the lower bound mentioned above, we can obtain F(θ)→∞, therefore F has a finite minimum under this anchor point constraint. At the minimum, the partial derivatives corresponding to the first N-1 free variables are zero, and the Nth term is determined by the sum of the winning probabilities of all nodes being 1:

[0077]

[0078] This result proves that any strictly positive target allocation share can be achieved in an ideal independent continuous uniform hash model, but it does not assert that the calibration solution is necessarily unique, nor does it guarantee that any numerical algorithm will converge. After fixing the Nth calibration bias to 0, θ is represented as a column vector composed of the first N-1 free biases, and the probability residual is defined as follows:

[0079]

[0080] Step 3: Numerical calibration, verification, and release. The configuration terminal can be initialized from zero bias or the calibration bias value of the old configuration, and the system of equations consisting of the first N-1 probability residuals is solved using the damped Newton iteration method. Let the residual column vector be composed of the first N-1 probability residuals. Then, each iteration solves for:

[0081]

[0082] in, For the iterative correction vectors of the first N-1 free biases, Step size factor Let be the probability residual vector. for about A Jacobian matrix whose elements satisfy Since the Nth calibration bias is fixed at 0, only the first N-1 free biases are updated during the iteration process. The Jacobian matrix can be obtained analytically by probability integral differentiation or by using the central difference approximation:

[0083]

[0084] in, Let be the j-th standard basis vector in the N-1 dimensional real vector space, i.e., the j-th component is 1 and the other components are 0; h is the central difference step size. The above difference formula is approximated by applying +h and -h perturbations only to the j-th free bias. A backtracking search is used to progressively reduce α, and updates are only accepted when the residual norm decreases. If the linear system is singular, the number of iterations exceeds the limit, or the computational accuracy is insufficient, calibration failure is reported, and substandard results must not be published as acceptable configurations.

[0085] The probability residuals of the parameters to be published are recalculated, and both absolute and relative errors are verified. For extremely small weights, the calibration cannot be considered effective solely based on a small absolute error.

[0086]

[0087] threshold , It should be preset according to business requirements; the floating-point residual is not equal to the rigorously certified upper bound of the integral error. The actual error also includes integral evaluation, parameter quantization, discrete hashing, hash bias, and finite sample fluctuation.

[0088] After calibration, a configuration snapshot is generated: Configuration snapshot = {node, weight, bias, hash specification, winning rule, version}.

[0089] The configuration snapshot includes a stable node identifier, preset weight value, calibration bias value, hash specification, stable seed, winning rule, and configuration version. The configuration generation end uniformly generates and verifies parameters; each routing end must not solve and round them separately. The configuration snapshot is atomically replaced during deployment, and only one version is read per request. Version consistency should be ensured during cross-platform switching based on the request version or switching protocol.

[0090] Under the ideal independent continuous uniform hash model, calibration guarantees that the node selection probability equals the target allocation share; the actual implementation is verified according to the declared error conditions:

[0091] Ideal conclusion: .

[0092] Actual implementation: Verify according to the declared error conditions.

[0093] A new configuration snapshot is not published when calibration fails. The old configuration is retained only if the old node set is still serviceable; if the old node has failed, it returns to unroutable or enters an explicitly marked emergency procedure, and the emergency result does not guarantee the target allocation share.

[0094] For example, when the preset weight ratio of the two nodes is 2:1, we can take...

[0095]

[0096] Their selection probabilities under the ideal model are 2 / 3 and 1 / 3, respectively. When the preset weight values ​​of the three nodes are in the ratio of 1:2:3, a set of approximate calibration bias values ​​are (−0.2545706617, −0.1029290408, 0), which should be recalibrated according to the actual storage precision when used.

[0097] The time complexity of traversing N nodes online is O(N), the additional space complexity of streaming maximum search is O(1), and the space complexity of configuration is O(N). Configuration calibration costs are accounted for separately: the reference implementation of directly calculating the winning probability of all nodes node-by-node and segment-by-segment is approximately O(N) per iteration. 4 The arithmetic operations, iterations, and Jacobian evaluations incur additional costs. This embodiment is suitable for clusters with moderate node size and infrequent changes in preset weight values; it does not guarantee that the configuration calibration can be used for real-time weight adjustment of any size.

[0098] Furthermore, this calculation process is entirely based on the session identifier and node identifier carried in the request, without relying on any globally stored state information. Each calculation is independent and has no side effects, achieving stateless routing decisions. During the request phase, only hash, addition, and comparison operations need to be performed on each candidate node, resulting in extremely low overall computational overhead, making it suitable for high-concurrency inference request scenarios.

[0099] (4) Compare the routing metrics in the routing metric set, select the optimal node as the target storage node and perform request forwarding.

[0100] After traversing all candidate nodes and calculating routing metrics, the system obtains a set of routing metric values ​​that correspond one-to-one with each node. This step compares these routing metric values ​​to determine the target storage node and complete the request forwarding.

[0101] Specifically, the system performs a maximum value lookup operation on the set of routing metric values, selecting the candidate node corresponding to the routing metric with the largest value, and determining it as the target storage node for this request. If multiple nodes have the same maximum routing metric value, the system selects the winner based on a stable lexicographical order of node identifiers, and does not rely on the returned order of the node list; if the candidate set is empty, it returns "unroutable". Ideal calibration guarantees the probability of a node being selected; the actual allocation ratio, cache usage, request popularity, and computational load under a limited number of sessions may deviate from the weight ratio, and it does not guarantee the elimination of overload or memory overflow.

[0102] According to the calibration during the configuration phase, under the ideal independent continuous uniform hash model, the probability that any candidate node is selected as the target storage node is equal to the proportion of that node's preset weight value to the sum of the preset weight values ​​of all candidate nodes. In actual implementation, the calibration results are verified according to preset absolute error thresholds and relative error thresholds. The validity of this property does not depend on the number of virtual nodes or the historical distribution of requests.

[0103] After determining the target storage node, the system forwards the current inference request to that node. Upon receiving the request, if the target storage node has a historical KV Cache previously generated by the session in its local storage system, it reuses it directly, performing inference calculations only on the newly added dialogue content in this request, and storing the newly generated KV Cache data locally. Due to the deterministic nature of the joint hash operation in step (2), requests initiated by the same session at different times are routed to the same target storage node when the configuration snapshot remains unchanged. Whether the cache can be obtained after a route change or node failure depends on other data recovery or recalculation mechanisms; this routing formula itself does not guarantee that the target node already possesses a complete historical cache.

[0104] In expansion scenarios, where at least one new storage node is added to the cluster, the new node set and target allocation share are recalibrated, and a new configuration snapshot is published. The calibration bias values ​​of existing nodes may change, and the scoring order among existing nodes may also change; it cannot be claimed that migration only applies to the new node. If the old calibration bias values ​​are fixed, although the order among the old nodes is maintained, the target allocation share of the new configuration generally cannot be guaranteed simultaneously.

[0105] In a scaling-down scenario, where an existing node in the cluster is removed due to failure or voluntary shutdown, the exiting node is deleted and the remaining nodes are recalibrated. Even if the original target node has not exited, the session may be rerouted. Routing cannot continue when the last node exits. Cache recovery or recalculation after a node failure is handled by a separate mechanism.

[0106] This solution guarantees determinism under a fixed configuration snapshot and the target allocation share after ideal calibration, but does not guarantee minimal migration or minimum cache byte transfer during scaling up or down. When all weights are scaled proportionally and the target allocation share remains unchanged, the original calibration bias value and configuration snapshot can be reused to maintain the routing unchanged.

[0107] The above description is only a specific embodiment of the present invention and the technical principles used. If changes are made according to the concept of the present invention, and the resulting functions do not exceed the spirit covered by the specification and drawings, they should still be within the protection scope of the present invention.

Claims

1. A stateless routing method for distributed caching, characterized in that... include: Step (1) Perform request parsing to confirm session identification information; Step (2) Traverse the candidate nodes and obtain the pseudo-random variables of the candidate nodes; Step (3) Obtain the preset weight value and calibration bias value of the candidate node. The calibration bias value is obtained by calibration during the configuration phase based on the preset weight values ​​of all candidate nodes. Calculate the routing index of the candidate node, wherein the routing index is the sum of the pseudo-random variable of the candidate node and the calibration bias value, to obtain a set of routing indices; Step (4) compares the routing metrics in the routing metric set, selects the optimal node as the target storage node, and performs request forwarding.

2. The stateless routing method for distributed caching according to claim 1, characterized in that, The session identification information is a data field used to uniquely identify a continuous multi-turn dialogue session.

3. The stateless routing method for distributed caching according to claim 2, characterized in that, The session identification information is a request feature for routing decisions. As one of the input factors for subsequent joint hash calculations, it enables requests initiated by the same session at any time to generate a deterministic numerical sequence bound to itself when participating in routing calculations.

4. The stateless routing method for distributed caching according to claim 1, characterized in that, Step (2) includes: (2.1) Obtain the list of available nodes and node identifiers; (2.2) Concatenate the request feature with the node identifier to construct a union string; (2.3) Perform a deterministic hash operation on the concatenated string to generate a fixed-width integer hash value, and verify that the hash value does not exceed the range that the fixed-width string can represent; (2.4) Normalize the integer hash value to generate pseudo-random variables for candidate nodes.

5. A stateless routing method for distributed caching according to claim 4, characterized in that, Each node in the node list described in step (2.1) has a globally unique and stable node identifier, which is used to uniquely distinguish each node at the network level. Changing the node address does not change its node identifier.

6. A stateless routing method for distributed caching according to claim 1, characterized in that, The preset weight value of the node is preset according to the node's storage capacity, bandwidth and / or load rate, and can reflect the node's carrying capacity in a true and linear manner. The larger the preset weight value, the more user sessions and the amount of accumulated KV cache data the node is expected to handle.

7. A stateless routing method for distributed caching according to claim 6, characterized in that, Under the ideal independent continuous uniform hash model, after configuration phase calibration, the probability that any request is routed to the node is equal to the proportion of the node's preset weight value to the sum of the preset weight values ​​of all nodes. In practice, the calibration results are verified according to the preset absolute error threshold and relative error threshold.

8. A stateless routing method for distributed caching according to claim 1, characterized in that, Step (4) includes: (4.1) Perform a maximum value search on the set of routing metrics, select the candidate node corresponding to the routing metric with the largest value, and determine it as the target storage node for this request; (4.2) If multiple nodes have the same maximum routing index, the system selects the target node according to the fixed lexicographical order of the node identifiers; (4.3) After determining the target storage node, the system forwards the current inference request to the node. After receiving the request, if the target storage node has a historical KV Cache previously generated by the session in its local storage system, it will reuse it directly, only perform inference calculation on the dialogue content added in the current request, and continue to store the newly generated KV Cache data in its local storage.

9. A stateless routing method for distributed caching according to claim 1, characterized in that, The calibration bias value is determined in the configuration phase by solving a calibration problem: in the configuration phase, the difference between the probability of each node being selected and the target allocation share of each node under the ideal independent continuous uniform hash model is used as the residual. Through iterative solution, the probability that any request is routed to the node under the ideal independent continuous uniform hash model is equal to the proportion of the preset weight value of the node to the sum of the preset weight values ​​of all nodes.

10. A stateless routing method for distributed caching according to claim 9, characterized in that, The calibration bias value, node set, preset weight value, hash specification, and winning rule are published together in the form of a configuration snapshot; the same version of the configuration snapshot is read during the request phase; when the node set changes or the preset weight value is adjusted, the configuration snapshot is recalibrated and a new configuration snapshot is published.

Images