Large model distributed reasoning acceleration method based on PD storage and calculation separation

Abstract

The invention relates to the technical field of computer data processing, and discloses a large-model distributed reasoning acceleration method based on PD storage and calculation separation. The method comprises the following steps: constructing a topology and a global fingerprint index adaptive to a storage and calculation separation architecture; reasoning the request based on an index route and driving a pre-filling node to execute incremental calculation and position correction; dynamically writing the incremental data into a multi-level storage medium according to the global popularity score; dynamically adjusting node roles according to cluster load characteristics; performing pipeline processing and video memory data replacement in cooperation with a network transmission time sequence; the system comprises a topology and index management module, a routing and computing control module, a hierarchical storage management module, a role adaptive scheduling module and an assembly line optimization module. According to the method, data multiplexing is realized through global index and position correction, and storage and calculation resource decoupling is realized by combining hierarchical storage and dynamic scheduling, so that the limitation of a video memory is broken through, and the reasoning throughput is improved.

Description

Technical Field

[0001] This invention relates to the field of computer data processing technology, specifically to a method for accelerating distributed inference in large models based on PD (Proof-of-Storage) separation. Background Technology

[0002] The reasoning process of a large language model mainly consists of two core stages: prefill and decoding. The prefill stage is responsible for processing the prompt word sequence input by the user in parallel to generate an initial key-value cache; the decoding stage generates new lexical units one by one in an autoregressive manner based on the key-value cache generated in the previous stage. The PD storage-computation separation architecture aims to optimize the running efficiency of large models in distributed clusters by decoupling the resource allocation of these two stages.

[0003] Existing distributed inference solutions typically employ tensor parallelism or pipeline parallelism strategies, binding model weights and key-value data generated during inference to the GPU memory of computing nodes. In conventional deployment models, nodes within a computing cluster are often statically assigned roles to handle specific tasks, or each node independently maintains complete context data. To maintain the continuity of inference services, the system typically relies on GPU memory to store all historical key-value pairs and occupies the corresponding storage space until the inference request ends.

[0004] Because existing technologies tightly couple computing and storage resources, and lack fine-grained data routing and hierarchical storage mechanisms for compute-storage separation architectures, large models suffer from significant shortcomings when handling long contexts or high-concurrency requests. On the one hand, due to the lack of global data indexing and location alignment methods, key-value data generated during the pre-filling stage is difficult to migrate and reuse at low cost between different nodes, resulting in repeated computation of historical information during subsequent inference, leading to wasted computing power. On the other hand, a single GPU memory storage structure cannot handle massive amounts of key-value data that grow linearly with sequence length. Once the GPU memory capacity reaches its limit, the system is often forced to interrupt services or discard contexts. Furthermore, static node configurations cannot cope with the real-time fluctuations in the ratio of pre-filling to decoding tasks in inference traffic, which can easily cause cluster load imbalance and reduce overall inference throughput. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method for accelerating distributed inference of large models based on PD storage-computation separation. It aims to solve the problems in existing technologies, such as the difficulty in reusing historical key-value data due to the lack of effective data interaction and storage mechanisms, resulting in repetitive calculations, the limitation of long context processing capabilities by a single memory level, and the cluster load imbalance caused by static node resource configuration.

[0006] To achieve the above objectives, the present invention provides the following technical solution: The first aspect of this invention provides a method for accelerating distributed inference of large models based on PD (Programmable Memory Deployment) separation. This method addresses the technical problem of low GPU memory utilization and dynamic load imbalance in large language models during distributed inference due to the mismatch in computational and storage resource requirements between the pre-filling and decoding stages. Specifically, the method includes the following steps: S1. Construct an interconnect topology adapted to the PD (Prefill-Decode) storage-compute separation architecture. Divide the distributed cluster's computing resources into two logical roles: prefill nodes for parallel computation tasks handling prompt word input, and decoding nodes for serial computation tasks handling autoregressive generation. Based on this, establish a global fingerprint index to maintain the physical distribution information of key-value data blocks. The construction process of this global fingerprint index includes: generating a content integrity fingerprint for verifying data uniqueness using a cryptographic hash algorithm for the original tag sequence corresponding to the key-value data block; and generating semantic feature vectors representing the semantic similarity of the data using a word embedding model combined with a locality-sensitive hash algorithm. By establishing a mapping relationship between the content integrity fingerprint, semantic feature vectors, and physical storage locations, an addressing basis is provided for data collaboration between prefill nodes and decoding nodes.

[0007] S2. Receive the large model inference request and perform routing decisions and incremental calculations based on the global fingerprint index. During the routing decision phase, to locate the optimal target decoding node, a routing affinity score is calculated. This score is obtained by weighting three parameters: the historical prefix matching length of the candidate decoding node, the current computing unit utilization, and the available inbound network bandwidth. The longer the historical prefix matching length, the closer the computing unit utilization is to the optimal load target, and the larger the available inbound network bandwidth, the higher the routing affinity score. The node with the highest score is selected as the target decoding node.

[0008] Subsequently, the pre-filled node performs key-value calculations only for the incremental portion of the inference request. To address the discontinuous positional encoding issue caused by data sharding in a compute-sharded architecture, positional correction is performed: by comparing the historical data fingerprint list stored by the target decoding node with the fingerprint sequence of the current request, the divergence point of the sequence is located; the phase offset of the incremental data relative to the original historical data is calculated based on the position index of the divergence point; and a phase rotation parameter is generated using a rotational positional encoding algorithm based on this phase offset, serving as a positional correction vector for the decoding node to perform phase alignment of the key vector of the historical data.

[0009] S3. During the transmission of incremental key-value data to the target decoding node, hierarchical storage mapping is performed based on the global heat score of the data. The global heat score is calculated as follows: the multi-head attention weight matrix of the large model network layer is aggregated, the attention weight values ​​of query terms to key-value terms are extracted, and a time decay factor is introduced to weight and sum the weight values ​​within the generated sequence window to characterize the access frequency and importance of the data block. According to the score, the data is written to different levels of storage media: data with a score higher than the first threshold is directly written to the first-level video memory; data with a score between the first and second thresholds is written to the second-level memory; data with a score lower than the second threshold is written to the third-level solid-state drive through remote direct memory access (RDMA) technology.

[0010] S4. Collect the operating status of the distributed cluster to extract cluster load characteristics, and dynamically adjust node roles based on these characteristics. The cluster load status is determined by calculating a resource reconfiguration urgency index. This index is calculated based on the ratio of the task queue length to the maximum allowed queue length of the decoder node resource group, and the deviation between the computing unit utilization of the pre-filled node resource group and the optimal load target. When the resource reconfiguration urgency index exceeds a preset threshold, the cluster load is determined to require adjustment. At this time, a data transmission channel is established using idle bandwidth to mirror and synchronize active data to the target pre-filled node to be switched. After synchronization, the role attribute of the target pre-filled node is updated to a decoder node, and the original decoder node is reset, thereby achieving dynamic reallocation of computing resources.

[0011] S5. The collaborative network transmission timing and the computation progress of the target decoding node are pipelined, and the video memory data is replaced based on the data retention value score. The data transmission timing is adjusted according to the real-time progress of the decoding computation to reduce the impact of communication latency. Simultaneously, to address the issue of limited video memory space, a data retention value score is calculated for each data block. This score is a weighted sum of the data's global heat score and the magnitude of the corresponding positional correction vector. When video memory space is insufficient, the retention value scores of data blocks within video memory are sorted, and the data block with the lowest score is identified as the data block to be replaced and downgraded and migrated to the second-level memory, thereby freeing up video memory space.

[0012] A second aspect of the present invention provides a large-scale distributed inference acceleration system based on PD storage-compute separation, comprising: The topology and index management module is used to build the storage-computing interconnection topology of the distributed cluster, maintain the global fingerprint index, and provide physical address query services to the routing and computing control module. The routing and computation control module connects to the topology and index management module, receives large model inference requests and distributes the requests to target decoding nodes based on routing affinity scores, driving pre-filled nodes to perform incremental computation; The hierarchical storage management module receives the calculation results generated by the routing and calculation control module, and writes the data stream to the target decoding node's video memory, RAM, or SSD based on the calculated global heat score; the role adaptive scheduling module collects cluster load characteristics and calculates the urgency of resource reconfiguration, and sends a node role switching instruction to the topology and index management module and triggers data preloading when the conditions are met. The role-adaptive scheduling module is used to collect cluster load characteristics and calculate the urgency of resource reconfiguration. When the conditions are met, it sends a node role switching instruction to the topology and index management module and triggers data preloading. The pipeline optimization module, in collaboration with the network transmission engine, adjusts the data transmission timing and calls the interface of the hierarchical storage management module to perform dynamic replacement of video memory data.

[0013] The third aspect of the present invention provides a computer device, including a processor, a communication interface, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the large-model distributed inference acceleration method based on PD storage-computation separation provided in the first aspect.

[0014] This invention provides a method for accelerating distributed inference in large models based on PD (Proof-of-Store) computation-in-store separation. It offers the following advantages: 1. This invention effectively solves the problem of difficult reuse of historical KV data in distributed inference by constructing an interconnection topology adapted to PD storage-compute separation and a global fingerprint index. It uses routing affinity scoring to schedule inference requests to the decoding node with the highest cache hit rate, and combines position correction vectors to solve the problem of discontinuous position encoding caused by data fragmentation. This allows the key-value pairs generated by incremental calculation to be directly concatenated with historical data. This mechanism avoids repeated calculation of long historical contexts, reduces the calculation latency in the pre-filling stage, and helps to improve the first character generation speed of large models under the storage-compute separation architecture.

[0015] 2. This invention adopts a hierarchical storage mapping mechanism based on semantic popularity, which breaks through the limitation of single-node memory capacity on the length of inference context for large models. By aggregating attention weights to calculate the global popularity score, data is dynamically distributed to three levels of media: video memory, RAM, and solid-state drives. Dynamic replacement is performed based on the data retention value score. This method ensures that core data with high-frequency access always resides in high-speed storage, while low-frequency data automatically sinks to lower levels. Without increasing the hardware cost of video memory, it supports inference tasks with longer sequences and optimizes the utilization of storage resources.

[0016] 3. This invention introduces a dynamic role switching and pipelined collaboration strategy based on load characteristics, which enhances the stability of the distributed cluster in large-scale concurrent scenarios. By real-time computing resource reconfiguration urgency, node roles are automatically switched when the pre-filling task and decoding task loads are unbalanced, avoiding local congestion or idleness of computing resources. Combined with pipelined processing of network transmission and computing progress, it effectively masks the data communication delay between nodes, enabling the system to maintain high throughput and response consistency under dynamically changing load pressure. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the method flow in an embodiment of the present invention; Figure 2 This is a schematic diagram of the system framework in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Please see the appendix Figure 1 This invention provides a method for accelerating distributed inference of large models based on PD (distributed storage and computation) separation. This method is executed through electronic devices coordinating Prefill computing nodes, Decode decoding nodes, distributed storage controllers, and network transmission engines, and includes the following steps: Step S1: Construct a storage-compute decoupled PD interconnection topology and global index.

[0020] First, during the system initialization phase, the electronic device divides the physical computing cluster into logical Prefill computing groups and Decode decoding groups according to a preset resource configuration strategy. Using the RDMA (Remote Direct Memory Access) network protocol, a direct data transmission channel is established between the computing memory space of the Prefill nodes and the multi-level storage address space of the Decode nodes, ensuring the decoupling of storage and computing at the physical layer and their interconnection at the logical layer.

[0021] Based on this, the electronic device initializes and maintains a distributed global fingerprint index in real time. This index does not store the actual key-value pair data, but records the unique fingerprint identifiers of all key-value cache data blocks in the cluster and their physical distribution information.

[0022] Specifically, the index records which node and which level of storage medium each data fingerprint currently resides in, including the first level of video memory, the second level of RAM, and the third level of solid-state drive, thus forming a globally unified virtual address space view.

[0023] Step S2: Perform PD routing and incremental prefill calculation based on storage residuals.

[0024] When the electronic device receives an inference request from the client, it first extracts the feature vector of the Prompt word in the request and queries the aforementioned global fingerprint index. Based on the query results, the electronic device selects Decode nodes that have historical key-value data highly similar to the current request features stored in the third-level solid-state drive as candidate nodes.

[0025] Taking into account the computational load of candidate nodes, available network bandwidth, and similarity of historical data, the electronic device uses a preset affinity scoring strategy to select the optimal Decode node and pre-assigns the decoding task of the request to that node.

[0026] After identifying the target Decode node, the electronic device schedules the Prefill node to perform prefill computation. At this point, the Prefill node does not perform a full computation of the requested prompt. Instead, it retrieves fingerprint information of historical data from the target Decode node via the network and compares it to identify the differences between the current request and historical data. Subsequently, the Prefill node performs incremental KVCache computation only on the token sequence of this difference, and simultaneously calculates a correction vector to correct historical data deviations, thereby helping to reduce the computational load in the prefill stage.

[0027] Step S3: Perform QoS-based hierarchical transmission based on semantic-network vertical mapping.

[0028] During the process of generating KVCache data at the Prefill node, the electronic device monitors and analyzes the AttentionMap in the large model network layer in real time, quantitatively evaluates the global heat score of the KVCache data block corresponding to each Token, and controls the network transmission engine to perform hierarchical transmission based on Quality of Service (QoS).

[0029] Specifically, for anchor data blocks with a heat score higher than the first threshold, the electronic device encapsulates them into a high-priority RDMA queue and controls them to be directly written to the first-level video memory of the target Decode node; for regular data blocks with a heat score between the first and second thresholds, they are encapsulated into a standard priority queue and written to the second-level memory of the target node; for long-tail cold data blocks with a heat score lower than the second threshold, they are encapsulated into a low-priority queue and directly written to the third-level solid-state drive of the target node via remote write operations. In this way, dual alignment of network transmission priority and storage disk writing level is achieved.

[0030] Step S4: Perform dynamic and seamless switching of PD roles based on the accompanying image mechanism.

[0031] The electronic device monitors the load status of the Prefill and Decode groups in the cluster in real time. When the system load meets the preset role reconfiguration conditions, such as the Prefill task queue being idle and the Decode task queue being congested, the electronic device locks the target node that is about to switch from the Prefill role to the Decode role.

[0032] During the window period before the official role-switching command is issued, the electronic device establishes a companion channel using the target node's idle inbound network bandwidth. Through this channel, the electronic device pre-images the high-frequency data fingerprints of the third-level storage layer from nearby active Decode nodes, transmits them in the background, and writes them to the target node's local solid-state drive. When the role-switching command is officially triggered, the electronic device controls the target node to directly mount the local solid-state drive containing the companion data as its third-level cache pool, thus eliminating the need for cold-start data loading and achieving a seamless and rapid role-switching.

[0033] Step S5: Perform hierarchically aligned pipeline decoding and intelligent replacement.

[0034] During the autoregressive decoding process to generate tokens, the electronic device collects the current decoding progress information in real time and feeds it back to the network transmission engine. Based on this progress information, the transmission engine dynamically adjusts the sending sequence of subsequent KVCache data to ensure that the arrival time of data from each layer in the Decode node's memory is strictly after the previous layer's calculation is completed and before the current layer's calculation begins, thus completely hiding cross-node data transmission latency on the timeline.

[0035] Meanwhile, when the memory capacity of the Decode node reaches its limit, the electronic device triggers a smart replacement mechanism. This mechanism combines the heat score calculated in step S3 with the magnitude of the correction vector generated in step S2, and prioritizes the elimination or migration of residual data blocks with low heat scores and minimal impact on inference accuracy to the next level of storage, thereby ensuring that limited memory resources always serve the most critical context data.

[0036] The specific implementation process of each of the above steps will be explained in detail below.

[0037] During step S1, the electronic device, in collaboration with the pre-filled computing node, the decoding computing node, and the distributed storage controller, performs the operations of constructing the storage-computation decoupled topology and initializing the global index. This process specifically includes the following sub-steps: Step S101: Establish a physical and logical decoupled in-memory interconnect topology.

[0038] The electronic device logically partitions the computing resources in the distributed cluster, defining them as pre-filled resource groups and decoding resource groups. This partitioning is based on the current role labeling of the nodes, rather than on static binding of physical hardware. The electronic device configures the cluster's high-speed interconnect network, specifically employing the Remote Direct Memory Access (RDMA) network protocol. By configuring queue pairs on the network interface cards (NICs), the electronic device establishes direct data paths between the computing memory of the pre-filled nodes and the various levels of storage media of the decoding nodes.

[0039] To achieve decoupling of storage and computation, electronic devices introduce a unified address space technology. The video memory, system memory, and local solid-state drive storage space of all decoding nodes within the cluster are mapped to a global virtual address space. In this space, each physical storage unit corresponds to a unique global virtual address. The electronic device maintains a global address mapping table in the metadata server, which records the correspondence between the global virtual address and the physical node identifier, local physical address, and remote access key.

[0040] Electronic devices register the aforementioned three-tier physical storage regions as memory regions accessible via remote direct memory access (RDBMI) through a memory registration operation. Specifically, for the third storage tier, namely the solid-state drive (SSD), the electronic device allows the network card to directly read and write to the SSD using peer-to-peer memory technology or a dedicated driver. This process bypasses the operating system kernel's file system layer and directly transfers data via bus addresses, thereby eliminating the copying overhead incurred by the central processing unit (CPU) during data transfer.

[0041] Step S102: Generate key-value data fingerprints based on semantic features.

[0042] In large model inference tasks, the generated key-value cache data is managed in units of data blocks.

[0043] Step S103: Construct a distributed global fingerprint index structure.

[0044] Based on the generated fingerprint, the electronic device initializes a distributed global fingerprint index. This index uses a distributed hash table or in-memory database cluster as the underlying storage engine. The purpose of the index is to maintain the mapping relationship between data fingerprints and physical storage locations.

[0045] Each entry in the index contains a key and a value. The key is the data fingerprint generated in step S102, and the value is a list of physical locations corresponding to that fingerprint. Each element in the physical location list records detailed node information containing the data block, specifically including: a unique identifier of the physical node storing the data block, a storage tier identifier (corresponding to video memory, RAM, or SSD), the starting global virtual address of the data block in the corresponding storage tier (corresponding to the global address mapping table in step S101), the number of times the data block has been referenced, and the timestamp of the last access.

[0046] Electronic devices store index data in shards on the cluster's metadata server, ensuring that any pre-populated node can concurrently query the index.

[0047] Step S104: Perform real-time consistency maintenance of the index.

[0048] Electronic devices initiate background maintenance threads to monitor data change events within the cluster. When new key-value cache writes occur, data is migrated between different storage tiers, or data is evicted, the storage controller generates change logs.

[0049] Electronic devices synchronously update the corresponding entries in the global fingerprint index based on the change log. For data migration events, electronic devices update the hierarchical identifier and address fields in the corresponding fingerprint entry; for data deletion events, electronic devices decrement the reference count, and when the reference count reaches zero, the fingerprint entry is physically deleted from the index. For index state conflicts caused by concurrent read and write operations on multiple nodes, electronic devices use a distributed lock mechanism to lock the index entries to ensure the consistency of the index data.

[0050] Through the steps S101 to S104 above, the electronic device constructs a shared key-value storage pool view that spans the boundaries of physical nodes at the logical level, and establishes a fast indexing mechanism from semantic fingerprints to physical addresses. This enables the scheduler to allocate subsequent tasks based on the actual distribution of data rather than just the idleness of computing resources, laying the data foundation for routing decisions based on storage residuals.

[0051] During step S2, the electronic device, in coordination with the pre-fill computing node, the decoding computing node, and the distributed storage controller, performs pre-fill-decoding routing and incremental pre-fill computation based on storage residuals. This process specifically includes the following sub-steps: Step S201: Calculate the routing affinity score of candidate decoding nodes. When a reasoning request initiated by a client is received, the electronic device first extracts the feature of the prompt words in the request and queries the global fingerprint index constructed in step S1 to identify all decoding nodes that hold historical key-value data similar to the current request in their local storage (including video memory, RAM, or SSD), and uses these as a set of candidate nodes. To select the optimal target decoding node, the electronic device comprehensively considers the reuse value of historical data, the current computational load of the node, and the network bandwidth status, and calculates the affinity score of each candidate node. The specific scoring calculation is based on the following formula: ; In the formula, For the first Routing affinity score of each candidate decoding node; The length of the historical prefix that is the same as the current request prompt word stored in the candidate node; The total length of the current request prompt; This represents the current computing unit utilization rate of the candidate node. This is the maximum utilization threshold for the computing unit; This represents the inbound network bandwidth currently available for this candidate node. The estimated bandwidth required to transmit the remaining difference data; , , These are preset historical data similarity weights, computational load weights, and network bandwidth weights, respectively, and satisfy the following conditions: .

[0052] The electronic device selects the candidate node with the highest affinity score as the target decoding node for this inference task. This routing strategy based on storage residuals utilizes the distribution characteristics of data in physical space to allocate computational tasks to the location of the data, thereby reducing the amount of newly generated computational data.

[0053] Step S202: Retrieve historical fingerprints and determine the incremental calculation range.

[0054] Once the target decoding node is identified, the electronic device schedules a pre-filled node to respond to the request. Instead of immediately initiating full computation, the pre-filled node reads a list of fingerprints of historical key-value data held by the target decoding node from its storage space via a remote direct memory access network.

[0055] The pre-filled node compares the data block fingerprint sequence corresponding to the current request prompt with the read historical fingerprint sequences item by item. The electronic device starts matching from the beginning of the sequence until it finds the first inconsistent fingerprint position, which is defined as the divergence point. The electronic device marks the sequence portion before the divergence point as the shared prefix region, where the data directly reuses the existing historical data of the target decoding node without recalculation; the sequence portion after the divergence point is marked as the incremental calculation region, where the data needs to be recalculated using new key values ​​by the pre-filled node.

[0056] Step S203: Perform differential incremental key value calculation.

[0057] The pre-fill node only loads the model parameters corresponding to the incremental calculation area and performs forward inference calculation on the prompt word tags in that area to generate new key-value cache data blocks. Compared to traditional full pre-filling, step S203 utilizes the existing inactive data on the target decoding node, i.e., the historical records stored in secondary or tertiary storage, and only calculates the differences in the new user input. In multi-turn dialogue scenarios, the dialogue history of the previous rounds often already exists in storage, and the pre-fill node only needs to calculate the key-value data corresponding to the latest round input, thereby reducing the computational power consumption and processing latency of the pre-filling stage.

[0058] Step S204: Generate historical deviation correction vector.

[0059] Because the attention mechanism in large models is position-sensitive, directly concatenating historical key-value data with newly calculated key-value data will introduce deviations in positional encoding. Therefore, electronic devices simultaneously calculate correction parameters for positional alignment while performing incremental computation.

[0060] The electronic device calculates the offset relative to the original historical data position encoding based on the location index of the bifurcation point. Utilizing the characteristics of the Rotated Position Encoding (RoPE) algorithm, the electronic device generates corresponding phase rotation parameters as a correction vector. Pre-filled nodes package the newly generated incremental key-value data along with this correction vector.

[0061] In the subsequent data transmission phase, this data is sent to the target decoding node. When loading historical data, the target decoding node applies this correction vector to perform phase rotation adjustment on the key vector of the historical key-value data, thereby ensuring that the concatenated full key-value data maintains continuity and consistency in semantic and positional information.

[0062] Through the above steps S201 to S204, the present invention utilizes the data retention characteristics of the storage layer in the storage-compute separation architecture, and transforms the pre-filling task into a task that combines storage retrieval and partial computation through routing algorithms and incremental computation, thereby improving the overall inference efficiency of the system.

[0063] During step S3, the electronic device, in coordination with the pre-filled computing node, the decoding computing node, and the network controller, performs a service quality-level transmission operation based on semantic-network vertical mapping. This process specifically includes the following sub-steps: Step S301: Quantify the global heat of key-value data based on attention map.

[0064] During the process of generating key-value cache data from pre-filled nodes, the electronic device monitors the attention maps in the large model's network layers in real time. Since the large model typically contains multiple network layers and attention heads, the electronic device aggregates the weight matrices of the multiple attention heads (e.g., calculating the average or taking the maximum activation value) to statistically determine the cumulative weight of each word in the sequence generation process and quantitatively evaluate the global heat score of each key-value data block.

[0065] Specifically, electronic devices focus on the normalized weight matrix obtained by the dot product of the query vector and the key vector, identifying frequently activated terms in long sequence generation. The specific global popularity score is calculated according to the following formula: ; In the formula, This is the global heat score for the current key-value data block. This represents the number of terms contained in the data block. Position in the aggregated attention graph Query term pair position Attention weight values ​​for key-value words; This represents the starting position of the current generation phase; The length of the generated sequence; This is a time decay factor used to reduce the weight of behaviors that have been observed in the past.

[0066] Using the above formula, electronic devices can filter out key information that has a significant impact on subsequent reasoning from a large amount of context.

[0067] Step S302: Establish a dual alignment mapping between semantic popularity and storage level.

[0068] The electronic device has a preset first threshold and a second threshold (the first threshold is greater than the second threshold). These two thresholds can be dynamically set based on the statistical distribution characteristics of the current batch of data block popularity scores (such as the 90th percentile and the 50th percentile) to adapt to the semantic distribution differences of different tasks.

[0069] Electronic devices categorize key-value data blocks into three types based on global heat scores: For data blocks with a global heat score higher than a first threshold, the electronic device defines them as anchor data blocks. Such data typically contains key subjects or core instructions that are frequently accessed during inference, and therefore are mapped to the first-level video memory of the target decoding node.

[0070] For data blocks with global heat scores between the first and second thresholds, the electronic device defines them as regular data blocks. This type of data is more active in recent conversations and is therefore mapped to the second-level system memory of the target decoding node.

[0071] For data blocks with a global heat score below the second threshold, the electronic device defines them as long-tailed cold data blocks. Such data mostly consists of background descriptions or distant history, with an extremely low access probability, and is therefore mapped to the third-level solid-state drive of the target decoding node.

[0072] Step S303: Configure a tiered transmission channel based on quality of service.

[0073] To ensure the timely delivery of anchor data blocks, the electronic device controls the network transmission engine to perform tiered transmission based on Quality of Service (QoS). The electronic device configures multi-level priority queues in the Remote Direct Memory Access (RDMA) network.

[0074] Specifically, the electronic device encapsulates anchor data blocks into a high-priority queue and marks them as the highest traffic category, ensuring they receive priority bandwidth resources during network congestion; it encapsulates regular data blocks into a standard priority queue; and it encapsulates long-tail cold data blocks into a low-priority queue, utilizing remaining bandwidth for background transmission. In this way, the electronic device achieves dual alignment between network transmission priority and storage disk stack level, preventing large amounts of cold data from preempting the transmission channels of critical hot data.

[0075] Step S304: Perform multi-level direct write and metadata synchronization.

[0076] The network adapter of the pre-filled node concurrently performs differentiated write operations according to the mapping relationship determined in step S302. For anchor data blocks, the network card writes them directly to the video memory of the decoding node through remote direct memory access write operations; for regular data blocks, they are written to system memory; for long-tail cold data blocks, the network card bypasses the central processing unit of the decoding node and directly writes them to the third-level solid-state drive through remote write operations in conjunction with peer-to-peer memory technology.

[0077] After transmission is complete, the electronic device synchronously updates the global fingerprint index, recording the physical level at which each data block is finally written to disk, so that subsequent decoding nodes can invoke the corresponding loading instructions based on the actual storage location of the data. If lower-level data (such as solid-state drives) needs to be accessed during inference, the decoding node will asynchronously load it into higher-level storage through a pipelined prefetch mechanism.

[0078] Through the above steps S301 to S304, the present invention constructs a vertical architecture where semantics is storage, ensuring that limited video memory resources are always used to serve the highest value anchor data, thereby expanding the model's context window capacity without sacrificing accuracy.

[0079] During step S4, the electronic device collaborative cluster scheduler and each computing node perform a pre-filling and decoding role dynamic seamless switching operation based on the accompanying image mechanism. This process specifically includes the following sub-steps: Step S401: Monitor bidirectional load and trigger role reconfiguration.

[0080] The electronic device periodically collects the task queue status of the pre-filled resource group and the decoding resource group in the cluster. To accurately capture moments of system load imbalance, the electronic device introduces a resource reconfiguration urgency index to quantitatively assess the necessity of converting pre-filled nodes into decoding nodes in the current cluster. The specific urgency index is calculated according to the following formula: ; In the formula, The urgency index for resource reallocation; The current average task queue length for the decoding resource group; This represents the maximum allowed queue length for the decoding node; This represents the current average computing unit utilization of the pre-filled resource group; The target value for optimal load utilization of pre-filled nodes; A penalty factor for estimated role switching overhead is used to prevent the system from oscillating frequently in a critical state; and These are the decoding load weighting coefficient and the pre-filled idle weighting coefficient, respectively.

[0081] When the calculated resource reconfiguration urgency index exceeds a preset trigger threshold, the electronic device determines that a favorable load scissors difference has occurred, satisfying the role reconfiguration conditions. The electronic device locks the target node to be switched from the idle pre-filled resource group and sends a pre-switching signal to it.

[0082] Step S402: Establish the accompanying image channel and filter high-frequency image data.

[0083] During the transition phase before the official role-switching instruction is issued, the electronic device does not immediately stop the current service of the target node. Instead, it utilizes its idle inbound network bandwidth to establish a dedicated background data transmission link, i.e., a mirroring channel, with a nearby active decoding node that is under heavy load. The electronic device queries the access logs of the active decoding node and, combined with the global popularity score generated in step S3, filters out the currently frequently accessed hot data set. This set typically contains system-wide prompt prefixes or contextual data of recent trending topics. The electronic device obtains the data content corresponding to this hot data set and its corresponding fingerprint mapping relationship as the mirroring data index to be mirrored.

[0084] Step S403: Perform background image data preloading.

[0085] The electronic device controls the active decoding node to asynchronously transmit the selected hot data blocks and their metadata to the target node through the accompanying image channel using remote direct memory access technology.

[0086] To avoid interfering with the ongoing finalization computations on the target node, this transfer process is configured as a low-priority background transfer. Upon receiving a data block, the target node does not load it into video memory or RAM, but instead writes it directly to its local third-level solid-state drive, creating a localized shadow cache pool. This process runs independently in the background until the data coverage of the shadow cache pool reaches a preset ready threshold.

[0087] Step S404: Atomic switching of execution state and activation of storage.

[0088] Once the shadow cache pool is ready and the target node has completed all remaining tasks, the electronic device formally issues the role switching command. The electronic device atomically updates the target node's role tag in the metadata server, changing it from a pre-populated node to a decoding node. Simultaneously, the target node logically registers its local solid-state drive containing the accompanying shadow data as its third-level storage layer and loads the received fingerprint mapping relationship into its local index table in memory. Because high-frequency hot data is pre-existing on the local solid-state drive and the index is already in effect, the newly launched decoding node does not need to undergo a time-consuming cold start process of fetching data from a remote source. It can immediately accept inference requests from the load balancer in a hot cache state, achieving a smooth and seamless role switching.

[0089] Through the above steps S401 to S404, the present invention utilizes a mathematical urgency assessment and accompanying mechanism to eliminate the decision lag and data migration overhead during the node scaling process, enabling the computing cluster to flexibly switch between pre-filling and decoding roles based on real-time traffic characteristics, thereby maximizing the utilization efficiency of hardware resources.

[0090] During step S5, the electronic devices, in collaboration with the decoding computing node, network transmission engine, and storage controller, perform hierarchically aligned pipelined decoding and intelligent substitution operations. This process specifically includes the following sub-steps: Step S501: Acquire real-time decoding layer progress. During the autoregressive decoding process to generate tokens, the decoding node uses a layer-by-layer computation mode to process the network layers of the large model. The electronic device monitors the current computation progress in real time through a monitoring interface embedded in the computing framework. The electronic device obtains the index number of the network layer currently being executed in the graphics card computing unit and, combined with the total number of layers in the model and the average computation time per layer, predicts the computation start time of the next network layer. This layer progress information is fed back in real time to the network transmission engine responsible for data scheduling, serving as a synchronization clock signal for data transmission.

[0091] Step S502: Perform pipeline timing alignment for computation and transmission.

[0092] To eliminate network latency caused by reading key-value cache data across nodes, the electronic device employs a data prefetching strategy. The network transmission engine dynamically adjusts the transmission timing of subsequent network layer key-value cache data based on received layer progress information. The electronic device constructs a fine-grained pipeline scheduling mechanism to ensure that the arrival time of the Nth layer's key-value cache data in the decoding node's memory is strictly controlled to be after the N-1th layer's computation is completed and before the Nth layer's computation begins. Through this precise alignment on the timeline, the electronic device completely covers time-consuming data transmission operations within the time window of the computation unit executing the previous layer's matrix operations, achieving parallel overlap between computation and communication, thereby eliminating computational latency caused by waiting for data loading.

[0093] Step S503: Calculate the data retention value of multi-dimensional feature fusion.

[0094] As the generated word sequence grows, the video memory capacity of the decoding node easily reaches its physical limit. At this point, the electronic device triggers an intelligent replacement mechanism to determine which data blocks should be retained in video memory. To quantify the contribution of data blocks to inference accuracy, the electronic device combines the global heat score generated in step S3 with the correction vector magnitude generated in step S2 to calculate the data retention value score for each data block. The specific scoring calculation is based on the following formula: ; In the formula: Score the value of data retention; This is the global heat score for the data block, representing its semantic importance; The magnitude of the correction vector corresponding to this data block represents the magnitude of the positional encoding adjustment or the sensitivity of the geometric features; and These are the preset semantic weight coefficients and positional geometric weight coefficients, respectively.

[0095] The above formula takes into account both semantic attention and positional geometric features. The larger the magnitude of the correction vector, the more helpful the positional deviation information contained in the data block is, and the higher the risk of accuracy loss caused by replacement, thus giving it higher retention value.

[0096] Step S504: Perform dynamic permutation based on value scoring.

[0097] The electronic device sorts all data blocks in the video memory based on a calculated data retention value score. The electronic device then selects the data blocks with the lowest scores as the replacement targets.

[0098] If the score of the object to be replaced is lower than the preset discard threshold, the electronic device directly releases the video memory space it occupies; if the score is higher than the discard threshold but space must be freed up, the electronic device downgrades and migrates it to the system memory or local solid-state drive of the decoding node, and simultaneously updates the storage location information in the global fingerprint index. At the same time, the electronic device writes newly generated or prefetched high-value data blocks into the freed-up video memory space, thereby ensuring that limited video memory resources always serve the most critical context data.

[0099] Through the above steps S501 to S504, the present invention establishes a pipeline mechanism that deeply couples computation and transmission, utilizes computation gaps to digest transmission overhead, and uses value scoring to ensure that the video memory always stores high-value data.

[0100] Please see the appendix Figure 2 This embodiment provides a large-scale distributed inference acceleration system based on PD storage-compute separation, including: The topology and index management module is used to construct the storage-computing interconnection topology of the distributed cluster, logically dividing the cluster nodes into pre-filled nodes and decoding nodes, and establishing communication links between physical nodes. This module also maintains a global fingerprint index, providing physical address lookup services to the routing and computation control module. In the specific implementation of index construction, this module uses a cryptographic hash algorithm to generate a content integrity fingerprint from the original tag sequence of key-value data blocks, and combines a word embedding model and a locality-sensitive hashing algorithm to generate semantic feature vectors. By establishing a mapping relationship between the above fingerprints and vectors and physical storage locations, a global addressing benchmark is provided for cross-node data retrieval.

[0101] The routing and computation control module, communicatively connected to the topology and index management module, receives large model inference requests and distributes them to target decoding nodes based on routing affinity scores. This module generates the routing affinity score by reading the historical prefix matching length, current computing unit utilization, and available inbound network bandwidth of candidate decoding nodes and performing a weighted summation of these parameters. Simultaneously, this module drives pre-filled nodes to perform incremental computation. For location encoding alignment in a storage-compute separation scenario, this module locates divergence points by comparing fingerprint sequences, calculates phase offsets, generates phase rotation parameters, and constructs a location correction vector, thereby ensuring the correct splicing of incremental and historical data.

[0102] The hierarchical storage management module receives the calculation results generated by the routing and computation control module and performs hierarchical storage mapping. Based on the calculated global heat score, this module streams data into the multi-level storage medium of the target decoding node. Specifically, the module aggregates the multi-head attention weight matrix of the large model network layers and introduces a time decay factor to calculate the global heat score. If the score is higher than a first threshold, the data is written as high-priority data to the first-level video memory. If the score is between the first and second thresholds, it is written to the second-level memory. If the score is lower than the second threshold, it is written as cold data to the third-level solid-state drive via Remote Direct Memory Access (RDMA).

[0103] The role-adaptive scheduling module is used to collect cluster load characteristics and calculate resource reconfiguration urgency. This module monitors in real time the task queuing length of the decoding node resource group and the computing unit utilization of the pre-filled node resource group, calculating a resource reconfiguration urgency index based on the deviation of these two values ​​from their respective baselines. When this index exceeds a preset threshold, the module sends a node role switching command to the topology and index management module, establishes a background transmission channel using idle bandwidth to synchronize active data mirroring, and updates the role attributes of the target node after synchronization to dynamically balance the cluster load.

[0104] The pipeline optimization module coordinates with the network transmission engine to adjust data transmission timing to mask transmission delays and calls the interface of the hierarchical storage management module to perform dynamic replacement of video memory data. When video memory space is insufficient, this module obtains the global heat score and the magnitude of the position correction vector of the data block, and generates a data retention value score through weighted calculation; then, it sorts the scores of the data blocks in video memory, and migrates the data blocks with the lowest scores to the second-level memory, thereby freeing up video memory resources to maintain the continuous operation of the inference process.

Claims

1. A method for accelerating distributed inference in large-scale models based on PD (Proof-of-Store) and compute-in-store separation, characterized in that: Includes the following steps: S1. Construct an interconnection topology adapted to the PD storage-compute separation architecture, divide the distributed cluster into pre-filled nodes and decoding nodes, establish a global fingerprint index of the physical distribution of mapped key-value data, and establish a benchmark for collaborative data interaction between nodes. S2. Receive a large model inference request, locate the target decoding node in the distributed cluster based on the global fingerprint index, and drive the pre-filled node to perform key-value calculation and position correction for the incremental part of the large model inference request to generate incremental key-value data. S3. During the process of transmitting the incremental key-value data to the target decoding node, the incremental key-value data is dynamically written into the multi-level storage medium of the target decoding node according to the global heat score of the data, thus completing the hierarchical storage mapping under the storage-computation separation architecture. S4. Collect the running status of the distributed cluster to extract cluster load characteristics, calculate the urgency of resource reconfiguration based on the cluster load characteristics, and dynamically adjust the node roles to maintain load balance accordingly. S5. The collaborative network transmission timing and the computation progress of the target decoding node are pipelined, and the memory data is replaced according to the data retention value score to maintain the continuous operation of the large model inference acceleration process.

2. The method for accelerating distributed inference in large models based on PD storage-computation separation according to claim 1, characterized in that, In step S1, establishing a global fingerprint index for the physical distribution of mapped key-value data specifically includes: For the original tag sequence corresponding to the key-value data block, a cryptographic hash algorithm is used to generate a content integrity fingerprint for accurate comparison; Semantic feature vectors for similarity retrieval are generated using word embedding models and locality-sensitive hashing algorithms. Establish the mapping relationship between the content integrity fingerprint, the semantic feature vector, and the physical storage location, and construct the global fingerprint index.

3. The method for accelerating distributed inference in large models based on PD storage-computation separation according to claim 1, characterized in that, In step S2, locating the target decoding node in the distributed cluster based on the global fingerprint index specifically includes: Query the global fingerprint index to obtain the list of candidate decoding nodes; Obtain the historical prefix matching length, current computing unit utilization, and available inbound network bandwidth of the candidate decoding node; The historical prefix matching length, the current computing unit utilization, and the available inbound network bandwidth are weighted and calculated to obtain the route affinity score. The candidate decoding node with the highest routing affinity score is selected as the target decoding node.

4. The method for accelerating distributed inference in large models based on PD storage-computation separation according to claim 1, characterized in that, In step S2, the correction of the incremental execution location for the large model inference request specifically includes: Obtain the list of historical data fingerprints stored by the target decoding node, and compare the list of historical data fingerprints with the fingerprint sequence of the large model inference request to locate the divergence point; Based on the position index of the divergence point in the sequence, calculate the phase offset of the incremental data relative to the position encoding of the original historical data; A rotational position encoding algorithm is used to generate phase rotation parameters based on the phase offset, which serve as a position correction vector for the target decoding node to perform phase alignment of the key vector of historical data.

5. The method for accelerating distributed inference in large models based on PD storage-computation separation according to claim 1, characterized in that, In step S3, dynamically writing the incremental key-value data into the multi-level storage medium of the target decoding node based on the global heat score of the data specifically includes: The multi-head attention weight matrix of the aggregated large model network layers is used to extract the attention weight values ​​of query terms to key terms, and a time decay factor is introduced for weighted summation to calculate the global popularity score. Data blocks whose global heat scores are higher than the first threshold are marked as high-priority data and directly written into the first-level video memory of the target decoding node; Data blocks whose global heat scores fall between the first threshold and the second threshold are marked as regular data and written into the second-level memory of the target decoding node; Data blocks with a global heat score below the second threshold are marked as cold data and written to the third-level solid-state drive of the target decoding node via remote direct memory access.

6. The method for accelerating distributed inference in large models based on PD storage-computation separation according to claim 1, characterized in that, In step S4, calculating the urgency of resource reconfiguration based on the cluster load characteristics specifically includes: Extract the task queuing length of the decoding node resource group and the computing unit utilization of the pre-filled node resource group from the cluster load characteristics; The resource reconfiguration urgency index is calculated based on the ratio of the task queue length to the maximum allowed queue length and the deviation between the computing unit utilization and the optimal load target. When the resource reconfiguration urgency index exceeds a preset threshold, a role switching process is triggered.

7. The method for accelerating distributed inference in large models based on PD storage-computation separation according to claim 6, characterized in that, In step S4, dynamically adjusting node roles to maintain load balancing specifically includes: Establish a background transmission channel using idle bandwidth to mirror and synchronize active data to the local storage of the target pre-filled node to be switched. After synchronization is complete, update the role attribute of the target pre-filled node to a decoding node, and release or reset the original decoding node to a pre-filled node.

8. The method for accelerating distributed inference in large models based on PD storage-computation separation according to claim 5, characterized in that, In step S5, the replacement of the video memory data based on the data retention value score specifically includes: Obtain the global popularity score of the data block and the magnitude of the corresponding position correction vector; calculate the data retention value score by weighted summing of the global popularity score and the magnitude of the position correction vector. When there is insufficient video memory space, the data retention value scores of the data blocks in the video memory are sorted, the data block with the lowest data retention value score is determined as the data block to be replaced, and the data block to be replaced is migrated to the second level of memory.

9. A large-scale distributed inference acceleration system based on PD (Proof-of-Store) computing separation, characterized in that: The method for accelerating distributed inference of large models based on PD storage-compute separation as described in any one of claims 1-8 includes: The topology and index management module is used to build the storage-computing interconnection topology of the distributed cluster, maintain the global fingerprint index, and provide physical address query services to the routing and computing control module. The routing and computation control module is used to connect to the topology and index management module, receive large model inference requests and distribute the large model inference requests to target decoding nodes based on routing affinity scores, and drive pre-filled nodes to perform incremental computation. The hierarchical storage management module is used to receive the calculation results generated by the routing and calculation control module, and write the data stream to the target decoding node's video memory, RAM, or solid-state drive according to the calculated global heat score. The role-adaptive scheduling module is used to collect cluster load characteristics and calculate the urgency of resource reconfiguration. When the conditions are met, it sends a node role switching instruction to the topology and index management module and triggers data preloading. The pipeline optimization module is used to coordinate with the network transmission engine to adjust the data transmission timing and call the interface of the hierarchical storage management module to perform dynamic replacement of video memory data.

10. A computer device, comprising a processor, a communication interface, a memory, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the large-model distributed inference acceleration method based on PD storage-computation separation as described in any one of claims 1-8.

Images