Heterogeneous storage-aware weight loading method and device for large language model reasoning and medium

Abstract

The application discloses a heterogeneous storage-aware weight loading method for large language model reasoning, equipment and medium, relates to the field of large language model reasoning, and comprises the following steps: detecting the local non-uniform memory access topology and the peripheral component interconnect standard device affinity relationship; based on the available state of each bucket in each storage source, the effective bandwidth of each storage source, the total bandwidth budget constraint of the network card, the local memory budget constraint and the source instance interference constraint, each bucket is allocated with a corresponding storage source by taking the bandwidth efficiency as the target; an independent pulling thread is created for each allocated bucket storage source; and each pulling thread pulls the weight tensor data in the bucket to the local graphics processor video memory in parallel. The application can avoid the bandwidth decline and delay increase caused by cross non-uniform memory access node transmission, and guarantee the efficiency of data transmission.

Description

Technical Field

[0001] This application relates to the field of large language model reasoning technology, and in particular to a heterogeneous storage-aware weight loading method, device and medium for large language model reasoning. Background Technology

[0002] Rapid startup of large language model inference services is a core capability of cloud-based Model-as-a-Service platforms. As the scale of model parameters continues to grow (the weight of mainstream models has reached hundreds of gigabytes or even more than one terabyte), the cold start time of inference services has become a major bottleneck, severely restricting critical business scenarios such as elastic scaling, model switching, and fault recovery.

[0003] In related technologies, weight loading typically adopts the following schemes: one is a sequential loading scheme based on disk or remote storage, which reads weight files one by one from the local solid-state drive or remote storage cluster, and copies them to the graphics processing unit (GPU) memory after being transferred through the host memory; the other is a weight update and cloning scheme based on parameter server, which distributes the weight data held by the central processing unit through a three-stage pipeline via broadcast or point-to-point mode.

[0004] However, the above solutions only support weight loading from a single storage source and a single transmission path. They cannot adapt to the various heterogeneous weight storage sources that actually exist in large model inference clusters, such as remote storage clusters, local host memory, distributed memory pools, and GPU memory of running instances. If a single transmission path is still used for sequential scheduling and loading of multiple heterogeneous sources, it will result in excessively long weight loading time. Summary of the Invention

[0005] This application provides a heterogeneous storage-aware weight loading method, device, and medium for large language model inference, to solve the problem of excessively long weight loading time from multiple heterogeneous storage sources during large language model inference services.

[0006] Firstly, this application provides a heterogeneous storage-aware weight loading method for large language model reasoning, the method comprising: Obtain the non-consistent memory access topology of the local machine, and determine the local machine based on the non-consistent memory access topology. The affinity between the graphics processors and network cards of the computer; The availability status of each bucket in each storage source is detected, and the corresponding storage source is allocated to each bucket based on the effective bandwidth of each storage source, the total bandwidth budget constraint of the network card, the local memory budget constraint, and the source instance interference constraint, with bandwidth efficiency as the priority. Here, a bucket is an independent unit divided by all weight tensors of the target large language model according to a preset bucketing rule, and the effective bandwidth of the storage source is determined by the affinity of the transmission path from the storage source to the image processor's video memory in the non-consistent memory access topology. Create a separate pull thread for each type of allocated bucket's storage source, and each pull thread pulls the weight tensor data in the bucket to the local graphics processor's video memory in parallel.

[0007] In a second aspect, this application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it performs the heterogeneous storage-aware weight loading method for large language model inference provided in the first aspect.

[0008] Thirdly, this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the heterogeneous storage-aware weight loading method for large language model inference provided in the first aspect.

[0009] The heterogeneous storage-aware weight loading method, device, and medium for large language model inference provided in this application have the following beneficial effects: By superimposing the parallel bandwidth of different hardware transmission paths, multi-source parallel loading across heterogeneous hardware paths is achieved, effectively improving overall bandwidth utilization and significantly shortening weight loading time. Simultaneously, before loading begins, the affinity between the local non-consistent memory access topology and peripheral component interconnection standard devices is detected. Based on the detection results, the optimally compatible network card and memory region are selected for each graphics processor to initiate data transmission, thereby avoiding bandwidth degradation and increased latency caused by transmission across non-consistent memory access nodes, ensuring efficient data transmission. Attached Figure Description

[0010] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0011] Figure 1 This is a system architecture diagram related to the embodiments of this application; Figure 2 A flowchart illustrating a heterogeneous storage-aware weight loading method for large language model inference provided in an embodiment of this application; Figure 3Another flowchart illustrating the heterogeneous storage-aware weight loading method for large language model inference provided in this application embodiment; Figure 4 A multi-source parallel pipeline timing diagram of the heterogeneous storage-aware weight loading method for large language model inference provided in the embodiments of this application; Figure 5 A flowchart illustrating a weight loading runtime control method based on real-time network card utilization feedback provided in this application embodiment; Figure 6 A schematic diagram of the control strategy involved in the weight loading runtime control method based on real-time network card utilization feedback provided in the embodiments of this application; Figure 7 A flowchart illustrating the application-layer bucket-level local host memory weight cache management method provided in this application embodiment; Figure 8 A schematic diagram illustrating the management strategies involved in the application-layer bucket-level local host memory weight cache management method provided in this application embodiment; Figure 9 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0012] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0013] The following explains the technical terms that appear in this application.

[0014] Large Language Model Inference Service: This refers to a service system that, after a Large Language Model (LLM) has been trained, deploys the model and provides efficient and stable prediction capabilities to the outside world. Users can input questions or instructions through an interface (such as an API), and the model generates output results such as text, answers, and summaries accordingly.

[0015] Weight tensor: A multidimensional array used in deep learning to represent the parameters (i.e., weights) of a neural network. For example, the weight tensor of a fully connected layer is in the form of [number of input features, number of output features], while the weight tensor of a convolutional layer is usually in the form of [number of output channels, number of input channels, kernel height, kernel width].

[0016] Model weights: refer to the parameter tensors of a large language model.

[0017] Bucket: In this application, it refers to the basic scheduling unit / transmission and buffering unit after the weight tensor of a large language model is divided at a fixed granularity. In this application, it is also referred to as "bucket".

[0018] Local host memory cache: refers to the model weight data cached in the local host memory of the inference node, and will be referred to as "local cache" in the following text.

[0019] Multi-source heterogeneity: The "multi-source heterogeneity" in this application has two levels of meaning: The first level is heterogeneity of storage media types (core feature), that is, multiple storage sources use completely different hardware transmission paths and protocols. For example, local host memory is transmitted to video memory via the peripheral component interconnect standard bus (PCI-Express, PCIe) without passing through the network card; distributed memory pools and remote storage clusters are transmitted via remote direct memory access (RDMA) network cards; and running instance video memory is transmitted across network cards via GPU Direct RDMA (GDR). These four paths use different hardware resources, among which the local PCIe path and the network card path are physically independent and can work in true parallel. The second level is parallelization of read locations (auxiliary feature), that is, reading data from multiple network nodes simultaneously. General distributed caching systems (such as multi-copy parallel reading of distributed file systems) only involve the second level—parallel reading of the same type of data blocks from multiple data nodes, but all reads are through the same type of network path, without involving heterogeneous parallelism of different hardware transmission paths, nor bandwidth contention management of inference communication traffic.

[0020] During the initialization phase, large language model inference services require loading massive amounts of model weights (i.e., model parameter tensor data, hundreds of gigabytes or even more than one terabyte) into the graphics processor's memory. In a normal model deployment process, the only authoritative source of model weights is a remote storage cluster (usually a centralized service based on object storage or a distributed file system), and all inference nodes obtain the complete weights from this single source. However, the remote loading of weights in the hundreds of gigabytes range is too time-consuming (usually taking tens of seconds to several minutes), severely restricting critical business scenarios such as elastic scaling and fault recovery. In engineering practice, a multi-level preheating and caching acceleration system has gradually evolved: After the initial load, the operating system caches the read weight file in the node's local host memory to form a local cache, namely the operating system page cache. During subsequent restarts, it can be directly transferred from local memory to video memory via the PCIe peripheral interconnect standard bus, which is much faster than remote storage. To further improve availability and loading speed, weights can also be pre-imported into a distributed memory pool composed of idle cluster memory and pulled to video memory via remote direct memory access network. In addition, when there are already running instances of the same model in the cluster, their video memory already contains complete weights, which can also serve as a high-speed data source. The combined effect of these optimization methods means that weight fragments of the same model objectively exist simultaneously in multiple heterogeneous storage sources such as remote storage clusters, local host memory, distributed memory pools, and video memory of running instances in actual production clusters, and the availability of each fragment changes dynamically over time.

[0021] However, current technical solutions lack the ability to make globally optimal loading decisions among these heterogeneous storage sources with varying bandwidths, dynamically changing availability, and inconsistent reliability. Specifically, this problem can be analyzed from three levels: resource constraints, management mechanisms, and scheduling decisions. (1) Resource constraint layer (heterogeneous storage sources have limited bandwidth and there is competition): Remote storage cluster throughput is limited and highly volatile: In production environments, remote storage clusters are typically shared across the entire data center as a centralized service. During peak business periods, despite the extremely high bandwidth of the physical links (such as high-speed remote direct memory access networks), the actual read throughput is often significantly volatile and far below the nominal bandwidth of the network interface card (NIC) due to contention among multiple tenants on the storage service side and gateway bottlenecks. The inference service side lacks direct optimization methods to improve the performance bottlenecks on the storage service side.

[0022] The remote data transmission path via the network interface card (NIC) lacks fine-grained bandwidth control: distributed memory pool loading, running instance memory transfer, and remote storage cluster access all require transmission through the remote direct memory access NIC. This NIC also carries cross-machine communication traffic for inference services (such as full-to-full communication in Expert Parallel (EP) deployments), with both types of traffic sharing the NIC's physical link. While existing hardware-level isolation solutions (such as InfiniBand virtual channels and traffic classification) can provide some bandwidth allocation, research indicates that implicit interference still exists at the NIC microarchitecture level (such as shared resources like NIC internal caches and arbitration logic). Furthermore, these solutions rely on specific hardware configurations, resulting in high deployment barriers. More importantly, the application layer lacks fine-grained proactive bandwidth control capabilities for weight loading scenarios, making it impossible to dynamically adjust loading traffic based on the real-time load of the inference service.

[0023] (2) Management mechanism layer (a semantic gap exists in local resource management): Uncontrolled eviction of local host memory cache: To accelerate repeated loading, weight files are typically pre-warmed into local host memory (operating system page cache) in production environments. However, this presents two serious problems: First, it consumes host memory space, competing with the memory required for inference runtime, such as key-value cache (KVCache); second, the operating system's page cache eviction policy is not controlled by the application layer. When memory is insufficient, it may blindly evict some weight pages, and the application layer cannot know which parts have been evicted. Once partial loss occurs, the entire file must be re-warmed, which is extremely costly. The essence of this problem lies in the fundamental semantic gap between the management semantics of the operating system's page cache and the weight loading requirements of the application layer.

[0024] (3) Scheduling decision-making layer (lacking global optimal scheduling capability): Direct inter-instance transfer has limitations and lacks coordination with other sources: While directly fetching weights from the GPU memory of an existing running instance is fast, there are two different implementation paths: one is based on a collection communication library, which requires triggering the GPU's compute core in the source instance, directly interfering with the inference service running on the source instance; the other is based on a lightweight remote direct memory access transfer engine, achieving zero-copy transfer of GPU memory through pure remote direct memory access, without triggering the GPU's compute core in the source instance, thus avoiding computational interference. However, regardless of the implementation path, the data flow still needs to occupy the network card's transmission bandwidth of the source instance, and without coordinated scheduling, it will compete for network resources with the source instance's own inference communication; in addition, both solutions only focus on a single transmission source and lack the ability to coordinate and schedule with other storage sources.

[0025] The lack of global collaborative scheduling capabilities among multiple storage sources (a core pain point): While multiple storage sources coexist in the same cluster, existing loading frameworks typically follow a linear logic, statically assigning only a single primary storage source for sequential reads. Because they cannot comprehensively consider the real-time availability, physical topology affinity, and network overhead of each storage source for globally optimal bucket allocation and multi-path concurrent loading, the parallel bandwidth of each level of cache is severely wasted. Furthermore, when a single storage source fails or degrades, there is a lack of automatic switching to other available sources to continue loading.

[0026] While parallel data retrieval from multiple data sources has been practiced in the field of distributed storage (such as multi-replica reading in distributed file systems), directly applying this approach to weight loading in large language model inference services faces the following unique technical obstacles that do not exist in general distributed storage scenarios: (1) Heterogeneity of storage source hardware paths. Multiple data sources in a general distributed storage system are homogeneous—all are network nodes accessed through the same type of network interface, with consistent transmission paths and protocols. However, the four storage sources addressed in this application use completely different hardware transmission paths: local host memory is transmitted to video memory via the peripheral component interconnect standard bus without passing through the network card; distributed memory pools and remote storage clusters are transmitted via remote direct memory access network cards; and the video memory of running instances is transmitted across network cards via direct remote memory access of the graphics processor. The bandwidth characteristics of different paths can vary by orders of magnitude (e.g., the bandwidth of the local PCIe path can reach 20 to 35 gigabytes per second, while the measured usable bandwidth of the remote storage cluster may only be a few gigabytes per second). Simple multi-threaded parallel fetching cannot handle such cross-path bandwidth difference scheduling.

[0027] (2) Competition Constraints for Shared Network Interface Card (NIC) Resources. The three remote sources—remote storage clusters, distributed memory pools, and running instance memory—all require transmission through the remote direct memory access NIC, sharing the same physical NIC bandwidth. More importantly, this NIC simultaneously carries the business communication traffic of the inference service (such as all-to-all communication in expert parallel deployments), a constraint unique to large language model inference scenarios. Simple multi-threaded parallel fetching can lead to NIC overload, interfering with cross-machine communication of the inference service—a competitive relationship of "loading traffic and business communication traffic sharing the NIC" does not exist in general distributed storage scenarios.

[0028] (3) Dynamic heterogeneity of storage source availability. The availability of each storage source changes dynamically over time and the mechanisms of change are different: the local host memory cache is affected by the operating system's page cache eviction policy and may be partially evicted at any time, and the application layer cannot perceive which buckets have been evicted; running instances may go online or offline at any time due to the elastic scaling of the inference service; distributed memory pool nodes may exit due to failure. This kind of bucket-level granularity of availability dynamic change requires the system to accurately detect the availability status of each bucket in each source before each load and to respond to source failures in real time during the loading process, which cannot be covered by the simple block redundancy and retry mechanism in general distributed storage.

[0029] (4) Significant impact of hardware topology on transmission performance. In a typical multi-GPU inference server, Non-Uniform Memory Access (NUMA) topology leads to significant differences in transmission performance between different GPUs and network cards—the transmission bandwidth within the same NUMA node can be 2 to 3 times that of cross-node transmission. Simple multi-threaded parallel fetching cannot utilize topology affinity to select the optimal transmission path, resulting in significant performance loss when cross-node access is frequent.

[0030] In a typical large-model inference cluster, there are actually several heterogeneous weight storage sources: (1) Remote storage cluster. This is usually a centralized storage service based on object storage or a distributed file system, which stores a complete master copy of the model weights. Inference nodes access the network via remote direct memory access (DMI) network cards, but the actual available throughput is limited by the multi-tenant concurrency of the storage service and the performance of the gateway, and is often far lower than the nominal bandwidth of the network card.

[0031] (2) Local host memory (operating system page cache). After the inference node loads weights from remote storage for the first time, the operating system caches the read file data in the host memory's page cache. When the inference service is restarted subsequently, if the page cache is hit, the data can be directly transferred from the host memory to the graphics processor's video memory via the peripheral component interconnect standard bus, which is much faster than remote storage. This path does not go through the network card, does not consume network card bandwidth, is physically independent of the remote path, and can work in parallel.

[0032] (3) Distributed Memory Pool. This utilizes the idle host memory of multiple nodes in the cluster, forming a distributed key-value storage service via a remote direct memory access network. Model weights enter the memory pool through two paths: one is proactive preloading, where the system operator imports frequently used model weights into the memory pool in advance during the deployment phase; the other is passive write-back, where after the inference node completes the weight loading from remote storage for the first time, it asynchronously writes the loading result back to the distributed memory pool, similar to a write-through caching strategy, to accelerate subsequent loading tasks of the same model. During loading, the data is pulled to the graphics processor's video memory via remote direct memory access. This solution provides higher throughput than a remote storage cluster, but loading traffic needs to be transmitted through the remote direct memory access network card.

[0033] (4) Graphics processor memory of running instances in the cluster. When other inference instances in the cluster have already loaded the same model, newly launched instances can directly pull weights from the memory of existing instances to their local memory through Graphics Processor Direct Remote Memory Access (GDR) technology, realizing cross-machine direct transfer between graphics processors.

[0034] In this application, local host memory is classified as "local storage source", while storage sources that do not require transmission via network card, such as remote storage clusters, distributed memory pools, and graphics processor memory of running instances in the cluster, are classified as "remote storage source".

[0035] The typical performance characteristics of the four storage sources mentioned above are compared as follows: Storage source Typical bandwidth range Delay characteristics reliability Does it occupy the network card? Main bottlenecks Remote storage cluster Hundreds of megabytes per second to several gigabytes per second High (on the order of tens of milliseconds) High (multiple replica redundancy) yes Storage service client performance Local host memory Ten to thirty gigabytes per second (PCIe) Low (below microseconds) (Controlled by the operating system) no Memory capacity, eviction policy Distributed memory pool Five to ten gigabytes per second (RDMA) Medium (several microseconds to tens of microseconds) (Node failure risk) yes Network bandwidth, memory pool capacity Runnable instance memory Ten to fifty gigabytes per second (GDR) Low (microseconds) Low (depends on instance lifecycle) yes Source instance load balancing, network interface contention In a typical eight-GPU inference server, the server usually contains two Non-Uniform Memory Access (NUMA) nodes, each equipped with four graphics processors and four remote direct memory access network interface cards (NICs), interconnected via a peripheral component interconnect standard switching chip. Hardware topology has a significant impact on data transmission performance.

[0036] Currently, there are four main technical solutions for loading weights in large language models. Each solution has its applicable scenarios, but also has corresponding limitations, as follows: Option 1: Sequential loading scheme based on disk or remote storage.

[0037] This approach reads weight files one by one from the local SSD or remote storage cluster when the inference engine starts, copies them to the graphics processor's memory after passing through the host memory. This approach is a common baseline implementation for various mainstream inference frameworks. However, due to the limited read bandwidth of a single data source, loading a large model of hundreds of gigabytes typically takes tens of seconds to several minutes.

[0038] Option 2: Weight update and cloning scheme based on parameter server.

[0039] This scheme is applied to scenarios involving the collaborative deployment of reinforcement learning and inference. After training, the updated weights need to be distributed to the inference instances. The parameter server and inference instances reside in the same cluster, holding the weight data in the central processing unit's memory, and the transmission is completed through the following two modes: Broadcast Mode: Suitable for synchronous hot updates of all inference instances. Weights are organized into buckets and transferred via a three-stage pipeline: ① Host-to-Device (H2D) Transfer: Buckets are loaded from CPU memory into GPU memory; ② Broadcast: Broadcasts to all parameter server worker processes via a collection communication library, with results written to a shared inter-process communication GPU memory buffer; ③ Reload: Each inference engine reads from the shared buffer and applies the weights. This three-stage overlapping execution maximizes transfer throughput.

[0040] Point-to-point mode: Suitable for dynamic cluster expansion, where new inference instances clone weights from existing instances. A lightweight remote direct memory access (RDA) transfer engine directly writes the weights from the existing instance's CPU memory to the new instance's GPU memory via the RDA network, without triggering the existing instance's GPU cores. This mode optimizes sender-receiver pair allocation at the bucket level, maximizing network bandwidth utilization for each path while minimizing total transmission time.

[0041] Both of the above modes are designed for hot update scenarios of weights. They do not involve the scheduling problem of parallel loading from multiple heterogeneous storage sources such as local cache and distributed memory pool when the inference service is cold started, nor do they involve the control of the total network card bandwidth budget between loading traffic and inference cross-machine communication traffic.

[0042] Option 3: Accelerated loading solution based on distributed memory pool.

[0043] The scheme pre-imports model weights into a distributed key-value storage service consisting of idle host memory in the cluster. When the inference instance is cold-started, the weights are pulled from the memory pool to the graphics processor memory via remote direct memory access. The scheme has the following characteristics and limitations in terms of technical architecture: (1) It only involves the transmission optimization from a single storage level (distributed memory pool) to the graphics processor memory, and does not involve the heterogeneous parallel collaboration between the local peripheral component interconnect standard bus path and the remote direct memory access network card path - all loading traffic is transmitted through the network card, and it is impossible to use the free peripheral component interconnect standard bus bandwidth of the local host memory cache for path offloading; (2) It lacks the application layer network card bandwidth active management capability. The loading traffic and the cross-machine communication traffic of the inference service (such as expert parallel all-to-all communication) have no active mitigation mechanism at the network card level, resulting in uncontrollable bandwidth contention in the multi-tenant cluster; (3) It does not involve the application layer active management of the local host memory cache, and it is impossible to achieve bucket-level accurate page fault detection and on-demand reloading. It does not have the ability to avoid full reloading when the cache part fails.

[0044] Option 4: Local multi-level storage optimization loading scheme based on direct input and output.

[0045] This solution targets serverless inference scenarios, employing direct input / output to bypass the operating system's page cache. It allocates a fixed memory region for weight data at the application layer and constructs a multi-layered storage pipeline between host memory and SSDs to accelerate loading. Designed for scenarios where the old model is completely unloaded during model switching, this solution does not involve globally optimal allocation and scheduling across heterogeneous storage sources, network interface card bandwidth management, or runtime coordination between online inference services and loading traffic.

[0046] The four existing technical solutions mentioned above each have their own focus in solving the problem of loading weights in large language models, but all of them have systemic shortcomings.

[0047] It is worth noting that in scenarios with multiple storage sources coexisting, parallel loading of multiple sources is not a sequential strategy of "utilizing each source in order of speed priority," but rather an inevitable choice based on the following three key physical constraints: (1) The data transmission paths are physically independent and can truly run in parallel: the transmission from the local host memory to the graphics processor is carried out through the peripheral component interconnect standard bus, and the transmission of remote data to the graphics processor is carried out through the remote direct memory access network card. These two paths use completely different hardware resources and run simultaneously without interfering with each other. If the sequential strategy is changed (the local cache is used up first, and then the remote path is switched), the peripheral component interconnect standard bus path will be completely idle during the remote loading phase, resulting in a significant waste of hardware resources.

[0048] (2) Each storage source has capacity or dependency constraints, and the sequential strategy cannot make full use of path complementarity: The local host memory (PCIe path) is limited by capacity and eviction, and the hit rate is usually between 30% and 80%, which can only cover a part of the model weights. This part of the data is transmitted through PCIe without occupying network card bandwidth, which is "free" local throughput. Buckets that miss the local cache must go through the remote path (network card) in any case. Since the remote path is already transmitting, the additional cost of making the local path work at the same time is zero. Although remote storage and P2P instances can hold the full weights, the former is constrained by multi-tenant bandwidth competition, and the latter depends on the existence of existing instances of the same model in the cluster. Both occupy network card bandwidth and compete with inference communication traffic. Therefore, the sequential strategy (using one path first and then switching to the other) wastes the idle capacity of the PCIe path in any scenario, while the value of the parallel strategy lies in path complementarity: the local PCIe path and the remote network card path use completely different physical resources and work simultaneously without interfering with each other.

[0049] (3) Capacity constraints dictate that multi-source parallelism is a necessary rather than optional optimization: Taking a 440-gigabyte hybrid expert model as an example, if the local host memory cache capacity is limited to 150 gigabytes, then regardless of speed priority, the remaining 290 gigabytes must be loaded from the remote path. In this case, both paths must work simultaneously to minimize the total loading time, and no sequential strategy can achieve the same loading speed.

[0050] Based on the above three points, we can conclude that the core value of multi-source parallel loading lies in simultaneously utilizing the full available bandwidth of both physically independent local and remote transmission paths to minimize the model weight loading time, thereby solving the core problem of excessively long cold start time for large language model inference services.

[0051] It should be noted that, in this application, "local fetch path" refers to the transmission path that needs to be taken from the local storage source to the local image processor's video memory, and "remote fetch path" refers to the transmission path that needs to be taken from the remote storage source to the local image processor's video memory.

[0052] Based on this, this application proposes a heterogeneous storage-aware weight loading method for large language model inference. This method conducts bucket-level availability detection for multiple heterogeneous storage sources using different hardware transmission paths. Combining hardware topology affinity information and multi-dimensional resource constraints (including at least network card total bandwidth budget constraints and local memory budget constraints), it generates a bucket-source allocation scheme that minimizes the total loading time, thereby significantly improving weight loading efficiency. This method is particularly suitable for cold start scenarios of large language model inference services.

[0053] Furthermore, to address the dynamic uncertainty of storage sources in cold start scenarios, this application also provides an application-layer bucket-level local host memory weighted cache management method to implement application-layer proactive management of local host memory cache in order to achieve accurate page fault loading.

[0054] Furthermore, to address the issue that network interface card (NIC) contention between loading traffic and inference communication traffic in cold start scenarios cannot be avoided through time isolation, this application proposes a weighted loading runtime control method based on real-time NIC utilization feedback.

[0055] It should be noted that the various weight loading schemes in the aforementioned existing technologies are all designed for hot weight update scenarios and cannot be directly adapted to the loading requirements of cold start scenarios. The reason is that the cold start scenario of large language model inference services faces the following unique technical obstacles compared to the hot update scenario: (1) The storage source has dynamic uncertainty. During the cold start process, the availability status of each heterogeneous storage source has significant dynamic uncertainty: the weight data in the local host memory cache may be partially evicted by the operating system without rules, and it is impossible to predict in advance which buckets will be evicted. It is necessary to detect and confirm bucket by bucket during runtime; the distributed memory pool may experience partial data unavailability due to node failure, operation and maintenance, etc.; the running inference instances of the same model in the cluster may not exist (such as the first instance cold start scenario), or even if they exist, they are in a high inference load state. This dynamic uncertainty of the availability of multiple storage sources does not exist in the hot update scenario - in the hot update scenario, the central processing unit memory of the parameter server always holds a complete copy of the model weights, and the availability of the data source has clear certainty. (2) Lack of a global multi-path coordination mechanism. In the existing technology, the point-to-point mode of Scheme 2 only optimizes the design for a single transmission path (direct transmission between CPU memory and GPU memory) and does not involve parallel collaborative scheduling across heterogeneous hardware paths. That is, it cannot achieve collaborative scheduling between the local peripheral component interconnection standard bus path and the remote direct memory access network card path. In the cold start scenario, it is necessary to complete the global optimal allocation of weight data between the above two physically independent transmission paths. At the same time, it is also necessary to deal with the constraint problem of the shared network card bandwidth of the three remote storage sources: remote storage cluster, distributed memory pool, and running instance memory. The core requirements and technical problems of this kind of multi-path coordination do not exist in the hot update scenario involving only a single path. (3) The characteristics of inference communication traffic competition are different. The weight loading operation in the hot update scenario can usually be executed within the controllable time window of the inference service, such as pausing the reception of new inference requests and performing batch updates during the interval of inference request processing. This can achieve partial isolation of the time window of weight loading traffic and inference communication traffic, and greatly reduce the network card bandwidth competition between the two. In a cold start scenario, the server where the node performing weight loading resides may have other running inference instances deployed. These instances will generate continuous cross-machine communication traffic. In the mainstream deployment mode of multi-instance shared servers, the network interface bandwidth competition between weight loading traffic and inference communication traffic cannot be avoided by time isolation. It is necessary to rely on proactive bandwidth management mechanisms for targeted coordination and control.

[0056] The specific embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0057] The method provided in the embodiments of this application is based on, as Figure 1The system architecture shown is implemented as follows. This system includes at least a resource detection module, a weight allocation decision module, and a multi-source parallel loading engine. In some preferred embodiments, a runtime control module and a local cache management module can be further added according to actual application requirements.

[0058] The core execution functions of each module are as follows (it should be noted that the functions implemented by each module are not limited to those described below): Resource detection module: Responsible for detecting the availability, bandwidth and health status of each storage source, detecting the hardware topology information of the local machine (including non-consistent memory access nodes, the affinity between peripheral component interconnect standard switching chips and graphics processors / network cards), and collecting the network card bandwidth usage of the inference service.

[0059] Weight allocation decision module: Based on the detection results output by the resource detection module and the hardware topology information, it executes the globally optimal weight bucketing and storage source allocation algorithm to generate the corresponding weight loading and allocation scheme.

[0060] Multi-source parallel loading engine: Based on the allocation scheme generated by the weight allocation decision module, it pulls weight data from multiple heterogeneous storage sources in parallel, and distributes the weight data to all graphics processors after aggregation and broadcasting.

[0061] Runtime control module: Monitors the traffic status and network card utilization of the inference service in real time, dynamically adjusts the rate limit threshold for remote loading traffic, and performs automatic degradation and storage source switching operations when a storage source failure is detected.

[0062] Local cache management module: Maintains a bucket-level cache index table of weighted cache in local host memory at the application layer. This index table can be refined to the cache status of each bucket, thereby replacing the blind management mechanism of the operating system's page cache and realizing accurate page fault detection and on-demand reloading of weighted cache.

[0063] In the first implementation scheme, the resource detection module, weight allocation decision module, and multi-source parallel loading engine can form a complete data flow and control flow closed loop. Their collaborative workflow is as follows: Before weight loading starts, the resource detection module completes the collection of the status of each storage source and the local hardware topology information, and synchronously transmits the collected detection results to the weight allocation decision module; the weight allocation decision module generates the globally optimal weight loading allocation scheme based on the detection results, and then sends the scheme to the multi-source parallel loading engine for execution; the multi-source parallel loading engine completes the weight retrieval and distribution from the multi-source heterogeneous storage sources according to the allocation scheme, thereby realizing weight loading under multi-source heterogeneous storage sources and effectively solving the technical problem of excessive weight loading time in multi-source heterogeneous storage source scenarios.

[0064] In the second implementation scheme, a more comprehensive closed-loop control system is further constructed by integrating a runtime control module, building upon the first scheme. Specifically, during the weighted loading operation, the multi-source parallel loading engine continuously feeds back real-time information such as actual data throughput and bucket loading completion status to the runtime control module. The runtime control module dynamically adjusts the remote loading traffic rate limiting threshold of the multi-source parallel loading engine based on the real-time network card utilization of the inference service and the health status of each storage source, and adaptively optimizes the storage source allocation strategy, thus forming a complete closed-loop control loop of "probe-decision-execution-feedback-adjustment". Based on this closed-loop control, this application can implement fine-grained token bucket rate limiting at the application layer based on real-time network card utilization. This method does not rely on specific hardware configurations, can dynamically adjust loading traffic according to inference load, effectively ensures the normal operation of the inference service, and achieves zero interference of weighted loading on the inference service.

[0065] In the third implementation scheme, based on the first or second implementation scheme, a local cache management module can be further integrated. This module can be deeply integrated into the entire weight loading process and work collaboratively with other modules: before weight loading starts, the local cache management module provides the weight allocation decision module with the accurate cache hit status of each bucket, providing reliable data support for weight bucketing and storage source allocation decisions; after the weight loading operation is completed, the local cache management module updates the weight cache index in the local host memory in a timely manner, providing accelerated support for subsequent weight loading tasks of the same model. Through the above methods, the local cache management module can achieve accurate page fault detection and on-demand reloading of the weight cache, completely avoiding the full re-warm-up operation when the local cache partially fails, and significantly reducing the time and resource costs of repeated weight loading.

[0066] In other implementations, to specifically address the technical issues of multi-tenant bandwidth contention constraints and the competition for network card resources between loading traffic and inference communication traffic, during the weight loading process of the large language model inference service, the runtime control module can monitor the traffic status and network card utilization of the inference service in real time, dynamically adjust the rate limiting threshold of remote loading traffic, and perform automatic degradation and storage source switching operations when a storage source failure is detected. This application names the method for implementing this process "weight loading runtime control method based on real-time network card utilization feedback", which will be described in detail below.

[0067] In other implementation schemes, to specifically address the technical issue of needing to fully reload weights when a portion of the local cache becomes invalid, an application-layer bucket-level local host memory weight cache management method can be implemented based on the local cache management module. This method will be described in detail below.

[0068] The following is combined Figure 2-4This application provides a detailed description of the heterogeneous storage-aware weight loading method for reasoning with large language models.

[0069] The heterogeneous storage-aware weight loading method for large language model inference provided in this application is applied to any computer node (hereinafter referred to as "inference node") in a distributed inference cluster. The inference node can be an electronic device with computer processing capabilities, such as a server, rack server, or edge computing node, or a computing device such as a laptop computer, to adapt to deployment scenarios of large language model inference clusters of different sizes.

[0070] like Figure 2 As shown, this heterogeneous storage-aware weight loading method for large language model inference includes the following steps: A10, obtain the non-consistent memory access topology of the local machine, and determine the non-consistent memory access topology. Determine the affinity between the graphics processors and network cards of this machine.

[0071] A20 detects the availability status of each bucket in each storage source, and combines the effective bandwidth of each storage source, the total bandwidth budget constraint of the network card, the local memory budget constraint, and the source instance interference constraint to allocate the corresponding storage source to each bucket with bandwidth efficiency as the priority. Among them, a bucket is an independent unit divided by all weight tensors of the target large language model according to a preset bucketing rule. The effective bandwidth of the storage source is determined by combining the affinity of the transmission path from the storage source to the image processor's video memory in the non-consistent memory access topology. The A30 creates an independent pull thread for each type of allocated bucket's storage source, and each pull thread pulls the weight tensor data in the bucket to the local graphics processor's video memory in parallel.

[0072] Specifically, please see Figure 3 The provided flowchart illustrates the weighted bucketing and storage allocation algorithm involved in this application. An inference node receives a model loading request, which includes a target model identifier. The inference node then determines the target large language model to be loaded (hereinafter referred to as the "target large model") based on this target model identifier and obtains the configuration information associated with the target large model from the cluster configuration center or model repository, such as the weight metadata of the target large model.

[0073] Among them, weight metadata is data used to describe the attribute information of weight tensors, such as the name and size of each weight tensor.

[0074] Therefore, inference nodes can bucket all weight tensors of the target large model based on weight metadata, grouping each weight tensor into a bucket. Each bucket contains multiple weight tensors. The size of each bucket is adapted to the requirements of the graphics processing unit (GPU) memory buffer budget and the target pipeline depth, and meets the registered memory unit and minimum alignment specifications for Remote Direct Memory Access (RDMA) / Graphics Processor Direct Memory Access (GDR) transfers.

[0075] In this application, the processing logic for bucketing the weight tensor is as follows: All weight tensors of the target large model are arranged in alphabetical order by name, and a greedy binning strategy is used to merge adjacent weight tensors into one bucket.

[0076] The weight tensors within each bucket are aligned to a preset granularity (e.g., 256 bytes). All weight tensors are divided into K buckets (K being the total number of buckets), forming a bucket set B = {b_1, b_2, ..., b_K}, with each bucket b_k having a size of s_k (in bytes).

[0077] The bucket size is determined using an adaptive strategy: Let the total size of the model weights be S, the budget available for loading buffers in the graphics processor memory be M_gpu_budget, and the target pipeline depth be N_target (ranging from 2 to 8). Then the maximum bucket size is bucket_size_max = M_gpu_budget / N_target, and the number of buckets is K = ⌈S / bucket_size_max⌉.

[0078] If K is too small (e.g., less than 10), the bucket size should be appropriately reduced to increase scheduling parallelism. The bucket size should also be coordinated with the registered memory units for remote direct memory access transfers and the minimum alignment requirements for graphics processor direct memory access to avoid additional copy overhead.

[0079] The bucket size should also be coordinated with the registered memory unit of the remote direct memory access transfer and the minimum alignment requirement of the graphics processor direct memory access. This means that the bucket size must be an integer multiple of the registered memory unit granularity of the remote direct memory access and the bucket size must be an integer of the minimum alignment granularity of the graphics processor direct memory access. This can be achieved through the following steps: First, the upper limit of a single slot’s capacity, bucket_size_cap, is calculated based on the budget M_gpu_budget available for the load buffer in the graphics processor’s video memory and the target pipeline depth N_target. Then, the lower limit of the bucket size is determined based on the model itself: bucket_size_base=max(1GiB,max_single_tensor_size), where max_single_tensor_size represents the size of the largest single weight tensor in the model. The purpose is to ensure that no weight tensor is forcibly split into two buckets. Next, take the common alignment granularity align_unit that simultaneously satisfies the RDMA registration granularity, GDR minimum alignment granularity, and 256B alignment requirements within the bucket, and obtain the aligned bucket size according to bucket_size=ceil(bucket_size_base / align_unit)×align_unit; if bucket_size>bucket_size_cap at this time, it means that the current N_target is too large and the single-slot memory is insufficient, and N_target should be reduced and recalculated. The final number of buckets is naturally obtained by K=ceil(S / bucket_size).

[0080] The target pipeline depth corresponds to the number of independent buffer slots pre-allocated in the graphics processor's memory for weight loading. The size of each slot is the same as the maximum bucket size, and one slot can independently carry the complete loading process of one bucket (including stages such as fetch, broadcast, and application).

[0081] In this application, the target pipeline depth is a preset baseline value during bucket initialization. The actual pipeline depth during the loading process is dynamically adjusted based on this baseline value, combined with real-time memory pressure and actual throughput. For example, it linearly decreases when there is insufficient free memory and exponentially increases when throughput does not meet expectations. However, the adjusted actual pipeline depth is still limited to a preset range (2 to 8) to ensure compatibility with the initial bucket size. How the target pipeline depth is applied to the bucket loading process will be described in detail in the following sections.

[0082] In some implementations, building affinity relationships between the local graphics processors and network interface cards based on a non-uniform memory access topology may include: Based on whether each graphics processor and each network interface card (NIC) is located on the same non-uniform memory access node and whether it needs to cross peripheral component interconnect standard switching chips, the affinity relationship between each graphics processor and each NIC is determined, and the corresponding affinity coefficient is generated.

[0083] Taking an eight-GPU inference server as an example, the server typically contains two non-uniform memory access nodes. Each node is equipped with four GPUs and four remote direct memory access network cards (NICs), interconnected via the same peripheral component interconnect standard switching chip. Transmission performance is optimal between GPUs and NICs within the same non-uniform memory access node. Transmission across non-uniform memory access nodes requires relaying through the processor interconnect bus, resulting in decreased bandwidth and increased latency. The specific detection process is as follows: (1) Enumerate all non-consistent memory access nodes on the local machine, and obtain the list of graphics processors and network cards attached to each node. (2) For each pair (graphics processor i, network card j), determine their topological affinity: Standard switching chips that interconnect with non-uniform memory access nodes and peripheral components offer optimal affinity. Mark it as "near end" and set the bandwidth coefficient λ=λ1.

[0084] For nodes with inconsistent memory access but standard switching chips for interconnecting peripheral components: second-best affinity. Mark it as "Mid-range" and set the bandwidth coefficient λ=λ2.

[0085] Cross-inconsistent memory access nodes: Poor affinity, marked as "remote", with a bandwidth factor of λ3.

[0086] (3) Output the affinity coefficient matrix Affinity[i][j]=λ The bandwidth coefficient λ reflects the ratio of the actual transmission bandwidth to the theoretical peak bandwidth at different topological locations, with λ1>λ2>λ3. As an example, λ1=1.0, λ2=0.7, and λ3=0.4.

[0087] The values ​​of the above coefficients are based on the following: In the same node and same switching chip scenario, the graphics processor and network card are located in the same peripheral component interconnection standard root complex, and data transmission does not need to pass through the processor interconnection bus, resulting in the least bandwidth loss, so λ is taken as 1.0; In the same node and cross switching chip scenario, data needs to pass through the interconnection between switching chips, resulting in certain latency and bandwidth overhead. The measured bandwidth is usually 60% to 80% of that at the near end, so the median value λ=0.7 is taken; In the cross node scenario, data needs to be transferred through the inter-processor interconnection bus (such as Hyperpath Interconnect (UPI)), resulting in the most significant bandwidth drop. The measured bandwidth is usually 30% to 50% of that at the near end, so λ=0.4 is taken.

[0088] In some implementations, during step A10, the availability status of each bucket in each storage source is first obtained. The availability status includes available and unavailable. Available indicates that the corresponding storage source holds the weight tensor data of the corresponding bucket, while unavailable indicates that the corresponding storage source does not hold the weight tensor data of the corresponding bucket.

[0089] Unlike the idealized design in existing technologies that assumes all storage sources hold the complete weights of the model, this application addresses the weight coverage differences of heterogeneous storage sources and detects the actual availability of the weight tensor data of each bucket in each storage source.

[0090] For each bucket b_k and each storage source t, the label available[k][t] indicates the availability status. The method for determining the availability status of each bucket in each storage source is as follows: For remote storage clusters (t=1): always "available" (holding full weight), i.e., available[k][1]=1.

[0091] For local host memory (t=2), query the bucket-level cache index table maintained by the local cache management module. If a record for bucket b_k exists in the index table, and the actual checksum of the bucket matches the checksum in the weight metadata, then mark available[k][2]=1; otherwise, mark available[k][2]=0. This part will be described in detail below. Of course, in other implementations, a caching scheme based on a user-space file system can be adopted: implement a lightweight file system in user space, store weight data as a memory-mapped file managed by user space, and bypass the kernel page cache mechanism. Alternatively, a cross-process caching scheme based on shared memory can be adopted: use the operating system's shared memory (such as POSIX shared memory objects) to store the weight cache, and multiple inference processes can share the same cache data, reducing memory consumption. Alternatively, a caching scheme based on persistent memory can be adopted: use non-volatile memory to store the weight cache, which has the access speed close to host memory and persistence characteristics, and the cache is not lost after restart. This application does not specifically limit the caching scheme.

[0092] For the distributed memory pool (t=3): Connect to the metadata service of the distributed memory pool, query the integrity verification result of bucket b_k. If the bucket has been fully cached and the verification passed, mark available[k][3]=1; otherwise mark available[k][3]=0.

[0093] For running instance memory (t=4): Query the cluster scheduler. If there is a running instance with the same parallel configuration as the target large model and the instance has an open remote memory access transmission port, mark available[k][4]=1; otherwise mark available[k][4]=0.

[0094] Then, the objective function is to minimize the sum of the loading times of all buckets (hereinafter referred to as "total loading time") T_load. In this embodiment, the inference node includes multiple weight loading units (also called "rank"), one weight loading unit corresponds to one graphics processor, and multiple weight loading units execute the fetch task in parallel. The calculation logic of the total loading time T_load is as follows: Assuming the set of weight loading units participating in parallel loading is N, the amount of data completed by the i-th weight loading unit via the local path is S_i(local), and the amount of data completed via the remote path is S_i(remote). The effective bandwidths corresponding to the local and remote pull paths are BW_i(local) and BW_i(remote), respectively. Then, the time consumed in the pull phase can be expressed as T_i(pull) = max(S_i(local) / BW_i(local), S_i(remote) / BW_i(remote)). If the loading process only considers the pull phase, then the loading time of the i-th weight loading unit is expressed as T_i = T_i(pull), and the total loading time of the entire loading group is T_load = max_(i∈N)T_i.

[0095] In other implementations, if the loading process also includes a broadcast and local application phase, the complete loading time of the i-th weighted loading unit can be expressed as T_i = T_i(pull) + T_i(bcast) + T_i(apply), where T_i(bcast) represents the time required for the i-th weighted loading unit to complete all bucket broadcasts with the entire loading group, and T_i(apply) represents the time required for the i-th weighted loading unit to complete all local applications. Finally, the maximum complete loading time of the entire loading group is taken to obtain the total loading time.

[0096] The calculation logic for S_i(remote) is as follows: First, the local available bucket sets of each weighted loading unit are aggregated to obtain the global local coverable bucket set. Then, the remaining buckets are defined as remote buckets. Next, based on the local hit rate and remote fetching capability of each weighted loading unit, the total amount of remote buckets is allocated to each weighted loading unit to obtain their respective remote fetching task volume. The specific calculation method will be explained in detail later.

[0097] In other implementations, S_i(remote) can also be obtained by averaging or other allocation algorithms based on the total amount of data in all remote buckets.

[0098] At the same time, the following constraints are established: (1) Uniqueness constraint: Each bucket is loaded from a storage source in this loading task. Specifically, define... This indicates whether the k-th bucket was loaded from storage source t, where t is the index of the storage source. t=1 represents the remote storage cluster, t=2 represents the local host memory, t=3 represents the distributed memory pool, and t=4 represents the GPU memory of the running instance. This indicates that the k-th bucket is loaded from storage source t. This indicates that the k-th bucket is not loaded from storage source t, and satisfies... .

[0099] If the amount of data allocated by storage source t before allocating the k-th bucket is D_t, the size of the k-th bucket is s_k, and the effective bandwidth of storage source t for the current bucket is BW_t, then the estimated completion time after allocating the k-th bucket to storage source t can be written as (D_t+s_k) / BW_t. Therefore, the storage source selection rule for the current bucket can be written as t*(k)=argmin_t((D_t+s_k) / BW_t). After each determination of t*(k), D_t is updated to D_t+s_k, and the next bucket is processed.

[0100] (2) Availability constraint: Buckets can only be loaded from available storage sources, i.e. .

[0101] (3) Completion time constraints for each storage source: The total amount of data allocated to each storage source shall not exceed the product of its effective bandwidth and total loading time, i.e.: y_t≤BW_t×T_load; in, .

[0102] In the above formula, y_t represents the amount of data allocated to storage source t, s_k represents the amount of data contained in the k-th bucket, BW_t is the effective bandwidth of storage source t, the calculation method of which will be explained later, and T_load is the total loading time. This constraint ensures that each storage source can complete the transmission of allocated data within the total loading time.

[0103] (4) Network interface card (NIC) total bandwidth budget constraint: The instantaneous total transmission rate of all storage sources transmitted through the NIC shall not exceed the NIC's total bandwidth budget, i.e.: y_1+y_3+y_4≤B_load×T_load; Here, B_load represents the total network interface card (NIC) bandwidth budget, the calculation method of which will be explained in detail later. y_1, y_3, and y_4 represent the amount of data allocated to the remote storage cluster, distributed memory pool, and running instance memory, respectively. This constraint transforms the non-linear rate constraint into a linear form, ensuring that the total transmission volume of the three types of storage sources (also known as "remote storage sources") transmitted via the NIC does not exceed the NIC's total throughput capacity.

[0104] (5) Local host memory budget constraint: The local cache usage shall not exceed the memory budget, and space shall be reserved for the key-value cache, that is: y_2+M_kv≤M_host; Where y_2 is the total amount of data allocated to the local host memory path, M_kv is the key-value cache reserved memory, and M_host is the total host memory capacity.

[0105] (6) Source instance interference constraint: The total bandwidth for pulling data from the memory of an already running instance shall not exceed its maximum allowable limit, i.e.: y_4≤B_src_allow×T_load Among them, B_src_allow is the maximum bandwidth that the source instance can currently use to send weight data to the outside world. Its calculation method will be explained in detail later.

[0106] (7) Graphics processor memory buffer constraints: N×bucket_size_max≤M_gpu_budget; Where N is the pipeline depth, bucket_size_max is the maximum bucket size, and M_gpu_budget is the budget in the graphics processor's memory that can be used for the load buffer.

[0107] It should be noted that the above constraints apply to the source allocation decision in a single weight loading task, rather than describing the physical storage distribution of the weight data. In actual deployments, the same bucket can exist in multiple storage sources simultaneously (e.g., both local and remote storage sources hold the complete data of the bucket), which is a normal state of redundant storage. The meaning of the above uniqueness constraint is: for each bucket that needs to be loaded, the system only pulls data from one of the optimal storage sources in this loading task to avoid redundant transmission and wasting bandwidth, rather than restricting its physical storage location.

[0108] The aforementioned constrained optimization problem involves integer decision variables and an objective function that minimizes the total loading time. It is equivalent to the classic constrained maximum completion time problem and belongs to the NP-hard problem class. In real-world production environments, the number of buckets K is typically in the tens to hundreds, and the number of storage sources is fixed at four types. Therefore, this application employs the following greedy approximation algorithm based on bandwidth efficiency to obtain an approximate optimal solution in polynomial time: / / Core algorithm idea: ListScheduling / / Maintain two estimated completion times: T_local (local PCIe path) and T_nic (network interface card path).

[0109] / / The goal is to make the two paths complete as simultaneously as possible (minimize makespan).

[0110] / / For each bucket, select the storage source that minimizes the overall estimated completion time after allocation.

[0111] / / Note: BW_t represents the actual measured effective bandwidth (the local path already includes the topology affinity coefficient λ). / / The network card path already includes the topology affinity coefficient λ, rather than the theoretical peak value.

[0112] Input: Bucket set B = {b1, ..., b_K}, effective bandwidth BW_t of each storage source, Availability matrix available[k][t], total network interface card bandwidth budget B_load Output: Allocation scheme assign:{1,...,K}→{1,2,3,4} (1) Sort the storage sources in descending order of effective bandwidth BW_t to obtain a priority list P (the higher the bandwidth, the higher the priority). (2) Traverse the bucket set in order of bucket number (to ensure the continuous loading order and facilitate pipeline scheduling); Since the buckets are designed to be nearly uniform in size, sorting them by size offers no substantial benefit. (3) Initialize the allocated data amount D_t of each storage source to 0, and the allocated total amount D_nic of the network interface card level to 0. (4) For each bucket b_k (traverse in bucket number order): (4a) Traverse each storage source t according to the priority list P: - If available[k][t]=0, skip (this bucket is not available in this source). / / For local host memory (t=2): available[k][2] encodes whether the bucket is in the cache. / / There is no additional capacity limit check—if it hits, use it; otherwise, use the remote path. / / For remote sources (t=1,3,4): The data physically exists intact; the constraint is the bandwidth budget, not the capacity. / / Bandwidth constraints have been implicitly handled through the D_nic tracing below. -If storage source t requires a network card (t∈{1,3,4}): / / T_nic_new: After adding bucket k to the network interface path, all network interface sources share the aggregation completion time of B_load. The numerator is the total amount of network interface card (NIC) paths allocated, and the denominator is the shared NIC bandwidth budget. When multiple remote sources are working simultaneously, each storage source may be fast individually, but the shared NIC may become a bottleneck. / / T_nic_new captures precisely this shared bottleneck scenario. T_nic_new=(D_nic+s_k) / B_load / / T_local: Estimated completion time of the current local PCIe path (completion time of storage sources without going through the network card) T_local=max(D_{t'} / BW_{t'}, For all sources t' that do not occupy a network card and D_{t'}>0) / / Path balancing check: If the local path has already been allocated, and bucket k is placed in the network interface path... / / The network card path completion time will significantly exceed that of the local path (the network card becomes the bottleneck). / / This skips the current network interface source and allows the local path to handle the bucket, thus balancing the completion time of the two paths. If T_local > 0 and T_nic_new > T_local × 1.2: Skipping (assigning it to a network interface card path will cause the network interface card path to lag significantly, becoming an overall bottleneck) / / 1.2×threshold is the default recommended value for "tolerance for slight imbalance": / / When the value is below 1.1, it is too sensitive to path imbalance, which leads to frequent path switching and increases scheduling overhead; / / When the value is above 1.5, the tolerance for imbalance is too high, and equilibrium may not be triggered even when there is severe backlog in the bottleneck path. / / This causes the overall completion time to deviate significantly from the theoretical optimal time.

[0113] / / This value can be adjusted according to the specific hardware configuration in actual deployment.

[0114] - Assign b_k to source t: assign[k]=t, D_t+=s_k If t∈{1,3,4}, then simultaneously update D_nic+=s_k. - Break out of the inner loop (4b) If none of the above conditions are met, allocate b_k to the remote storage cluster. (t=1, fallback source, always available) (5) Return the allocation scheme assign Algorithm Complexity Analysis: The storage source sorting time is O(4log4), i.e., constant time. The main loop traverses K buckets in bucket number order, checking at most 4 storage sources per bucket; therefore, the main loop time complexity is O(K). The total time complexity is O(K), and the space complexity is O(K) for the storage allocation scheme and availability matrix. Compared to the exponentially exact solution of integer linear programming, this greedy algorithm can complete the decision in milliseconds, meeting the real-time requirements of online scheduling.

[0115] Actual decision time estimation for different bucket numbers: The algorithm's decision process for each bucket involves only integer comparisons (availability checks, threshold comparisons) and integer addition (accumulating allocated data), without floating-point operations or dynamic memory allocation. On modern processors, a single bucket allocation decision takes approximately tens of nanoseconds. The following are typical decision time estimates for different scales: Number of buckets K Typical scenarios Approximate number of basic operations Estimated total decision time 100 Medium-sized model (tens of gigabytes), coarse-grained bucketing 400 times (100 barrels x 4 sources) tens of microseconds 1000 Extremely large models (hundreds of gigabytes), fine-grained bucketing 4000 times Hundreds of microseconds 5000 Extreme scenarios (terabyte-scale model or extremely fine-grained binning) 20,000 times Millisecond level (approximately 1 to 2 milliseconds) Even in the extreme scenario of K=5000, the total decision time (about 1 to 2 milliseconds) is still much less than the minimum transmission time of a single bucket (calculated with a minimum bucket size of about 100 megabytes and a maximum transmission bandwidth of about 50 gigabytes per second, the transmission time of a single bucket is about 2 milliseconds; when the typical bucket size is hundreds of megabytes, it is tens to hundreds of milliseconds), and the allocation decision will not become a performance bottleneck of the loading pipeline.

[0116] Of course, in other implementations, the above allocation algorithm can also be modeled as an integer linear programming problem, using the branch and bound method to find the exact optimal solution. This approach is suitable for scenarios with a small number of buckets (e.g., less than a few hundred) and can be solved in seconds. Alternatively, a heuristic solution based on genetic algorithms can be used: by using operations such as crossover and mutation, a better allocation scheme is searched in the solution space, suitable for complex scenarios with a larger number of storage sources. Alternatively, an adaptive solution based on reinforcement learning can be used: the allocation problem is modeled as a Markov decision process, learning the optimal allocation strategy based on historical loading records and continuously optimizing it with increasing usage.

[0117] The following section details the specific implementation methods of the total network interface card bandwidth budget B_load, effective bandwidth BW_t, and the upper limit of the bandwidth currently available for sending weight data from the source instance, B_src_allow, involved in this application. It should be noted that B_load, BW_t, and B_src_allow are calculated based on a single weight loading unit / graphics processor.

[0118] In some implementations, the total network interface card (NIC) bandwidth budget B_load is determined through the following steps: The total physical bandwidth of the network interface card and the bandwidth utilization of the network interface card for the current inference service are used to calculate the total network interface card bandwidth budget that can be used for weight loading. The calculation formula is as follows: B_load=B_nic×(1-U_inf)×α.

[0119] Wherein, B_nic represents the total physical bandwidth of the network interface card, U_inf represents the bandwidth utilization rate of the network interface card for the current inference service (also referred to as "NIC bandwidth utilization rate" in this application), and α is a safety factor (ranging from 0.5 to 0.9), used to reserve margin to cope with sudden fluctuations in inference traffic.

[0120] Preferably, the network card bandwidth utilization U_inf is collected through the Remote Direct Memory Access (RDMA) network card performance counter. The statistical interface natively provided by the RDMA network card hardware directly records the network card's transmit and receive bytes, bandwidth utilization, queue length and other indicators.

[0121] In other implementations, where the performance counters of the virtualized network interface cards (NICs) of some cloud servers are not enabled, the cumulative data transmission and reception volume of the NICs is read through the network device statistics interface provided by the operating system kernel, and the NIC bandwidth utilization rate per unit time is calculated accordingly.

[0122] The U_inf collection interval is consistent with the monitoring cycle of runtime regulation. For example, if the regulation cycle is set to 100ms, the collection interval is also set to 100ms, so that the latest network card bandwidth utilization data is used for each regulation, avoiding the use of outdated data that may lead to regulatory decision errors.

[0123] It should be noted that the total network card bandwidth budget is the total upper limit shared by all remote storage sources (remote storage clusters, distributed memory pools, and running instance memory) that need to be transmitted through the network card.

[0124] In some implementations, the effective bandwidth of each storage source is the maximum bandwidth that the storage source can actually use to transfer weight data to the graphics processing unit (GPU) memory. The core calculation logic is as follows: Query the first available transmission bandwidth of each storage source, where the first available transmission bandwidth is the upper limit of the bandwidth that the physical path of the storage source itself can provide, which is used to characterize the basic transmission capacity of the corresponding storage source without considering sharing contention and runtime constraints. Based on the affinity of the transmission paths from each storage source to the image processor's video memory in the non-uniform memory access topology, a corresponding affinity coefficient is introduced to correct the topology loss of the first available transmission bandwidth, thereby obtaining the second available transmission bandwidth. The effective bandwidth of the storage source is determined at least based on the second available transmission bandwidth. For remote storage sources whose transmission path is via a network card, the effective bandwidth is also determined in conjunction with the share of the total bandwidth budget of the network card that it shares. For remote storage sources such as running instance video memory, the effective bandwidth is also determined in conjunction with the upper limit of the bandwidth that the source instance can send weight data to.

[0125] Specifically, for different types of storage sources, their basic transmission capacity is obtained through the resource detection module and denoted as the first available transmission bandwidth: Remote storage cluster (t1): Its first available transmission bandwidth is the measured read throughput. Specifically, the resource detection module sends a detection request to the server of the remote storage cluster to obtain the measured read throughput as the first available transmission bandwidth of the storage source.

[0126] Local host memory (t2): Its first available transmission bandwidth is the nominal bandwidth of the PCIe bus between the local host memory and the graphics processing unit (GPU) (an inherent parameter determined by hardware specifications). Specifically, the resource detection module, in conjunction with the local cache management module, executes two auxiliary queries to determine the availability of the local path. The first is a physical path capability judgment, i.e., whether the PCIe bus from the local host memory to the GPU has the nominal bandwidth (a fixed condition determined by hardware and topology). The second layer is a local availability judgment of the bucket, i.e., whether the bucket still exists in the local host memory cache, whether the integrity check has passed, and whether the current local memory budget allows it to continue to be retained or read. Only when both of these conditions are met can the bucket participate in the subsequent weight loading and allocation process using local host memory as the source.

[0127] Distributed memory pool (t3): Its first available transmission bandwidth is the RDMA base bandwidth between the distributed memory pool and the local node. Specifically, the resource detection module establishes a communication connection with the distributed memory pool metadata service and performs three query operations in sequence: first, it verifies whether the target large model has been imported into the memory pool; second, it queries the integrity verification results of each bucket of the target large model; finally, on the premise that the bucket is fully cached, it reads the upper limit of the RDMA transmission bandwidth allocated by the memory pool to the local node, and uses this upper limit value as the first available transmission bandwidth.

[0128] Running instance memory (t4): Its first available transfer bandwidth is the base bandwidth of GDR between the running instance memory and the local GPU, which is the upper limit of the physical path's inherent capacity.

[0129] Then, an affinity coefficient is introduced to correct the first available transmission bandwidth: For local host memory (t2), distributed memory pool (t3), and running instance video memory (t4): Second available transmission bandwidth = First available transmission bandwidth × λ; Among them, λ is determined according to the NUMA node affiliation relationship between the GPU and the network card / memory involved in the transmission path (same node, same switch chip λ=1.0, same node, different switch chips λ=0.7, different nodes λ=0.4).

[0130] For the remote storage cluster (t1): its first available transmission bandwidth is the measured value (which already includes link loss), and no additional topology correction is required. The measured read throughput is directly used as the second available transmission bandwidth.

[0131] Then, depending on whether the transmission path from the storage source to the image processor (also referred to as the "pull path" in this application) of the bucket to be pulled passes through the network card, the final effective bandwidth of the storage source is calculated for each scenario: Scenario 1: Pulling data from remote storage sources via network interface cards (such as remote storage clusters, distributed memory pools, or the memory of running instances). a. Based on the preset allocation principle, the total network card bandwidth budget B_load is allocated to each remote storage source to obtain the network card bandwidth share of each remote storage source.

[0132] Assuming the set of remote storage sources accessible to the i-th weight loading unit via the network interface card (NIC) is R_i, and the total bandwidth budget of the NIC available for weight loading by the i-th weight loading unit is B_{load,i}, then the allocation is performed through the following steps: First, calculate the candidate capability value C_{(i,t)} for each remote storage source (where t represents a single remote storage source in R_i). Where: For remote storage sources such as remote storage clusters and distributed memory pools, the candidate capability value is directly taken as the second available transmission bandwidth of the storage source after topology correction, that is, C_{(i,t)} is equal to the second available transmission bandwidth of the storage source; for remote storage sources such as running instance video memory, the candidate capability value must be constrained by both the second available transmission bandwidth of the storage source and the upper limit of the source instance's transmission, that is, C_{(i,t)} takes the minimum value of the two. Then, allocate network card bandwidth share according to the proportion of candidate capability values: first calculate the proportion q_{(i,t)} of the candidate capability value of a single remote storage source to the total candidate capability value of all remote storage sources, and then multiply this proportion by the total network card bandwidth budget B_{load,i} of the current rank_i to obtain the network card bandwidth share B_{(i,t)} of the remote storage source. If a remote storage source is unavailable for the current bucket, malfunctions, or has insufficient remaining loading tasks resulting in unused bandwidth, the unused bandwidth will be redistributed to the remaining available remote storage sources according to their current candidate capacity values.

[0133] b. For remote storage sources such as remote storage clusters and distributed memory pools, the minimum value of the second available transmission bandwidth of the network interface card bandwidth share is taken as its effective bandwidth. For remote storage sources such as running instance video memory, its effective bandwidth is also constrained by the current sending limit that the source instance can provide to the outside world. That is, the effective bandwidth of running instance video memory is the minimum value of its shared network interface card bandwidth share, the second available transmission bandwidth, and the current sending limit constraint that the source instance can provide to the outside world.

[0134] The current sending limit that the source instance can provide externally is determined through the following steps: Query whether there are running instances in the cluster that have loaded the target large model and have the same parallel configuration (such as GPU model and model accuracy); if a matching instance exists, further check whether the instance has opened the remote memory access transmission service port and whether the port is in a connected state; then, through the instance load monitoring interface, obtain the instance's current inference load rate, memory usage rate and other multi-dimensional indicators, evaluate the instance's maximum allowed loading bandwidth ratio B_src, and obtain the upper limit of the bandwidth that the source instance can send weight data to the outside world by BW_src_allow=B_src×B_nic,src, where B_nic,src is the nominal bandwidth of the network card corresponding to the running instance.

[0135] In some implementations, the maximum allowed loading bandwidth ratio B_src is determined based on at least one of the following metrics: When the request queue depth is zero and the graphics processor utilization is less than k1, the load level is idle, and the maximum allowed loading bandwidth ratio B_src is b1 in this case. When the request queue depth is less than d1 and the graphics processor utilization is greater than or equal to k1 and less than k2, the load level is light load, and the maximum allowed load bandwidth ratio B_src is b2 in this case. When the request queue depth is greater than or equal to d1 and less than or equal to d2, or the graphics processor utilization is greater than or equal to k2 and less than k3, the load level is light load, and the maximum allowable load bandwidth ratio B_src is b3 in this case. When the request queue depth is greater than d2 and the graphics processor utilization is greater than d3, the load level is heavy load. In this case, the maximum allowed loading bandwidth ratio B_src is b4 or loading is disabled. Where 0 <d1<d2,k1<k2<k3,b1> b2>b3>b4.

[0136] As an example, the values ​​of d1 and d2 are 10 and 50 respectively; the values ​​of k1, k2 and k3 are 30%, 60% and 85% respectively; and the values ​​of b1, b2, b3 and b4 are 100%, 70%, 40% and 10% respectively.

[0137] The request queue depth refers to the number of inference requests that the source instance has received but not yet completed processing, which is used to characterize the current queuing pressure of the source instance; this metric, along with GPU utilization and network card utilization, serves as input for calculating the maximum allowed loading bandwidth ratio B_src.

[0138] Scenario 2: Fetching data from local storage sources (local host memory) without going through the network card. The transfer between the local host memory and the GPU is completed based on the PCIe bus, which does not occupy network card resources. Therefore, it is not subject to the shared constraints of the total network card bandwidth budget, and the second available transmission bandwidth is directly used as the effective bandwidth of the storage source.

[0139] It should be noted that the second available transmission bandwidth is essentially the maximum physical transmission capacity of the storage source after excluding topology loss, while the final effective bandwidth is the actual usable bandwidth that combines hardware capabilities and network card bandwidth quota. Through the two-layer calculation logic of "topology correction + quota constraint", it can be ensured that the weighted loading traffic fully utilizes the hardware transmission potential without crowding out the network card bandwidth resources of the inference service.

[0140] Optionally, after determining the storage source allocated to each bucket, a corresponding allocation table can be generated. This allocation table maintains the mapping relationship between each bucket and the storage source, identifying that each bucket pulls from a unique storage source. Then, the inference node can create an independent pull thread for each type of allocated bucket's storage source based on this allocation table. The pull threads in this embodiment include: Remote storage fetch thread: Reads weights from the storage cluster into the host memory buffer via remote direct memory access, and then transmits them to the graphics processor's video memory via the peripheral component interconnect standard bus. A token bucket algorithm is used to limit the transmission rate.

[0141] Local memory fetch thread: Transfers memory directly to the graphics processor's video memory from the cache area managed by the local cache management module via the peripheral component interconnect standard bus. This path bypasses the network card and is not limited by the network card's total bandwidth budget. It selects a memory area on the same non-uniform memory access node as the target graphics processor to obtain optimal bandwidth.

[0142] Distributed memory pool fetch thread: Fetches weights from the distributed memory pool via remote direct memory access. Prioritizes using the graphics processor's direct remote memory access to write data directly to video memory, avoiding intermediaries in host memory. Initiates transmission using a network interface card (NIC) compatible with the target graphics processor. Uses a token bucket algorithm for rate limiting.

[0143] Inter-instance fetch thread: Uses a lightweight remote direct memory access transfer engine to directly read weights from the source instance's GPU memory into local GPU memory. Selects an affinity network interface card, uses token bucket rate limiting, and the rate limiting threshold is constrained by the source instance's load.

[0144] Each fetch thread performs weight loading independently and in parallel.

[0145] Optionally, considering that multiple weight loading units in the inference node load the same large model in parallel, if round-robin allocation is used (for example, 1000 remote buckets are evenly distributed among 3 weight loading units, with each weight loading unit being assigned 333, 333, and 334 remote buckets respectively), an imbalance will occur where "weight loading units with more local cache complete local loading quickly, but spend a lot of time waiting for remote fetching; weight loading units with less local cache load locally quickly, but have many remote fetching tasks." The total loading time is determined by the slowest weight loading unit.

[0146] Based on this, this application proposes a cross-node multi-source aggregation strategy, the processing logic of which includes: The key information is synchronized among the weighted loading units participating in parallel loading. The key information includes at least the set of buckets available locally by each weighted loading unit, the local cache hit rate, the effective bandwidth of the local path, the effective bandwidth of the remote path aggregation, and the unified bucket number order of all weighted loading units. Based on the local available bucket set synchronized by each weighted loading unit, the global local overwriteable bucket set is calculated, which is the union of all local available buckets of all weighted loading units. Remove the globally local overwriteable bucket set from all bucket sets to be loaded to obtain the remote bucket set. The remote buckets in the remote bucket set cannot be loaded through the local fetch path on any weighted loading unit and must be fetched from the remote storage source by the cooperation of each weighted loading unit. The remote allocation weight of each weight loading unit is calculated based on the local cache hit rate, local path effective bandwidth, and remote path aggregate effective bandwidth of each weight loading unit. Based on the remote allocation weights of each weighted loading unit and the total data volume of the remote bucket set, the amount of remote fetch task data allocated to each weighted loading unit is determined.

[0147] Among them, the locally available bucket set is used to identify whether each bucket can be loaded through the local pull path on the corresponding weight loading unit. The local cache hit rate is used to characterize the degree of dependence of the weight loading unit on the remote storage source. The higher the local cache hit rate, the fewer remote storage source pull tasks it needs to undertake. The effective bandwidth of the local path of the remote storage source and the effective bandwidth of the remote path aggregation reflect the local loading capability and remote pull capability of the weight loading unit, respectively.

[0148] Specifically, let L_i represent the local available bucket set of the i-th weight loading unit (also referred to as a "bitmap" in this application), and p_i represent the local cache hit rate of the i-th parallel execution node, which is obtained by calculating the proportion of available buckets in the local cache of the corresponding parallel execution node. Assuming that the total set of buckets to be loaded is B, based on the local available bucket set synchronized by each weight loading unit, calculate the global locally coverable bucket set L_global=∪_(i∈Q)L_i (Q is the number of weight loading units participating in parallel loading), then the remote bucket set R=B∖L_global can be defined.

[0149] Wherein, BW_i(local) represents the effective bandwidth of the local path of the i-th weight loading unit. When the corresponding bucket is available in the local host memory cache and the integrity check passes, and the current local memory budget is sufficient, its effective bandwidth of the local path is the nominal bandwidth of the PCIe bus between the local host memory and the graphics processor (GPU).

[0150] Here, BW_i(remote) is the aggregated effective bandwidth of the remote path obtained by summing the effective bandwidth of all available remote storage sources under rank_i. If a remote source is unavailable, faulty, or has insufficient remaining tasks, its unused bandwidth share will be redistributed according to the proportion of the candidate capacity values ​​of the remaining available sources and included in the final aggregation result.

[0151] Let rank_i represent the remote allocation weight of the i-th weight loading unit. Its calculation formula can be expressed as: w_i=((1-p_i)·BW_i(remote)) / Σ_(j∈N)((1-p_j)·BW_j(remote)), where (1−p_i) is positively correlated with the dependence of rank_i on the remote source, and BW_i(remote) is positively correlated with the remote fetching efficiency of the weight loading unit. If the total amount of data in the remote bucket set R is S_R, then the amount of data to be fetched from the remote storage source allocated to rank_i is S_i(remote)=w_i×S_R.

[0152] In this case, the total loading time T_load also needs to be determined in conjunction with the amount of remote fetch data S_i(remote) allocated to each weighted loading unit.

[0153] Specifically, assuming that for rank_i, the amount of data completed through the local pull path (also called "local pull task data amount") is S_i(local), and combined with the allocated remote pull task data amount S_i(remote), since the local path and the remote path can work in parallel, the time T_i(pull) of the pull phase of rank_i is: T_i(pull)=max(S_i(local) / BW_i(local),S_i(remote) / BW_i(remote)). If the weight loading process includes broadcast and local application phases, then the complete loading time of rank_i is: T_i = T_i(pull) + T_i(bcast) + T_i(apply), and the total loading time of all weight loading units is T_load = max_(i∈Q)T_i, where T_i(bcast) is the time taken for rank_i to complete the cross-node broadcast of all buckets with all weight loading units, and T_i(apply) is the time taken for rank_i to complete the local application of all weight data.

[0154] In some implementations, considering the significant differences in bandwidth characteristics among different storage sources, concurrent fetching from multiple sources may result in a difference between the actual arrival order of buckets and their numbering order: the local PCIe path, with its lower latency, may complete later buckets first; while the remote fetch path, with its higher latency, may lag behind the local fetch path. However, the subsequent broadcasting and weight application phases must be executed in strict bucket numbering order (to ensure the consistency of model parameters in the inference engine).

[0155] To address this, this application introduces an ordered waiting mechanism, the logic of which is as follows: initialize the expected number to 0; set each bucket to enter the ready queue after it is pulled; when the bucket number at the head of the ready queue is equal to the expected number, the bucket at the head of the queue is taken out and broadcast to other inference nodes; otherwise, it is temporarily stored in the ready queue to wait for the preceding bucket to complete the pulling.

[0156] Specifically, each bucket enters the ready queue after being fetched. The system maintains a cursor for the current desired bucket number. Only when the bucket number at the head of the ready queue equals the desired number is the bucket removed and enter the broadcast phase; otherwise, it is temporarily stored in the queue waiting for the preceding bucket to complete fetching. After the data of each bucket has fully arrived at the graphics processor's loading buffer, the bucket data is broadcast from the owner node to all target nodes participating in this loading. The inference engine is notified through inter-process communication (IPC) mechanism. The inference engine directly reads the weight data through shared video memory and applies it to the model parameters.

[0157] This mechanism is a necessary order-preserving mechanism after out-of-order arrivals are introduced by multi-source parallel fetching (single-source sequential loading naturally preserves order and does not require this mechanism). Its innovation lies in its synergistic integration with multi-source allocation algorithms and pipeline scheduling, rather than the order-preserving process itself. (1) Order Preservation Granularity and Semantics. Unlike the event time window order preservation for unbounded data streams in the general stream processing framework, the order preservation in this application is for bounded model bucket sequences. The purpose of order preservation is to ensure the determinism of subsequent set communication broadcasts—all inference nodes involved in loading must perform broadcast operations in the same bucket order; otherwise, set communication will result in deadlock or data misalignment, leading to incorrect model parameter loading.

[0158] (2) Coupling design with multi-source allocation algorithm. The greedy allocation algorithm traverses the bucket set in bucket number order. This design is not accidental: the sequential traversal of bucket numbers ensures the continuity of allocation decisions and subsequent pipeline scheduling (adjacent buckets tend to be allocated to the same storage source, reducing path switching overhead), and simplifies the implementation of ordered waiting (sequential allocation makes the degree of out-of-order arrival bounded, and the waiting depth of the ready queue controllable). This coupling design of allocation algorithm and order preservation mechanism does not exist in general stream processing frameworks.

[0159] (3) Integration with pipeline slot management. The ordered waiting mechanism directly controls the timing of pipeline slot release—only after the bucket with the desired number has completed its broadcast is the memory buffer slot it occupies released for use by subsequent buckets. If out-of-order buckets wait for the preceding bucket for a long time, the slot will be occupied and cannot be allocated to new fetch tasks, affecting the overall loading throughput. Therefore, the design of ordered waiting needs to be considered in conjunction with pipeline depth and slot management strategies.

[0160] Based on this, after each bucket has been retrieved and the sequence conditions are met, they are all put into the production line. Please see below. Figure 4 The provided timing diagram for the multi-source parallel pipeline. The pipeline consists of three stages executed sequentially: pull completion confirmation, cross-card broadcast, and weight application, as detailed below: Phase 1 (Pull Completion Confirmation): Perform integrity verification on the pulled weighted data buckets to ensure that the bucket data has completely reached the graphics processor (GPU) loading buffer. After the verification is successful, proceed to the next phase.

[0161] The second phase (cross-card broadcast): After completing the selection of weight data sources and bucket allocation, each weight loading unit (rank) participating in parallel loading executes the weight data retrieval completion confirmation, aggregate communication broadcast, and local application operations sequentially according to a unified bucket number order. The broadcast operation only occurs within the newly started loading group (i.e., all weight loading units participating in the loading). The running inference instance (source instance) providing the weight data does not participate in the aggregate communication operation to avoid consuming source instance resources and interfering with its inference service. Regarding the implementation of the broadcast, a suitable aggregate communication implementation can be selected based on the network topology between nodes, ensuring that the broadcast efficiency adapts to the actual deployment environment.

[0162] The third stage (weight application): The inference engine is notified through the inter-process communication (IPC) mechanism. The inference engine directly reads the weight data through shared memory and applies it to the model parameters, thus completing the loading and application of the single-bucket weights.

[0163] To improve loading efficiency and avoid wasting video memory resources, in some implementations, this embodiment also performs fine-grained management of the graphics processing unit (GPU) video memory, including: Initialize N independent buffer slots for each graphics processor's video memory, with the storage space of each buffer slot being the same size as a single bucket; For each type of storage source, the pull thread is divided into N sub-threads, and each sub-thread is mapped and bound to a buffer slot in the graphics processor's video memory. Control the parallel loading of N sub-threads from each storage source.

[0164] Specifically, in this embodiment, N independent buffer slots are initialized for each image processor's video memory. The storage space size of each buffer slot is consistent with the maximum size (bucket_size_max) of a single weight data bucket, ensuring that it can accommodate any weight data bucket. For each type of storage source's pull thread, N parallel sub-threads are further divided, with each sub-thread mapped and bound to a buffer slot to achieve parallel loading of weight data buckets.

[0165] The number of buffer slots N (i.e., pipeline depth) is automatically determined based on the GPU memory buffer budget, and the calculation formula is as follows: N=max(2,min(floor(M_gpu_budget / bucket_size_max),8)).

[0166] Where M_gpu_budget is the budget available in GPU memory for loading buffers, and bucket_size_max is the maximum size of a single weighted data bucket.

[0167] It should be noted that N does not redefine a new pipeline depth, but rather the actual number of buffer slots that can be used at runtime under the initial bucketing constraints. That is, the bucketing result is determined first based on M_gpu_budget, bucket_size, and N_target, and here the number of slots N actually used is determined based on the available video memory at runtime, but N still needs to fall within the range allowed by initialization, i.e., the range [2, 8].

[0168] The upper and lower limits of the pipeline depth N are set based on the following: Lower limit N=2: Ensure that each stage of the pipeline can be executed in parallel and overlapped. At least one slot is required for the current bucket to be pulled, and another slot is required for the previous bucket to be broadcast or weighted, so as to avoid degenerating into serial execution. Maximum N=8: Considering the GPU memory budget constraints, the total storage space of 8 buffer slots usually accounts for 5% to 10% of the total GPU memory. This will not squeeze the key-value cache (KVCache) and activation value storage space required for inference runtime, thus avoiding impacting inference performance.

[0169] The buffer slots are managed through a semaphore mechanism to prevent data conflicts caused by multiple child threads occupying the same slot at the same time, and to ensure that each weighted data bucket can be accurately loaded into the corresponding buffer slot.

[0170] In some implementations, the heterogeneous storage-aware weight loading method for large language model inference may further include: During the parallel execution of each fetch thread, the availability status of buckets in the local host storage at the application level is managed.

[0171] And / or, adjust the traffic rate of each storage source loading weight at the application layer.

[0172] Specifically, the management of the availability status of application-level local host storage buckets can be implemented using any of the following schemes: (1) Establish a bucket-level cache index table at the application layer to record the cache status (or "availability status") of at least one bucket of the corresponding large language model on the local machine, and fix the cached buckets in the local host memory through a memory locking system call. (2) Caching scheme based on user-space file system: Implement a lightweight file system in user space, store weight data as a memory-mapped file managed by user space, and bypass the kernel page cache mechanism.

[0173] (3) Cross-process caching scheme based on shared memory: Use the operating system's shared memory (such as POSIX shared memory objects) to store weight cache. Multiple inference processes can share the same cache data, reducing memory usage.

[0174] (4) Cache scheme based on persistent memory: using non-volatile memory to store weighted cache, which has access speed close to host memory and persistence characteristics, and the cache is not lost after restart.

[0175] Specifically, adjusting the traffic rate of each storage source's loading weight at the application layer can be achieved using any of the following methods: (1) Adjust the weight loading traffic rate of each storage source based on the real-time bandwidth utilization of the network card.

[0176] (2) Feedback adjustment scheme based on proportional-integral-derivative (PID) controller: The response delay of the inference service is used as the feedback signal to automatically adjust the weight loading flow rate of each storage source.

[0177] (3) Hardware-level bandwidth isolation scheme based on virtual channels in unlimited bandwidth networks: This scheme utilizes the virtual channel function of unlimited bandwidth network switches to allocate inference communication traffic to high-priority virtual channels and weighted traffic to low-priority channels, achieving strict bandwidth isolation through hardware. The isolation effect of this scheme is guaranteed by hardware, with extremely low latency overhead. However, it requires both the switch and the network card to support virtual channel configuration, resulting in a high deployment threshold. Furthermore, the isolation strategy is relatively fixed after configuration, making it difficult to dynamically adjust according to service load.

[0178] (4) Hardware-level management scheme based on network service quality (QoS) tags: By utilizing the traffic classification and priority queuing functions of network switches, different service quality tags are set for loading traffic and inference traffic, and bandwidth isolation is implemented by hardware.

[0179] (5) Adaptive scheme based on congestion control protocol: Combine the congestion control mechanism of remote direct memory access network (such as data center quantized congestion notification protocol) to automatically adjust the weight loading traffic rate of each storage source according to the network congestion status.

[0180] The heterogeneous storage-aware weight loading method for large language model inference provided in this embodiment achieves multi-source parallel loading across heterogeneous hardware paths by superimposing the parallel bandwidth of different hardware transmission paths, effectively improving overall bandwidth utilization and significantly shortening weight loading time. Simultaneously, before loading begins, it detects the non-consistent memory access topology and the affinity of peripheral component interconnection standard devices on the local machine. Based on the detection results, it selects the optimally compatible network card and memory region for each graphics processor to initiate data transmission, thereby avoiding bandwidth degradation and increased latency caused by transmission across non-consistent memory access nodes and ensuring high-efficiency data transmission.

[0181] Considering the significant dynamic uncertainty in the availability of each storage source during cold start scenarios (especially the startup of the first inference instance and batch startup of multiple instances), it is necessary to continuously monitor the running status of the inference service during weight loading to dynamically adjust the loading strategy. Based on this, this application also proposes a weight loading runtime control method based on real-time network interface card (NIC) utilization feedback. This method dynamically adjusts the bandwidth share allocated to each storage source and / or the pull path of the weight tensor data that has not yet been pulled by each storage source based on real-time monitored NIC traffic during the weight data loading process of multiple storage sources, so as to ensure that cross-machine communication of the inference service is not affected.

[0182] The following is combined Figure 5-6This application provides a detailed description of the weight loading runtime control method based on real-time network card utilization feedback.

[0183] Please see Figure 5 The weight loading runtime control method based on real-time network card utilization feedback provided in this embodiment includes: B10, during the process of pulling weight tensor data of the target large model in parallel from various storage sources based on the allocation table, collects the real-time network card traffic of the local machine, and determines the real-time bandwidth utilization of the network card based on the real-time network card traffic. The allocation table is used to maintain the mapping relationship between each weight tensor and the storage source, so as to identify that each weight tensor is pulled from the unique corresponding storage source.

[0184] B20 dynamically adjusts the bandwidth allocated to each storage source and / or the retrieval path of unretrieved weight tensor data in each storage source based on real-time bandwidth utilization.

[0185] It should be noted that the above allocation table can be the allocation table generated by the heterogeneous storage-aware weight loading method for large language model inference provided in the previous embodiment. In this case, the weight tensors in the allocation table are in units of buckets and establish a unique mapping relationship with the storage source. In other embodiments, other reasonable allocation methods can also be used to allocate corresponding pull paths to each weight tensor, as long as it is ensured that each weight tensor (or "bucket") corresponds to only one storage source and pull path, and that the bandwidth allocation and path selection can be dynamically adjusted according to the real-time utilization of the network card, without being limited to a single allocation mode.

[0186] In addition, the generation of the allocation table can be flexibly adjusted. If storage source failure or abnormal network card traffic occurs during the weight loading process, the mapping relationship between buckets / weight tensors and storage sources in the allocation table can be updated in real time to ensure that each weight tensor / bucket can be loaded through the optimal path, while avoiding excessive network card resources and ensuring the normal operation of the inference service.

[0187] For ease of explanation, the following text will use "bucket" as the smallest weighted scheduling unit for illustration.

[0188] In some implementations, in step B10 above, the real-time traffic of the local network card can be collected at preset intervals (e.g., 100 milliseconds), and the real-time bandwidth utilization rate U_inf of the network card can be obtained by calculating the ratio of the real-time traffic of the network card to the total physical bandwidth of the network card.

[0189] Please see Figure 6 , Figure 6This diagram illustrates various control strategies involved in the weighted runtime control method based on real-time network card utilization feedback provided in this application. In this embodiment, the network card bandwidth dynamic control strategy is implemented through the above step B20. In some implementations, the network card bandwidth dynamic control strategy may include: The total network interface card (NIC) bandwidth budget that can be used for weight loading is adjusted based on real-time bandwidth utilization, where the total NIC bandwidth budget serves as the upper limit of the token generation rate of the token bucket. Based on the total network interface card (NIC) bandwidth budget, allocate corresponding bandwidth share percentages to each remote storage source, and within the upper limit of the token generation rate, allocate token resources to each remote storage source according to the bandwidth share percentage. Here, a remote storage source refers to a storage source whose pull path passes through the NIC. And / or, If the real-time bandwidth utilization exceeds the preset upper limit, the pull path for weight tensors that have not yet been pulled from the remote storage source will be switched to the local pull path. The local pull path refers to the local host memory path that does not go through the network card.

[0190] In this embodiment, the token bucket algorithm is used, with the total network interface card (NIC) bandwidth budget B_load as the upper limit of the token generation rate. This algorithm uniformly limits the weighted loading traffic of all remote storage sources transmitted through the NIC (including remote storage clusters, distributed memory pools, and running instance memory). When real-time bandwidth utilization increases, the upper limit of the token generation rate is simultaneously reduced (i.e., the total NIC bandwidth budget is reduced), and the bandwidth allocation for each remote storage source is proportionally reduced according to its corresponding bandwidth share (determined by preset allocation rules). Conversely, when NIC real-time utilization decreases, the upper limit of the token generation rate is increased (i.e., the total NIC bandwidth budget is increased), and the bandwidth allocation for each remote storage source is proportionally increased according to its corresponding bandwidth share. This adapts to the dynamic uncertainty of storage sources in cold start scenarios, ensuring reasonable allocation of bandwidth resources.

[0191] More specifically, if the real-time bandwidth utilization U_inf is greater than the preset upper limit θ_high, then the system enters protection mode. In protection mode, the rate limiting threshold for all pull threads of remote storage sources is reduced to the total network card bandwidth budget B_load × 0.5. For buckets that have not yet started pulling, they are preferentially reallocated to the local host memory path. If the real-time bandwidth utilization U_inf is less than the preset lower limit θ_low, then the system enters acceleration mode and restores to B_load (without additional rate limiting, making full use of the available bandwidth budget; that is, restoring from 0.5 of the protection mode to the full budget). If θ_low ≤ real-time bandwidth utilization U_inf ≤ θ_high, then the current mode remains unchanged (hysteresis interval, avoiding frequent switching).

[0192] The calculation method of the total network card bandwidth budget B_load has been described in the relevant content of the above-mentioned heterogeneous storage-aware weight loading method embodiment for large language model inference, and will not be repeated here.

[0193] The preset upper limit value θ_high is determined based on the following: In typical Remote Direct Memory Access (RDA) networks, the Data Center Quantized Congestion Notification Protocol (DCQCN) congestion feedback mechanism triggers when the network interface card (NIC) utilization approaches saturation; beyond this point, network latency increases rapidly and non-linearly. Real-world testing shows that in typical InfiniBand environments, the Explicit Congestion Notification (ECN) marking rate increases sharply after NIC utilization exceeds 70%, with 85% being a typical congestion trigger point. Therefore, θ_high is set between 0.7 and 0.85. A conservative 0.7 can be used for initial deployment, gradually adjusted based on actual inference latency fluctuations. In Remote Direct Memory Access (RoCEv2) networks based on Aggregated Ethernet, due to differences in congestion control mechanisms compared to InfiniBand networks (RoCEv2 relies on Priority-Based Flow Control (PFC) and ECN working together, and congestion triggering behavior is more significantly affected by switch buffer configuration), it is recommended to calibrate the specific value of θ_high through network latency benchmark testing during actual deployment. The preset lower limit value θ_low should maintain a sufficient hysteresis interval with θ_high (a difference of no less than 30 percentage points is recommended) to ensure the stability of mode switching. The value range is 0.2 to 0.4. The design principle of the hysteresis interval is similar to the temperature control hysteresis in industrial control—if the gap between the upper and lower limits is too small, the system will frequently oscillate between protection mode and acceleration mode near the critical utilization rate. The speed limit adjustment overhead brought by each switch will actually reduce the load throughput. The minimum gap of 30 percentage points has been verified in actual deployment as an empirical value that can effectively suppress control oscillations.

[0194] like Figure 6 As shown, the weight loading runtime control method based on real-time network card utilization feedback in this embodiment also executes fault detection and handling strategies.

[0195] The fault detection and handling strategy may include the following steps: B30 monitors the health status of each storage source; if the health status of any storage source is faulty, the pull task of the faulty storage source is suspended, the buckets that have not been pulled are reallocated to other available storage sources, and the pull path of the corresponding bucket is updated.

[0196] In some implementations, if a weight tensor that has not been fully fetched is available in a storage source whose fetch path does not pass through a network interface card (NIC), the unfetched weight tensor is allocated to that storage source. Conversely, if the fetch path of a storage source in a faulty state passes through a NIC, the bandwidth allocated to that faulty storage source is released.

[0197] In other embodiments, if the health status of any of the storage sources is detected to be faulty, a recovery probe is initiated after a first cooling-off period of a set duration; if the recovery probe is successful, the bandwidth allocated to the storage source is gradually increased according to an exponential increment strategy until its originally set bandwidth share is reached.

[0198] In this application, four health states are defined for the storage source: available, probing, faulty, and recovering. The state transition rules are as follows: If an access timeout is detected or the error rate rises to an error threshold, the health status of the storage source is changed from "available" to "probing".

[0199] If h consecutive probes fail, the health status of the storage source is changed from "probing" to "faulty". As an example, h equals 3.

[0200] If a recovery probe is automatically initiated after the first cooling-off period (the initial cooling-off period is 30 seconds, which doubles exponentially after consecutive recovery failures, with a maximum of 5 minutes), then the health status of the storage source is determined to have changed from "faulty" to "recovering".

[0201] If the recovery probe is successful, the health status of the storage source is determined to change from "recovering" to "available", and the allocation weight is gradually increased.

[0202] If the recovery probe fails, the system will re-enter the fault state and extend the cooling period, and the health status of the storage source will be changed from "recovering" to "fault".

[0203] The cooling-off period parameter setting is based on the following: The initial 30-second cooling-off period is a trade-off between recovery times for different fault types. Brief network jitter (such as switch port flipping or remote direct memory access connection timeouts) typically recovers within 10 seconds, but premature retries (e.g., 5 seconds) increase invalid probe overhead before the jitter subsides. Node restarts (such as restarting a crashed distributed memory pool node process) typically take 60 to 120 seconds, but a 30-second initial probe can promptly detect scenarios requiring rapid recovery (such as process-level restarts rather than node-level restarts). The cooling-off period uses an exponentially increasing strategy (30 seconds initially, 60 seconds later, 120 seconds later, with a maximum of 300 seconds), balancing rapid recovery detection with avoiding continuous invalid probes of permanent fault sources.

[0204] Fault detection employs a tiered timeout strategy, setting differentiated timeout parameters based on the response characteristics of different storage sources: Storage source First timeout Retry strategy Fault diagnosis Remote storage cluster 500 milliseconds Index retreat, maximum 5 seconds Three consecutive timeouts Distributed memory pool 50 milliseconds Retry immediately at fixed intervals Three consecutive timeouts Runnable instance memory 1000 milliseconds (considering source instance load fluctuations) Index retreat, maximum 10 seconds Three consecutive timeouts If a storage source is determined to be in a faulty state during the retrieval process, at least one of the following operations will be performed: (1) Mark the storage source as "faulty".

[0205] (2) Reassign the unfinished buckets / weight tensors of the storage source to available storage sources (preferably storage sources that do not occupy network cards).

[0206] (3) If the faulty storage source is a storage source via the network card, release its bandwidth for use by other remote storage sources via the network card.

[0207] (4) Record fault event logs.

[0208] (5) Start the cooldown timer, and enter the recovery detection after the cooldown expires.

[0209] It should be noted here that the responsibility for fault recovery is as follows: the fault repair of the underlying storage service (such as the restart of remote storage inference nodes or the replacement of distributed memory pool nodes) is completed by the storage system itself, and this application is not responsible for repairing underlying faults. The "recovery probe" in this application only refers to: after the cooling period, retrying to connect and verify the availability of the storage source; once the underlying service recovers, the source is automatically reinstated into the available source pool upon successful probe, and its allocation weight is gradually increased (gradual recovery to avoid the newly recovered source immediately bearing the full load). For permanent faults (such as the P2P source instance being completely offline or the distributed memory pool node no longer restarting), if the recovery probe fails more than a preset number of times (such as 10 times), the source will be permanently removed from the available list of this loading task, and loading will be ensured by relying on other available sources (the backup remote storage cluster is always available).

[0210] like Figure 6 As shown, the weight loading runtime control method based on real-time network card utilization feedback in this embodiment also executes a pipeline depth adaptive adjustment strategy, which can include the following steps: During the weight loading process, B40 monitors the actual throughput of local weight loading and the memory pressure of the graphics processor in real time; based on the actual throughput and memory pressure, it dynamically adjusts the pipeline depth; where the pipeline depth corresponds to the number of independent buffer slots pre-allocated in the graphics processor's memory for weight loading, and each buffer slot is mapped and bound one-to-one with the sub-threads of different loading stages of each memory source fetching thread.

[0211] It should be noted that steps B10~B20 above can be executed simultaneously with B30 and B40.

[0212] More specifically, the dynamic adjustment of the pipeline depth adopts a hybrid strategy that combines exponential growth and linear reduction, taking into account both the performance exploration efficiency and the running stability. The specific execution logic is as follows: First, calculate the throughput ratio Rthroughput: Based on the pulling time of the most recent g buckets (where g is 3 to 5), count the actual pulling throughput per unit time, and calculate the ratio of this actual throughput to the theoretical maximum throughput to obtain the throughput ratio Rthroughput.

[0213] Then, continuously scan the free space distribution of the GPU video memory in real time to determine the size of the largest continuous free video memory block.

[0214] After that, define the initial value of the pipeline depth N to be preset as 2 to 4 according to the total capacity of the GPU video memory and bucketsize_max, with an upper limit of 8 and a lower limit of 2.

[0215] If the following two conditions are met in the pipeline depth exponential growth logic, then perform the pipeline depth growth operation: N = min(N × 2, 8). Rapidly explore the performance upper limit through the exponential growth mode to improve the weighted data parallel pulling efficiency.

[0216] Condition 1: Rthroughput < 0.7 (indicating that the current throughput has not reached the theoretical upper limit and there is room for performance improvement); Condition 2: The size of the largest continuous free block in the GPU video memory > bucketsize_max × 1.5 (where 1.0 times bucketsize_max is used to carry new slots, and 0.5 times bucketsize_max is used to reserve redundant space for video memory fragmentation to avoid memory allocation conflicts).

[0217] If the following condition is met in the pipeline depth linear reduction logic: The size of the largest continuous free block in the GPU video memory < bucketsize_max × 1.2 (indicating that the current free space in the video memory is not sufficient to stably maintain the operation of the existing slots and there is a risk of video memory overflow), then perform the pipeline depth reduction operation: N = max(N - 1, 2). Smoothly reduce the parallelism through the linear reduction mode to avoid loading interruptions or system crashes caused by insufficient video memory space and ensure running stability.

[0218] The weight loading runtime regulation method based on the real-time utilization rate feedback of the network card provided in this embodiment dynamically adjusts the bandwidth allocated to each storage source and / or adjusts the pulling path of the weight tensors that have not started pulling in each storage source according to the real-time load of the inference service, and supports automatic degradation in case of storage source failures. It does not require specific hardware support and can be dynamically adjusted according to the inference load to ensure zero interference to the inference service.

[0219] Given that the availability of each storage source changes dynamically over time and the mechanisms of change vary, the local host memory cache is affected by the operating system's page cache eviction policy and may be partially evicted at any time without the application layer being aware of which specific buckets have been evicted. Running instances may go online or offline at any time due to the elastic scaling of the inference service; distributed memory pool nodes may fail due to faults. This bucket-level granularity of dynamic availability changes requires the system to accurately detect the availability status of each bucket in each storage source before each load and to respond to storage source failures in real time during the loading process, which cannot be covered by the simple block redundancy and retry mechanisms in general distributed storage.

[0220] Based on this, this application also proposes an application-layer bucket-level local host memory weighted cache management method, which serves as the accurate data foundation for the multi-source perception system and replaces the uncontrollable mechanism of the operating system page cache.

[0221] Please see Figure 7 The application-layer bucket-level local host memory weight cache management method provided in this embodiment includes the following steps: C10 retrieves the bucket-level cache index table established and maintained by the application layer.

[0222] The bucket-level cache index table records the local cache status of at least one bucket of the corresponding large language model, with the model identifier as the dimension. A bucket is an independent transmission and caching unit formed by dividing all weight tensors of the corresponding large language model according to a preset bucketing rule.

[0223] C20 uses a memory locking system call to pin locally cached buckets in the local host memory, where the total amount of memory locked is constrained by the local host cache budget.

[0224] C30, when loading the target large language model, queries the cache status bucket by bucket based on the bucket-level cache index table, and sets locally cached buckets to be loaded from local host memory, and evicted buckets to be reloaded from other remote storage sources.

[0225] The application-layer bucket-level local host memory weight cache management method provided in this embodiment can be applied to... In the weight loading process of the heterogeneous storage-aware weight loading method for large language model inference described above, the bucket-level cache index table can serve as the basic data for the allocation algorithm. This table determines the local cache availability of each bucket, thereby marking the available[k][2] status and achieving the optimal decision of "prioritizing local bucket loading and supplementing remote buckets on demand." Furthermore, the "precise page fault detection and on-demand supplementation loading strategy" (i.e., setting buckets that are already cached locally and whose verification values ​​pass to load from local host memory, and supplementing evicted buckets from other remote storage sources) involved in this application-layer bucket-level local host memory weight cache management, as well as the "memory lock degradation and recovery mechanism," "cache preheating strategy," "proactive eviction strategy when budget exceeds limits," and "preheating pause and breakpoint resumption strategy" described later, can also be applied to the weight loading process of the heterogeneous storage-aware weight loading method for large language model inference.

[0226] It should be noted that the application-layer bucket-level local host memory weight cache management method provided in this embodiment is not limited to the heterogeneous storage-aware weight loading method for large language model inference provided in the above embodiment. It can be applied to the weight loading service of any other large language model inference service. This application does not make any specific limitations on this.

[0227] As an example, the bucket-level cache index table involved in step C10 is represented as follows: CacheIndex={ model_id:{ bucket_0:{status:"cached",addr:0x...,size:...,checksum:...}, bucket_1:{status:"evicted",addr:null}, bucket_2:{status:"cached",addr:0x...,size:...,checksum:...}, ... } }

[0228] Optionally, in addition to recording the local cache status of the bucket, the bucket-level cache index table can also record its starting address, data size, and checksum in the host memory.

[0229] Optionally, the checksum uses a hierarchical check strategy: By default, a lightweight fast hash algorithm (such as xxHash64) is used, which has the characteristics of computation speed close to host memory bandwidth and excellent uniformity of hash value distribution.

[0230] When a suspicious collision is detected in the fast hash value (i.e. different buckets calculate the same fast hash value), or in a critical data integrity scenario (including but not limited to the first loading of the model, the first caching after the weight data is updated, and the inference service involves high-security business), switch to a cryptographic hash algorithm (such as the SHA-256 algorithm) for secondary verification.

[0231] For buckets reloaded from remote storage sources: After the data is written to the local host memory, the complete fast hash value (with encrypted hash values ​​added if necessary) is immediately calculated and written to the corresponding entry in the bucket-level cache index table. For buckets that hit the local cache, the pre-stored hash value in the bucket-level cache index table is directly reused to complete the verification, eliminating the need for repeated calculations and further reducing the performance overhead of the verification process.

[0232] In some implementations, the execution of step C20 above requires that a memory lock execution condition be met. The logic for determining whether the memory lock execution condition is met is as follows: The memory lock execution condition is determined to be met when the main process of the target large language model inference service has CAP_IPC_LOCK capability and the single-process memory lock limit set by the system is not lower than the local host cache budget.

[0233] Among them, the main process of the target large language model inference service is responsible for managing the scheduling and execution of the resource detection module, weight allocation decision module, runtime control module and local cache management module.

[0234] CAP_IPC_LOCK is a process capability of the Linux system that allows processes to lock physical memory pages to prevent them from being swapped to virtual memory by the operating system.

[0235] The memory lock limit is a threshold set by the Linux system through a resource limitation mechanism (the `RLIMIT_MEMLOCK` parameter), representing the maximum amount of data a single process is allowed to lock in physical memory. The Linux system's memory lock limit is divided at the individual process level; that is, the system limits the total amount of physical memory that each independent process can lock. All memory lock operations of the inference service main process are included in that process's locked memory quota. Its subordinate pull threads / child threads inherit the main process's memory lock status but do not consume additional quotas.

[0236] In step C20 above, instead of relying on the operating system's page caching mechanism, the application layer directly uses the memory locking system call (mlock) to fix the cached buckets in host memory, preventing them from being evicted by the operating system. The total amount of memory locked is constrained by the memory budget. M_cache=M_host-M_kv-M_sys Where M_cache is the cache budget, M_host is the total host memory capacity, M_kv is the key-value cache reserved memory (taken as the peak estimated amount occupied by the key-value cache during inference runtime, using a conservative estimate to ensure that the cache will not squeeze the memory required for inference runtime; in actual deployment, this value can be determined through historical monitoring data or the pre-allocated configuration of the inference engine), and M_sys is the system reserved memory.

[0237] In some implementations, if the main process does not meet the conditions for memory lock execution, a memory lock downgrade and recovery mechanism is executed, the execution logic of which includes: Before a bucket is loaded, the integrity of the bucket is checked based on the checksum of the bucket recorded in the maintained bucket-level cache index table. Buckets that pass the checksum are marked as available, and buckets that fail the checksum are marked as unavailable. They are then allocated to a remote storage source by a preset allocation algorithm. If the memory lock execution condition is detected again during service operation, the system will revert to memory lock mode and re-execute the memory lock operation on the cached buckets.

[0238] Specifically, without memory locking, cached data is affected by the operating system's page cache eviction policy, and the cache hit rate depends on the host memory pressure: when memory pressure is low (e.g., ample host memory and no other large memory processes competing for memory), the hit rate can approach 100%, with minimal performance loss; when memory pressure is high (e.g., increased key-value cache usage or other inference instances running on the same machine), the hit rate may drop from nearly 100% to 30% to 60%, corresponding to an increase in loading time of approximately 1.5 to 3 times. However, even in degraded mode, the cache index remains effective, requiring only verification of the cache integrity of each bucket before each load (an additional overhead of approximately 1% to 3% of the total loading time). Buckets that fail the verification are marked as unavailable and allocated to other remote storage sources by an allocation algorithm, which can employ the bandwidth-efficiency-first allocation algorithm mentioned earlier.

[0239] When the system detects that the memory locking condition is not met and performs a degradation, it records an alarm log and sets a degradation status flag. The operation and maintenance monitoring system can query the current cache management mode ("memory locking mode" or "unlocked degradation mode") through the standard indicator interface, so that operation and maintenance personnel can promptly investigate deployment issues such as container security configuration or system resource limitations.

[0240] The system re-checks memory lock availability conditions at startup and at checkpoints after each loaded task (checking CAP_IPC_LOCK capability and RLIMIT_MEMLOCK limit). Once the conditions are met (e.g., the corresponding capability is granted after the container security configuration is updated), it automatically upgrades from degraded mode to memory lock mode and re-executes memory lock operations on cached buckets without restarting the service.

[0241] In this embodiment, when loading weights, a precise page fault detection and on-demand reloading strategy is implemented. That is, the bucket-level cache index table is used to identify which buckets are available in the local cache, and the cached buckets with valid verification values ​​are loaded from the local host cache. It also identifies which buckets need to be reloaded from other storage sources, eliminating the need for a full re-warm-up due to partial cache missing. For each bucket b_k: If CacheIndex[model_id][bucket_k].status=="cached": Validate the checksum; if it passes, mark available[k][2]=1 (local memory is available). otherwise: Mark available[k][2]=0 (will be allocated to other available sources by the allocation algorithm). In some implementations, the application-layer bucket-level local host memory weighted cache management method also executes an active eviction strategy when the budget is exceeded. That is, when the absolute value of the difference between the total cache amount of the buckets locked in the local host memory and the local host cache budget is less than a preset difference threshold, the buckets in the local host cache are evicted according to the strategy of redundancy backup priority and large-size bucket priority.

[0242] Specifically, buckets in the local host cache are evicted according to the following rules: (1) Prioritize eliminating buckets that have been fully backed up by the distributed memory pool; (2) Next, eliminate buckets whose data volume meets the preset large size requirements; When eviction is performed, the status of the corresponding bucket in the bucket-level cache index table is updated to "eviction", and the memory lock of the bucket in the local host memory is released, thus freeing up the corresponding memory space.

[0243] In some implementations, the application-layer bucket-level local host memory weighted cache management method also performs a cache preheating strategy, which includes: asynchronously performing cache preheating during idle periods of the inference service, preloading frequently used buckets from remote storage sources into local host memory and updating the bucket-level cache index table.

[0244] In practice, frequently used buckets from the remote storage source are added to a preheating queue, and the buckets in the preheating queue have priorities. Based on the priority order of the buckets in the preheating queue, the buckets in the preheating queue are preloaded into the local host memory in turn, and the bucket-level cache index table is updated synchronously.

[0245] The priority of buckets in the preheating queue is determined by a combination of the historical usage frequency and business priority of the corresponding large language model. Buckets with higher historical usage frequency and higher business priority have higher priority in the preheating queue and are given priority in preheating.

[0246] The rules for determining the running status of the inference service are as follows: if the duration of the request queue depth of the target large language model being 0 exceeds the preset first duration (e.g., 5 seconds), and the utilization rate of the local graphics processor is lower than the preset idle utilization rate threshold (e.g., 10%), the inference service is determined to be in an idle state; otherwise, the inference service is determined to be in a non-idle state.

[0247] In some implementations, the application-level bucket-level local host memory weighted cache management method also executes a warm-up pause and breakpoint resume strategy, the execution logic of which is as follows: During the warm-up process, the operational status of the inference service is monitored in real time; If the inference service changes from an idle state to a non-idle state, wait for the bucket currently being preheated to complete preheating, then pause the entire preheating operation and record the current pause breakpoint position. Once the inference service is detected to be idle again, the preheating operation of the next bucket in the preheating queue will continue from the last pause point until the preheating of all buckets in the preheating queue is completed.

[0248] The application-layer bucket-level local host memory weight cache management method provided in this embodiment establishes a bucket-level cache index table at the application layer. This table records the cache status of at least one bucket of the corresponding large language model locally. When the target large language model is loaded, the cache status is queried bucket by bucket based on the bucket-level cache index table. Cached buckets are loaded from local host memory, while evicted buckets are reloaded from other remote storage sources. This method accurately identifies missing buckets and only reloads them from remote storage sources, avoiding the hundreds of gigabytes of invalid duplicate loading caused by the inability to locate missing weight fragments in existing solutions. This significantly reduces remote data transmission and shortens cold start time. Simultaneously, by using an application-layer memory locking system call, the cached weight data is fixed in local host memory, eliminating reliance on the operating system's page cache. This solves the shortcomings of existing solutions where the operating system autonomously evicts weight data and the application layer cannot intervene, ensuring the stability and predictability of the local cache status, reducing the overhead of bucket-by-bucket probing during cold starts, and further improving cold start loading efficiency.

[0249] To illustrate that local host memory paths and remote paths via network cards can work in parallel, this section simplifies the loading process into two categories: local paths and remote paths. This is used to analyze the impact of increasing the local hit rate p on the total loading time. However, in actual implementation, the bucket partitioning algorithm, storage source selection algorithm, remote allocation weight, and rank-based parallel loading rules described above are still followed.

[0250] Two-path parallel model: The pull / transmission path of the weight tensor is simplified into two categories: the local path without network card (local host memory cache transmitted via PCIe, effective bandwidth is BW_local) and the remote path via network card (remote storage cluster, distributed memory pool, and running instance memory, constrained by network card bandwidth budget, aggregate effective bandwidth is BW_remote). Let the local cache hit rate be p (i.e., the proportion of available buckets in local host memory to the total weights of the model), and the total size of the model weights be S.

[0251] Single-source loading time (using only remote paths): T_single=S / BW_remote.

[0252] Multi-source parallel loading time (local path and remote path work in parallel): T_multi=max(p×S / BW_local,(1-p)×S / BW_remote).

[0253] Speedup ratio: R(p) = T_single / T_multi.

[0254] When p is small (remote path is the bottleneck), R(p)≈1 / (1-p); when p is large (local path is the bottleneck), R(p)=BW_local / (p×BW_remote). The speedup ratio reaches its theoretical maximum when both paths are completed simultaneously (i.e., perfect load balancing), and the optimal cache hit rate is: p*=BW_local / (BW_local+BW_remote).

[0255] The corresponding maximum speedup ratio is: R_max=(BW_local+BW_remote) / BW_remote.

[0256] The value of multi-source parallelism does not depend on the premise that "local is faster than remote". The key characteristic of the above formula is that the allocation algorithm takes the measured bandwidth BW_t as input and automatically allocates the workload proportionally, so that each path completes simultaneously (minimizing makespan). If the measured bandwidth of RDMA > the effective bandwidth of PCIe, the algorithm automatically allocates more to the remote path and less to the local path, and vice versa. The two essential values ​​of multi-source parallelism are unrelated to relative speed: The local path does not consume network card bandwidth: Regardless of the speed of RDMA, the local PCIe path is "transparent" to the network card. When the inference service requires network card communication (such as the expert parallel all-to-all communication), the weight loading of the local path does not cause network card competition, which is equivalent to "releasing" the corresponding proportion of network card bandwidth to the inference service.

[0257] Adaptive allocation is close to optimal at any hit rate: In a multi-source system, regardless of the bandwidth ratio, the algorithm automatically adjusts to balance the loads of the two paths; while fixed policies (such as pure remote or pure local) will degrade significantly when the parameters change.

[0258] The following uses two typical deployment scenarios to illustrate the multi-source parallel effects in different network environments.

[0259] Scenario A: Traditional network environment (RDMA bandwidth < PCIe effective bandwidth) Typical configuration: 1×100Gbps InfiniBand network card, the measured available RDMA bandwidth for weight loading is BW_remote ≈ 10GB / s (after deducting the occupancy of the inference service and safety factor), and the measured effective bandwidth of PCIe Gen4 is BW_local ≈ 20GB / s. Taking the example of loading a mixed expert large model of about 440GB: Optimal hit rate: p* = 20 / (20 + 10) ≈ 0.67, at this time the maximum acceleration ratio R_max = 30 / 10 = 3.0.

[0260] Cache hit rate p = 0 (no local cache): T_multi = 440 / 10 = 44 seconds, and the acceleration ratio is 1.0.

[0261] Cache hit rate p = 50%: T_multi = max(220 / 20, 220 / 10) = max(11, 22) = 22 seconds, and the acceleration ratio is 2.0.

[0262] Cache hit rate p ≈ 67% (optimal point): T_multi = max(295 / 20, 145 / 10) = max(14.7, 14.5) ≈ 14.7 seconds, and the acceleration ratio is 3.0.

[0263] In this scenario, the local path is faster, and the main benefit of multi-source parallelism is to shorten the loading time.

[0264] Scenario B: High-speed network environment (RDMA bandwidth > PCIe effective bandwidth) Typical configuration: 2×400Gbps RDMA network cards (a part of the bandwidth has been reserved for the inference service), the measured available RDMA bandwidth for weight loading is BW_remote ≈ 50GB / s, and the measured effective bandwidth of PCIe Gen5 is BW_local ≈ 35GB / s. Taking the same 440GB model as an example: Optimal hit rate: p*=35 / (35+50)≈0.41, at which point the maximum acceleration ratio R_max=85 / 50=1.7.

[0265] Cache hit rate p=0 (no local cache): T_multi=440 / 50=8.8 seconds, speedup ratio 1.0.

[0266] Cache hit rate p=41% (optimal point): T_multi=max(180 / 35,260 / 50)=max(5.1,5.2)≈5.2 seconds, speedup ratio 1.7.

[0267] Meanwhile, the local path handles approximately 41% of the data transmission (180GB), effectively freeing up approximately 180 / 440×100%≈41% of the network card bandwidth for cross-machine communication of the inference service.

[0268] In this scenario, RDMA is faster, and the acceleration of loading time by multiple sources in parallel is limited (about 1.7×), but its main value lies in freeing up network card bandwidth for inference communication: if the local path is not used, using RDMA for the entire process will occupy an additional 35GB / s of network card bandwidth, which may significantly interfere with expert parallel full-to-full communication.

[0269] The following explains why loading a multi-source heterogeneous large language model cannot rely solely on optimizing the local cache hit rate: Local DRAM is limited by capacity and cannot achieve 100% hit rate (it needs to share host memory with KVCache, system memory, and other concurrent models).

[0270] After exceeding the optimal point p*, the local path becomes the bottleneck (in scenario B, p*≈41%, after exceeding this value, the local path is slower than the remote path).

[0271] The adaptive allocation of multi-source systems is close to optimal at any hit rate, without the need for precise control of the hit rate.

[0272] Both scenarios above demonstrate that multi-source parallel loading has positive value in different network environments. The difference lies only in the form of the main benefit (accelerated loading vs. freeing up network card bandwidth), and this value is unrelated to whether "local is faster than remote".

[0273] The following section presents the expected performance metrics of the technical solution in a typical deployment scenario to verify the effectiveness of each core module. The following experiments are based on a typical eight-card inference server (dual non-consistent memory access nodes, four graphics processors per node, and four remote direct memory access network cards) loading a hybrid expert model of approximately 440 gigabytes.

[0274] Experiment 1: Comparison of loading times under different configurations. The following compares the expected loading times of three modes—pure remote storage loading, pure local cache loading, and multi-source parallel loading (as per this application)—under different local cache hit rates (network configuration: 1×100Gbps unlimited bandwidth network card; actual measured available remote direct memory access bandwidth is approximately 10 Gigabytes per second; effective bandwidth of the local peripheral component interconnect standard bus is approximately 20 Gigabytes per second): Cache hit rate Pure remote loading time Pure local loading time Multi-source parallel loading time Compared to pure long-range acceleration 0% (No local cache) Approximately 44 seconds not applicable Approximately 44 seconds 1.0× 30% Approximately 44 seconds not applicable Approximately 31 seconds 1.4× 50% Approximately 44 seconds not applicable Approximately 22 seconds 2.0× 67% (optimal point) Approximately 44 seconds not applicable Approximately 15 seconds 3.0× 80% Approximately 44 seconds Approximately 18 seconds Approximately 18 seconds 2.4× 100% (complete hit) Approximately 44 seconds Approximately 22 seconds Approximately 22 seconds 2.0× In the table above, once the cache hit rate exceeds the optimal point (approximately 67%), the local peripheral component interconnect standard bus path becomes the bottleneck, and the loading time is determined by the local path; multi-source parallel loading is not inferior to single-path loading at any hit rate.

[0275] Experiment 2: Network Interface Card (NIC) Utilization Fluctuations and Mode Switching Characteristics. During loading, a sudden change in NIC utilization for the inference service is simulated, and the response behavior of the runtime control module is observed: event Network card utilization changes Regulation and response Response delay Impact of loading throughput Sudden increase in inference load It rose from 40% to 75% (exceeding θ_high=0.7). Switch to protection mode, speed limit reduced to 50%. 1 to 2 acquisition cycles (100 to 200 milliseconds) Remote loading throughput is reduced by approximately 50%, while local paths remain unaffected. Inference load decline It decreased from 75% to 35% (below θ_low=0.4) Switch to acceleration mode and restore full budget. 1 to 2 collection cycles Remote loading throughput restored to budget limit Fluctuations within the hysteresis interval Fluctuating between 45% and 65% Maintain the current model not applicable No fluctuations, remaining stable Experiment 3: Failover Time Statistics. Simulate different storage source failure scenarios and record the time from failure occurrence to successful failover: Fault Scenario Fault detection time Bucket weight distribution time Total switching time Impact on loading progress Distributed memory pool node failure Approximately 150 milliseconds (3 x 50 milliseconds timeout) Tens of microseconds (greedy redistribution) Approximately 200 milliseconds Affected buckets are switched to a remote storage cluster or existing running instances continue loading. Runnable instances have been taken offline. Approximately 3 seconds (3 x 1000 milliseconds timeout) tens of microseconds Approximately 3 seconds Affected buckets switch to other available sources Remote storage cluster timeout Approximately 1.5 seconds (3 x 500 milliseconds timeout) tens of microseconds Approximately 1.5 seconds Affected buckets will be preferentially switched to local cache or distributed memory pool. Experiment 4: Real-world measurement of the decision-making time of the greedy algorithm. The total decision-making time of the allocation algorithm was measured on a standard server processor under different numbers of buckets: The experimental results above show that: multi-source parallel loading can effectively shorten loading time under different cache hit rates; the runtime control module can respond to inference load changes in the hundreds of milliseconds; the fault switching mechanism can complete source switching in the seconds to ensure loading continuity; and the decision overhead of the greedy algorithm is negligible compared to the data transmission time.

[0276] See Figure 9 , Figure 9 This is a structural block diagram of an electronic device provided in an embodiment of this application. In this embodiment, the electronic device 10 includes a memory 101, a processor 100, and a computer program 102 stored in the memory 101 and executable on the processor 100. When the processor 100 executes the computer program 102, it implements the heterogeneous storage-aware weight loading method for large language model inference, the weight loading runtime control method based on real-time network card utilization feedback, and the application layer bucket-level local host memory weight cache management method provided in the above embodiment.

[0277] The electronic device can be a mobile computing device (such as a mobile phone), a desktop computer, a laptop, a PDA, and a cloud server, etc. This electronic device may include, but is not limited to, a processor 100 and a memory 101. Those skilled in the art will understand that... Figure 9This is merely an example of an electronic device and does not constitute a limitation on electronic devices. It may include more or fewer components than shown in the illustration, or combinations of certain components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0278] The processor 100 may be a microcontroller unit (MCU) or a central processing unit (CPU). It may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0279] In some embodiments, memory 101 may be an internal storage unit of an electronic device, such as a hard disk or memory. In other embodiments, memory 101 may be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Furthermore, memory 101 may include both internal and external storage units of the electronic device. Memory 101 is used to store operating systems, applications, bootloaders, data, and other programs, such as program code for computer programs. Memory 101 can also be used to temporarily store data that has been output or will be output.

[0280] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon. These computer-readable program instructions are used to execute the heterogeneous storage-aware weight loading method for large language model inference, the weight loading runtime control method based on real-time network card utilization feedback, and the application-layer bucket-level local host memory weight cache management method described in the above embodiments. The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof. The aforementioned computer-readable storage medium can be included in an electronic device; it can also exist independently, not assembled into an electronic device.

[0281] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0282] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Modules described in the embodiments of this application can be implemented in software or hardware. The names of modules do not, in some cases, constitute a limitation on the unit itself.

[0283] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A heterogeneous storage-aware weight loading method for reasoning in large language models, characterized in that, include: Obtain the non-consistent memory access topology of the local machine, and determine the local machine based on the non-consistent memory access topology. The affinity between the graphics processors and network cards of the computer; The availability status of each bucket in each storage source is detected, and the corresponding storage source is allocated to each bucket based on the effective bandwidth of each storage source, the total bandwidth budget constraint of the network card, the local memory budget constraint, and the source instance interference constraint, with bandwidth efficiency as the priority. Here, a bucket is an independent unit divided by all weight tensors of the target large language model according to a preset bucketing rule, and the effective bandwidth of the storage source is determined by the affinity of the transmission path from the storage source to the image processor's video memory in the non-consistent memory access topology. Create a separate pull thread for each type of allocated bucket's storage source, and each pull thread pulls the weight tensor data in the bucket to the local graphics processor's video memory in parallel.

2. The heterogeneous storage-aware weight loading method for large language model inference as described in claim 1 The method, characterized in that, The method for determining the affinity between each graphics processor and each network interface card (NIC) of the local machine based on non-uniform memory access topology includes: The affinity between each graphics processor and each network card is determined based on whether each graphics processor and each network card are located on the same or different memory access nodes of the local machine, and whether they need to be interconnected across peripheral components using standard switching chips. And / or, the method further includes: The affinity coefficient between each graphics processor and each network interface card (NIC) is determined based on the affinity relationship between each graphics processor and each NIC.

3. The heterogeneous storage-aware weight loading method for large language model inference as described in claim 1 The method is characterized by, The method further includes: Receive a model loading request, wherein the model loading request includes the target model identifier; The target large language model to be loaded and its associated weight configuration information are determined based on the target model identifier. The weight configuration information includes the name of the weight tensor of the target large language model. The weight tensors are arranged in lexicographical order by name, and a greedy binning strategy is used to divide all the weight tensors of the target large language model into K bins, where K is a positive integer.

4. The heterogeneous storage-aware weight loading method for large language model inference as described in claim 1, characterized in that, The method further includes: The total physical bandwidth of the network interface card and the bandwidth utilization of the network interface card for the current inference service are used to calculate the total bandwidth budget of the network interface card that can be used for weight loading.

5. The heterogeneous storage-aware weight loading method for large language model inference as described in claim 1, characterized in that, The method further includes: Obtain the first available transmission bandwidth for each storage source, where the first available transmission bandwidth is the upper limit of bandwidth that the physical path of the storage source itself can provide; Based on the affinity of the transmission paths from each storage source to the image processor's video memory in the non-uniform memory access topology, the first available transmission bandwidth is corrected for topology loss to obtain the second available transmission bandwidth. The effective bandwidth of the storage source is determined at least based on the second available transmission bandwidth. For storage sources whose transmission path is via a network interface card (NIC), the effective bandwidth is also determined based on the share of the total bandwidth budget of the NIC that it shares. For remote storage sources such as running instance memory, the effective bandwidth is also determined in conjunction with the upper limit of the bandwidth that the source instance can send weighted data to.

6. The heterogeneous storage-aware weight loading method for large language model inference as described in claim 1, characterized in that, The method further includes: The key information is synchronized among the weighted loading units participating in parallel loading. The key information includes at least the set of buckets available locally by each weighted loading unit, the local cache hit rate, the effective bandwidth of the local path, and the effective bandwidth of the remote path aggregation. Among them, one weighted loading unit is bound to one graphics processor. Based on the locally available bucket set synchronized by each weighted loading unit, a global locally overwhelmable bucket set is obtained. Remove the globally local overwriteable bucket set from all bucket sets to be loaded to obtain the remote bucket set. The remote buckets in the remote bucket set cannot be loaded from the local storage source on any weighted loading unit and must be pulled from the remote storage source by the cooperation of each weighted loading unit. The remote allocation weight of each weight loading unit is calculated based on the local cache hit rate, local path effective bandwidth, and remote path aggregate effective bandwidth of each weight loading unit. Based on the remote allocation weight of each weighted loading unit and the total data volume of the remote bucket set, determine the amount of remote fetch task data allocated to each weighted loading unit. And / or, the objective function is constructed based on the maximum time taken during the pull phase of all weighted loading units involved in the loading process, and the time taken during the pull phase of each weighted loading unit is determined based on the local pull task data volume, the remote pull task data volume, the effective bandwidth of the local path, and the effective bandwidth of the remote path aggregation for each weighted loading unit.

7. The heterogeneous storage-aware weight loading method for large language model inference as described in claim 1, characterized in that, The method further includes: During the parallel execution of each fetch thread, the availability status of buckets in the application-level local host cache is managed.

8. The heterogeneous storage-aware weight loading method for large language model inference as described in claim 1, characterized in that, The method further includes: During the parallel execution of each fetch thread, the traffic rate of each storage source loading weight is adjusted based on the real-time network card traffic.

9. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored on the memory and executable on the processor, the computer program being configured to implement the heterogeneous storage-aware weight loading method for large language model reasoning as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, performs the heterogeneous storage-aware weight loading method for large language model inference as described in any one of 1 to 8.

Images