Localized large-scale language model service methods and related equipment
By adjusting the five-dimensional parallel strategy in real time, the problem of resource and load mismatch in large-scale language model inference services was solved, achieving the technical effect of optimizing throughput and latency under high availability.
Patent Information
- Application Number
- CN202511400694.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-28
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2045-09-28
AI Technical Summary
Existing large-scale language model inference services cannot adapt to the dynamic computing characteristics at different stages in terms of parallel strategy configuration, resulting in resource and load mismatch, resource waste or performance bottlenecks, and service interruption is required when making adjustments, making it difficult to optimize throughput and latency under high availability requirements.
By acquiring request load metrics in real time, the five-dimensional parallel strategy, including data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism, and expert parallelism, is dynamically adjusted to achieve uninterrupted reconfiguration and ensure precise matching of computing resources and workload.
Maintaining low response latency and high throughput under fluctuating load conditions ensures service continuity, avoids resource waste and communication bottlenecks, and minimizes computing costs.
Smart Images

Figure CN121478468B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of computer and communication technology, and more specifically, to a method and related equipment for localized large-scale language model service. Background Technology
[0002] In large-scale language model inference services, various parallel computing strategies are commonly employed to cope with the enormous computational and memory overhead, such as data parallelism, tensor parallelism, pipelined parallelism, sequence parallelism, and expert parallelism. However, existing large-scale language model inference services suffer from significant technical shortcomings when applying these parallel strategies. Specifically, the combination and degree of parallelism of these strategies are usually statically configured and fixed during service startup or model loading. This static strategy cannot adapt to the dynamic computational characteristics of different stages in the large-scale language model inference process. For example, the pre-filling stage is typically computationally intensive, while the decoding stage is more constrained by communication and memory bandwidth. Fixed parallel strategies inevitably lead to resource and load mismatches when handling tasks with different computational characteristics, resulting in resource waste or performance bottlenecks, making it difficult to simultaneously optimize throughput and latency. Although some existing technologies can achieve single-dimensional dynamic scaling based on data parallelism, for deeper parallel dimensions such as tensor parallelism and pipelined parallelism, adjustments often require service interruption and restart, which is unacceptable for online services requiring high availability. Therefore, how to uniformly and dynamically reconstruct multi-dimensional parallel strategies, including data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism, and expert parallelism, based on real-time changing request load without interrupting service, so that computing resources are always precisely matched with workload characteristics, is a technical problem that urgently needs to be solved in the field of large-scale language model inference services. Summary of the Invention
[0003] The embodiments of this application provide a localized large-scale language model service method and related equipment, which can, at least to a certain extent, reconstruct multi-dimensional parallel strategies, including data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism and expert parallelism, according to the real-time changing request load without interrupting the service, so that computing resources are always precisely matched with the characteristics of the workload.
[0004] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.
[0005] According to one aspect of the embodiments of this application, a method for providing a localized large-scale language model service is provided, comprising: acquiring request load metrics of an inference service in real time; determining a five-dimensional parallel strategy within a computation iteration cycle based on the request load metrics, wherein the five-dimensional parallel strategy includes parallelism in five dimensions: data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism, and expert parallelism; and reconstructing the current parallel execution mode of the service into the five-dimensional parallel strategy without interruption and executing it.
[0006] According to one aspect of the embodiments of this application, a localized large-scale language model service apparatus is provided, the localized large-scale language model service apparatus comprising: a real-time load acquisition module, used to acquire request load indicators of an inference service in real time; a five-dimensional parallel strategy module, used to determine a five-dimensional parallel strategy within a computation iteration cycle based on the request load indicators, the five-dimensional parallel strategy including five dimensions of parallelism: data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism, and expert parallelism; and a non-disruptive reconstructing module, used to non-disruptibly reconstruct the parallel execution mode of the current service into the five-dimensional parallel strategy and execute it.
[0007] According to one aspect of the embodiments of this application, a computer program product is provided, including one or more computer programs that, when executed by one or more processors, implement the steps of the localized large-scale language model service method as described in the above embodiments.
[0008] According to one aspect of the embodiments of this application, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the localized large-scale language model service method as described in the above embodiments.
[0009] According to one aspect of the embodiments of this application, an electronic device is provided, including: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the localized large-scale language model service method as described in the above embodiments.
[0010] In some embodiments of this application, the optimal five-dimensional parallelism combination can be automatically switched under different request modes, including short context, high concurrency, long context, and expert-unbalanced load. The parallelism adjustment process does not affect ongoing inference tasks, ensuring service continuity. By jointly scheduling across multiple dimensions, memory waste or communication bottlenecks caused by excessive scaling in a single dimension are avoided. Maintaining low average response latency and superior throughput under high-fluctuation load scenarios ensures that the system can dynamically switch between multiple parallel dimensions under load fluctuations, minimizing overall computational costs while guaranteeing service quality.
[0011] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description
[0012] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings: Figure 1 A schematic diagram of an exemplary system architecture to which the technical solutions of the embodiments of this application can be applied is shown.
[0013] Figure 2 The illustration shows a flowchart of a localized large-scale language model service method provided in an embodiment of this application.
[0014] Figure 3 The diagram illustrates a flowchart of another localized large-scale language model service method provided in an embodiment of this application.
[0015] Figure 4 It shows according to Figure 3 A flowchart illustrating a specific implementation of step S300 in the localized large-scale language model service method shown in the corresponding embodiment.
[0016] Figure 5 A diagram of a localized large-scale language model service system provided in an embodiment of this application is shown.
[0017] Figure 6 A flowchart of an inference execution method based on sequential parallel communication provided in an embodiment of this application is shown.
[0018] Figure 7 The diagram shows a structural schematic of a localized large-scale language model service device provided in an embodiment of this application.
[0019] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation
[0020] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.
[0021] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.
[0022] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.
[0023] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.
[0024] Figure 1 A schematic diagram of an exemplary system architecture to which the technical solutions of the embodiments of this application can be applied is shown.
[0025] like Figure 1 As shown, the system architecture may include terminal devices (such as...) Figure 1 The device shown includes one or more of a smartphone 101, tablet 102, and portable computer 103 (which could also be a desktop computer, etc.), a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the terminal device and the server 105. The network 104 can include various connection types, such as wired communication links, wireless communication links, etc.
[0026] It should be understood that Figure 1The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, there can be any number of terminal devices, networks, and servers. For example, server 105 could be a server cluster composed of multiple servers.
[0027] Users can interact with server 105 via network 104 using terminal devices to receive or send messages, etc. Server 105 can be a server providing various services. For example, a user can upload request load metrics for inference services to server 105 using terminal device 103 (or terminal device 101 or 102). Server 105 can determine a five-dimensional parallel strategy within the computation iteration cycle based on the request load metrics. The five-dimensional parallel strategy includes parallelism in five dimensions: data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism, and expert parallelism. The current parallel execution mode of the service is then reconstructed without interruption to the five-dimensional parallel strategy and executed.
[0028] It should be noted that the localized large-scale language model service method provided in this application embodiment is generally executed by server 105, and correspondingly, the localized large-scale language model service device is generally located in server 105. However, in other embodiments of this application, the terminal device may also have similar functions to the server, thereby executing the localized large-scale language model service scheme provided in this application embodiment.
[0029] The implementation details of the technical solutions in the embodiments of this application are described in detail below: Figure 2 A flowchart of a localized large-scale language model service method according to an embodiment of this application is shown. This localized large-scale language model service method can be executed by a server, which may be... Figure 1 The server shown. (Refer to...) Figure 2 As shown, this localized large-scale language model service method includes at least: S100, obtains real-time request load metrics for the inference service.
[0030] S200, based on the requested load index, determine a five-dimensional parallel strategy within the calculation iteration cycle. The five-dimensional parallel strategy includes parallelism in five dimensions: data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism, and expert parallelism.
[0031] S300, reconstruct the current parallel execution mode of the service without interruption into the five-dimensional parallel strategy and execute it.
[0032] In this embodiment, the optimal five-dimensional parallelism combination can be automatically switched under different request modes, including short context, high concurrency, long context, and expert-unbalanced load. The parallelism adjustment process does not affect the ongoing inference task, ensuring service continuity. By jointly scheduling across multiple dimensions, memory waste or communication bottlenecks caused by over-expansion of a single dimension are avoided. Maintaining low average response latency and optimal throughput under high-fluctuation load scenarios ensures that the system can dynamically switch between multiple parallel dimensions under load fluctuations, minimizing overall computational costs while ensuring service quality.
[0033] In S100, the policy engine module continuously monitors various online operational metrics of the inference service. These metrics include, but are not limited to, request arrival rate, request feature distribution, and system resource level. Request arrival rate refers to the number of requests per unit time; request feature distribution includes, for example, the distribution of the length of the input sequence and the estimated length of the output sequence; and system resource level includes, for example, the GPU memory utilization, computing power utilization, and inter-node communication bandwidth of each computing instance.
[0034] The strategy engine analyzes this real-time data to accurately determine whether the current service is in a pre-filling intensive or decoding intensive state. Decoding intensive services include, for example, multi-turn dialogues and low-concurrency streaming generation, while pre-filling intensive services include, for example, processing long texts and high-concurrency batch processing.
[0035] It's important to note that the strategy engine assesses the overall resource status of all current computing instances, including remaining GPU memory, compute load, and available network bandwidth. Simultaneously, it examines the requests in the waiting queue and selects a subset of requests based on the length of their input text and a predicted output length. The selection principle prioritizes maximizing overall processing efficiency without exceeding current system resource constraints, while minimizing the risk of subsequent requests being blocked for extended periods due to a single long request or a large batch of requests. This is the first step in request processing, essentially acting as an intelligent request gateway. The system doesn't simply process all queued requests in a first-come, first-served manner; instead, it comprehensively considers the current system state and the characteristics of the requests to determine which requests can proceed to the next stage.
[0036] That is, during operation, real-time metrics related to requests are continuously collected. These metrics include request arrival rate (QPS), input context length distribution, target token length distribution, time ratio of pre-filling stage to decoding stage, memory usage of computing instances, bandwidth utilization, and communication latency. These metrics serve as the basic data input for dynamically adjusting the parallelism.
[0037] Before executing S200, this application will also perform instance allocation. That is, after determining the subset of requests to be processed, specific computing resources need to be allocated to this subset.
[0038] Specifically, the system first searches for completely idle compute instances and prioritizes allocating requests to them. If no completely idle instances are found, the system initiates a warm migration mechanism. It checks all instances currently processing decoding tasks and identifies the one with the least amount of GPU memory currently used for key-value cache. Then, the system performs a token-level migration, moving a small portion of its key-value cache data to other instances, quickly freeing up enough space to pre-populate new requests. This migration is small-scale and non-blocking, hence the name "migration."
[0039] After the resources are allocated, an optimal parallel execution plan needs to be determined for this batch of requests. At this point, S200 is executed.
[0040] In S200, within a computational iteration cycle, for example in batch processing units, a five-dimensional parallel strategy adapted to the current request is determined based on real-time request load metrics. The five-dimensional parallel strategy includes parallelism in five dimensions: data parallelism (DP), tensor parallelism (TP), pipeline parallelism (PP), sequence parallelism (SP), and expert parallelism (EP).
[0041] Among these parameters, data parallelism determines how request batches are distributed across multiple machines; tensor parallelism determines how tensors are partitioned across different devices for single-layer computation; pipeline parallelism determines how model layers are distributed across multi-machine pipelines for concurrent computation; sequence parallelism determines the segmented computation of long sequence tokens across different devices; and expert parallelism (in hybrid expert models, it determines how expert modules are distributed across devices). Through the joint optimization of these five parameters, the system can dynamically select the optimal combination under different load conditions. For example, in short-context, high-QPS scenarios, DP is prioritized, while in long-context scenarios, SP and TP are prioritized to ensure performance.
[0042] Specifically, the requests are first clustered based on the similarity of their input and output lengths, forming different length clusters. For each length cluster, the unified parallel orchestrator performs a complex computational solution. It attempts different combinations of data parallelism, tensor parallelism, pipelined parallelism, sequence parallelism, and expert parallelism, and, considering different batch sizes, uses a cost model to predict which combination will achieve the minimum latency per iteration. The goal of this process is to find the most efficient parallel execution method for this batch of requests given the current resources.
[0043] For example, in high-concurrency pre-filling scenarios, the depth of tensor parallelism and pipelined parallelism might be increased to improve computational efficiency; while in decoding-intensive scenarios, tensor parallelism or pipelined parallelism might be reduced to lower communication overhead, and the number of data parallel replicas might be increased to improve total throughput. Ultimately, the system will output a specific combination of five-dimensional parallelism.
[0044] Specifically, in some embodiments, the specific implementation of step S200 can be found in the following embodiments. This embodiment is based on... Figure 2 According to the detailed description of step S200 in the localized large-scale language model service method shown in the corresponding embodiment, step S200 may include the following steps: Based on the request load metrics and the current resource status of the computing instance, the optimal combination of five-dimensional parallel strategies is calculated using a revenue-cost model according to the optimization objective.
[0045] In this embodiment, a unified benefit-cost model is used to compare the combinations of five parallel dimensions, avoiding only local optimization and dynamically selecting the optimal solution based on different load characteristics (short context high concurrency, long context low concurrency, MoE expert hotspots, etc.). By introducing the real-time state of computing instances, the parallel strategy selection avoids relying solely on static configuration. The benefit-cost model can be extended with different weights or objectives to adapt to latency-first, energy-first, or mixed objective scenarios.
[0046] Specifically, within the computation iteration cycle, not only are real-time request load metrics for the inference service acquired, but the current resource status of the computing instances is also collected. This includes GPU memory capacity and current utilization, computing core utilization, communication bandwidth utilization and link latency, local cache hit rate, and load balancing between instances. These data, along with the request load metrics, serve as inputs to the revenue-cost model. The computing core utilization can be the SM utilization rate of the GPU or Tensor Processing Unit (TPU).
[0047] Meanwhile, the aforementioned optimization objectives may include, but are not limited to, latency objectives, throughput objectives, energy consumption objectives, and fairness objectives, which can be determined according to the service scenario. Among them, the latency objective is to reduce the average response time or P99 latency of requests; the throughput objective is to maximize the number of tokens completed per unit time; the energy consumption objective is to reduce the total energy consumption of GPUs and TPUs; and the fairness objective is to ensure that the performance of requests from different users or with different priorities is not unbalanced.
[0048] The aforementioned benefit-cost model can quantitatively evaluate different five-dimensional parallel combinations, which can specifically include a benefit function R, a cost function C, and an objective function F = RC.
[0049] Here, the benefit function R represents the performance improvement under a certain parallel combination, which is calculated by weighting latency improvement, throughput increase, and energy consumption reduction. The cost function C represents the computation and communication costs when switching to this combination, including memory usage, communication overhead, and load migration costs. The objective function F=RC comprehensively measures the actual benefits of different combinations.
[0050] It should be noted that the revenue function R consists of three parts: latency revenue RL, throughput revenue RT, and energy consumption revenue RE; the cost function C consists of three parts: communication cost CC, migration cost CM, and memory cost CV, which conform to the following formula: RL = (Baseline delay – Current delay) / Baseline delay RT = (Current throughput - Baseline throughput) / Baseline throughput RE = (Baseline Energy Consumption - Current Energy Consumption) / Baseline Energy Consumption R = α·RL + β·RT + γ·RE CC = Link Delay × Data Transmission Volume CM = Key-value data migration time CV = Additional video memory usage / Total video memory capacity C = CC + CM + CV Where α, β, and γ are weighting coefficients.
[0051] Specifically, when calculating the optimal five-dimensional parallel strategy combination, the process first substitutes each candidate combination of data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism, and expert parallelism into the benefit-cost model to calculate the corresponding objective function value F. Under the premise of ensuring service constraints, the combination with the largest objective function value is selected as the optimal five-dimensional parallel strategy within the iteration cycle. It should be noted that service constraints include, for example, preventing memory overflow and ensuring communication latency does not exceed a threshold. The final optimal five-dimensional parallel strategy combination is applied to subsequent inference processes and recalculated in the next iteration cycle to achieve dynamic optimization.
[0052] In other embodiments, hardware characteristics of different GPUs and TPUs can be collected, including peak computing power, memory size, bandwidth performance, etc. When calculating candidate strategies, the benefit-cost model introduces a hardware weighting factor so that the strategy can be matched with the hardware characteristics. For example, tensor parallelism is prioritized on high-bandwidth GPUs and sequence parallelism is prioritized on GPUs with large memory. This mechanism ensures that different hardware clusters can obtain optimal performance under a unified framework.
[0053] In S300, once the five-dimensional parallel strategy is determined, the currently running parallel execution mode is directly switched to the five-dimensional parallel strategy. During the switching process, there is no need to restart the inference process, and the existing computation state and key-value cache data are preserved without loss, ensuring uninterrupted continuous response to requests. This refactoring mechanism enables the service to adaptively adjust to load fluctuations and always maintain optimal operating conditions.
[0054] In some embodiments of this application, such as Figure 3 As shown, prior to S300, the localized large-scale language model service method further includes: S400 constructs a unified virtualized key-value cache pool spanning multiple compute instances, where key-value cache data is managed with pagination and indexing at the token granularity to support cross-compute instance referencing.
[0055] In this embodiment, key-value data is paginated at the token level and shared across instances, avoiding the high overhead of overall migration. When adjusting the parallel strategy, the parallel mode can be switched without moving the entire cache data. The cross-instance cache pool enables on-demand sharing of GPU memory, reducing resource waste caused by cache occupancy in a single instance. The application layer and scheduling layer do not need to be aware of the underlying distribution, reducing system complexity.
[0056] Specifically, a logically unified key-value cache pool is first abstracted among multiple computing instances. The local memory space of each instance is virtualized as part of the cache pool. Through a unified address mapping table, each instance can access the entire cache pool at the logical level without being aware of the underlying physical distribution.
[0057] The key-value cache data is paginated at the token granularity. Each pagination unit corresponds to a set of token key-value representations. A unique logical index number is assigned to each pagination unit. The storage location and the computing instance to which the pagination unit belongs are recorded through the index table to achieve fast location and access.
[0058] When a computing instance needs to access key-value data that is not local, it can find the target instance through the index table and initiate a call. The cache pool supports direct references across instances, avoiding overall data migration. Data access is mainly based on on-demand retrieval, reducing redundant transmission.
[0059] This embodiment provides a unified cache pool interface for inference services, including operations such as allocation, querying, and recycling. The user side and the upper-layer scheduling logic do not need to know which specific computing instance the data is stored in, thus achieving transparent management.
[0060] Following the above steps, the localized large-scale language model service method further includes driving a recycling strategy based on the priority of requests, the access frequency of key-value cache pages, and their lifecycle.
[0061] In this embodiment, the eviction policy can be driven by request priority, access frequency of key-value cache pages, and lifecycle. Furthermore, a tiered storage mechanism can be introduced. For example, newly generated, frequently accessed tokens can be temporarily cached in a low-bit format (e.g., FP8 / INT4) in the high-speed cache, while less frequently accessed key-value cache pages can be compressed and moved to slower but larger host memory, thereby maximizing effective memory capacity. This embodiment avoids unlimited cache pool growth through lifecycle and access frequency management, ensuring high-priority request caches are retained and guaranteeing low latency and stability for critical business operations. Tiered management of hot and cold data reduces redundant usage and improves cache space reuse. Automated execution of the cache eviction policy avoids manual intervention that could interrupt inference tasks.
[0062] Specifically, this embodiment maintains metadata information for each cached page in the virtualized cache pool, including: request identifier, priority level, access count, last access time, lifecycle start time, and expected expiration time. The cache pool is periodically scanned to collect the above metadata and update statistical information for subsequent eviction decisions.
[0063] When the cache pool space is insufficient, cached pages corresponding to high-priority requests are retained first, while cached pages of low-priority requests are reclaimed first if resources are scarce, in order to ensure the computing needs of critical tasks.
[0064] The frequency of access to cached pages determines their hotness or coldness. Pages with high access frequency are marked as hot data and are retained with priority, while cold data that has not been accessed for a long time is put into a candidate recycling list and is released with priority when space is insufficient.
[0065] Each cached page is configured with lifecycle parameters when it is generated, including minimum retention time and maximum lifespan. Cached pages that exceed the lifecycle limit are automatically entered into the recycling queue. The lifecycle strategy ensures that data will not be prematurely cleaned up due to extremely low access frequency, nor will new requests be blocked due to long-term data occupation.
[0066] When the memory pressure is high, the system comprehensively considers three factors: priority, access frequency, and lifespan to calculate the priority of cache page reclamation. Under the premise of ensuring the stability of high-priority requests, low-value cache pages are released first. After reclamation is completed, the virtualization cache pool index table is updated to ensure cache consistency and traceability.
[0067] It should be noted that the virtualized key-value cache pool described above employs a multi-level caching mechanism. The first-level cache is stored in the local instance's video memory, resulting in the lowest access latency. The second-level cache is stored in a shared cache pool across instances, with slightly higher latency, but it supports on-demand cross-instance access. Third-level cache overflow data is stored in host memory or high-bandwidth storage devices for storing cold data in long-context scenarios. During the decoding phase, the system prioritizes accessing the first-level cache. If a cache miss occurs, it accesses the second and third-level caches through the index table, ensuring cache utilization and storage scalability.
[0068] Specifically, in some embodiments, the specific implementation of step S300 can be found in [reference needed]. Figure 4 . Figure 4 It is based on Figure 3 According to the detailed description of step S300 in the localized large-scale language model service method shown in the corresponding embodiment, step S300 may include the following steps: S310, In the pre-filling stage, parallelism reduction is performed, and the generated key-value cache data is directly distributed to the target computing instance according to the five-dimensional parallel strategy. The target computing instance is the computing instance to which the decoding stage is ultimately assigned.
[0069] S320, during the decoding phase, multi-host multi-shard decoding is performed on each target computing instance, and the local attention computing results are obtained as needed through the virtualized key-value cache pool.
[0070] In this embodiment, key-value cache data is directly placed into the target instance during the decoding stage, eliminating the need for redistribution or copying during stage switching. This embodiment employs a multi-host, multi-shard decoding mode, effectively distributing computational pressure during long sequence generation. Simultaneously, relying on a virtualized key-value cache pool, the decoding instance can directly access the required data when needed, avoiding overall data migration. By combining reduced pre-filling parallelism with distributed decoding, end-to-end latency is significantly shortened. Adapted parallel strategies are used in both the pre-filling and decoding stages to improve overall GPU resource utilization efficiency.
[0071] In S310, parallelism reduction is performed during the pre-filling phase. It reuses existing communication paths and hardware resources from the sequential parallel computation process, writing the generated key-value cache data directly and sequentially to the target instance group specified for the subsequent decoding phase of the request, achieving zero additional migration overhead. Specifically, the system reuses the hardware links and data flows of communication operations such as All-Reduce or All-Gather inherent in the sequential parallel computation process. At the final stage of these communications, the unified parallel orchestrator controls the data flow, distributing and writing the key-value cache data generated for each token directly to the GPU memory of the target computing instance that will perform the decoding task, rather than simply aggregating it, according to the parallel strategy determined in the subsequent decoding phase. This avoids a separate, large-scale migration of key-value cache data after the pre-filling computation is completed, thus achieving zero additional migration.
[0072] In the model pre-filling stage, the input context is typically long, and the computational load is concentrated on self-attention and forward propagation operations. According to the five-dimensional parallel strategy, parallelism contraction is performed, meaning that higher sequence parallelism and tensor parallelism are used in this stage to distribute the long sequence computation across multiple computational instances. During computation, the system synchronously generates corresponding key-value caches. The generated key-value cache data is no longer centrally stored but is instead distributed sequentially through the sequence-parallel communication link, directly to the target computational instance specified by the five-dimensional parallel strategy. This target computational instance is the one ultimately responsible for performing inference in the decoding stage, thus avoiding subsequent additional migrations.
[0073] Specifically, in some embodiments, the specific implementation of step S310 can be found in the following embodiments. This embodiment is based on... Figure 4 According to the detailed description of step S310 in the localized large-scale language model service method shown in the corresponding embodiment, step S310 in the localized large-scale language model service method may include the following steps: In response to a request, perform sequential parallel computations to generate key-value cache data.
[0074] By utilizing the communication links in the sequence parallel computing process, the generated key-value cache data is directly distributed and stored on the target computing instance.
[0075] In this embodiment, the KV cache is directly distributed to the decoding target instance upon generation, eliminating the need for secondary data transfer during the switch between the pre-filling and decoding phases. The communication links in the parallel sequence computation are reused, eliminating the need to establish new transmission channels and reducing communication overhead. KV data can be placed in the target location during the pre-filling phase and directly invoked in-place during the decoding phase, shortening startup latency. The virtualized cache pool index update mechanism ensures data location transparency and improves system management efficiency.
[0076] Specifically, in the pre-filling stage, the input long sequence is divided into multiple sequence segments, and sequence-parallel computation is performed using multiple computation instances according to a five-dimensional parallel strategy. During the computation process, the intermediate results corresponding to each sequence segment generate key-value cache data, including key tensors and value tensors. The key-value cache data carries index information during generation to identify its position in the complete context sequence.
[0077] During sequence-parallel computation, the system needs to transfer intermediate results of sequence segments between different computation instances to ensure the continuity of forward propagation and attention computation. This embodiment utilizes this communication link to directly distribute the generated key-value cache data to the target computation instance, which is the final destination instance for the decoding stage predetermined in the five-dimensional parallel strategy. During distribution, the system does not need to perform additional data migration; instead, it writes the key-value cache to the target instance's local storage along the way. After distribution, the target instance registers the key-value cache data to a virtualized key-value cache pool for subsequent decoding steps to access as needed.
[0078] After receiving the key-value cache data, the target computing instance stores it in its local video memory space. Through the index management mechanism of the virtualization cache pool, it updates the key-value paging index table and marks the current instance location where the key-value data is stored. This index update ensures that any computing instance can locate and access the corresponding data through the cache pool interface during the subsequent decoding stage.
[0079] In the traditional model, when switching from pre-filling to decoding, if the parallel strategy changes (e.g., from 4-GPU tensor parallelism to 2-GPU tensor parallelism + 2-way data parallelism), the following blocking process occurs: computation, service pause, global migration, and service resumption. Specifically, during computation, four GPUs perform pre-filling calculations, with each GPU storing a portion of the key-value cache. When the service is paused, the system suspends processing of that request. During global migration, a separate data migration task is initiated to collect the KV caches from the four GPUs, and then re-split and distribute them to the target GPUs according to the new 2+2 strategy. This process is time-consuming and does not generate any effective computation, making it the main cause of service interruptions. When the service resumes, the decoding phase begins after the data is properly stored.
[0080] Before prefilling begins, the Unified Parallel Orchestrator (UPO) has already calculated the optimal parallel strategy and target GPU group for the subsequent decoding stage based on load prediction. During prefilling computation, especially when using sequential parallelism, GPUs naturally need to communicate to exchange intermediate computation results (e.g., after attention computation followed by an All-Reduce operation). This solution cleverly reuses this existing communication link. When a GPU computes the key-value cache for a token, it doesn't simply store it locally. Instead, according to pre-issued instructions from the UPO, the GPU directly sends this key-value cache data to its final destination GPU in the subsequent decoding stage during the sequential parallel communication process. Therefore, when the entire prefilling computation task ends, all requested KV cache data has been distributed smoothly, accurately, and according to the parallel strategy required for the decoding stage, landing on the correct target GPUs. The system can seamlessly proceed to the first step of decoding without any pauses or additional data migration. In other words, this application transforms the serial mode of computation followed by migration into a pipeline mode of computation and migration simultaneously, thereby eliminating interruptions.
[0081] In S320, during the decoding phase, the system performs multi-host, multi-shard decoding. If additional computing power is needed, new computing instances can be dynamically added. The new instance does not need to load the entire historical key-value cache for the request; it only needs to receive the query vector for the current decoding step and some model layer parameters necessary for computation from the master instance. The new instance performs attention computation locally in parallel and then returns the results to the master instance. This communication process can overlap with the computation process to hide communication latency. The key to this step is that the new instance does not need to load or replay the entire historical key-value cache for the request from the beginning. It only needs to receive the query vector for the current decoding step and the model layer weight parameters necessary for performing attention and feedforward network computations via the network. The new instance completes some computations locally (such as local attention and FFN layer computations) and then returns the results to the master instance for aggregation. The entire process achieves plug-and-play computing instances with extremely low scaling overhead.
[0082] That is, when entering the decoding stage, the newly generated tokens are sequentially input into the model for step-by-step reasoning. The decoding process adopts a multi-host, multi-shard approach, where different computing instances share the decoding computation of a part of the model layer or part of the sequence. Each target computing instance obtains the key-value data required for local attention computation as needed by calling the virtualized key-value cache pool. If the target instance does not have the required key-value data locally, it can directly access the data stored in the remote instance through the cross-instance indexing mechanism without the need for overall migration. After the computation is completed, each instance sends the attention computation results and the corresponding intermediate tensors back to the assigned decoding group to ensure the consistency of the output tokens.
[0083] After each round of decoding, the results calculated by each instance are collected and concatenated to form a complete output token. The key-value representation of the new token is immediately appended to the virtualized key-value cache pool so that subsequent decoding steps can continue to call it. The five-dimensional parallel strategy remains unchanged throughout the entire decoding process until the request is completed.
[0084] Specifically, in some embodiments, the specific implementation of step S320 can be found in the following embodiments. This embodiment is based on... Figure 4 According to the detailed description of step S320 in the localized large-scale language model service method shown in the corresponding embodiment, step S320 may include the following steps: Dynamically add the new compute instance to the currently requested parallel compute group.
[0085] Through the virtualized key-value cache pool, the new computing instance receives the query vector of the current decoding step and the necessary model layer weights, participates in local attention calculation, and sends the calculation results back.
[0086] In this embodiment, new computing instances can be added at any time without restarting the inference process, improving system flexibility. Simultaneously, new instances acquire data on demand through a virtualized cache pool, avoiding the complete migration of historical key-value pairs and reducing memory and communication overhead. Furthermore, new instances are plug-and-play, ensuring uninterrupted decoding and maintaining request continuity. Multi-host, multi-shard decoding achieves horizontal scaling with the addition of new instances, effectively improving overall computing power.
[0087] Specifically, when the request load continues to increase during the decoding phase, or when the existing computing instance resources are insufficient, the scheduler can add one or more new computing instances to the current request's parallel computing group. When a new instance is added, there is no need to interrupt the existing decoding task. The parallel group management mechanism automatically allocates sharded tasks to it. The process of adding a new instance is completed entirely online and does not affect the continuity of the decoding process.
[0088] After a new compute instance joins the parallel group, it obtains the query vector required for the current decoding step through the virtualized key-value cache pool interface. Simultaneously, the scheduler distributes the necessary model layer weights corresponding to this decoding step to the new instance. Because the virtualized cache pool maintains a global index, the new instance can directly locate and access the key-value cache data required for its assigned shard on demand, without needing to migrate the entire historical cache.
[0089] The new instance uses the received query vector, weight parameters, and key-value data provided by the virtualization cache pool to perform local attention computation. After completion, the new instance sends the result tensor back to the decoding parallel group to generate the predicted distribution of the current token. The returned result is aggregated with other shard results within the decoding parallel group to ensure the consistency of the output tokens.
[0090] After completing the computation, the new instance stores the newly generated key-value data locally and registers it in the virtualization cache pool. The index table is updated in real time with the new storage location of the key-value data to ensure that subsequent decoding steps can directly access it.
[0091] In some embodiments of this application, when the system detects fluctuations in the distribution of request context lengths (e.g., a sudden increase from an average of 1K tokens to 8K tokens), a strategy adjustment is triggered. If the proportion of the pre-filling and decoding stages exceeds a set threshold (e.g., the pre-filling proportion is greater than 70%), the strategy favors increasing sequence parallelism and tensor parallelism; if the decoding stage accounts for a high proportion, the strategy favors increasing data parallelism and expert parallelism to improve throughput. Through this adaptive switching mechanism, the system can quickly adjust its strategy under load fluctuations to maintain stable performance.
[0092] Before implementing S320, state virtualization needs to be implemented. The core of this is a unified virtualized key-value cache pool. All requested key-value caches are treated as a logical whole, with pagination and indexing at the token level, and physically distributed across any GPU instance. Instances can efficiently access each other's key-value cache pages on demand.
[0093] When a compute instance needs to be added (e.g., scaling up from 2 cards to 3 cards) to accelerate decoding, the new instance does not need to load the entire historical key-value cache for that request. This is a significant overhead saving. The master instance only needs to broadcast the query vector Q for the current decoding step to the new instance. Using this Q vector, the new instance retrieves the necessary historical key-value data pages for calculating the attention score on demand through a virtualized key-value cache pool. The new instance completes its assigned portion of the computation (e.g., a portion of the attention head or FFN layer computation) and sends the results back to the master instance for aggregation.
[0094] The new instance can be put into computation of the next token almost immediately, and the entire scaling process is smooth and non-blocking for the service. Because the state (key-value cache) is globally virtualized and accessible on demand, large-scale state migration is avoided, thus enabling uninterrupted parallel reconstruction. The process of removing an instance is similar; UPO simply needs to stop assigning tasks to it.
[0095] In other words, S320 transforms the heavy operation of global state reconfiguration into a light operation of adding a stateless computing unit and accessing shared state on demand, thereby achieving uninterrupted operation.
[0096] In summary, in S300, pre-filled parallelism reduction solves the problem of switching parallel strategies across stages from one computationally intensive stage to another communication-intensive stage; decoding multi-host decoding solves the problem of continuous and dynamic adjustment of parallel resources within the same computation stage due to load changes.
[0097] In some embodiments, when the large language model is a hybrid expert model, the following steps are also performed in parallel with step S300: Monitor the load of each expert in the hybrid expert model in real time.
[0098] When it is detected that the load of a certain expert is consistently higher than a preset threshold, the weight corresponding to that expert is copied or migrated online to one or more computing instances with lower load, and the request routing is adjusted to balance the load among experts.
[0099] In this embodiment, real-time monitoring and dynamic governance mechanisms prevent individual experts from becoming performance bottlenecks. Expert weight replication or migration is completed during runtime, without affecting the normal execution of inference requests. Dynamic routing table adjustments distribute requests across multiple expert replicas, reducing average latency. Replicas expand under high load and are recycled under low load, ensuring efficient and elastic use of computing resources. Specifically, the invocation frequency and computational load of each expert are monitored in real time. A request processing quota within an iteration cycle can be set for each expert using a gated budget vector, quickly suppressing hotspots and preventing a single expert from slowing down the entire batch. When the load of an expert consistently exceeds a preset threshold, the unified parallel orchestrator takes more proactive measures. It replicates the model weights corresponding to the hotspot expert online to one or more idle computing instances with lower loads and updates the routing table, diverting some subsequent requests destined for that expert to new replicas. When the system load is low, online migration can also be performed to optimize the long-term physical layout of experts and balance the load across computing instances.
[0100] Specifically, during operation, load metrics for each expert in the hybrid expert model are continuously collected, including the number of currently received requests, average token processing rate, memory usage and bandwidth utilization, and P99 request latency. These metrics are statistically analyzed within the calculation iteration cycle to create a load profile for each expert. When the load of an expert consistently exceeds a preset threshold, subsequent governance operations are triggered. It should be noted that a consistently higher-than-preset-threshold load for an expert is, for example, when its average QPS percentage exceeds twice the global average.
[0101] When a hot expert is detected, the system scheduler can alleviate the load using two strategies: replication and migration. Specifically, the replication strategy copies the weight corresponding to the expert to one or more low-load computing instances, forming multiple replicas; the migration strategy migrates the entire weight of the expert to an idle or low-load computing instance.
[0102] Weight data is distributed via distributed storage or high-speed communication links. New replica instances become available immediately after weight loading is complete, while existing compute instances continue to provide services until the routing policy is updated. In other words, weight replication or migration is completed online and does not affect the current inference service.
[0103] Update the routing table of the hybrid expert gating module or scheduler. For newly arriving requests, dynamically adjust the routing ratio according to the number of expert replicas and computing resources to ensure that expert requests that were originally concentrated in the load are evenly distributed to multiple replica instances.
[0104] When the load on expert replicas tends to be balanced and the overall system is under low load, redundant expert replicas are gradually reclaimed. The reclamation strategy is executed based on the replica's utilization rate and lifecycle parameters to avoid long-term resource idleness.
[0105] In some embodiments of this application, during strategy switching, if a new combination of parallelism is detected that causes memory overflow or communication latency to exceed a threshold, the switching is immediately interrupted, and the system reverts to the previous version of the five-dimensional parallel strategy to ensure uninterrupted requests. The rollback mechanism is implemented by saving the strategy parameters from the most recently stable run, providing rapid recovery capabilities.
[0106] like Figure 5 As shown, this embodiment provides a localized large-scale language model service system, which includes: an API gateway, a policy controller, a unified parallel orchestrator, and multiple elastic instances. These modules work together to dynamically determine a five-dimensional parallel strategy based on the real-time request load metrics of the inference service, and to reconstruct the parallel execution mode of the service without interruption without restarting.
[0107] The API gateway is set up at the system entry point to receive external inference requests and perform functions such as authentication and priority determination, request forwarding, and interface management.
[0108] Specifically, authentication and priority determination involves verifying the identity of the caller to confirm the legitimacy of the request and determining the priority of the request based on a preset service level policy; request forwarding involves sending authenticated requests to the policy controller in priority order; and interface management supports multiple access protocols and provides a unified access interface to ensure the scalability and security of the inference service.
[0109] The policy controller receives request information from the API gateway and collects real-time load metrics for the inference service, including request arrival rate, input sequence length, token throughput, latency metrics, and current compute instance resource usage. The policy controller's functions include target constraint management, scheduling decision generation, and feedback monitoring.
[0110] Among them, target constraint management determines the current system optimization direction based on service level targets, cost and energy consumption constraints; scheduling decision generation combines load indicators and optimization targets to generate a global scheduling plan of five-dimensional parallel strategy and sends the plan to a unified parallel orchestrator; feedback monitoring monitors the actual execution results after the strategy is issued and triggers a rollback mechanism when an anomaly occurs.
[0111] The unified parallel orchestrator is the core scheduling module of the system. Its function is to transform the global scheduling plan issued by the policy controller into an executable parallelism configuration. Its main functions include: We jointly analyze the parallel requirements across five dimensions: data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism, and expert parallelism.
[0112] Parallel causal graphs are used to model the dependencies and conflicts between different dimensions, ensuring feasibility at the execution level.
[0113] Based on the optimization objective, the five-dimensional parallelism is weighted according to priority, and the optimal solution that balances benefit and cost is selected from multiple candidate strategies.
[0114] The parsed five-dimensional parallelism configuration is distributed to multiple elastic instances to achieve uninterrupted service reconstruction.
[0115] The system contains multiple elastic instances, each consisting of an execution core, a communication core, and a virtualized key-value cache pool.
[0116] The execution core is responsible for inference computation, including matrix multiplication, attention calculation, and feedforward network operations, ensuring the model runs correctly on the instance. The communication core is used for data exchange between instances, especially in tensor parallelism, sequence parallelism, and expert parallelism, supporting efficient communication across multiple hosts and shards. The virtualized key-value cache pool manages paging and indexing at the token level, supporting shared access across instances. During inference, key-value cache data generated in the pre-filling stage can be directly allocated to the target instance and referenced as needed in the decoding stage, avoiding latency and memory overhead caused by overall migration.
[0117] When a request arrives, the API gateway completes authentication and priority determination before passing the request to the policy controller. The policy controller generates a five-dimensional parallel policy based on the real-time load. The unified parallel orchestrator parses and optimizes the policy and distributes the configuration to multiple elastic instances. The elastic instances execute inference computations according to the allocated parallelism and collaborate with the communication core and virtualized key-value cache pool to achieve efficient cross-instance collaboration. The entire process requires no service interruption, and the system can dynamically reconstruct the five-dimensional parallelism within an iteration cycle, thereby achieving low-latency, high-throughput, and high-resource-utilization inference services under localized deployment conditions.
[0118] like Figure 6 As shown, this embodiment provides an inference execution method based on sequence parallel communication, which mainly includes the following steps: First, the system enters the parallel communication phase, in which the requested input sequence is divided into multiple segments and distributed to different computing instances for parallel processing. Each computing instance only retains key-value cache data corresponding to its local sequence segment.
[0119] Secondly, the system predicts the decoding target based on the context length and load characteristics of the input request. During the prediction process, the scheduling module estimates the computational pressure and cache requirements that may be generated in subsequent decoding stages, thereby providing a basis for adjusting parallelism and cache layout.
[0120] Subsequently, the system selects the storage location for the key-value cache along the communication path. Specifically, the key-value cache data generated during the pre-filling stage is selectively written to disk or distributed at the token granularity in the sequential parallel communication link, enabling the cached data to be directly referenced by the target computing instance during the decoding stage, avoiding large-scale migration.
[0121] After the cache layout is complete, the system performs parallelism reduction. That is, at the end of the pre-filling stage, the original high parallelism is reduced to a lower parallelism suitable for the decoding task. In this way, computing resources can be fully utilized in the pre-filling stage, while communication overhead can be reduced in the decoding stage, achieving a balance between latency and throughput.
[0122] Finally, the decoding-ready stage begins. In this stage, each computing instance collaborates to complete attention calculations and subsequent generation steps by accessing a pre-laid key-value cache, thereby ensuring the continuity of the inference process and the efficient use of computing resources.
[0123] Through the above embodiments, the system can complete the prediction and cache layout of the decoding target in the pre-filling stage based on the parallel communication of the sequence, and realize the dynamic reduction of parallelism when switching stages, thereby effectively reducing the additional overhead caused by key-value cache migration and improving the performance and stability of localized inference of large-scale language models.
[0124] The following describes an apparatus embodiment of this application, which can be used to execute the localized large-scale language model service method in the above embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the localized large-scale language model service method described above.
[0125] Figure 7 A block diagram of a localized large-scale language model service apparatus according to an embodiment of this application is shown.
[0126] Reference Figure 7 As shown, a localized large-scale language model service device 500 according to an embodiment of this application includes a real-time load acquisition module 510, a five-dimensional parallel strategy module 520, and an uninterrupted reconstruction module 530.
[0127] The load acquisition module 510 is used to acquire the request load index of the inference service in real time; the five-dimensional parallel strategy module 520 is used to determine the five-dimensional parallel strategy within the calculation iteration cycle based on the request load index. The five-dimensional parallel strategy includes five dimensions of parallelism: data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism, and expert parallelism; and the uninterrupted reconstruction module 530 is used to reconstruct the current parallel execution mode of the service into the five-dimensional parallel strategy without interruption and execute it.
[0128] In some feasible embodiments of this application, the five-dimensional parallel strategy module 520 specifically includes: a benefit-cost model submodule, which is used to calculate the optimal combination of five-dimensional parallel strategies based on the request load index and the current resource status of the computing instance, according to the optimization objective, through the benefit-cost model.
[0129] In some feasible embodiments of this application, the localized large-scale language model service apparatus further includes: The virtualized cache pool module is used to build a unified virtualized key-value cache pool that spans multiple compute instances. Key-value cache data is managed by paging and indexing at the token granularity to support references across compute instances.
[0130] In some feasible embodiments of this application, the uninterrupted reconstruction module 530 specifically includes: a parallelism shrinking submodule, used to perform parallelism shrinking during the pre-filling stage, directly distributing the generated key-value cache data to the target computing instance according to a five-dimensional parallel strategy, wherein the target computing instance is the computing instance to which the decoding stage ultimately belongs; and a multi-host decoding submodule, used to perform multi-host multi-shard decoding on each target computing instance during the decoding stage, and to obtain local attention calculation results as needed through the virtualized key-value cache pool.
[0131] In some feasible embodiments of this application, the parallelism reduction submodule specifically includes: a key-value cache data generation unit, used to generate key-value cache data in response to a request to perform sequential parallel computation; and a key-value cache data distribution unit, used to directly distribute and store the generated key-value cache data on the target computing instance using the communication link during the sequential parallel computation process.
[0132] In some feasible embodiments of this application, the multi-host decoding submodule specifically includes: an instance dynamic addition unit, used to dynamically add a new computing instance to the currently requested parallel computing group; and a vector weight receiving unit, used to enable the new computing instance to receive the query vector of the current decoding step and the necessary model layer weights through the virtualized key-value cache pool, participate in local attention calculation, and return the calculation results.
[0133] In some feasible embodiments of this application, the localized large-scale language model service device further includes: a recycling strategy driving module, used to drive the recycling strategy according to the priority of the request, the access frequency of the key-value cache page, and the life cycle.
[0134] In this embodiment, the optimal five-dimensional parallelism combination can be automatically switched under different request modes, including short context, high concurrency, long context, and expert-unbalanced load. The parallelism adjustment process does not affect the ongoing inference task, ensuring service continuity. By jointly scheduling across multiple dimensions, memory waste or communication bottlenecks caused by over-expansion of a single dimension are avoided. Maintaining low average response latency and optimal throughput under high-fluctuation load scenarios ensures that the system can dynamically switch between multiple parallel dimensions under load fluctuations, minimizing overall computational costs while ensuring service quality.
[0135] Figure 8 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown.
[0136] It should be noted that, Figure 8 The computer system of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.
[0137] like Figure 8As shown, the computer system includes a GPU 1801, which can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) 1802 or a program loaded from storage portion 1808 into random access memory (RAM) 1803, such as performing the methods described in the above embodiments. Various programs and data required for system operation are also stored in RAM 1803. The GPU 1801, ROM 1802, and RAM 1803 are interconnected via bus 1804. An input / output (I / O) interface 1805 is also connected to bus 1804.
[0138] The following components are connected to I / O interface 1805: an input section 1806 including a keyboard, mouse, etc.; an output section 1807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1808 including a hard disk, etc.; and a communication section 1809 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1809 performs communication processing via a network such as the Internet. A drive 1810 is also connected to I / O interface 1805 as needed. Removable media 1811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1810 as needed so that computer programs read from them can be installed into storage section 1808 as needed.
[0139] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1809, and / or installed from removable medium 1811. When the computer program is executed by GPU 1801, it performs various functions defined in the system of this application.
[0140] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.
[0141] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0142] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.
[0143] In another aspect, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods described in the above embodiments.
[0144] This application also provides a computer program product storing at least one instruction, which is loaded and executed by the processor as described above. Figures 1-4 The method described in the illustrated embodiment can be found in the following document for a detailed execution process. Figures 1-4 The specific details of the illustrated embodiments will not be elaborated here.
[0145] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.
[0146] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of this application.
[0147] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.
[0148] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.
Claims
1. A method for localizing a large-scale language model service, the method comprising: include: Obtain real-time request load metrics for the inference service; Based on the requested load metric, a five-dimensional parallel strategy is determined within the computation iteration cycle. The five-dimensional parallel strategy includes parallelism in five dimensions: data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism, and expert parallelism. A unified virtualized key-value cache pool spanning multiple computing instances is constructed, wherein the virtualized key-value cache pool is formed by virtualizing the local video memory space of each computing instance as part of the pool; the key-value cache data is divided into multiple page units with token granularity, each page unit is assigned a unique logical index number, and the storage location and computing instance to which each page unit belongs are recorded through an index table to support cross-computing instance referencing; During the pre-filling stage, parallelism reduction is performed to reuse communication paths and hardware resources in the sequential parallel computing process, and the generated key-value cache data is directly distributed to the target computing instance according to the five-dimensional parallel strategy. The target computing instance is the computing instance to which the decoding stage ultimately belongs. During the decoding phase, multi-host, multi-shard decoding is performed on each target computing instance, and local attention computing results are obtained as needed through the virtualized key-value cache pool.
2. The localized large-scale language model serving method of claim 1, wherein, The step of determining a five-dimensional parallel strategy within the calculation iteration cycle based on the requested load metric specifically includes: Based on the request load metrics and the current resource status of the computing instance, the optimal combination of five-dimensional parallel strategies is calculated using a revenue-cost model according to the optimization objective.
3. The localized large-scale language model serving method of claim 1, wherein, The process of performing multi-host, multi-shard decoding on each target computing instance and obtaining local attention computation results on demand through the virtualized key-value cache pool specifically includes: Dynamically add the new compute instance to the currently requested parallel compute group; Through the virtualized key-value cache pool, the new computing instance receives the query vector of the current decoding step and the necessary model layer weights, participates in local attention calculation, and sends the calculation results back.
4. The localized large-scale language model serving method of claim 1, wherein, Following the construction of a unified virtualized key-value cache pool spanning multiple computing instances, the localized large-scale language model service method further includes: The recycling strategy is driven by the priority of requests, the access frequency of key-value cached pages, and their lifecycle.
5. The localized large-scale language model serving method of claim 1, wherein, When the large-scale language model is a hybrid expert model, the localized large-scale language model service method further includes: Real-time monitoring of the load on each expert in the hybrid expert model; When it is detected that the load of a certain expert is consistently higher than a preset threshold, the weight corresponding to that expert is copied or migrated online to one or more computing instances with lower load, and the request routing is adjusted to balance the load among experts. 6.A local large-scale language model service apparatus, characterized by, The localized large-scale language model service device includes: The real-time load acquisition module is used to acquire request load metrics for the inference service in real time. The five-dimensional parallel strategy module is used to determine the five-dimensional parallel strategy within the calculation iteration cycle based on the request load index. The five-dimensional parallel strategy includes parallelism in five dimensions: data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism, and expert parallelism. The virtualized cache pool module is used to build a unified virtualized key-value cache pool spanning multiple computing instances. The virtualized key-value cache pool is constructed by virtualizing the local video memory space of each computing instance as part of the pool. The key-value cache data is divided into multiple page units with token granularity. Each page unit is assigned a unique logical index number, and the storage location and the computing instance to which each page unit belongs are recorded through an index table to support cross-computing instance referencing. The uninterrupted refactoring module is used to perform parallelism reduction during the pre-filling stage to reuse communication paths and hardware resources in the sequential parallel computing process, and to directly distribute the generated key-value cache data to the target computing instance according to the five-dimensional parallel strategy. The target computing instance is the computing instance to which the decoding stage ultimately belongs. During the decoding stage, multi-host multi-shard decoding is performed on each target computing instance, and the local attention calculation results are obtained as needed through the virtualized key-value cache pool.
7. A computer readable medium having stored thereon a computer program, characterized in that When the computer program is executed by a processor, it implements the localized large-scale language model service method as described in any one of claims 1 to 5.
8. An electronic device, comprising: include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the localized large-scale language model service method as described in any one of claims 1 to 5.
9. A computer program product comprising one or more computer programs that, when executed by one or more processors, implement the steps of the localized large-scale language model service method according to any one of claims 1 to 5.
Citation Information
Patent Citations
Large model automatic multi-dimensional parallel training system based on heterogeneous GPU
CN120387025A
Data management method, device and equipment and readable storage medium
CN120499269A
Large model pre-training system based on distributed parallel processing
CN120509482A
Memory pooling method and system for model reasoning acceleration and computer program product
CN120525063A
