Localized large-scale language model service method and related equipment

Abstract

The embodiment of the invention provides a localized large-scale language model service method and related equipment. The localized large-scale language model service method comprises the following steps: acquiring a request load index of an inference service in real time; according to the request load index, a five-dimensional parallelism strategy is determined in a calculation iteration period, and the five-dimensional parallelism strategy comprises parallelism degrees of five dimensions including data parallelism degree, tensor parallelism degree, pipeline parallelism degree, sequence parallelism degree and expert parallelism degree; and reconstructing the parallel execution mode of the current service into the five-dimensional parallel strategy without interruption, and executing the five-dimensional parallel strategy. According to the technical scheme provided by the embodiment of the invention, the system can be dynamically switched among a plurality of parallel dimensions under the condition of load fluctuation, and the overall calculation cost is minimized on the premise of ensuring the service quality.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers and communications, in particular to a localized large-scale language model service method and related equipment. BACKGROUND

[0002] In the inference service of large language models, in order to cope with huge computing and memory overhead, the industry generally adopts various parallel computing strategies, such as data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism, and expert parallelism. However, the existing large-scale language model inference service has significant technical defects when applying these parallel strategies. Specifically, the combination mode and parallel degree of these parallel strategies are usually statically configured and fixed at the service start or model loading. This static strategy cannot adapt to the dynamic computing characteristics of different stages in the large-scale language model inference process. For example, the pre-filling stage is usually computationally intensive, while the decoding stage is more limited by communication and memory bandwidth. The fixed parallel strategy inevitably causes resource and load mismatch when processing tasks with different computing characteristics, resulting in resource waste or performance bottlenecks, making it difficult to optimize throughput and latency simultaneously. Although some existing technologies can achieve single-dimensional dynamic scaling based on data parallelism, for deeper parallel dimensions such as tensor parallelism, pipeline parallelism, etc., once adjustment is needed, the service often needs to be interrupted and restarted, which is unacceptable for online services that require high availability. Therefore, how to dynamically reconstruct the multi-dimensional parallel strategies including data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism, and expert parallelism without interrupting the service according to the real-time changing request load, so that the computing resources are always accurately matched with the workload characteristics, is a technical problem that needs to be solved in the current large-scale language model inference service field. SUMMARY

[0003] Embodiments of the present application provide a localized large-scale language model service method and related equipment, which can at least to some extent dynamically reconstruct the multi-dimensional parallel strategies including data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism, and expert parallelism without interrupting the service according to the real-time changing request load, so that the computing resources are always accurately matched with the workload characteristics.

[0004] Other characteristics and advantages of the present application will become apparent from the following detailed description, or will be learned by practice of the present application.

[0005] According to an aspect of some embodiments of the present application, there is provided a method for local large-scale language model service, comprising: acquiring a request load index of inference service in real time; determining a five-dimensional parallel strategy according to the request load index, the five-dimensional parallel strategy comprising parallel degrees of five dimensions of data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism and expert parallelism; and reconstructing a parallel execution mode of current service to the five-dimensional parallel strategy without interruption and executing.

[0006] According to an aspect of some embodiments of the present application, there is provided a device for local large-scale language model service, comprising: a load real-time acquisition module configured to acquire a request load index of inference service in real time; a five-dimensional parallel strategy module configured to determine a five-dimensional parallel strategy according to the request load index, the five-dimensional parallel strategy comprising parallel degrees of five dimensions of data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism and expert parallelism; and a non-interruption reconstruction module configured to reconstruct a parallel execution mode of current service to the five-dimensional parallel strategy without interruption and execute.

[0007] According to an aspect of some embodiments of the present application, there is provided a computer program product comprising one or more computer programs to perform the steps of the method for local large-scale language model service as described in the above embodiments when the one or more computer programs are executed by one or more processors.

[0008] According to an aspect of some embodiments of the present application, there is provided a computer-readable medium having stored thereon a computer program, the computer program, when executed by a processor, implementing the method for local large-scale language model service as described in the above embodiments.

[0009] According to an aspect of some embodiments of the present application, there is provided an electronic device, comprising: one or more processors; a storage device configured to store one or more programs, the one or more programs, when executed by the one or more processors, causing the one or more processors to implement the method for local large-scale language model service as described in the above embodiments.

[0010] In the technical solutions provided in some embodiments of the present application, the optimal five-dimensional parallelism combination can be automatically switched in different request modes, including short context, high concurrency, long context, and expert unbalanced load. The parallelism adjustment process does not affect the ongoing inference task, ensuring the continuity of the service. By jointly scheduling in multiple dimensions, memory waste or communication bottlenecks caused by over-expansion of a single dimension are avoided. In the high fluctuation load scenario, the average response time is kept low and the throughput capacity is optimal, which can ensure that the system can dynamically switch in multiple parallel dimensions in the case of load fluctuation, and realize the minimization of the overall computing cost on the premise of ensuring the quality of service.

[0011] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0012] The accompanying drawings, which are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application and, together with the specification, serve to explain the principles of the present application. It is obvious that the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained from these drawings without creative labor for those skilled in the art. In the drawings: Figure 1 A schematic diagram of an exemplary system architecture to which the technical solutions of the embodiments of the present application can be applied is shown.

[0013] Figure 2 A flowchart of a localized large-scale language model service method provided by an embodiment of the present application is shown.

[0014] Figure 3 A flowchart of another localized large-scale language model service method provided by an embodiment of the present application is shown.

[0015] Figure 4 A flowchart of a step S300 in the localized large-scale language model service method shown in the corresponding embodiment is shown. Figure 3 A flowchart of a step S300 in the localized large-scale language model service method shown in the corresponding embodiment is shown.

[0016] Figure 5 A system diagram of a localized large-scale language model service system provided by an embodiment of the present application is shown.

[0017] Figure 6 A flowchart of an inference execution method based on sequence parallel communication provided by an embodiment of the present application is shown.

[0018] Figure 7 A structural schematic diagram of a localized large-scale language model service device provided by an embodiment of the present application is shown.

[0019] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0020] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0021] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0022] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0023] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0024] Figure 1 A schematic diagram of an exemplary system architecture to which the technical solutions of the embodiments of this application can be applied is shown.

[0025] like Figure 1 As shown, the system architecture may include terminal devices (such as...) Figure 1 The device shown includes one or more of a smartphone 101, tablet 102, and portable computer 103 (which could also be a desktop computer, etc.), a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the terminal device and the server 105. The network 104 can include various connection types, such as wired communication links, wireless communication links, etc.

[0026] It should be understood that Figure 1The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, there can be any number of terminal devices, networks, and servers. For example, server 105 could be a server cluster composed of multiple servers.

[0027] Users can interact with server 105 via network 104 using terminal devices to receive or send messages, etc. Server 105 can be a server providing various services. For example, a user can upload request load metrics for inference services to server 105 using terminal device 103 (or terminal device 101 or 102). Server 105 can determine a five-dimensional parallel strategy within the computation iteration cycle based on the request load metrics. The five-dimensional parallel strategy includes parallelism in five dimensions: data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism, and expert parallelism. The current parallel execution mode of the service is then reconstructed without interruption to the five-dimensional parallel strategy and executed.

[0028] It should be noted that the localized large-scale language model service method provided in this application embodiment is generally executed by server 105, and correspondingly, the localized large-scale language model service device is generally located in server 105. However, in other embodiments of this application, the terminal device may also have similar functions to the server, thereby executing the localized large-scale language model service scheme provided in this application embodiment.

[0029] The implementation details of the technical solutions in the embodiments of this application are described in detail below: Figure 2 A flowchart of a localized large-scale language model service method according to an embodiment of this application is shown. This localized large-scale language model service method can be executed by a server, which may be... Figure 1 The server shown. (Refer to...) Figure 2 As shown, this localized large-scale language model service method includes at least: S100, obtains real-time request load metrics for the inference service.

[0030] S200, based on the requested load index, determine a five-dimensional parallel strategy within the calculation iteration cycle. The five-dimensional parallel strategy includes parallelism in five dimensions: data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism, and expert parallelism.

[0031] S300, reconstruct the current parallel execution mode of the service without interruption into the five-dimensional parallel strategy and execute it.

[0032] In this embodiment, the optimal five-dimensional parallelism combination can be automatically switched in different request modes, including short context, high concurrency, long context, and expert uneven load. The parallelism adjustment process does not affect the ongoing inference task, ensuring the continuity of the service. By jointly scheduling in multiple dimensions, the waste of video memory or communication bottleneck caused by over-expansion of a single dimension is avoided. In the high fluctuation load scenario, the system can dynamically switch in multiple parallel dimensions and minimize the overall computing cost while ensuring service quality.

[0033] In S100, the policy engine module continuously monitors various online running indicators of the inference service. These indicators include but are not limited to request arrival rate, request feature distribution, system resource water level. Among them, the request arrival rate is the number of requests per unit time, the request feature distribution is for example the distribution of the input sequence length and the estimated output sequence length of the request, and the system resource water level is for example the GPU memory occupancy rate, the computing power utilization rate of each computing instance, and the communication bandwidth between nodes.

[0034] The policy engine analyzes these real-time data to accurately determine whether the current service is in a pre-filling intensive or decoding intensive state. Among them, decoding intensive is for example multi-round dialogue, low concurrency stream generation, and pre-filling intensive is for example processing long text, high concurrency batch processing.

[0035] It should be noted that the policy engine will evaluate the overall resource situation of all computing instances, including the remaining memory, computing power load and available network bandwidth. At the same time, it will look at the requests in the waiting queue and select a subset of requests according to the length of their input text and the prediction of the output length. The principle of selection is to maximize the total processing efficiency without exceeding the current system resource constraints, and to minimize the situation of long-term blocking of subsequent requests caused by a single long request or a large number of requests. This is the first step of request processing, which is equivalent to an intelligent request gateway. The system does not simply process all queued requests in the order of first come first served, but will comprehensively consider the current system state and the characteristics of the request to determine which requests can enter the next stage.

[0036] That is, during the running process, real-time indicators related to requests are continuously collected, including request arrival rate (QPS), input context length distribution, generated target token length distribution, time ratio of pre-filling stage to decoding stage, memory usage rate of computing instances, bandwidth utilization rate and communication latency, etc. The above indicators are used as basic data input for dynamic adjustment of parallelism.

[0037] Before S200 is executed, the application will also perform instance allocation. That is, after determining the subset of requests to be processed, the specific computing resources need to be allocated to this subset.

[0038] Specifically, first, completely idle computing instances are sought, and requests are preferentially allocated to them. If there are no completely idle instances, the system will start a warm migration mechanism. It will check all instances that are processing decoding tasks to find the instance that currently stores the key-value cache that occupies the least video memory. Then, the system will perform a token-level migration to migrate a small amount of key-value cache data there to other instances, thereby quickly freeing up enough space to meet the pre-filling needs of new requests. This migration is small-scale and non-blocking, so it is called migration.

[0039] After the resource allocation is complete, an optimal specific parallel execution scheme needs to be determined for this batch of requests, at which time S200 is executed.

[0040] In S200, within a computing iteration period, for example, in batch processing units, based on real-time request load indicators, a five-dimensional parallel strategy is determined to adapt to the current request. The five-dimensional parallel strategy includes data parallelism (DP), tensor parallelism (TP), pipeline parallelism (PP), sequence parallelism (SP), and expert parallelism (EP) in five dimensions.

[0041] Among them, data parallelism determines how the request batch is allocated among multiple machines; tensor parallelism determines the tensor splitting method among different devices for single-layer computation; pipeline parallelism determines how model layers are distributed in a multi-machine pipeline for concurrent computation; sequence parallelism determines the segmented computation of long sequence tokens among different devices; and expert parallelism determines how expert modules are distributed across devices in a hybrid expert model. Through the joint optimization of the above five-dimensional parameters, the system can dynamically select the optimal combination under different load conditions, for example, preferentially increasing DP in short context high QPS scenarios, and preferentially increasing SP and TP in long context scenarios, to ensure performance.

[0042] Specifically, first, the batch of requests will be clustered according to the similarity of input and output lengths to form different length clusters. For each length cluster, the unified parallel orchestrator will perform a complex solution calculation. It will try different combinations of data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism, and expert parallelism, and combine different batch sizes to predict which combination can achieve the minimum single iteration delay through a cost model. The goal of this process is to find the most efficient parallel execution method for this batch of requests under the current resources.

[0043] For example, in a high-concurrency pre-filling scenario, the depth of tensor parallelism and pipeline parallelism can be increased to improve computational efficiency; while in a decoding-intensive scenario, the tensor parallelism or pipeline parallelism can be reduced to reduce communication overhead, and the number of data parallel copies can be increased to improve the total throughput. Finally, the system outputs a specific five-dimensional parallelism combination.

[0044] Specifically, in some embodiments, the specific implementation of step S200 can refer to the following embodiments. The present embodiment is based on the method of localizing a large-scale language model service shown in the corresponding embodiments Figure 2 The details of step S200 in the method of localizing a large-scale language model service shown in the corresponding embodiments can include the following steps: Based on the request load indicators and the current resource state of the computing instance, the optimal five-dimensional parallelism strategy combination is calculated through a benefit-cost model according to the optimization target.

[0045] In this embodiment, among the combinations of the five parallel dimensions, comparison is made through a unified benefit-cost model to avoid local optimization, and the optimal solution can be dynamically selected according to different load characteristics (high concurrency for short context, low concurrency for long context, MoE expert hotspot, etc.). By introducing the real-time state of the computing instance, the selection of the parallel strategy is only dependent on the static configuration. The benefit-cost model can extend different weights or targets to adapt to latency priority, energy consumption priority, or mixed target scenarios.

[0046] Specifically, within the calculation iteration period, not only the real-time request load indicators of the inference service are obtained, but also the current resource state of the computing instance is collected, specifically including the memory capacity and current occupancy rate, the utilization rate of computing cores, the utilization rate of communication bandwidth and link delay, the local cache hit rate, and the load balancing degree among instances. These data and the request load indicators are jointly used as inputs of the benefit-cost model. Among them, the utilization rate of computing cores can be the SM occupancy rate of GPU or Tensor Processing Unit (TPU).

[0047] Meanwhile, the optimization target mentioned above can include but is not limited to latency target, throughput target, energy consumption target, fairness target, etc., which can be determined according to the service scenario. Among them, the latency target is to reduce the average response time or P99 latency of the request; the throughput target is to maximize the number of tokens completed per unit time; the energy consumption target is to reduce the total energy consumption of GPU or TPU; and the fairness target is to ensure that the performance of different users or different priority requests is not imbalanced.

[0048] The benefit-cost model mentioned above can quantitatively evaluate different five-dimensional parallel combinations, which can specifically include a benefit function R, a cost function C, and a target function F = R-C.

[0049] wherein the revenue function R represents the performance improvement value under a certain parallel combination, which is weighted calculation combined with delay improvement, throughput increase and energy consumption reduction. The cost function C represents the calculation and communication cost when switching to the combination, including memory occupation, communication overhead, load migration cost. The objective function F = R-C comprehensively measures the actual revenue of different combinations.

[0050] It should be noted that the revenue function R is composed of three parts, including delay revenue RL, throughput revenue RT and energy consumption revenue RE; the cost function C is composed of three parts, including communication cost CC, migration cost CM and memory cost CV, which conforms to the following formula: RL = (baseline latency - current latency) / baseline latency RT = (current throughput - baseline throughput) / baseline throughput RE = (baseline energy consumption - current energy consumption) / baseline energy consumption R = a·RL + b·RT + g·RE CC = link delay x transmission data volume CM = key-value data migration time CV = additional memory occupation / total memory capacity C = CC + CM + CV wherein a, b and g are weight coefficients.

[0051] Specifically, when calculating the optimal five-dimensional parallel strategy combination, first, the revenue-cost model is substituted into the candidate data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism and expert parallelism combination space one by one, and the corresponding objective function value F is calculated. Under the premise of guaranteeing the service constraint condition, the combination with the maximum objective function value is selected as the optimal five-dimensional parallel strategy in the iteration period. It should be noted that the service constraint condition is, for example, memory overflow, communication delay does not exceed the threshold. The finally determined optimal five-dimensional parallel strategy combination will be applied to the subsequent reasoning process, and will be recalculated in the next iteration period to achieve dynamic optimization.

[0052] In some other embodiments, the hardware characteristics of different GPUs and TPUs can also be collected, including computing power peak, memory size, bandwidth performance, etc. The revenue-cost model introduces a hardware weighting factor when calculating the candidate strategy, so that the strategy can match the hardware characteristics, for example, preferentially increasing the tensor parallelism on a high-bandwidth GPU, and preferentially increasing the sequence parallelism on a large-memory GPU. This mechanism ensures that different hardware clusters can all obtain optimal performance under a unified framework.

[0053] In S300, after the five-dimensional parallel strategy is determined, the currently running parallel execution mode is directly switched to the five-dimensional parallel strategy. In the execution switching process, the inference process does not need to be restarted, and the existing calculation state and key value cache data are maintained without loss, ensuring continuous response of the request without interruption. The reconstruction mechanism enables the service to adaptively adjust when the load fluctuates and always maintain the best running state.

[0054] In some embodiments of the present application, as shown in Figure 3 Before S300, the localized large-scale language model service method further includes: S400, a unified virtualized key value cache pool spanning multiple computing instances is constructed, wherein the key value cache data is managed by paging and indexing at a token granularity to support cross-computing instance reference.

[0055] In this embodiment, the key value data is paged at a token granularity and shared across instances, avoiding high overhead caused by overall migration. When the parallel strategy is adjusted, the cache data does not need to be moved as a whole to switch the parallel mode. The cross-instance cache pool enables the video memory space to be shared on demand, reducing resource waste caused by cache occupation in a single instance. The application layer and the scheduling layer do not need to be aware of the underlying distribution, reducing the complexity of the system.

[0056] Specifically, a logical unified key value cache pool is first abstracted among multiple computing instances, and the local video memory space of each instance is virtualized as a part of the cache pool. Through a unified address mapping table, each instance can access the entire cache pool at a logical level without being aware of the underlying physical distribution.

[0057] The key value cache data is paged at a token granularity, and each paging unit corresponds to a group of token key values. Each paging unit is assigned a unique logical index number, and the storage location and the corresponding computing instance of each paging unit are recorded in the index table to realize fast positioning and access.

[0058] When a computing instance needs to access key value data that is not local, the target instance can be found through the index table and a call is initiated. The cache pool supports cross-instance direct reference, avoiding overall data migration. Data access is mainly based on on-demand retrieval, reducing redundant transmission.

[0059] This embodiment provides a unified cache pool interface for inference services, including allocation, query, and recycling operations. The user side and the upper layer scheduling logic do not need to care about where the data is specifically stored in a computing instance, thereby realizing transparent management.

[0060] After the above steps, the localized large-scale language model service method further includes: driving a recycling strategy according to the priority of the request, the access frequency and the life cycle of the key value cache page.

[0061] In the present embodiment, the recycling strategy can be driven according to the priority of the request, the access frequency and the life cycle of the key-value cache page. In addition, a hierarchical storage mechanism can also be introduced, for example, the key-value cache of a newly generated and frequently accessed token can be temporarily stored in the cache in a low-bit format (such as FP8 / INT4), while the cold and long-term unaccessed key-value cache page can be compressed and sunk into the slower but larger capacity host memory, thereby maximizing the effective video memory capacity. Through life cycle and access frequency management, the present embodiment avoids unlimited growth of the cache pool, preserves the cache of high-priority requests, and guarantees low latency and stability of critical business. The cold and hot data hierarchical management reduces redundant occupation and improves cache space reuse rate. The cache recycling strategy is automatically executed to avoid manual intervention that may interrupt the inference task.

[0062] Specifically, the present embodiment maintains metadata information for each cache page in the virtualized cache pool, including request identification, priority level, access count, recent access time, life cycle start time, and expected expiration time. The cache pool is periodically scanned to collect the above metadata and update statistical information for subsequent recycling decisions.

[0063] When the cache pool space is insufficient, the cache pages corresponding to high-priority requests are preferentially retained. For the cache pages of low-priority requests, if resources are tight, they are preferentially recycled to ensure the computing needs of critical tasks.

[0064] According to the access frequency of the cache page, the degree of coldness and hotness is determined. The pages with high access frequency are marked as hot data and are preferentially retained. The cold data that has not been accessed for a long time enters the candidate recycling list and is preferentially released when the space is insufficient.

[0065] Each cache page is set with a life cycle parameter when it is generated, including minimum retention time and maximum survival time. The cache page exceeding the upper limit of the life cycle automatically enters the recycling queue. The life cycle strategy ensures that data is not cleaned up too early due to extremely low access frequency, and new requests are not blocked due to long-term data occupation.

[0066] When the video memory pressure is large, the system considers the priority, access frequency and life cycle three factors to calculate the recycling priority of the cache page. On the premise of guaranteeing the stability of high-priority requests, the low-value cache page is preferentially released. After recycling is completed, the virtualized cache pool index table is updated to ensure cache consistency and traceability.

[0067] It should be noted that in the above virtualized key-value cache pool, a multi-level cache mechanism is adopted, the first level cache is stored in the local instance video memory, the access delay is the lowest; the second level cache is stored in the shared cache pool across instances, the delay is slightly higher, but can support on-demand cross-instance access; the third level cache overflow data is stored in the host memory or high bandwidth storage device, used for cold data saving in long context scenarios. In the decoding stage, the system preferentially accesses the first level cache, and if it is not hit, it accesses the second and third level caches through the index table to ensure cache utilization and storage scalability.

[0068] Specifically, in some embodiments, the specific implementation of step S300 can refer to Figure 4 . Figure 4 is according to Figure 3 The details of step S300 in the localized large-scale language model service method are described in the corresponding embodiment, and step S300 can include the following steps: S310, in the pre-filling stage, the parallelism contraction is performed, and the generated key-value cache data is directly distributed to the target computing instance according to the five-dimensional parallel strategy, and the target computing instance is the final computing instance in the decoding stage.

[0069] S320, in the decoding stage, each target computing instance performs multi-host multi-shard decoding, and obtains local attention calculation results through the virtualized key-value cache pool.

[0070] In this embodiment, the key-value cache data is directly placed in the target instance in the decoding stage in the pre-filling stage, and does not need to be redistributed or copied when the stage is switched. This embodiment adopts a multi-host multi-shard decoding mode, which effectively distributes the computing pressure when generating long sequences. At the same time, relying on the virtualized key-value cache pool, the decoding instance can directly access the required data when needed, avoiding overall moving. By combining the contraction of pre-filling parallelism and distributed decoding, the end-to-end delay of the request is significantly shortened. By using the adaptive parallel strategy in the pre-filling and decoding stages respectively, the use efficiency of the overall GPU resources is improved.

[0071] In S310, in the pre-filling stage, parallelism contraction is performed. It reuses the communication path and hardware resources that already exist in the sequence parallel computing process, and writes the generated key-value cache data directly and on the way to the target instance group designated for the subsequent decoding stage of the request, achieving zero additional migration overhead. Specifically, the system reuses the hardware links and data streams of the communication operations such as All-Reduce or All-Gather inherent in the sequence parallel computing process. In the final link of these communications, the unified parallel arranger controls the data flow, and instead of simply aggregating, the key-value cache data generated by each token is directly distributed and written into the memory of the target computing instance that will perform the decoding task according to the parallel strategy determined in the subsequent decoding stage. This avoids a separate and large-scale migration of key-value cache data after the pre-filling calculation is completed, thereby achieving zero additional migration.

[0072] That is, in the model pre-filling stage, the input context is usually long, and the calculation is concentrated in the self-attention and forward propagation operations. According to the five-dimensional parallel strategy, parallelism contraction is performed, that is, a higher sequence parallelism and tensor parallelism are used in this stage to distribute the long sequence calculation among multiple computing instances. During the calculation process, the system synchronously generates the corresponding key-value cache, and the generated key-value cache data is no longer stored centrally, but is directly distributed to the target computing instance designated by the five-dimensional parallel strategy, which is the instance ultimately responsible for performing inference in the decoding stage, thereby avoiding additional migration in the subsequent stage.

[0073] Specifically, in some embodiments, the specific implementation of step S310 can refer to the following embodiments. The present embodiment is based on the five-dimensional parallel strategy Figure 4 The details of step S310 in the localized large-scale language model service method shown in the corresponding embodiment can include the following steps: In response to the request to perform sequence parallel calculation, generate key-value cache data.

[0074] Using the communication link in the sequence parallel computing process, the generated key-value cache data is directly distributed and stored on the target computing instance.

[0075] In this embodiment, the KV cache is directly distributed to the decoding target instance when it is generated, without the need for secondary migration when switching between the pre-filling and decoding stages. Reusing the communication link in the sequence parallel computing process eliminates the need to establish new transmission channels, reducing communication overhead. The KV data can complete target landing in the pre-filling stage, and the decoding stage directly calls on site, shortening the start-up delay. Through the virtual cache pool index update mechanism, the data location is transparent, and the system management efficiency is improved.

[0076] Specifically, in the pre-filling stage, the input long sequence is divided into multiple sequence segments, and sequence parallel computing is performed using multiple computing instances according to a five-dimensional parallel strategy. During the computing process, the intermediate result corresponding to each sequence segment generates key-value cache data, including a key tensor and a value tensor, which is generated with index information to identify its position in the complete context sequence.

[0077] During the sequence parallel computing process, the system itself needs to pass the intermediate result of the sequence segment between different computing instances to ensure the continuity of the forward propagation and attention calculation. In this embodiment, the generated key-value cache data is no longer stored in the temporary instance, but is directly distributed to the target computing instance through the link, which is the final attribution instance of the decoding stage determined in advance in the five-dimensional parallel strategy. During the distribution process, the system does not need to perform additional data migration, but writes the key-value cache to the local storage of the target instance along the way. After the distribution is completed, the target instance registers the key-value cache data to the virtualized key-value cache pool for subsequent decoding steps to call as needed.

[0078] After the target computing instance receives the key-value cache data, it stores it in the local video memory space, updates the key-value paging index table through the index management mechanism of the virtualized cache pool, and marks the current instance location where the key-value data is stored. This index update ensures that any computing instance in the subsequent decoding stage can locate and access the corresponding data through the cache pool interface.

[0079] In the traditional mode, when switching from pre-filling to decoding stage, if the parallel strategy changes (for example, from 4-card tensor parallel to 2-card tensor parallel + 2-way data parallel), the following blocking process will occur: calculation, service suspension, global migration, and service recovery. Specifically, during the calculation, four GPUs complete the pre-filling calculation, and each GPU stores part of the key-value cache. When the service is suspended, the system suspends processing of the request. During global migration, a separate data migration task is started to collect the KV cache on the four GPUs, and then according to the new 2+2 strategy, the KV cache is re-divided and distributed to the target GPU. This process is time-consuming and does not produce any effective calculation, which is the main reason for service interruption. When the service is restored, after the data is settled, the decoding stage begins.

[0080] Before the pre-filling starts, the unified parallel orchestrator has already calculated the optimal parallel strategy and target GPU group for the subsequent decoding stage according to the load prediction. During the pre-filling calculation, especially when sequence parallel is used, communication between GPUs is needed to exchange intermediate calculation results (for example, All-Reduce operation after attention calculation). The scheme ingeniously reuses this existing communication link. When a GPU calculates the key-value cache of a certain token, it does not simply store it locally. Instead, according to the instructions previously issued by the UPO, the GPU directly sends the key-value cache data to the final destination GPU in the subsequent decoding stage during the sequence parallel communication process. Therefore, when the entire pre-filling calculation task is completed, all the requested KV cache data has been conveniently and accurately distributed according to the parallel strategy required by the decoding stage and positioned on the correct target GPU. The system does not need any pause or additional data migration and can directly and seamlessly enter the first step of decoding. That is, the serial mode of first calculation and then migration is changed to the pipeline mode of calculation and migration completed at the same time, thereby eliminating the interruption.

[0081] In S320, in the decoding stage, the system performs multi-host multi-shard decoding. If additional computing power is needed, new computing instances can be dynamically added. The new instance does not need to load the entire historical key-value cache of the request. It only needs to receive the query vector of the current decoding step and the necessary part of the model layer parameters for execution from the master instance. The new instance performs attention calculation locally, and then returns the result to the master instance, and this communication process can overlap with the calculation process to hide the communication delay. The key of this step is that the new instance does not need to load or replay the entire historical key-value cache of the request since the beginning. It only needs to receive the query vector of the current decoding step and the model layer weight parameters necessary for attention calculation and feedforward network calculation through the network. The new instance completes partial calculation (such as local attention and FFN layer calculation) locally, and then returns the result to the master instance for aggregation. The entire process realizes plug-and-play of the computing instance, with extremely low expansion overhead.

[0082] That is, when entering the decoding stage, the newly generated tokens are sequentially input into the model for step-by-step inference, and the decoding process adopts a multi-host multi-shard manner, that is, different computing instances share part of the model layer or part of the sequence decoding calculation, and each target computing instance calls the virtualized key-value cache pool as needed to obtain the key-value data required for local attention calculation. If the target instance does not have the required key-value data locally, it can directly access the data stored in the remote instance through the cross-instance indexing mechanism without the need for overall migration. After the calculation is completed, each instance returns the attention calculation result and the corresponding intermediate tensor to the assigned decoding group to ensure the consistency of the output tokens.

[0083] After each round of decoding is completed, the results calculated by each instance are collected and spliced to form a complete output token, and the key value representation of the new token is immediately appended to the virtualized key value cache pool for subsequent decoding steps to continue calling. The five-dimensional parallel strategy remains unchanged throughout the decoding process until the request is completed.

[0084] Specifically, in some embodiments, the specific implementation of step S320 can refer to the following embodiments. The present embodiment is based on Figure 4 The detailed description of step S320 in the localized large-scale language model service method shown in the corresponding embodiments can include the following steps: Dynamically adding a new computing instance to the parallel computing group of the current request.

[0085] Through the virtualized key value cache pool, the new computing instance receives the query vector and the necessary model layer weight of the current decoding step, participates in local attention calculation, and returns the calculation result.

[0086] In the present embodiment, the new computing instance can access at any time without restarting the inference process, improving the flexibility of the system. At the same time, the new instance obtains data on demand through the virtualized cache pool, avoiding the overall migration of historical key values and reducing the memory and communication overhead. The new instance can also be plug-and-play, and the decoding process is not interrupted, ensuring the continuity of the request. Multi-host multi-shard decoding realizes horizontal expansion with the addition of new instances, effectively improving the overall computing power.

[0087] Specifically, when the request load continues to increase in the decoding phase, or the original computing instance resource is insufficient, the scheduler can add one or more new computing instances to the parallel computing group of the current request. The new instance does not need to interrupt the existing decoding task when it is added, and automatically allocates shard tasks for it through the parallel group management mechanism. The access process of the new instance is completely completed online, without affecting the continuity of the decoding process.

[0088] After the new computing instance joins the parallel group, it obtains the query vector required by the current decoding step through the virtualized key value cache pool interface. At the same time, the scheduler issues the necessary model layer weight corresponding to the decoding step to the new instance. Since the virtualized cache pool maintains a global index, the new instance can directly locate and access the key value cache data required by its responsible shard on demand, without the need to migrate the entire historical cache.

[0089] The new instance performs local attention calculation using the received query vector, weight parameter, and key value data provided by the virtualized cache pool. After completing the calculation, the new instance returns the result tensor to the decoding parallel group for generating the prediction distribution of the current token. The returned result is aggregated with other shard results in the decoding parallel group to ensure the consistency of the output token.

[0090] The new instance stores the newly generated key-value data locally and registers to the virtualized cache pool after finishing the computation. The index table updates the new storage location of the key-value data in real time, ensuring that the subsequent decoding steps can be directly accessed.

[0091] In some embodiments of the present application, when the system detects that the request context length distribution fluctuates (for example, from an average of 1K tokens to a sudden increase of 8K tokens), a strategy adjustment is triggered. If the proportion of pre-filling and decoding stages exceeds a certain threshold (for example, the pre-filling proportion is greater than 70%), the strategy is biased to increase the sequence parallelism and tensor parallelism; if the decoding stage proportion is high, the strategy is biased to increase the data parallelism and expert parallelism to improve the throughput. Through this adaptive switching mechanism, the system can quickly adjust the strategy under load fluctuations to maintain stable performance.

[0092] Before performing S320, virtualization of the state needs to be implemented. The core is to unify the virtualized key-value cache pool. All requested key-value caches are considered as a logical whole, and are paged and indexed by tokens, and can be physically distributed on any GPU instance. Instances can access each other's key-value cache pages as needed and efficiently.

[0093] When a computing instance needs to be added (for example, from 2 cards to 3 cards) to speed up decoding, the newly added instance does not need to load the complete and complete history key-value cache of the request. This is a huge saving of overhead. The master instance only needs to broadcast the query vector Q of the current decoding step to this new instance. The new instance obtains the necessary history key and value data pages for computing attention scores through the virtualized key-value cache pool based on the Q vector. The new instance completes the part of the computation it is assigned to (for example, a part of the attention head or FFN layer computation), and returns the result to the master instance for aggregation.

[0094] The new instance can almost immediately enter the computation of the next token, and the entire expansion process is smooth and non-blocking to the service. Because the state (key-value cache) is globally virtualized and can be accessed on demand, large-scale state migration is avoided, thereby realizing parallel reconstruction without interruption. The process of removing an instance is also the same, and the UPO only needs to stop assigning tasks to it.

[0095] That is, S320 changes the heavy operation of global state reconfiguration to the light operation of adding a stateless computing unit that accesses shared state on demand, thereby achieving interruption-free.

[0096] In summary, in S300, the pre-filled parallelism shrink solves the cross-stage, one-time parallel strategy switching problem from one computation-intensive stage to another communication-intensive stage; and the decoding multi-host decoding solves the persistent, dynamic parallel resource adjustment problem that needs to be made due to load changes within the same computation stage.

[0097] In some embodiments, when the large language model is a hybrid expert model, the following steps are also executed in parallel with step S300: Real-time monitoring of the load of each expert in the hybrid expert model.

[0098] When it is detected that the load of a certain expert is continuously higher than a preset threshold, the weight corresponding to the expert is copied or migrated online to one or more computing instances with lower load, and the request routing is adjusted to balance the load among experts.

[0099] In the present embodiment, through the real-time monitoring and dynamic governance mechanism, the single expert is avoided from becoming a performance bottleneck, and the expert weight copying or migration is completed in operation, without affecting the normal execution of inference requests. Through dynamic routing table adjustment, the requests are distributed among multiple expert replicas, the average delay is reduced, the replicas are expanded in high load and recycled in low load, and the efficiency and flexibility of the use of computing resources are guaranteed. Specifically, the calling frequency and computation load of each expert are monitored in real time. Through the gating budget vector, a request processing quota within an iteration period can be set for each expert, so as to quickly suppress hotspots and avoid a single expert slowing down the processing speed of the entire batch. When it is monitored that the load of a certain expert is continuously higher than a preset threshold, the unified parallel arranger will take more active measures. It will copy the model weight corresponding to the hotspot expert to one or more idle computing instances with lower load online, and update the routing table to distribute a part of the subsequent requests sent to the expert to the new replica. When the system load is low, online migration can also be performed to optimize the long-term physical layout of the experts and balance the load of each computing instance.

[0100] Specifically, in the running process, the load indicators of each expert in the hybrid expert model are continuously collected, including the number of currently received requests, the average token processing rate, the memory occupancy and bandwidth utilization, and the P99 request delay. The above indicators are counted and analyzed within a computation iteration period to form the load portrait of each expert. When the load of a certain expert is continuously higher than a preset threshold, the subsequent governance operation is triggered. It should be noted that the load of a certain expert is continuously higher than a preset threshold, for example, its average QPS proportion exceeds twice the global average.

[0101] When the hotspot expert is detected, the system scheduler can take two ways of replication strategy and migration strategy to relieve the load. Specifically, the replication strategy copies the weight corresponding to the expert to one or more low-load computing instances to form multiple copies; the migration strategy migrates the weight of the expert to an idle or low-load computing instance.

[0102] The weight data is distributed through distributed storage or high-speed communication link, and the new copy instance is available immediately after completing the weight loading. The original computing instance continues to provide services until the routing strategy update is completed. That is, the weight replication or migration is completed in an online state, which does not affect the current inference service.

[0103] The routing table of the gating module or the scheduler of the updated hybrid expert is updated. For newly incoming requests, the routing proportion is dynamically adjusted according to the number of expert copies and the computing resource situation, so as to ensure that the requests of the originally load-concentrated expert are evenly distributed to multiple copy instances.

[0104] When the load of the expert copy tends to be balanced and the overall system is in a low-load state, the redundant expert copy is gradually recycled. The recycling strategy is based on the usage rate and the life cycle parameters of the copy to avoid long-term idling of resources.

[0105] In some embodiments of the present application, if it is detected during the policy switching process that the new parallel degree combination causes the video memory to overflow or the communication delay to exceed the threshold, the current switching is immediately interrupted, and the five-dimensional parallel strategy of the previous version is rolled back to ensure that the request is not interrupted. The rollback mechanism is implemented by saving the policy parameters of the last stable running, and has the ability of fast recovery.

[0106] As shown in Figure 5 The embodiment provides a localized large-scale language model service system, which comprises an API gateway, a policy controller, a unified parallel orchestrator and a plurality of elastic instances. The above-mentioned modules cooperate with each other, can dynamically determine the five-dimensional parallel strategy according to the real-time request load index of the inference service, and perform uninterrupted reconstruction on the parallel execution mode of the service without restarting.

[0107] The API gateway is arranged at the system entrance and is used for receiving external inference requests and completing functions such as authentication and priority determination, request forwarding, interface management and the like.

[0108] Specifically, the authentication and priority determination is to verify the identity of the calling party, confirm whether the request is legal, and determine the priority of the request according to the preset service level strategy; the request forwarding is to send the authenticated request to the policy controller according to the priority order; the interface management is to support multiple access protocols and provide a unified access interface, so as to ensure the scalability and security of the inference service.

[0109] The policy controller receives the request information from the API gateway and collects the load indicators of the inference service in real time, including the request arrival rate, input sequence length, token throughput, delay indicators, and resource occupation of the current computing instance. The functions of the policy controller include target constraint management, scheduling decision generation, feedback monitoring, etc.

[0110] Among them, the target constraint management is to determine the current system optimization direction according to the service level target, cost and energy consumption constraint; the scheduling decision generation is to generate a global scheduling plan of five-dimensional parallel strategy in combination with the load indicators and optimization target, and send the plan to the unified parallel orchestrator; the feedback monitoring is to monitor the actual execution result after the strategy is issued, and trigger the rollback mechanism when an exception occurs.

[0111] The unified parallel orchestrator is the core scheduling module of the system, which converts the global scheduling plan issued by the policy controller into an executable parallel degree configuration. Its main functions include: Jointly analyze the parallel requirements of data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism and expert parallelism in five dimensions.

[0112] Model the dependency and conflict relationship between different dimensions using parallel causal graphs to ensure feasibility at the execution level.

[0113] According to the optimization target, weight the five-dimensional parallel degree according to the priority, and select the scheme with the optimal balance between benefits and costs from multiple candidate strategies.

[0114] Distribute the analyzed five-dimensional parallel degree configuration to multiple elastic instances to realize the non-interrupted reconstruction of the service.

[0115] The system contains multiple elastic instances, each of which is composed of an execution core, a communication core and a virtualized key-value cache pool.

[0116] Among them, the execution core is responsible for inference calculation, including matrix multiplication, attention calculation and feedforward network operations, to ensure the normal operation of the model on the instance; the communication core is used for data exchange between instances, especially in tensor parallelism, sequence parallelism and expert parallelism, the communication core supports efficient communication of multiple hosts and multiple shards; the virtualized key-value cache pool is managed by paging and indexing at the token granularity, supporting cross-instance shared access. In the inference process, the key-value cache data generated in the pre-filling stage can be directly allocated to the target instance, and can be referenced on demand in the decoding stage, avoiding the delay and memory overhead caused by overall migration.

[0117] When the request arrives, the API gateway completes the authentication and priority determination and delivers the request to the policy controller; the policy controller generates a five-dimensional parallel strategy according to the real-time load; the unified parallel orchestrator analyzes and optimizes the strategy and delivers the configuration to multiple elastic instances; the elastic instances perform inference calculation according to the assigned parallel degree and cooperate with the virtualized key-value cache pool through the communication core to realize efficient cooperation across instances. The entire process does not need to interrupt the service, and the system can complete the dynamic reconstruction of the five-dimensional parallel degree within the iteration period, thereby realizing low-latency, high-throughput and high-resource utilization of the inference service under the condition of localized deployment.

[0118] As shown in Figure 6 The embodiment provides an inference execution method based on sequence parallel communication, mainly including the following steps: Firstly, the system enters the sequence parallel communication stage, in which the input sequence of the request is divided into multiple segments and distributed to different computing instances for parallel processing, and each computing instance only retains the key-value cache data corresponding to the local sequence segment.

[0119] Secondly, the system performs decoding target prediction according to the context length and load characteristics of the input request. In the prediction process, the scheduling module estimates the computing pressure and cache demand that may be generated in the subsequent decoding stage, thereby providing a basis for parallel degree adjustment and cache layout.

[0120] Subsequently, the system selects the storage location of the key-value cache on the communication path. Specifically, the key-value cache data generated in the pre-filling stage is selectively written to disk or distributed according to the token granularity in the sequence parallel communication link, so that the cache data can be directly referenced by the target computing instance in the decoding stage, avoiding large-scale migration.

[0121] After the cache layout is completed, the system performs parallel degree contraction. That is, at the end of the pre-filling stage, the original high parallel degree is reduced to a smaller parallel degree suitable for decoding tasks. In this way, not only can the computing power resources be fully utilized in the pre-filling stage, but also the communication overhead can be reduced in the decoding stage, achieving a balance between delay and throughput.

[0122] Finally, the decoding ready stage is entered, in which each computing instance cooperates to complete the attention calculation and subsequent generation steps by accessing the pre-laid key-value cache, thereby ensuring the continuity of the inference process and the efficient use of computing resources.

[0123] Through the above embodiment, on the basis of sequence parallel communication, the system can complete the prediction and cache layout of the decoding target in the pre-filling stage, and realize dynamic reduction of the parallel degree at the stage switching time, thereby effectively reducing the additional overhead caused by key-value cache migration and improving the performance and stability of large-scale language model localization inference.

[0124] The device embodiments of the present application are described below, which can be used to perform the localized large-scale language model service method in the above embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the above embodiments of the localized large-scale language model service method of the present application.

[0125] Figure 7 A block diagram of a localized large-scale language model service device according to an embodiment of the present application is shown.

[0126] Referring to Figure 7 As shown, the localized large-scale language model service device 500 according to an embodiment of the present application includes a load real-time acquisition module 510, a five-dimensional parallel strategy module 520, and a non-interrupt reconstruction module 530.

[0127] The load real-time acquisition module 510 is configured to acquire a request load indicator of an inference service in real time. The five-dimensional parallel strategy module 520 is configured to determine a five-dimensional parallel strategy according to the request load indicator within a calculation iteration period, the five-dimensional parallel strategy including parallel degrees of five dimensions of data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism, and expert parallelism. The non-interrupt reconstruction module 530 is configured to non-interruptively reconstruct a parallel execution mode of a current service into the five-dimensional parallel strategy and perform the five-dimensional parallel strategy.

[0128] In some possible embodiments of the present application, the five-dimensional parallel strategy module 520 specifically includes a benefit-cost model submodule configured to calculate an optimal five-dimensional parallel strategy combination through a benefit-cost model according to an optimization target based on the request load indicator and a current resource state of a calculation instance.

[0129] In some possible embodiments of the present application, the localized large-scale language model service device further includes: A virtualization cache pool module configured to construct a unified virtualization key-value cache pool spanning multiple calculation instances, wherein key-value cache data is managed in a token granularity for paging and indexing to support cross-calculation instance references.

[0130] In some possible embodiments of the present application, the non-interrupt reconstruction module 530 specifically includes a parallel degree contraction submodule configured to perform parallel degree contraction in a pre-population stage to directly distribute generated key-value cache data to target calculation instances according to the five-dimensional parallel strategy, the target calculation instances being final attribution calculation instances in a decoding stage; and a multi-host decoding submodule configured to perform multi-host multi-shard decoding on the target calculation instances in the decoding stage to obtain local attention calculation results on demand through the virtualization key-value cache pool.

[0131] In some possible embodiments of the present application, the parallelism degree contraction submodule specifically comprises: a key-value cache data generation unit, configured to generate key-value cache data in response to a request execution sequence parallel computing; and a key-value cache data distribution unit, configured to directly distribute and store the generated key-value cache data to a target computing instance by using a communication link in the sequence parallel computing process.

[0132] In some possible embodiments of the present application, the multi-host decoding submodule specifically comprises: an instance dynamic joining unit, configured to dynamically join a new computing instance to a parallel computing group of a current request; and a vector weight receiving unit, configured to enable the new computing instance to receive a query vector and necessary model layer weights of a current decoding step by the virtualized key-value cache pool, participate in local attention computing, and return a computing result.

[0133] In some possible embodiments of the present application, the localized large-scale language model service apparatus further comprises: a recycling strategy driving module, configured to drive a recycling strategy according to a priority of a request, an access frequency and a life cycle of a key-value cache page.

[0134] In this embodiment, the optimal five-dimensional parallelism combination can be automatically switched under different request modes, and the request modes include short context, high concurrency, long context, and expert unbalanced load. The parallelism degree adjustment process does not affect the ongoing inference task, and ensures the continuity of the service. By jointly scheduling in multiple dimensions, memory waste or communication bottlenecks caused by excessive expansion in a single dimension are avoided. In a high fluctuation load scenario, a relatively low average response time and a relatively optimal throughput capacity are maintained, so that the system can dynamically switch in multiple parallel dimensions under the condition of load fluctuation, and the overall computing cost is minimized under the premise of ensuring the quality of service.

[0135] Figure 8 A structural schematic diagram of a computer system of an electronic device suitable for implementing embodiments of the present application is shown.

[0136] It should be noted that, Figure 8 The computer system of the electronic device shown is only an example, and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0137] As Figure 8As shown, the computer system includes a GPU 1801 which can perform various appropriate actions and processes in accordance with a program stored in a Read-Only Memory (ROM) 1802 or a program loaded from the storage section 1808 into a Random Access Memory (RAM) 1803, such as performing the methods described in the above embodiments. In the RAM 1803, various programs and data required for the operation of the system are also stored. The GPU 1801, the ROM 1802, and the RAM 1803 are connected to each other through a bus 1804. An Input / Output (I / O) interface 1805 is also connected to the bus 1804.

[0138] Connected to the I / O interface 1805 are an input section 1806 including a keyboard, a mouse, etc.; an output section 1807 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), etc., and a speaker, etc.; a storage section 1808 including a hard disk, etc.; and a communication section 1809 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1809 performs communication processing via a network such as the Internet. A drive 1810 is also connected to the I / O interface 1805 as necessary. A removable media 1811 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 1810 as necessary, so that a computer program read therefrom is installed in the storage section 1808 as necessary.

[0139] In particular, in accordance with embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program comprising computer programs for performing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 1809, and / or installed from the removable media 1811. When the computer program is executed by the GPU 1801, various functions defined in the system of the present application are performed.

[0140] It should be noted that the computer-readable medium in the embodiments of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus or device. In this application, the computer-readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer-readable computer programs. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, transmit, propagate or transport a program for use by or in connection with an instruction execution system, apparatus or device. The computer programs contained in the computer-readable medium can be transmitted by any suitable medium, including but not limited to wireless, wired, or the like, or any suitable combination thereof.

[0141] The flowcharts and block diagrams in the drawings illustrate the possible implementation architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In the flowcharts or block diagrams, each block can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different orders than that shown in the drawings. For example, two blocks that are shown in succession can actually be executed substantially in parallel, and sometimes in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams or flowcharts, and the combination of blocks in the block diagrams or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0142] The units described in the embodiments of the present application can be implemented by software, or by hardware, or by a combination of software and hardware. The units described can also be located in a single processor. In some cases, the names of the units do not constitute a limitation on the units themselves.

[0143] As another aspect, the present application also provides a computer readable medium, which can be included in the electronic device described in the above embodiments, or can exist separately without being assembled into the electronic device. The computer readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to implement the method described in the above embodiments.

[0144] The present application also provides a computer program product, which stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the method of the above embodiments. The specific implementation process can refer to the specific description of the above embodiments, and will not be repeated here. Figures 1-4 Figures 1-4 The present application also provides a computer program product, which stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the method of the above embodiments. The specific implementation process can refer to the specific description of the above embodiments, and will not be repeated here.

[0145] It should be noted that although several modules or units of the device for action execution are mentioned in the above detailed description, such division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into several modules or units.

[0146] From the above description of the embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.) or a network, and includes several instructions to make a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) execute the methods according to the embodiments of the present application.

[0147] Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the embodiments disclosed herein. The present application is intended to cover any variations, uses, or adaptations of the application following the general principles thereof and including such departures from the present disclosure as come within known use or custom in the art.

[0148] ​It is to be understood that the application is not limited to the precise construction already described above and shown in the drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application. The scope of the application should only be limited by the claims appended hereto.

Claims

1. A method for localizing large-scale language model services, characterized in that, include: Obtain real-time request load metrics for the inference service; Based on the requested load metric, a five-dimensional parallel strategy is determined within the computation iteration cycle. The five-dimensional parallel strategy includes parallelism in five dimensions: data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism, and expert parallelism. The current parallel execution mode of the service is reconstructed without interruption into the five-dimensional parallel strategy and executed.

2. The method for providing localized large-scale language model services as described in claim 1, characterized in that, The step of determining a five-dimensional parallel strategy within the calculation iteration cycle based on the requested load metric specifically includes: Based on the request load metrics and the current resource status of the computing instance, the optimal combination of five-dimensional parallel strategies is calculated using a revenue-cost model according to the optimization objective.

3. The method for providing localized large-scale language model services as described in claim 1, characterized in that, Before reconstructing the current parallel execution mode of the service into the five-dimensional parallel strategy without interruption and executing it, the localized large-scale language model service method further includes: Build a unified virtualized key-value cache pool spanning multiple compute instances, where key-value cache data is managed with pagination and indexing at the token level to support cross-compute instance referencing.

4. The method for providing localized large-scale language model services as described in claim 3, characterized in that, The process of reconstructing the current parallel execution mode of the service into the five-dimensional parallel strategy and executing it specifically includes: During the pre-filling stage, parallelism reduction is performed, and the generated key-value cache data is directly distributed to the target computing instance according to the five-dimensional parallel strategy. The target computing instance is the computing instance to which the decoding stage ultimately belongs. During the decoding phase, multi-host, multi-shard decoding is performed on each target computing instance, and local attention computing results are obtained as needed through the virtualized key-value cache pool.

5. The method for providing localized large-scale language model services as described in claim 4, characterized in that, The parallelism reduction process involves directly distributing the generated key-value cache data to the target computing instance according to a five-dimensional parallel strategy, specifically including: In response to a request, perform sequential parallel computation to generate key-value cache data; By utilizing the communication links in the sequence parallel computing process, the generated key-value cache data is directly distributed and stored on the target computing instance.

6. The method for providing localized large-scale language model services as described in claim 4, characterized in that, The process of performing multi-host, multi-shard decoding on each target computing instance and obtaining local attention computation results on demand through the virtualized key-value cache pool specifically includes: Dynamically add the new compute instance to the currently requested parallel compute group; Through the virtualized key-value cache pool, the new computing instance receives the query vector of the current decoding step and the necessary model layer weights, participates in local attention calculation, and sends the calculation results back.

7. The method for providing localized large-scale language model services as described in claim 3, characterized in that, Following the construction of a unified virtualized key-value cache pool spanning multiple computing instances, the localized large-scale language model service method further includes: The recycling strategy is driven by the priority of requests, the access frequency of key-value cached pages, and their lifecycle.

8. The method for providing localized large-scale language model services as described in claim 1, characterized in that, When the large-scale language model is a hybrid expert model, the localized large-scale language model service method further includes: Real-time monitoring of the load on each expert in the hybrid expert model; When it is detected that the load of a certain expert is consistently higher than a preset threshold, the weight corresponding to that expert is copied or migrated online to one or more computing instances with lower load, and the request routing is adjusted to balance the load among experts.

9. A localized large-scale language model service device, characterized in that, The localized large-scale language model service device includes: The real-time load acquisition module is used to acquire request load metrics for the inference service in real time. The five-dimensional parallel strategy module is used to determine the five-dimensional parallel strategy within the calculation iteration cycle based on the request load index. The five-dimensional parallel strategy includes parallelism in five dimensions: data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism, and expert parallelism. The uninterrupted refactoring module is used to refactor the current parallel execution mode of the service into the five-dimensional parallel strategy without interruption and then execute it.

10. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the localized large-scale language model service method as described in any one of claims 1 to 8.

11. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the localized large-scale language model service method as described in any one of claims 1 to 8.

12. A computer program product comprising one or more computer programs that, when executed by one or more processors, implement the steps of the localized large-scale language model service method according to any one of claims 1 to 8.

Images