Distribution methods, distribution devices, electronic devices and storage media

Abstract

Embodiments of this disclosure provide an allocation method, allocation apparatus, electronic device, and storage medium. The allocation method includes: acquiring multiple configuration combinations, wherein each configuration combination includes configuration information for a first instance and configuration information for a second instance, the first instance and the second instance being used to perform a pre-fill task phase and a decoding task phase of the same inference task, respectively; for at least one configuration combination, performing a target operation, wherein the target operation includes: generating a test data stream for the configuration combination; performing a test process through the first instance and the second instance based on the configuration combination and the test data stream; and determining a target configuration combination from the multiple configuration combinations according to a preset performance index corresponding to the at least one configuration combination during the test process. This method not only significantly reduces the cost of manual debugging but also adapts to different input lengths and request distributions, achieving efficient dynamic resource allocation.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of artificial intelligence, and more specifically to an allocation method, allocation device, electronic device, and storage medium. Background Technology

[0002] Large Language Model (LLM) is a deep neural network model based on the Transformer architecture. By pre-training on a large amount of text data, it has powerful language understanding and generation capabilities.

[0003] Traditional LLM inference architectures typically treat the entire inference process as a single unit, making it difficult to fully utilize hardware resources and easily leading to performance bottlenecks and a degraded user experience. With the rapid increase in the size of model parameters, large-scale language models have significantly increased demands on computing resources, GPU memory capacity, and memory bandwidth during the inference phase, resulting in high inference latency, low throughput, and high deployment costs, severely limiting their availability and scalability in real-world scenarios. Summary of the Invention

[0004] This disclosure provides at least one embodiment of an allocation method, comprising: acquiring a plurality of configuration combinations, wherein each of the plurality of configuration combinations includes configuration information for a first instance and configuration information for a second instance, the first instance and the second instance being used to perform a pre-filling task phase and a decoding task phase of the same inference task, respectively; for at least one of the plurality of configuration combinations, performing a target operation, wherein the target operation includes: generating a test data stream according to the configuration combination; performing a test process through the first instance and the second instance based on the configuration combination and the test data stream; and determining a target configuration combination from the plurality of configuration combinations according to a preset performance index corresponding to the at least one configuration combination during the test process.

[0005] In the allocation method provided in at least one embodiment of this disclosure, after the test process is executed through the first instance and the second instance based on the configuration combination and the test data stream, the target operation further includes: after the test process is completed, calculating the preset performance indicators corresponding to the configuration information for the first instance and the configuration information for the second instance in the configuration combination during the test process, wherein the preset performance indicators include at least one of throughput indicators or latency indicators.

[0006] In at least one embodiment of the allocation method provided in this disclosure, the target operation further includes: in response to the existence of a preset performance indicator that meets a preset invalid condition, determining the configuration information corresponding to the preset performance indicator as invalid configuration information; or in response to the existence of a preset performance indicator that does not meet the preset invalid condition, determining the configuration information corresponding to the preset performance indicator as valid configuration information.

[0007] The allocation method provided in at least one embodiment of this disclosure further includes: in response to the existence of invalid configuration information, determining potential invalid configuration combinations from configuration combinations that have not yet been tested based on the invalid configuration information, and skipping the execution of the target operation for the potential invalid configuration combinations.

[0008] In at least one embodiment of the allocation method provided in this disclosure, generating a test data stream for the configuration combination includes: determining the amount of data in the test data stream according to the configuration combination; and generating the test data stream based on the amount of data.

[0009] In at least one embodiment of the allocation method provided in this disclosure, the step of performing a test process through the first instance and the second instance based on the configuration combination and the test data stream includes: applying the configuration combination to the first instance and the second instance; the first instance receiving and processing the test data stream, generating intermediate data, and transmitting the intermediate data to the second instance; and the second instance receiving and processing the intermediate data to generate result data.

[0010] In at least one embodiment of the allocation method provided in this disclosure, determining a target configuration combination from the plurality of configuration combinations based on preset performance indicators corresponding to the at least one configuration combination during the testing process includes: excluding invalid configuration information from the at least one configuration combination to obtain a plurality of first valid configuration information for the first instance and a plurality of second valid configuration information for the second instance; determining a plurality of valid configuration combinations based on the plurality of first valid configuration information and the plurality of second valid configuration information; calculating the throughput indicators of the plurality of valid configuration combinations respectively, and determining the valid configuration combination with the highest throughput indicator as the target configuration combination.

[0011] The allocation method provided in at least one embodiment of this disclosure further includes: determining a target allocation ratio based on a preset performance index corresponding to the target configuration combination, wherein the target allocation ratio is used to indicate the numerical ratio between the number of the first instance and the number of the second instance.

[0012] In at least one embodiment of the allocation method provided in this disclosure, the target configuration combination includes first target configuration information for the first instance and second target configuration information for the second instance, wherein determining the target allocation ratio according to the preset performance index corresponding to the target configuration combination includes: determining the target allocation ratio according to the ratio of the throughput index corresponding to the second target configuration information to the throughput index corresponding to the first target configuration information.

[0013] In the allocation method provided in at least one embodiment of this disclosure, the type of configuration information includes at least one of the following: computational parallelism strategy, batch size, storage space allocation strategy, cache scheduling strategy, or word generation concurrency.

[0014] In the allocation method provided in at least one embodiment of this disclosure, the configuration information is determined based on hardware resource information.

[0015] In the allocation method provided in at least one embodiment of this disclosure, the first instance and the second instance are used to perform reasoning tasks of a large-scale language model.

[0016] At least one embodiment of this disclosure provides an allocation apparatus, comprising: a configuration module configured to acquire a plurality of configuration combinations, wherein each of the plurality of configuration combinations includes configuration information for a first instance and configuration information for a second instance, the first instance and the second instance being used to perform a pre-filling task stage and a decoding task stage of the same inference task, respectively; a generation module configured to generate a test data stream for at least one of the plurality of configuration combinations; an execution module configured to perform a test process through the first instance and the second instance, based on the configuration combination and the test data stream, for at least one of the plurality of configuration combinations; and a solving module configured to determine a target configuration combination from the plurality of configuration combinations according to a preset performance index corresponding to the at least one configuration combination during the test process.

[0017] At least one embodiment of this disclosure provides an electronic device, including: at least one processor; at least one memory including one or more computer program modules; wherein the one or more computer program modules are stored in the at least one memory and configured to be executed by the at least one processor, and the one or more computer program modules are used to implement the allocation method provided in at least one embodiment of this disclosure.

[0018] At least one embodiment of this disclosure provides a non-transitory computer-readable storage medium having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by at least one processor, perform the allocation method provided in at least one embodiment of this disclosure.

[0019] The allocation method, allocation device, electronic device, and storage medium provided in at least one embodiment of this disclosure solve core challenges in LLM deployment, such as long-tail latency and low resource utilization, by splitting traditional monolithic inference into independent pre-filled and decoding instances and combining dynamic configuration search and hardware-aware optimization. Furthermore, an automated configuration search scheme is proposed, capable of dynamically exploring the optimal parallel configuration of pre-filled and decoding instances. By establishing a performance prediction model and combining it with a search algorithm, the optimal configuration combination and ratio can be automatically solved under different load conditions, maximizing overall throughput. Further, this scheme combines closed-loop feedback of traffic generation and performance analysis, which can quickly eliminate invalid configurations and lock in the optimal solution, significantly reducing the cost of manual parameter tuning. In addition, the modular design improves system flexibility, enabling it to adapt to different hardware environments and load scenarios. Compared to traditional static tuning methods, this scheme not only significantly reduces the cost of manual debugging but also adapts to different input lengths and request distributions, achieving efficient dynamic resource allocation. It is applicable to long text generation and high-concurrency inference scenarios, combining performance, efficiency, and scalability. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure.

[0021] Figure 1 This is a schematic diagram of a PD separation architecture for large-scale language model inference;

[0022] Figure 2 A flowchart illustrating an allocation method provided in at least one embodiment of this disclosure;

[0023] Figure 3 A schematic diagram illustrating the first token delay test results provided in at least one embodiment of this disclosure;

[0024] Figure 4 This is a schematic diagram illustrating the inter-Token latency test results provided in at least one embodiment of this disclosure;

[0025] Figure 5 A schematic block diagram of a dispensing device provided for at least one embodiment of this disclosure;

[0026] Figure 6 A schematic block diagram of another dispensing device provided for at least one embodiment of the present disclosure;

[0027] Figure 7 A schematic block diagram of an electronic device provided for at least one embodiment of this disclosure;

[0028] Figure 8 A schematic block diagram of another electronic device provided for at least one embodiment of this disclosure;

[0029] Figure 9 This is a schematic block diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0031] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms “first,” “second,” and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as “comprising” or “including” mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as “connected” or “linked” are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as “upper,” “lower,” “left,” and “right” are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described objects changes.

[0032] The present disclosure will now be described through several specific embodiments. To keep the following description of the embodiments of the present disclosure clear and concise, detailed descriptions of known functions and known components may be omitted. When any component of an embodiment of the present disclosure appears in more than one drawing, the component is represented by the same or similar reference numerals in each drawing.

[0033] Large-scale language models are deep neural network models based on the Transformer architecture. By pre-training on a large amount of text data, they have powerful language understanding and generation capabilities.

[0034] Traditional LLM inference architectures typically treat the entire inference process as a single unit, making it difficult to fully utilize hardware resources and easily leading to performance bottlenecks and a degraded user experience. With the rapid increase in the size of model parameters, large-scale language models have significantly increased demands on computing resources, GPU memory capacity, and memory bandwidth during the inference phase, resulting in high inference latency, low throughput, and high deployment costs, severely limiting their availability and scalability in real-world scenarios.

[0035] To address the aforementioned issues, the prefill-decode (PD) separation technique was proposed and applied to large-scale language model inference. This technique divides the inference process into two logically independent stages: a computationally intensive prefill stage and a memory-intensive decode stage. By decoupling these two stages, targeted optimizations are achieved. In other words, resources are allocated and system design is optimized based on the computational characteristics of each stage, ultimately significantly improving the overall inference efficiency of the model.

[0036] The pre-filling stage handles user input prompts, preprocesses the text, and generates the first output token and its corresponding key-value cache. In natural language processing, a token is the basic unit of text segmentation; it can be a character, word, or sub-word. A complete sentence typically consists of multiple tokens and is the fundamental unit for model understanding and language generation. Since the input usually contains long context sequences, the pre-filling stage requires a complete forward propagation computation, involving numerous matrix operations, making it a computationally intensive task. Specifically, after the input text is converted into a token sequence by the tokenizer, the model performs a complete forward propagation, generating the first output token. Simultaneously, it calculates and caches the corresponding key (K) and value (V) matrices for the self-attention modules in each Transformer layer, forming a key-value cache for reuse in the decoding stage.

[0037] Here, a lexical unit is the smallest unit into which text data is segmented. It can be a single character, an entire word, or a phrase. A complete sentence consists of multiple lexical units. In natural language processing, lexical units are the basic units by which models process and understand text.

[0038] The decoding phase employs an autoregressive approach to generate output tokens one by one. This phase processes only a single token per step, resulting in relatively low computational cost, but it requires frequent access to and updates to the key-value cache, placing high demands on memory bandwidth; thus, it is a memory-intensive task. In each generation step, the model performs a single-step forward computation based solely on the tokens and key-value cache from the previous step. It avoids repeatedly calculating attention keys from historical contexts, instead directly reading the stored K and V matrices from the key-value cache and appending the K and V matrices corresponding to the newly generated tokens to the cache. By storing and reusing the K and V matrices of historical tokens, redundant computations in the self-attention mechanism are avoided, significantly improving inference efficiency.

[0039] Figure 1 This is a schematic diagram of a PD separation architecture for large-scale language model inference.

[0040] For example, such as Figure 1 As shown, this architecture deploys the pre-filling stage and the decoding stage in the inference process in different computing instances. The core is to completely decouple the two stages, splitting the inference process into two independent and runnable computing instances to improve overall inference efficiency, resource utilization and throughput, and avoid the problem of computing resource contention in traditional hybrid deployments.

[0041] like Figure 1 As shown, the controller is responsible for receiving external requests and distributing them to the corresponding pre-filled instances. Upon receiving a request, the pre-filled instance performs parallel computation on the prompt words to generate a complete KVCache, which is then sent to the decoding instance via a high-speed transmission mechanism (such as RDMA or NVLink). The decoding instance is responsible for the efficient loading of the KVCache and, based on this, executes subsequent autoregressive lexical generation, thus achieving efficient operation of the inference process. Figure 1 The PD separation architecture shown can be configured with multiple pre-filled instances and multiple decoding instances, and the number of pre-filled instances can differ from the number of decoding instances. This multi-instance deployment mode better handles high-concurrency request scenarios, further enhancing the architecture's scalability and processing capabilities.

[0042] The instance mentioned above refers to a resource unit that manages a complete copy of the model weights; that is, an instance is a collection of resources that hosts the model's execution. For example... Figure 1 As shown, each instance may include the model (e.g., an LLM model), hardware device (e.g., a GPU), and runtime environment (e.g., a parallel runtime) required to perform the inference task.

[0043] In practical applications, the computational characteristics and resource requirements of the pre-filling and decoding stages differ significantly. This difference means that the resource allocation between the pre-filling and decoding stages directly impacts the latency, throughput, and utilization of the inference process. Different application scenarios have varying resource allocation requirements for the pre-filling and decoding stages. For example, in real-time dialogue scenarios, users are sensitive to response latency, necessitating priority for low latency and high response speed in the decoding stage. Conversely, in batch text generation tasks, where inputs are longer and requests are concentrated, the computational pressure on the pre-filling stage is greater, requiring more computing power to improve overall throughput. Therefore, it is necessary to adjust the resource allocation ratio between the pre-filling and decoding stages according to the performance metric emphasized in different application scenarios.

[0044] Currently, two main methods are used for resource allocation (PD ratio) between the pre-filling and decoding phases: static allocation and dynamic allocation. Static allocation refers to pre-allocating cluster resources proportionally to the pre-filling and decoding phases during the initial system deployment phase; for example, allocating 70% of resources to the pre-filling phase and 30% to the decoding phase. Dynamic allocation, on the other hand, predicts the resource allocation between the pre-filling and decoding phases based on historical request data during model runtime using algorithms or machine learning models, thus dynamically adjusting resources. While dynamic allocation offers greater flexibility, its implementation complexity and operational costs are correspondingly increased.

[0045] Static allocation schemes face significant challenges in the practical deployment of PD separation systems. Since the pre-filling and decoding stages typically share the same computing environment, their deep coupling in terms of parallel strategies and concurrency configurations makes system tuning extremely complex. Parallel strategies include GPU kernel function configuration, batch size, and parallel modes (tensor parallelism / pipeline parallelism / data parallelism / expert parallelism, etc.), while concurrency configurations include thread allocation and KV cache management.

[0046] This strong coupling results in a large search space for optimization. For example, increasing the parallelism of the pre-filling stage may consume a large amount of memory bandwidth, thereby crowding out memory bandwidth in the decoding stage; conversely, pipeline design adjustments made to optimize the decoding stage may negatively impact the computational throughput of the pre-filling stage. Currently, parameter tuning mainly relies on repeated trial-and-error based on manual experience. This not only requires running complete benchmark tests multiple times, which is time-consuming and labor-intensive, but is also prone to incomplete experimental data due to system fluctuations or task timeouts. The final configuration scheme is often only a locally optimal solution, and its global optimality is difficult to verify. This problem severely restricts the performance of the PD separation architecture in static allocation scenarios, and a systematic and automated method is urgently needed to solve for the optimal resource allocation and achieve efficient and stable inference service deployment.

[0047] This allocation method involves multiple areas such as distributed computing scheduling (e.g., GPU / CPU heterogeneous collaboration), KV Cache transmission optimization (e.g., RDMA, NVLink technology application), and parameter search. It can provide low-latency, high-throughput inference solutions for AI service scenarios such as real-time dialogue and batch text generation, effectively improving the efficiency and practicality of deploying large models in production environments.

[0048] The allocation method provided in at least one embodiment of this disclosure belongs to the field of efficient inference optimization technology for LLM models, focusing on computation-memory decoupling architecture and automated resource scheduling. By splitting traditional monolithic inference into independent pre-filled instances and decoding instances, and combining dynamic configuration search and hardware-aware optimization, it solves core challenges in LLM deployment such as long-tail latency and low resource utilization. This allocation method proposes an automated configuration search scheme that can dynamically explore the optimal parallel configuration of pre-filled instances and decoding instances. By establishing a performance prediction model and combining it with a search algorithm, the optimal configuration combination and optimal ratio can be automatically solved under different load conditions, thereby maximizing overall throughput.

[0049] Furthermore, this solution combines closed-loop feedback between traffic generation and performance analysis, enabling it to quickly eliminate invalid configurations and pinpoint the optimal solution, significantly reducing the cost of manual parameter tuning. In addition, the modular design enhances system flexibility, allowing it to adapt to different hardware environments and load scenarios.

[0050] Compared to traditional static optimization methods, this solution not only significantly reduces the cost of manual debugging, but also adapts to different input lengths and request distributions, achieving efficient dynamic resource allocation. It is suitable for long text generation and high-concurrency inference scenarios, combining performance, efficiency, and scalability.

[0051] Figure 2 This is a flowchart illustrating an allocation method provided for at least one embodiment of the present disclosure. This method can, for example, be applied to a distributed inference system. In embodiments of the present disclosure, the distributed inference system is configured to perform distributed inference on a model, that is, to distribute the inference task across multiple devices for collaborative completion. For example, the distributed inference system may include multiple computing nodes, and each computing node may include multiple devices.

[0052] For example, the device may include a graphics processing unit (GPU), a tensor processing unit (TPU), a general-purpose graphics processing unit (GPGPU), a deep learning processing unit (DPU), an accelerated processing unit (APU), a neural network processing unit (NPU), an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA), etc. Of course, the embodiments of this disclosure are not limited to these, and the device may be any other type.

[0053] For example, such as Figure 2 As shown, the allocation method provided in at least one embodiment of this disclosure may include steps S101 to S103.

[0054] Step S101: Obtain multiple configuration combinations, wherein each configuration combination includes configuration information for a first instance and configuration information for a second instance, and the first instance and the second instance are respectively used to perform the pre-fill task stage and the decoding task stage of the same inference task.

[0055] Step S102: For at least one of the multiple configuration combinations, perform a target operation, wherein the target operation includes: generating a test data stream for the configuration combination; and performing a test process through a first instance and a second instance based on the configuration combination and the test data stream.

[0056] Step S103: Determine the target configuration combination from multiple configuration combinations based on the preset performance indicators corresponding to at least one configuration combination during the testing process.

[0057] For example, the system can pre-configure a configuration database, which stores different configurations for various application scenarios. These configurations can originate from historical deployment configurations, standardized template configurations, etc. In step S101, multiple configuration combinations can be obtained as candidates by querying the configuration database, which serves as the basic set for the optimization search space. For example, each configuration combination can include configuration information for a first instance (also called P-specific configuration) and configuration information for a second instance (also called D-specific configuration). The first instance and the second instance are used to perform the pre-filling task stage and the decoding task stage of the same inference task (e.g., inference task of a large-scale language model), respectively. The first instance is a computational instance used to perform the pre-filling stage in the inference task, and the second instance is a computational instance used to perform the decoding stage in the inference task. These two instances jointly complete the same inference task, but they differ significantly in functionality, resource requirements, and optimization objectives, therefore, they need to be configured separately. The P-specific configuration is the configuration used for the pre-filling task stage, and the D-specific configuration is the configuration used for the decoding task stage.

[0058] For example, the configuration information types include at least one of the following: computational parallelism strategy, batch size, storage space allocation strategy, cache scheduling strategy, or token generation concurrency. The above configuration information types are applicable to both the first and second instances, meaning they can be used in the configuration information for the pre-filling and decoding stages, respectively, but their specific values ​​can be differentiated according to their respective needs.

[0059] For example, computational parallelism strategies can include parallelism modes and their corresponding degrees of parallelism. Parallelism modes include tensor parallelism, pipelined parallelism, data parallelism, and expert parallelism, etc. Each mode is equipped with specific parallelism parameters (tensor parallelism / pipeline parallelism / data parallelism / expert parallelism) to describe how the model is partitioned and the granularity of resource allocation in a distributed environment.

[0060] For example, tensor parallelism represents the number of devices a single tensor of the model (such as a weight matrix, activation values, etc.) is split across in tensor parallelism mode. For example, a tensor parallelism of 2 means that tensors involved in core operations such as matrix multiplication in the model are split into two parts along a specific dimension (such as rows or columns), stored and computed by two separate devices, and finally completed through inter-device communication.

[0061] For example, pipeline parallelism represents the number of devices allocated to the model after it is split into layers in a pipelined parallel mode. For example, when the pipeline parallelism is 4, the multi-layered structure of the model is split into 4 consecutive segments, and each device is responsible for processing one segment. Data flows between devices in sequence, forming a "pipeline" computation.

[0062] For example, data parallelism represents the number of devices involved in data splitting and parallel computation in a data parallel mode. For instance, when the data parallelism is 8, the input data is evenly split into 8 parts, and each of the 8 devices loads a complete copy of the model, independently processes 1 / 8 of the data and calculates the result (inference process) or gradient (training process), and finally aggregates the result or synchronizes the gradient through communication.

[0063] For example, expert parallelism represents the number of devices to which the "expert layer" of a Mixture of Experts (MoE) model is split in expert parallelism mode. For instance, when the expert parallelism is 16, the multiple expert sub-networks (such as 32 experts) contained in the model are evenly distributed across 16 devices, with each device responsible for the computation of 2 experts. After the active experts are dynamically selected through a gating network, the data only flows to the corresponding device for processing.

[0064] By selecting appropriate parallel modes and corresponding degrees of parallelism, a distributed inference architecture suitable for the target task and hardware environment can be constructed, thereby optimizing resource utilization and inference performance.

[0065] For example, batch size refers to the number of input samples (or requests) processed in parallel during a single forward propagation, which can affect the model's computational efficiency, memory usage, throughput, and latency.

[0066] In practical applications, the appropriate batch size and computational parallelism strategy can be selected based on the specific scenario and performance requirements. For example, when the input sequence length is 2048 and the output sequence length is 512, it falls under the category of long sequence processing, which places high demands on computing resources and GPU memory usage. In this case, the batch size can be set to a smaller range (e.g., 4-16) to avoid excessive single-processing load, and a tensor parallelism strategy (tensor parallelism of 4 or 8) can be combined to use multi-GPU collaborative computing to reduce latency. Conversely, if the input and output sequences are shorter, a larger batch size can be used to improve throughput efficiency, and the parallelism can be appropriately reduced to optimize resource utilization while ensuring performance.

[0067] For example, storage space allocation strategy refers to the planning and management of storage resources such as GPU memory. One example is the memory allocation strategy for KV Cache (that is, the proportion of memory reserved for KV Cache during system startup or instance initialization). By setting a reasonable storage space allocation strategy, not only can runtime memory fragmentation problems be effectively avoided, but memory utilization and system throughput can also be improved.

[0068] For example, cache loading strategies can refer to the loading and unloading strategies employed by the KV cache to optimize memory usage efficiency and access latency. For instance, a KV cache loading strategy refers to how to efficiently load cached K and V matrices from host storage (e.g., CPU memory) to device storage (e.g., GPU memory), which can include preloading, on-demand loading, and chunked loading. A KV cache unloading strategy refers to temporarily transferring inactive KV cache data to host storage (e.g., CPU memory) when device storage (e.g., GPU memory) is insufficient, which can include priority-based unloading and activity-based unloading. Essentially, cache loading strategies manage the lifecycle of the KV cache. By properly configuring loading and unloading strategies, inference systems can support higher concurrency, longer context, and lower latency inference services with limited hardware resources.

[0069] Token generation concurrency refers to the number of requests that generate output tokens concurrently during the LLM inference process, affecting the system's throughput, latency, and resource utilization.

[0070] For example, P-specific configurations are typically optimized for the computationally intensive characteristics of the pre-filling phase, and typical configuration items may include computational parallelism strategies, batch sizes, and storage allocation strategies.

[0071] For example, D-specific configurations are typically optimized for the memory-intensive characteristics of the decoding stage. Typical configuration items may include cache loading strategies and word generation concurrency.

[0072] In some examples, global configuration information can also be obtained from the configuration database. This global configuration information may include shared parameters such as input sequence length, output sequence length, and model path. These parameters have a common constraint effect on the pre-filling and decoding stages, ensuring that the first instance (pre-filling example) and the second instance (decoding instance) run collaboratively in a consistent task context and model environment, avoiding computational errors, cache mismatches, or communication failures caused by inconsistent global configurations. Based on this, the inference system implements a hierarchical configuration management architecture by dividing the configuration into global configuration information, P-specific configuration, and D-specific configuration, thereby improving the flexibility, maintainability, and scalability of configuration management.

[0073] In some examples, configuration information can be dynamically determined by combining underlying hardware resource information to match resource configuration with physical device capabilities. For instance, when obtaining configuration combinations, hardware resource information of the current deployment environment (such as GPU model, memory capacity, memory bandwidth, number of cores, interconnect capabilities, etc.) can be obtained and analyzed in real time, and used as constraints and optimization basis for generating configuration combinations to avoid performance bottlenecks caused by hardware limitations.

[0074] In step S102, a target operation can be performed on at least one of the multiple configuration combinations. The target operation may include steps S201 to S202. That is, steps S201 to S202 are performed on each of the at least one configuration combination. It should be noted that step S102 does not limit the target operation to be performed on every single configuration combination obtained in step S101.

[0075] Step S201: Generate a test data stream for this configuration combination.

[0076] Step S202: Based on the configuration combination and test data stream, execute the test process through the first instance and the second instance.

[0077] In step S201, a matching test data stream can be generated in real time as test cases according to the test scenario or test requirements to cover the performance of the first and second instances under different load pressures under the configuration combination.

[0078] For example, one embodiment of step S201 may include: determining the concurrency scale and data length of the test data stream based on the test scenario or test requirements; and generating the test data stream based on the concurrency scale and data length. For example, the test data stream can be dynamically generated to match the test scenario or test requirements by adjusting the concurrency scale and data length to simulate real load. For example, the test data stream can be obtained by simulating real input using randomly generated data (e.g., random word sequences) based on the set concurrency scale and data length.

[0079] For example, concurrency can refer to the rate at which data is distributed after generation or the number of sessions executed in parallel. It can simulate the number of user sessions running simultaneously in the real world, used to simulate different levels of load pressure. Some examples include requests per second, the number of concurrent users, or the number of sessions. For instance, low concurrency can result in 5-20 requests per second, medium concurrency in 20-100 requests per second, and high concurrency in more than 100 requests per second. It should be noted that the above is just one example and can be set according to the actual application scenario.

[0080] For example, data length can represent the length of the input or output data for a single request, and can be set according to the specific business scenario deployed by the user. For instance, in generative tasks, the input data length can be reflected in the length of the prompt words, and the output data length can be reflected in the length of the generated text. Length can be represented by the number of lemmas. For example, in a chat scenario, it can be set to short text mode, with both the input and output data containing 50-100 lemmas; in a document generation scenario, it can be set to long text mode, with the input data containing approximately 200 lemmas and the output data containing more than 500 lemmas. It should be noted that the above are just some examples; the specific data length needs to be set according to the actual application scenario.

[0081] For example, one embodiment of step S201 could include: determining the amount of data in the test data stream based on the configuration combination; and generating the test data stream based on the data amount. Alternatively, the test data stream could be generated based on the concurrency scale, data length, and data amount determined above.

[0082] For example, to avoid an excessively large search space affecting testing efficiency, the amount of data in the test data stream (the number of data samples generated each time or the total amount of data) can be dynamically determined based on the current test configuration combination. For instance, by combining parameters such as batch size or parallelism in the configuration combination, the number of data samples generated each time and the total amount of data in the test data stream can be reasonably set, so that the test load matches the system capacity, improving the targeting and execution efficiency of the test.

[0083] For example, when generating test data streams, the number of data samples generated each time can be set with reference to the batch size. For instance, assuming a batch size of 64, 64 data samples of similar length can be generated each time, with minor random fluctuations allowed. The total data volume can also be set with reference to the batch size; for example, to obtain stable and reliable latency metrics later, the total data volume can be set to more than 10 times the batch size.

[0084] Since the batch sizes of the P-specific and D-specific configurations in the current test may differ, the average of their batch sizes can be used as a reference. The parallelism can also be adjusted accordingly to improve the rationality and efficiency of the testing process.

[0085] In step S202, based on the current test configuration combination and the test data stream generated in step S201, the first instance and the second instance can be driven to collaboratively execute an end-to-end inference test process to evaluate the system performance under different resource configurations.

[0086] An example of step S202 may include: applying a configuration combination to a first instance and a second instance; the first instance receiving and processing a test data stream, generating intermediate data, and passing the intermediate data to the second instance; and the second instance receiving and processing the intermediate data to generate result data.

[0087] For example, the P-specific configuration from the configuration combination can be applied to the first instance (pre-filling example), and the D-specific configuration from the configuration combination can be applied to the second instance (decoding instance). Furthermore, global configuration information can be applied to both the first and second instances simultaneously. Next, the configured first and second instances can be used to process the test data stream. Specifically, the first instance receives and processes the test data stream (i.e., performs the pre-filling operation), generates intermediate data, and passes the intermediate data to the second instance; the intermediate data can be, for example, a KV cache. The second instance receives and processes the intermediate data (i.e., performs the decoding operation), generating result data; the result data can be, for example, tokens, or a complete text result generated word-by-word.

[0088] For example, the pre-fill instance and the decoding instance are designed as two logically independent computational instances that can be deployed and executed separately, supporting a serialized deployment architecture. This means they can run on different physical or virtual nodes and interact with each other via an interface. During runtime, the pre-fill instance simulates a real KV cache transfer process. After completing the pre-fill phase tasks, the pre-fill instance sends the generated KV cache to the pre-bound decoding instance, thus accurately assessing cross-instance communication overhead. The decoding instance can directly load the pre-generated KV cache and perform autoregressive decoding. This loosely coupled architecture between the pre-fill and decoding instances allows them to be optimized for computationally intensive and memory-intensive tasks respectively, and supports horizontal scaling (e.g., pre-fill instances process long prompts in parallel across multiple nodes, while decoding instances handle high-concurrency requests in a distributed manner), improving system flexibility and adapting to different hardware environments and load scenarios.

[0089] In step S103, the test process corresponding to each configuration combination can be quantitatively evaluated and compared according to preset performance indicators, and the optimal configuration scheme that meets the performance target can be selected to obtain the target configuration combination. For example, preset performance indicators may include throughput indicators that reflect the ability to process requests or generate content per unit time, or latency indicators that measure the response speed and generation efficiency of user requests. Throughput indicators may include, for example, the number of requests processed per second (QPS) and the number of tokens output per second (TPS), or other equivalent indicators may be selected according to actual needs, and this embodiment of the disclosure does not limit this. Latency indicators may include, for example, the first token delay (TTFT) and the inter-token latency (ITL), or other equivalent indicators may be selected according to actual needs, and this embodiment of the disclosure does not limit this.

[0090] For example, various strategies can be flexibly adopted based on actual optimization needs. For instance, a single-objective strategy can be used, focusing on a single key performance indicator, such as selecting the configuration combination with the highest QPS or TPS as the target combination, or selecting the configuration combination with the lowest average TTFT or average ITL as the target configuration combination. Alternatively, a multi-objective strategy can be used, selecting multiple performance indicators and assigning weight coefficients to each, evaluating them using a weighted comprehensive scoring function, and selecting the configuration combination with the highest score as the target configuration combination. Furthermore, indicator constraints can be set to filter out invalid configuration combinations, and then further apply single-objective or multi-objective strategies to the remaining configuration combinations. By combining single-objective strategies, multi-objective strategies, and indicator constraint filtering, the target configuration combination that best meets actual needs can be flexibly identified from a large number of configuration combinations, improving the scalability of resource allocation. In some examples, P-specific configurations and D-specific configurations from multiple configuration combinations can also be recombined, for example, selecting the highest-performing P-specific configuration and the highest-performing D-specific configuration to combine into a target configuration combination.

[0091] The allocation method provided in at least one embodiment of this disclosure may further include step S104, which may be performed after step S103.

[0092] Step S104: Determine the target ratio based on the preset performance index corresponding to the target configuration combination, wherein the target ratio is used to indicate the numerical ratio between the number of first instances and the number of second instances.

[0093] For example, since the target configuration combination has been determined in step S103, the target allocation ratio can be further determined based on the preset performance indicators corresponding to the target configuration combination. An example of the target allocation ratio is Np:Nd, where Np represents the number of first instances and Nd represents the number of second instances.

[0094] For example, the target configuration combination may include first target configuration information for a first instance and second target configuration information for a second instance. An example of step S104 may include: determining a target ratio based on the ratio of the throughput metric corresponding to the second target configuration information to the throughput metric corresponding to the first target configuration information.

[0095] For example, the target allocation ratio can be determined based on the throughput metrics described above. One example is Np:Nd = QPSd:QPSp, where QPSd is the QPS of the second instance using the target configuration combination (i.e., the QPS metric corresponding to the second target configuration information), and QPSp is the QPS of the first instance using the target configuration combination (i.e., the QPS metric corresponding to the first target configuration information). For example, if a single second instance can complete 20 requests per second for decoding and generation (QPSd=20), and a single first instance can handle 10 requests per second for pre-filling calculations (QPSp=10), then Np:Nd = QPSd:QPSp = 2:1. This means that to match the processing capabilities of decoding instances and pre-filling instances, for every 1 decoding instance deployed, 2 pre-filling instances need to be deployed. Depending on actual needs, the obtained target allocation ratio can be processed (normalized, truncated, rounded, etc.) to avoid inapplicability (e.g., exceeding hardware resource limits). The above is just one example; other throughput metrics (e.g., TPS) can also be used to calculate the ratio based on actual needs.

[0096] In the allocation method provided by at least one embodiment of the present disclosure, determining the target allocation ratio can indicate how to allocate computing resources during actual deployment. By reasonably allocating the number of instances, a dynamic balance between resources and tasks can be achieved, so that the processing capabilities of the pre-filling stage and the decoding stage are matched, thereby further improving resource utilization and inference efficiency.

[0097] In at least one embodiment of this disclosure, the target operation in step S102 may further include step S203. Step S203 may be executed after step S202.

[0098] Step S203: After the test process is completed, calculate the preset performance indicators corresponding to the configuration information for the first instance and the configuration information for the second instance in the configuration combination during the test process.

[0099] For example, during testing, runtime performance data for the first and second instances under load scenarios can be recorded separately. After the test, the configuration information (i.e., P-specific configuration and D-specific configuration) used for the first and second instances in the current test configuration combination can be evaluated for performance. Specifically, statistical values ​​related to preset performance indicators can be calculated and extracted based on the recorded runtime performance data to quantify the actual performance of the two instances. Preset performance indicators may include at least one of throughput indicators or latency indicators, as described above, and will not be repeated here.

[0100] Figure 3 This is a schematic diagram illustrating the first token delay test results provided for at least one embodiment of this disclosure.

[0101] For example, such as Figure 3 As shown, each curve represents the test result of a test data stream. The horizontal axis represents the processing steps (Step), with each processing step corresponding to one input request data. The vertical axis represents the first token delay time (in milliseconds). Test data stream 1 has a concurrency of 64, an input data length of 2048, and an output data length of 1024; test data stream 2 has a concurrency of 32, and its input / output data lengths are the same as test data stream 1. Figure 3 As can be seen, the initial token latency curve of test data stream 1 is generally higher than that of test data stream 2, reflecting the increased resource contention and scheduling latency caused by high concurrency. This curve is used to determine the preset performance indicators corresponding to the configuration information of the pre-populated instance.

[0102] Figure 4 This is a schematic diagram illustrating the inter-Token latency test results provided in at least one embodiment of this disclosure.

[0103] For example, such as Figure 4 As shown, each curve represents the test result of a test data stream. The horizontal axis represents the processing steps (Step), with each processing step corresponding to one input request data. The vertical axis represents the latency between tokens (in milliseconds). Test data stream 1 has a concurrency of 64, an input data length of 2048, and an output data length of 1024; test data stream 2 has a concurrency of 32, and its input / output data lengths are the same as test data stream 1. Figure 4 As can be seen, the inter-token latency test curve for test data stream 1 is generally higher than that for test data stream 2, reflecting the increased resource contention and scheduling latency caused by high concurrency. This curve is used to determine the preset performance indicators corresponding to the configuration information of the decoding instance.

[0104] In at least one embodiment of this disclosure, the target operation may further include step S204.

[0105] Step S204: In response to the existence of a preset performance indicator that meets the preset invalidity condition, the configuration information corresponding to the preset performance indicator is determined to be invalid configuration information.

[0106] In some examples, the default invalidation condition can be set based on the high percentile latency. One example of high percentile latency is the 90th percentile first token latency (P90 TTFT), which means that in a test, 90% of the requests have a first token latency of no more than this value. For example, when P90 TTFT = 500ms, it means that 90% of the requests have a first token latency of less than or equal to 500ms.

[0107] In step S204, when P90 TTFT is greater than or equal to a preset threshold (e.g., P90 TTFT > 1s), it can be determined that the first token delay meets the preset invalidity condition. It should be noted that the preset threshold can be set according to actual needs, and this embodiment does not limit it.

[0108] Besides P90 TTFT, examples of high percentile delays may also include the 99th percentile first token delay (P99TTFT), the 90th percentile inter-token delay (P90 ITL), the 99th percentile inter-token delay (P99 ITL), etc., and this disclosure does not limit these embodiments.

[0109] In some examples, the preset invalidation condition can be set based on the average value of latency-related metrics. For example, when the average latency of the first token is greater than or equal to a preset threshold, it can be determined that the latency of the first token meets the preset invalidation condition; when the average latency between tokens is greater than or equal to a preset threshold, it can be determined that the latency between tokens meets the preset invalidation condition.

[0110] In some examples, the preset invalidation condition can also be set based on the throughput metric. For instance, when the throughput metric is less than a preset threshold, it can be determined that the throughput metric meets the preset invalidation condition.

[0111] For example, when a preset performance indicator is determined to meet a preset invalid condition, the configuration information corresponding to the preset performance indicator can be determined as invalid configuration information and marked as "invalid" for subsequent processing.

[0112] Here is an example: Step 203 yields the first token delay for the P-specific configuration and the inter-token delay for the D-specific configuration in the current test configuration combination. In this example, P90 TTFT and P90 ITL are selected as the benchmarks for judging the preset invalidity conditions. If P90 TTFT > 1s, the P-specific configuration in the current test configuration combination is determined to be invalid configuration information; if P90 ITL > 30ms, the D-specific configuration in the current test configuration combination is determined to be invalid configuration information.

[0113] In at least one embodiment of this disclosure, the target operation may further include step S205.

[0114] Step S205: In response to the existence of a preset performance indicator that does not meet the preset invalidity condition, the configuration information corresponding to the preset performance indicator is determined as valid configuration information.

[0115] For step S205, please refer to the partial description of step S204 above. For example, when it is determined that a certain preset performance indicator does not meet the preset invalidity condition, the configuration information corresponding to the preset performance indicator can be determined as valid configuration information and marked as "valid" for subsequent process processing.

[0116] Here is an example: Step 203 yields the first token delay corresponding to the P-specific configuration in the current test configuration combination, and the inter-token delay corresponding to the D-specific configuration. In this example, P90 TTFT and P90 ITL are selected as the evaluation criteria for the preset invalidity condition. If P90 TTFT ≤ 1s, the P-specific configuration in the current test configuration combination is determined to be valid configuration information; if P90 ITL ≤ 30ms, the D-specific configuration in the current test configuration combination is determined to be valid configuration information.

[0117] In the allocation method provided by at least one embodiment of the present disclosure, valid configuration information and invalid configuration information can be filtered out through real-time performance analysis, thereby making a preliminary judgment on configuration feasibility and improving iteration efficiency.

[0118] The allocation method provided in at least one embodiment of this disclosure may further include step S105.

[0119] Step S105: In response to the existence of invalid configuration information, determine potential invalid configuration combinations from the configuration combinations that have not yet been tested based on the invalid configuration information, and skip performing the target operation for the potential invalid configuration combinations.

[0120] In step S105, combined with the invalid configuration information fed back in step S204, other potentially invalid configuration combinations can be inferred, and the test execution of these potentially invalid configuration combinations can be skipped to achieve search path pruning.

[0121] Taking batch size as an example, a larger value usually leads to higher memory usage and computational pressure. Therefore, when configuration information with a batch size of 32 is identified as invalid, it can be inferred that configuration information with a batch size exceeding 32 belongs to potentially invalid configuration information. Configuration combinations composed of potentially invalid configuration information should also be considered potentially invalid and can be pruned during the search process, that is, skipping the test of potentially invalid configuration combinations to avoid resource waste and improve search efficiency.

[0122] The pruning strategy based on batch size described above is just one example. This pruning mechanism can also be extended to other configuration types, which can be set according to actual needs.

[0123] For example, the pruning mechanism described above can also be applied to pruning test data stream parameters. Taking concurrency as an example, if invalid configuration information has already been generated when the concurrency is 64, it can be inferred that higher concurrency will lead to more severe performance degradation. Therefore, there is no need to perform tests on larger concurrency levels (such as 96, 128, etc.). Instead, the corresponding test paths can be marked as potentially invalid and skipped.

[0124] The allocation method proposed in at least one embodiment of this disclosure achieves a closed-loop feedback between test stream generation and performance analysis through a pruning mechanism based on invalid configuration information. This not only effectively avoids the waste of computing resources caused by performing tests on invalid configurations, but also dynamically optimizes the search strategy based on historical data accumulation, reducing the cost of manual parameter tuning. For example, by prioritizing the exploration of configuration combinations with lower memory usage or lower resource requirements, the convergence speed to the feasible solution region is accelerated, thereby improving the overall tuning efficiency.

[0125] In the allocation method provided in at least one embodiment of this disclosure, an example of step S103 may include steps S301 to S303.

[0126] Step S301: Eliminate invalid configuration information from at least one configuration combination to obtain multiple first valid configuration information for the first instance and multiple second valid configuration information for the second instance.

[0127] Step S302: Determine multiple valid configuration combinations based on multiple first valid configuration information and multiple second valid configuration information.

[0128] Step S303: Calculate the throughput index of multiple effective configuration combinations respectively, and determine the effective configuration combination with the highest throughput index as the target configuration combination.

[0129] In step S301, configuration information marked as "invalid" can be excluded and configuration information marked as "valid" can be retained based on the execution result of step S204 or step S205. "Multiple first valid configuration information for the first instance" can be understood as multiple valid P-specific configurations, and "multiple second valid configuration information for the second instance" can be understood as multiple valid D-specific configurations.

[0130] In step S302, valid P-specific configurations and valid D-specific configurations can be paired and combined. For example, assuming there are M valid P-specific configurations and N valid D-specific configurations, pairing and combining them can yield M×N valid configuration combinations.

[0131] In step S303, the throughput metrics (e.g., QPS) of the M×N effective configuration combinations can be calculated respectively. The effective configuration combination with the highest throughput metric is then identified as the optimal configuration combination. To further improve the solution efficiency, this process can be combined with heuristic algorithms (e.g., greedy search, Bayesian optimization, or reinforcement learning) to reduce unnecessary computational overhead and accelerate the search for the optimal solution by prioritizing the exploration of potentially better configuration spaces.

[0132] After determining the target configuration combination based on the above steps S301 to S303, step S104 described above can be executed to determine the target ratio, and finally the optimal configuration combination and optimal ratio can be output.

[0133] The following is an example of an allocation method proposed in at least one embodiment of this disclosure.

[0134] First, multiple configuration combinations are retrieved from the configuration database. Each combination includes P-specific configurations and D-specific configurations, and may also include global configurations. Next, one of these configuration combinations is selected as the current test configuration combination. A test data stream is generated for this current test configuration combination. Specifically, the concurrency scale and data length of the test data stream are determined based on the test scenario or test requirements; the data volume in the test data stream is determined based on the current test configuration combination; and the test data stream is generated based on the concurrency scale, data length, and data volume. Next, based on the current test configuration combination and the generated test data stream, the test process is executed through pre-populated instances and decoding instances. Specifically, the current test configuration combination is applied to pre-populated instances and decoding instances; the pre-populated instance receives and processes the test data stream, generates intermediate data, and passes the intermediate data to the second instance; the decoding instance receives and processes the intermediate data, generating result data. After the test process is completed, the preset performance metrics (e.g., QPS, TPS, TTFT, ITL) corresponding to the P-specific configuration and D-specific configuration in the current test configuration combination are calculated during the test. It is then determined whether the preset performance metrics corresponding to the P-specific configuration and the D-specific configuration meet the preset invalidation conditions. In response to the existence of a preset performance indicator that meets a preset invalidity condition, the configuration information corresponding to the preset performance indicator is determined to be invalid configuration information; in response to the existence of a preset performance indicator that does not meet the preset invalidity condition, the configuration information corresponding to the preset performance indicator is determined to be valid configuration information. In response to the existence of invalid configuration information, potential invalid configuration combinations are determined from the configuration combinations that have not yet been tested, and the above tests are skipped for the potential invalid configuration combinations. After performing the above tests on all non-potential invalid configuration combinations, invalid configuration information can be excluded from all non-potential invalid configuration combinations to obtain multiple first valid configuration information for the first instance and multiple second valid configuration information for the second instance; multiple valid configuration combinations are determined based on the multiple first valid configuration information and multiple second valid configuration information; the throughput index of each of the multiple valid configuration combinations is calculated, and the valid configuration combination with the highest throughput index is determined as the target configuration combination. The target configuration combination includes first target configuration information for pre-populating instances and second target configuration information for decoding instances, and the target ratio is determined based on the ratio of the throughput index corresponding to the second target configuration information to the throughput index corresponding to the first target configuration information.

[0135] In the allocation method provided in at least one embodiment of this disclosure, by solving based on the performance analysis results, the allocation of computing resources and memory resources can be dynamically balanced, and the system throughput can be maximized under the premise of strictly satisfying latency constraints.

[0136] It should also be noted that the execution order of the various steps of the allocation method in the various embodiments of this disclosure is not limited. Although the execution process of each step has been described in a specific order above, this does not constitute a limitation on the embodiments of this disclosure. The various steps in the allocation method can be executed serially or in parallel, which can be determined according to actual needs.

[0137] For example, compared to the above description, the allocation method provided in at least one embodiment of this disclosure may include more or fewer steps, and the embodiments of this disclosure do not limit this.

[0138] Figure 5 This is a schematic block diagram of an allocation device provided for at least one embodiment of the present disclosure. The allocation device may, for example, be a component of a distributed inference system.

[0139] For example, such as Figure 5 As shown, the allocation device 500 provided in at least one embodiment of this disclosure includes a configuration module 501, a generation module 502, an execution module 503, and a solution module 504.

[0140] In some examples, configuration module 501 is configured to obtain multiple configuration combinations, wherein each of the multiple configuration combinations includes configuration information for a first instance and configuration information for a second instance, the first instance and the second instance being used to perform the pre-fill task phase and the decoding task phase of the same inference task, respectively.

[0141] In some examples, generation module 502 is configured to generate a test data stream for at least one of a plurality of configuration combinations.

[0142] In some examples, execution module 503 is configured to perform a test process through a first instance and a second instance, based on at least one of a plurality of configuration combinations and a test data stream.

[0143] In some examples, the solver module 504 is configured to determine a target configuration combination from multiple configuration combinations based on preset performance metrics corresponding to at least one configuration combination during the test.

[0144] For example, in at least one embodiment of this disclosure, the generation module 502 is further configured to determine the amount of data in the test data stream according to the configuration combination; and generate the test data stream based on the amount of data.

[0145] For example, in at least one embodiment of this disclosure, the execution module 503 is further configured to apply the configuration combination to the first instance and the second instance; the first instance receives and processes the test data stream, generates intermediate data, and transmits the intermediate data to the second instance; the second instance receives and processes the intermediate data and generates result data.

[0146] For example, the allocation device 500 provided in at least one embodiment of this disclosure may further include a performance analysis module. The performance analysis module is configured to, for at least one of a plurality of configuration combinations, calculate, after the test process is completed, preset performance indicators corresponding to the configuration information for the first instance and the configuration information for the second instance in the configuration combination during the test process, wherein the preset performance indicators include at least one of throughput indicators or latency indicators.

[0147] For example, in at least one embodiment of this disclosure, the performance analysis module is further configured to determine the configuration information corresponding to the preset performance indicator as invalid configuration information in response to the existence of a preset performance indicator that meets the preset invalid condition; or to determine the configuration information corresponding to the preset performance indicator as valid configuration information in response to the existence of a preset performance indicator that does not meet the preset invalid condition.

[0148] For example, in at least one embodiment of this disclosure, the generation module 502 is further configured to determine potential invalid configuration combinations from configuration combinations that have not yet been tested based on invalid configuration information in response to the existence of invalid configuration information. Correspondingly, the execution module 503 is further configured to skip performing the target operation for the potential invalid configuration combinations.

[0149] For example, in at least one embodiment of this disclosure, the solving module 504 is further configured to exclude invalid configuration information from at least one configuration combination to obtain multiple first valid configuration information for a first instance and multiple second valid configuration information for a second instance; determine multiple valid configuration combinations based on the multiple first valid configuration information and the multiple second valid configuration information; calculate the throughput index of the multiple valid configuration combinations respectively, and determine the valid configuration combination with the highest throughput index as the target configuration combination.

[0150] For example, in at least one embodiment of this disclosure, the solving module 504 is further configured to determine a target ratio based on a preset performance index corresponding to the target configuration combination, wherein the target ratio is used to indicate the numerical ratio between the number of first instances and the number of second instances.

[0151] For example, in at least one embodiment of this disclosure, the target configuration combination includes first target configuration information for a first instance and second target configuration information for a second instance, and the solving module 504 is further configured to determine the target ratio based on the ratio of the throughput index corresponding to the second target configuration information to the throughput index corresponding to the first target configuration information.

[0152] For example, in at least one embodiment of this disclosure, the type of configuration information includes at least one of the following: computational parallelism strategy, batch size, storage space allocation strategy, cache scheduling strategy, or word generation concurrency.

[0153] For example, in at least one embodiment of this disclosure, the configuration information is determined based on hardware resource information.

[0154] For example, in at least one embodiment of this disclosure, the first instance and the second instance are used to perform reasoning tasks for large-scale language models.

[0155] It should be noted that the above-mentioned modules and units can be implemented by software, hardware, firmware or any combination thereof. For example, the configuration module, generation module, execution module and solving module can be implemented as configuration circuit, generation circuit, execution circuit and solving circuit respectively. The embodiments of this disclosure do not limit their specific implementation methods.

[0156] It should be understood that the dispensing device 500 provided in at least one embodiment of this disclosure can be used to implement the aforementioned dispensing method and can also achieve similar technical effects as the aforementioned dispensing method, which will not be elaborated here.

[0157] It should be noted that in the embodiments of this disclosure, the distribution device 500 may include more or fewer modules or units, and the connection relationship between the various modules or units is not limited and can be determined according to actual needs. The specific configuration of each module or unit is not limited; it can be constructed from analog devices according to circuit principles, or from digital chips, or in other suitable ways.

[0158] Figure 6 This is a schematic block diagram of another allocation device provided for at least one embodiment of the present disclosure. This allocation device may, for example, be a component of a distributed inference system.

[0159] For example, such as Figure 6 As shown, the allocation apparatus provided in at least one embodiment of this disclosure may include a configuration module, a generation module, an execution module, a performance analysis module, and a solution module. The functions of each module can be referred to the descriptions in the above embodiments. The configuration module can provide P-specific configuration, D-specific configuration, and global configuration; details can be found in the above method embodiments, and will not be repeated here. The execution module can support the operation of pre-filled instances and decoding instances, and can configure only one pre-filled instance and one decoding instance. The generation module can generate test data streams for the configuration combinations provided by the configuration module and prune test paths based on feedback from the performance analysis module. The performance analysis module includes a P performance analysis unit and a D performance analysis unit. The P performance analysis unit is used for performance analysis of the first instance (pre-filled instance), and the D performance analysis unit is used for performance analysis of the second instance (decoding instance). The results of the performance analysis are sent to the solution module to solve for the optimal configuration combination and optimal ratio; details can be found in the above method embodiments, and will not be repeated here.

[0160] The allocation apparatus provided in at least one embodiment of this disclosure is equivalent to an automated configuration search platform, which can dynamically explore the optimal parallel configuration of pre-filled instances and decoding instances. By establishing a performance prediction model and combining it with a search algorithm, the optimal configuration combination and optimal ratio can be automatically solved under different load conditions, thereby maximizing the overall throughput. Furthermore, this solution combines closed-loop feedback of traffic generation and performance analysis, which can quickly eliminate invalid configurations and lock in the optimal solution, significantly reducing the cost of manual parameter tuning. In addition, the modular design improves the system flexibility, enabling it to adapt to different hardware environments and load scenarios. Compared with traditional static tuning methods, this solution not only significantly reduces the cost of manual debugging, but also adapts to different input lengths and request distributions, achieving efficient dynamic resource allocation. It is suitable for long text generation and high-concurrency inference scenarios, combining performance, efficiency, and scalability.

[0161] Figure 7 This is a schematic block diagram of an electronic device provided for at least one embodiment of the present disclosure.

[0162] For example, such as Figure 7 As shown, the electronic device 700 includes at least one processor 701 and at least one memory 702. The at least one memory 702 includes one or more computer program modules. These computer program modules are stored in the memory 702 and configured to be executed by the at least one processor 701. The one or more computer program modules include instructions for performing the allocation method described above. When executed by the at least one processor 701, they can perform one or more steps of the allocation method provided in at least one embodiment of this disclosure. The memory 702 and the processor 701 can be interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0163] For example, processor 701 can be a central processing unit (CPU), digital signal processor (DSP), graphics processing unit (GPU), general-purpose graphics processing unit (GPGPU), artificial intelligence (AI) accelerator, or other form of processing unit with data processing and / or program execution capabilities, such as a field-programmable gate array (FPGA); for example, the central processing unit (CPU) can be an x86, ARM, or RISC-V architecture. Processor 701 can be a general-purpose processor or a special-purpose processor, capable of controlling other components in electronic device 700 to perform desired functions.

[0164] For example, memory 702 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc.

[0165] Figure 8 This is a schematic block diagram of another electronic device provided for at least one embodiment of the present disclosure.

[0166] The electronic devices in at least one embodiment of this disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable multimedia players (PMPs), in-vehicle terminals (e.g., in-vehicle navigation terminals), wearable electronic devices, and fixed terminals such as digital TVs and desktop computers. Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0167] The electronic device includes at least one processor and a memory. The processor may be referred to as processing device 801 as described below, and the memory may include at least one of read-only memory (ROM), random access memory (RAM), and storage device 808 as described below. The memory is used to store programs for performing the methods described in the various method embodiments above; the processor is configured to execute the programs stored in the memory. The processor may include a central processing unit (CPU) or other forms of processing unit having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0168] like Figure 8 As shown, the electronic device 800 may include a processing unit 801 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage device 808 into a random access memory (RAM). Various programs and data required for the operation of the electronic device 800 are also stored in RAM 803. The processing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. Input / output (I / O) interfaces are also connected to the bus 804.

[0169] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, displays, speakers, vibrators, etc.; storage devices 808 including, for example, magnetic tapes, hard disks, etc.; and communication devices 809. Communication device 809 allows electronic device 800 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 8 An electronic device 800 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0170] In particular, according to at least one embodiment of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, at least one embodiment of this disclosure includes a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by a processing device 801, it performs the functions defined in the methods of at least one embodiment of this disclosure.

[0171] It should be noted that the computer-readable medium described above in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In at least one embodiment of this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In at least one embodiment of this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, radio frequency (RF), etc., or any suitable combination thereof.

[0172] The aforementioned computer-readable medium may be included in the aforementioned electronic device 800; or it may exist independently and not assembled into the electronic device 800.

[0173] Figure 9 This is a schematic block diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure.

[0174] For example, such as Figure 9 As shown, a non-transitory computer-readable storage medium 900 stores computer-readable instructions 901, which, when executed by at least one processor, perform one or more steps of the above-described allocation method.

[0175] For example, the storage medium may include a memory card for a smartphone, a storage component for a tablet computer, a hard drive for a personal computer, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other suitable storage media. For example, the readable storage medium may also be... Figure 7 The memory 702 in the memory is described in the foregoing content and will not be repeated here.

[0176] Although the present disclosure has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to the embodiments of the present disclosure, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present disclosure are within the scope of protection claimed by the present disclosure.

[0177] The following points should be noted regarding this disclosure:

[0178] (1) The accompanying drawings of the embodiments of this disclosure only involve the structures involved in the embodiments of this disclosure. Other structures can be referred to the general design.

[0179] (2) For clarity, the thickness of layers or regions in the drawings used to describe embodiments of the present disclosure is enlarged or reduced, i.e., these drawings are not drawn to actual scale.

[0180] (3) Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0181] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. The scope of protection of this disclosure should be determined by the scope of protection of the claims.

[0182] Although the present disclosure has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to the embodiments of the present disclosure, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present disclosure are within the scope of protection claimed by the present disclosure.

[0183] The following points should be noted regarding this disclosure:

[0184] (1) The accompanying drawings of the embodiments of this disclosure only involve the structures involved in the embodiments of this disclosure. Other structures can be referred to the general design.

[0185] (2) For clarity, the thickness of layers or regions in the drawings used to describe embodiments of the present disclosure is enlarged or reduced, i.e., these drawings are not drawn to actual scale.

[0186] (3) Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0187] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. The scope of protection of this disclosure should be determined by the scope of protection of the claims.

Claims

1. A method of dispensing, characterized by, The distribution method is applied to a pre-filling-decoding separation system in which a pre-filling task phase and a decoding task phase of a same inference task are respectively deployed on different instances, and the distribution method comprises: obtaining a plurality of configuration combinations, wherein each configuration combination in the plurality of configuration combinations comprises configuration information for a first instance and configuration information for a second instance, the first instance and the second instance being respectively used for performing a pre-filling task phase and a decoding task phase of a same inference task; for at least one configuration combination in the plurality of configuration combinations, performing a target operation, wherein the target operation comprises: generating a test data stream for the configuration combination; based on the configuration combination and the test data stream, performing a test process by the first instance and the second instance; determining a target configuration combination from the plurality of configuration combinations according to a preset performance indicator corresponding to the at least one configuration combination in the test process.

2. The dispensing method of claim 1, wherein, after the test process based on the configuration combination and the test data stream is performed by the first instance and the second instance, the target operation further comprises: after the test process ends, calculating a preset performance indicator corresponding to the configuration information for the first instance and the configuration information for the second instance in the test process, respectively, wherein the preset performance indicator comprises at least one of a throughput indicator or a delay type indicator.

3. The dispensing method of claim 2, wherein, The target operation further comprises: in response to there being a preset performance indicator satisfying a preset invalid condition, determining the configuration information corresponding to the preset performance indicator as invalid configuration information; or in response to there being a preset performance indicator not satisfying a preset invalid condition, determining the configuration information corresponding to the preset performance indicator as valid configuration information.

4. The dispensing method of claim 3, wherein, The distribution method further comprises: in response to there being invalid configuration information, determining a potential invalid configuration combination from configuration combinations that have not been tested according to the invalid configuration information, and skipping performing the target operation on the potential invalid configuration combination.

5. The dispensing method of claim 1, wherein, The generating of the test data stream for the configuration combination comprises: determining an amount of data in the test data stream according to the configuration combination; generating the test data stream based on the amount of data.

6. The method of distributing according to claim 1, wherein, The performing of the test process by the first instance and the second instance based on the configuration combination and the test data stream comprises: applying the configuration combination to the first instance and the second instance; the first instance receives and processes the test data stream, generates intermediate data, and passes the intermediate data to the second instance; the second instance receives and processes the intermediate data to generate result data.

7. The method of distributing according to claim 3, wherein, The determining of the target configuration combination from the plurality of configuration combinations according to the preset performance indicator corresponding to the at least one configuration combination in the test process comprises: excluding invalid configuration information from the at least one configuration combination to obtain a plurality of first valid configuration information for the first instance and a plurality of second valid configuration information for the second instance; determining a plurality of valid configuration combinations according to the plurality of first valid configuration information and the plurality of second valid configuration information; The throughput indicators of the plurality of effective configuration combinations are calculated respectively, and an effective configuration combination with the highest throughput indicator is determined as the target configuration combination.

8. The method of distributing according to claim 1, wherein, The allocation method further comprises: determining a target ratio according to a preset performance indicator corresponding to the target configuration combination, wherein the target ratio is used to indicate a numerical ratio between the number of the first instances and the number of the second instances.

9. The dispensing method of claim 8, wherein, The target configuration combination comprises first target configuration information for the first instances and second target configuration information for the second instances, wherein the determining the target ratio according to the preset performance indicator corresponding to the target configuration combination comprises: determining the target ratio according to a ratio of a throughput indicator corresponding to the second target configuration information to a throughput indicator corresponding to the first target configuration information.

10. The dispensing method of claim 1, wherein, The type of the configuration information comprises at least one of a calculation parallel strategy, a batch size, a storage space allocation strategy, a cache scheduling strategy, or a token generation concurrency.

11. The dispensing method of claim 1, wherein, The configuration information is determined according to hardware resource information.

12. The dispensing method of claim 1, wherein, The first instances and the second instances are used to perform inference tasks of a large-scale language model.

13. A dispensing device characterized by The allocation apparatus is applied to a prefill-decoding separation system in which a prefill task phase and a decoding task phase of a same inference task are respectively deployed in different instances, and the allocation apparatus comprises: a configuration module configured to obtain a plurality of configuration combinations, wherein each configuration combination in the plurality of configuration combinations comprises configuration information for a first instance and configuration information for a second instance, the first instance and the second instance being respectively used to perform a prefill task phase and a decoding task phase of a same inference task; a generation module configured to, for at least one configuration combination in the plurality of configuration combinations, generate test data flow for the configuration combination; an execution module configured to, for at least one configuration combination in the plurality of configuration combinations, perform a test process through the first instance and the second instance based on the configuration combination and the test data flow; a solving module configured to determine a target configuration combination from the plurality of configuration combinations according to a preset performance indicator corresponding to the at least one configuration combination in the test process.

14. An electronic device, comprising: The electronic device comprises: at least one processor; at least one memory including one or more computer program modules; wherein the one or more computer program modules are stored in the at least one memory and configured to be executed by the at least one processor, and the one or more computer program modules are used to implement the allocation method in any one of claims 1-12.

15. A non-transitory computer-readable storage medium, comprising: The non-transitory computer readable storage medium has computer readable instructions stored thereon, wherein the computer readable instructions are executed by at least one processor to perform the allocation method in any one of claims 1-12.

Images