Inference request scheduling method and device, equipment, medium and distributed inference system

Abstract

The invention relates to a reasoning request scheduling method, device and equipment, a medium and a distributed reasoning system, and relates to the technical field of intelligent calculation. The method comprises the following steps: in response to a reasoning request of a user side, if intermediate cache data corresponding to the reasoning request does not exist in local storage and storage nodes of a distributed shared storage pool, determining a corresponding reasoning service demand according to the reasoning request; determining the demand matching degree corresponding to each reasoning processing node combination according to the reasoning service demand and the communication link attribute between the nodes in each reasoning processing node combination; and according to the demand matching degree, scheduling the reasoning request to a pre-filling node and a decoding node contained in a target reasoning processing node combination in each reasoning processing node combination for reasoning processing. By adopting the method, the memory use efficiency can be improved, the memory overflow risk can be reduced, the high-concurrency reasoning service requirement can be met, and the performance bottleneck in a high-load scene can be improved.

Description

Technical Field

[0001] This application relates to the field of intelligent computing technology, and in particular to a reasoning request scheduling method, system, device, network equipment, computer-readable storage medium, computer program product, and distributed reasoning system. Background Technology

[0002] In the field of intelligent computing technology, when faced with inference requests, enterprises traditionally use a method of fine-tuning and distilling a basic model to obtain a more powerful industry model, and then perform further inference based on the industry model. This significantly increases users' post-training needs (including the need to fine-tune and distill the model and the need for inference services based on the model). Moreover, in the post-training process, only intermediate cached data (KVCache, Key Value Cache) is passed between P (Prefill) nodes and D (Decode) nodes, which can effectively protect the security of the original data.

[0003] However, during large model inference, GPU (Graphics Processing Unit) resources (such as memory) are limited. The intermediate cache data generated during inference will consume a lot of GPU memory, which leads to memory competition between intermediate cache data and inference computation, limiting the efficient use of memory and inference performance. Especially under high load scenarios, memory overflow or cache duplication problems are likely to occur. Summary of the Invention

[0004] Therefore, it is necessary to provide a reasoning request scheduling method, system, apparatus, network device, computer-readable storage medium, and computer program product to address the aforementioned technical problems.

[0005] Firstly, this application provides a reasoning request scheduling method, including:

[0006] In response to an inference request from a user, if the local nodes and storage nodes of the distributed shared storage pool do not contain intermediate cache data corresponding to the inference request, then the corresponding inference service requirement is determined based on the inference request.

[0007] Based on the inference service requirements and the communication link attributes between nodes in each inference processing node combination, determine the requirement matching degree corresponding to each inference processing node combination;

[0008] Based on the required matching degree, the inference request is scheduled to the pre-filled nodes and decoding nodes included in the target inference processing node combination in each of the inference processing node combinations for inference processing.

[0009] In one embodiment, the communication link attributes between nodes in the inference processing node combination include at least two of the following attributes: network bandwidth, latency, and communication cost between the pre-filled node and the decoding node in the inference processing node combination, as well as node load information of the pre-filled node and the decoding node in the inference processing node combination.

[0010] The step of determining the demand matching degree corresponding to each of the inference processing node combinations based on the inference service requirements and the communication link attributes between nodes in each inference processing node combination includes:

[0011] Based on the communication link attributes between nodes in each of the inference processing node combinations, the attribute values ​​corresponding to each attribute information in the communication link attributes are obtained;

[0012] Based on the inference service requirements, the weight values ​​corresponding to each attribute information in the communication link attributes are obtained;

[0013] Based on the attribute values ​​and weight values ​​corresponding to each attribute information, the demand matching degree corresponding to each combination of inference processing nodes is determined.

[0014] In one embodiment, the method further includes:

[0015] Query the local node of the distributed shared storage pool;

[0016] If the intermediate cache data corresponding to the inference request exists in the storage location corresponding to the local node of the distributed shared storage pool, then the intermediate cache data is sent to the user terminal.

[0017] In one embodiment, the method further includes:

[0018] If the storage location corresponding to the local node of the distributed shared storage pool does not contain the intermediate cache data corresponding to the inference request, then query whether the storage nodes included in the distributed shared storage pool contain the intermediate cache data corresponding to the inference request.

[0019] If any storage node in the distributed shared storage pool contains intermediate cache data corresponding to the inference request, the address of the storage node is sent to the user terminal; the user terminal is used to initiate a download request for the intermediate cache data to the storage node based on the received address of the storage node.

[0020] In one embodiment, scheduling the inference request to the pre-filled nodes and decoding nodes included in the target inference processing node combination in each of the inference processing node combinations for inference processing based on the demand matching degree includes:

[0021] Based on the required matching degree, the inference request is scheduled to a pre-filled node included in the target inference processing node combination; the pre-filled node is used to respond to the inference request and perform inference processing, sending the intermediate cache data generated during the inference processing to the decoding node included in the target inference processing node combination and the storage node of the distributed shared storage pool; the decoding node is used to perform inference processing based on the intermediate cache data, sending the tokens generated during the inference processing to the user terminal; the tokens are used by the user terminal to obtain the intermediate cache data corresponding to the inference request.

[0022] Secondly, this application also provides a distributed inference system, including: a decision scheduler, a pre-filled pool, a decoding pool, and a distributed shared storage pool; the pre-filled pool includes multiple pre-filled nodes; the decoding pool includes multiple decoding nodes; and the distributed shared storage pool includes local nodes and multiple storage nodes.

[0023] The decision scheduler is configured to perform the following steps:

[0024] In response to an inference request from a user, if the local nodes and storage nodes of the distributed shared storage pool do not contain intermediate cache data corresponding to the inference request, then the corresponding inference service requirement is determined based on the inference request; based on the inference service requirement and the communication link attributes between nodes in each inference processing node combination, the requirement matching degree corresponding to each inference processing node combination is determined; based on the requirement matching degree, the inference request is scheduled to the pre-filled nodes and decoding nodes included in the target inference processing node combination for inference processing.

[0025] Thirdly, this application also provides a reasoning request scheduling apparatus, comprising:

[0026] The response module is used to respond to the inference request from the user. If the local node and storage node of the distributed shared storage pool do not have the intermediate cache data corresponding to the inference request, the corresponding inference service requirement is determined according to the inference request.

[0027] The determination module is used to determine the demand matching degree corresponding to each of the inference processing node combinations based on the inference service requirements and the communication link attributes between nodes in each inference processing node combination.

[0028] The scheduling module is used to schedule the inference request to the pre-filled nodes and decoding nodes included in the target inference processing node combination in each inference processing node combination according to the demand matching degree.

[0029] Fourthly, this application also provides a network device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0030] In response to an inference request from a user, if the local nodes and storage nodes of the distributed shared storage pool do not contain intermediate cache data corresponding to the inference request, then the corresponding inference service requirement is determined based on the inference request; based on the inference service requirement and the communication link attributes between nodes in each inference processing node combination, the requirement matching degree corresponding to each inference processing node combination is determined; based on the requirement matching degree, the inference request is scheduled to the pre-filled nodes and decoding nodes included in the target inference processing node combination for inference processing.

[0031] Fifthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0032] In response to an inference request from a user, if the local nodes and storage nodes of the distributed shared storage pool do not contain intermediate cache data corresponding to the inference request, then the corresponding inference service requirement is determined based on the inference request; based on the inference service requirement and the communication link attributes between nodes in each inference processing node combination, the requirement matching degree corresponding to each inference processing node combination is determined; based on the requirement matching degree, the inference request is scheduled to the pre-filled nodes and decoding nodes included in the target inference processing node combination for inference processing.

[0033] Sixthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0034] In response to an inference request from a user, if the local nodes and storage nodes of the distributed shared storage pool do not contain intermediate cache data corresponding to the inference request, then the corresponding inference service requirement is determined based on the inference request; based on the inference service requirement and the communication link attributes between nodes in each inference processing node combination, the requirement matching degree corresponding to each inference processing node combination is determined; based on the requirement matching degree, the inference request is scheduled to the pre-filled nodes and decoding nodes included in the target inference processing node combination for inference processing.

[0035] The aforementioned inference request scheduling method, apparatus, network device, computer-readable storage medium, and computer program product, in response to a user's inference request, if the distributed shared storage pool and storage nodes outside the distributed shared storage pool do not contain intermediate cache data corresponding to the inference request, then determine the corresponding inference service requirement based on the inference request; determine the requirement matching degree corresponding to each inference processing node combination based on the inference service requirement and the communication link attributes between nodes in each inference processing node combination; and schedule the inference request to the pre-filled nodes and decoding nodes included in the target inference processing node combination for inference processing based on the requirement matching degree. Compared to traditional methods, this solution introduces a distributed shared storage pool to store intermediate cache data, reducing the GPU memory usage of intermediate cache data and thus alleviating the memory contention between inference computation and cache data. Furthermore, it performs intelligent scheduling based on inference service requirements and inter-node communication link attributes, scheduling inference processing node combinations that include pre-filled nodes and decoding nodes. It selects target inference processing node combinations that match the requirements of inference services and schedules inference requests to the pre-filled nodes and decoding nodes included in the target inference processing node combination for inference processing. This helps alleviate GPU memory contention, improves memory utilization efficiency, and reduces the risk of memory overflow, thereby meeting the needs of high-concurrency inference services, supporting the growth of enterprise post-training requirements, and improving performance bottlenecks in high-load scenarios. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is an application environment diagram of the inference request scheduling method in one embodiment;

[0038] Figure 2 This is a flowchart illustrating the inference request scheduling method in one embodiment;

[0039] Figure 3 This is a schematic diagram of distributed inference request scheduling for a PDS-separated architecture in one embodiment.

[0040] Figure 4 This is a flowchart illustrating the inference request scheduling method in another embodiment;

[0041] Figure 5 This is a schematic diagram of the distributed inference process of a distributed inference system in one embodiment;

[0042] Figure 6 This is a structural block diagram of the inference request scheduling device in one embodiment;

[0043] Figure 7 This is a diagram of the internal structure of a network device in one embodiment. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0045] It should be noted that the terms "comprising" and "having," and any variations thereof, as used in this application, are intended to cover non-exclusive inclusion. The term "multiple" as used in this application refers to two or more. The term "and / or" as used in this application refers to one of the solutions, or any combination of multiple solutions.

[0046] The terminology used in the embodiments of this application is as follows:

[0047] GPU: A GPU is a microprocessor specifically designed to perform graphics calculations in personal computers, workstations, game consoles, and some mobile devices (such as tablets and smartphones). GPUs have hundreds or thousands of cores and can perform massive amounts of computation in parallel.

[0048] PD separation architecture: refers to deploying the Prefill and Decode stages of large model inference on different servers in the same intelligent computing center.

[0049] The Prefill stage primarily processes all contextual information from user input and generates necessary caches to optimize subsequent generation efficiency.

[0050] Decode phase: Based on the context representation obtained in the Prefill phase, new tokens are generated one by one.

[0051] KVCache (Key Value Cache) system: A caching system based on a key-value storage structure, used for storing and quickly retrieving data. The key is a unique identifier, and the value can be any type of data (such as strings, objects, binary data, etc.).

[0052] SLA (Service-Level Agreement): This is an agreement between a customer and a service provider that specifies the service level and quality standards.

[0053] The inference request scheduling method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, the decision scheduler can include a monitor, a Prefill scheduler, a KVCache scheduler, and a Decode scheduler. The Prefill scheduler communicates with the Prefill pool, the KVCache scheduler communicates with the distributed shared storage pool, and the Decode scheduler communicates with the Decode pool. The monitor can be used to monitor and obtain information about the distributed shared storage pool, the load of P nodes and D nodes, the network status between P nodes and D nodes, and the quality of service information for user requests. The Prefill pool can be a high-performance computing chip. It can be used for reusing intermediate cache data (multiple inference requests may correspond to the same intermediate cache data, which can be reused), providing parallel deployment strategies for Prefill, and providing a local KVCache pool to store intermediate cache data generated during inference. The Decode pool can be a high-performance memory chip. It can be used to provide parallel deployment strategies for Decode and a local KVCache pool to store intermediate cache data generated during inference. The distributed shared storage pool can be used to provide a global Prefix KVCache reuse mechanism (intermediate cache data corresponding to multiple inference requests may be the same; if so, the intermediate cache data can be reused), provide a multi-level storage pool for shared KVCache across video memory / RAM / SSD (Solid State Drive), and perform cross-node data transfer. The local video memory KVCache pool in the Prefill pool can transfer its stored intermediate cache data to the local video memory KVCache pool in the Decode pool. Both the local video memory KVCache pools in the Prefill pool and the local video memory KVCache pools in the Decode pool can perform bidirectional transfer with the distributed shared storage pool.

[0054] In one exemplary embodiment, such as Figure 2 As shown, an inference request scheduling method is provided, which can be applied to... Figure 1 The decision scheduler in the process may include the following steps:

[0055] Step 201: In response to the inference request from the user, if the local storage and storage nodes of the distributed shared storage pool do not have intermediate cache data corresponding to the inference request, then determine the corresponding inference service requirements based on the inference request.

[0056] The distributed shared storage pool comprises multiple storage nodes that can form a distributed shared storage network. This network employs a multi-level storage mechanism: local HBM (High Bandwidth Memory) + host CPU (Central Processing Unit) + host SSD. It stores global prefix KVCache information and provides a unified KVCache query interface. The KVCache query interface of the distributed shared storage network is used to query each storage node in the distributed shared storage pool to determine if intermediate cache data corresponding to inference requests exists on each node. The storage location corresponding to a local node in the distributed shared storage pool can refer to the local HBM. The intermediate cache data corresponding to an inference request can refer to the intermediate cache data calculated based on the inference request. Inference requests can refer to long input / short output sequences, long input / long output sequences, short input / short output sequences, short input / long output sequences, user-acceptable TTFT (Time To First Token), and user-acceptable TPOT (Time Per Output Token). The first token return time indicates the latency at which the first token returns to the user; the time per output token indicates the time interval between subsequent tokens returning to the user. Inference service requirements can refer to requirements such as low latency, high bandwidth, and low cost.

[0057] For example, in response to an inference request from a user, the local node of the distributed shared storage pool is first queried to determine whether the intermediate cache data corresponding to the inference request exists in the storage location corresponding to the local node. If not, the storage nodes of the distributed shared storage pool are queried to determine whether the intermediate cache data corresponding to the inference request exists. If it is determined that none of the storage nodes of the distributed shared storage pool contain the intermediate cache data corresponding to the inference request, the corresponding inference service requirement is determined based on the inference request.

[0058] Step 202: Determine the demand matching degree corresponding to each inference processing node combination based on the inference service requirements and the communication link attributes between nodes in each inference processing node combination.

[0059] The demand matching degree corresponding to the inference processing node combination can be used to indicate the network status between the pre-filled nodes and the decoding nodes in the inference processing node combination, as well as the degree of matching between the node load of the pre-filled nodes and the decoding nodes and the inference service requirements.

[0060] For example, based on the communication link attributes between nodes in the inference processing node combination, multi-dimensional attribute information of the inference processing node combination is determined; the multi-dimensional attribute information includes attribute values ​​of multiple dimensions; based on the inference service requirements, the weight value of each attribute value is determined; based on the weight value of each attribute value, the attribute value is weighted to obtain the requirement matching degree corresponding to the inference processing node combination.

[0061] Step 203: Based on the demand matching degree, schedule the inference request to the pre-filled nodes and decoding nodes included in the target inference processing node combination for inference processing.

[0062] The Prefill pool contains multiple prefill nodes that can form a distributed Prefill network; the Decode pool contains multiple decoding nodes that can form a distributed Decode network. A distributed Prefill network / distributed Decode network / distributed shared storage network can refer to a distributed system composed of multiple nodes used to perform tasks such as inference, storage, querying, and / or computation. Conversely, a Prefill pool / Decode pool / distributed shared storage pool can refer to a resource management unit used to centrally schedule and allocate the resources required by the distributed system to perform tasks.

[0063] For example, the target inference processing node combination can refer to the inference processing node combination corresponding to the highest demand matching degree; it can also refer to the inference processing node combination with the lower node load among the inference processing node combination corresponding to the highest demand matching degree and the second highest demand matching degree; or it can refer to the inference processing node combination with the lowest node load among multiple inference processing node combinations with a demand matching degree higher than a preset threshold. Inference requests are scheduled to pre-filled nodes included in the target inference processing node combination for inference processing to obtain intermediate cache data. This intermediate cache data is then sent to the decoding nodes included in the target inference processing node combination. The decoding nodes included in the target inference processing node combination are used to continue inference processing based on this intermediate cache data.

[0064] In the above inference request scheduling method, in response to a user's inference request, if there is no intermediate cache data corresponding to the inference request in the distributed shared storage pool and storage nodes outside the distributed shared storage pool, then the corresponding inference service requirement is determined according to the inference request; the requirement matching degree corresponding to each inference processing node combination is determined according to the inference service requirement and the communication link attributes between nodes in each inference processing node combination; and the inference request is scheduled to the pre-filled nodes and decoding nodes included in the target inference processing node combination for inference processing according to the requirement matching degree. Compared to traditional methods, this solution introduces a distributed shared storage pool to store intermediate cache data, reducing the GPU memory usage of intermediate cache data and thus alleviating the memory contention between inference computation and cache data. Furthermore, it performs intelligent scheduling based on inference service requirements and inter-node communication link attributes, scheduling inference processing node combinations that include pre-filled nodes and decoding nodes. It selects target inference processing node combinations that match the requirements of inference services and schedules inference requests to the pre-filled nodes and decoding nodes included in the target inference processing node combination for inference processing. This helps alleviate GPU memory contention, improves memory utilization efficiency, and reduces the risk of memory overflow, thereby meeting the needs of high-concurrency inference services, supporting the growth of enterprise post-training requirements, and improving performance bottlenecks in high-load scenarios.

[0065] In an exemplary embodiment, the communication link attributes between nodes in the inference processing node combination include at least two of the following attributes: network bandwidth, latency, and communication cost between the pre-filled nodes and the decoding nodes in the inference processing node combination, and node load information of the pre-filled nodes and the decoding nodes in the inference processing node combination; in step 202, determining the demand matching degree corresponding to each inference processing node combination based on the inference service requirements and the communication link attributes between nodes in each inference processing node combination may include:

[0066] Based on the communication link attributes between nodes in each inference processing node combination, the attribute values ​​corresponding to each attribute information in the communication link attributes are obtained; based on the inference service requirements, the weight values ​​corresponding to each attribute information in the communication link attributes are obtained; based on the attribute values ​​and weight values ​​corresponding to each attribute information, the requirement matching degree corresponding to each inference processing node combination is determined.

[0067] Among them, the network bandwidth, latency, and communication cost between the pre-filled node and the decoding node can be used to indicate the network status between the pre-filled node and the decoding node; the node load information of the pre-filled node and the decoding node can be used to indicate the node load status of the pre-filled node and the decoding node, which can be obtained in the inference script based on the GPU utilization; the demand matching degree corresponding to the combination of inference processing nodes can be used to indicate the quality of service information of user demand that can be obtained when scheduling inference requests to nodes in the combination of inference processing nodes for inference processing.

[0068] For example, a multi-dimensional data set is obtained based on the communication link attributes between nodes in the inference processing node combination; this multi-dimensional data set This includes the attribute values ​​corresponding to each attribute information in the communication link attributes. , This indicates the number of attribute information items contained in the communication link attributes; based on the inference service requirements, weights are assigned to each attribute information item in the communication link attributes to obtain a weight vector; this weight vector... This includes the weight values ​​corresponding to each attribute information in the communication link attributes. The higher the weight value, the higher the priority of the user's need for that attribute information; according to The process involves calculating the demand matching degree for each combination of inference processing nodes. These combinations are then sorted from highest to lowest demand matching degree, with the first one selected as the initial inference processing node combination. If the node load information of the initial combination indicates a high load, the first two combinations are selected as candidate inference processing node combinations. If two candidate combinations have similar demand matching degrees, the one with the lower load is chosen as the target inference processing node combination. By considering node load information in addition to demand matching degree calculations, the SLA requirements for user inference can be maximized.

[0069] In an exemplary embodiment, the method of this application embodiment may further include a first query sending step, which may include:

[0070] Query the local node of the distributed shared storage pool; if the storage location corresponding to the inference request exists at the local node of the distributed shared storage pool, then send the intermediate cache data to the user.

[0071] In this context, a local node can refer to a node in the distributed shared storage pool used to store intermediate cache data corresponding to each inference request. The storage location corresponding to a local node in the distributed shared storage pool can refer to a type of storage medium or storage chip within the distributed shared storage pool.

[0072] For example, in response to a user's inference request, the local node of the distributed shared storage pool is queried based on the Prompt prefix to determine whether the intermediate cache data corresponding to the inference request exists in the storage location corresponding to the local node. If so, the intermediate cache data is sent directly to the user.

[0073] In one exemplary embodiment, the method of this application embodiment may further include a second query sending step, which may include:

[0074] If the storage location corresponding to the local node of the distributed shared storage pool does not contain intermediate cache data for the inference request, then the system queries whether any of the storage nodes included in the distributed shared storage pool contain intermediate cache data for the inference request. If any of the storage nodes included in the distributed shared storage pool contains intermediate cache data for the inference request, then the system sends the address of the storage node to the user. The user then uses the received address of the storage node to initiate a download request for the intermediate cache data from the storage node.

[0075] The address of the storage node can be used to indicate the address where the intermediate cache data corresponding to the inference request is located.

[0076] For example, in response to a user's inference request, the local node of the distributed shared storage pool is first queried to determine whether the intermediate cache data corresponding to the inference request exists at the storage location corresponding to the local node. If not, the storage nodes of the distributed shared storage pool are then queried to determine whether the intermediate cache data corresponding to the inference request exists. If it is determined that any storage node of the distributed shared storage pool contains the intermediate cache data corresponding to the inference request, the address of the storage node is sent to the user. After receiving the address of the storage node, the user directly initiates a download request for the intermediate cache data to the distributed shared storage pool. In response to the download request, the distributed shared storage pool sends the intermediate cache data to the user.

[0077] In an exemplary embodiment, step 203, scheduling the inference request to the pre-filled nodes and decoding nodes included in the target inference processing node combination for inference processing based on the demand matching degree, may include:

[0078] Based on the demand matching degree, the inference request is scheduled to the pre-filled node included in the target inference processing node combination; the pre-filled node is used to respond to the inference request and perform inference processing, and sends the intermediate cache data generated during the inference processing to the decoding node and the storage node of the distributed shared storage pool included in the target inference processing node combination; the decoding node is used to perform inference processing based on the intermediate cache data, and sends the tokens generated during the inference processing to the user terminal; the tokens are used by the user terminal to obtain the intermediate cache data corresponding to the inference request.

[0079] The time when the user receives the token can be used to indicate the delay between the pre-filled node and the decoding node.

[0080] It should be noted that user inference needs are diverse. To maximize the fulfillment of user inference SLA requirements and improve user experience, inference requests from different users need to be scheduled to the optimal resource pool nodes. The pre-filled nodes and decoding nodes included in the target inference processing node combination refer to the optimal pre-filled nodes and decoding nodes determined for the user's inference request. Scheduling inference requests to the pre-filled nodes included in the target inference processing node combination can maximize the fulfillment of user inference SLA requirements.

[0081] For example, such as Figure 3 As shown in the diagram, this embodiment provides a distributed inference request scheduling schematic for a PDS (Power Distribution System) separation architecture. The PDS separation architecture deploys the Prefill stage of large model inference in intelligent computing center 1, the Decode stage in intelligent computing center 2, and the storage stage in intelligent computing center 3, thereby expanding computing power through the collaboration of multiple DCs (Data Centers). Specifically, intelligent computing center 1 manages multiple P nodes (i.e., prefill nodes), intelligent computing center 2 manages multiple D nodes (i.e., decode nodes), and intelligent computing center 3 manages multiple S nodes (i.e., storage nodes).

[0082] As an example, such as Figure 3 As shown, the pre-filled node in the target inference processing node combination is P node 11 managed by Intelligent Computing Center 1, the decoding node in the target inference processing node combination is D node 23 managed by Intelligent Computing Center 2, and the storage node used to store the intermediate cache data corresponding to the inference request is S node 32 managed by Intelligent Computing Center 3. The inference request is scheduled to P node 11; through P node 11, according to the Prompt prefix, in... Figure 1The system queries the local KVCache pool of the Prefill pool. During the query, it determines whether the intermediate cache data corresponding to the inference request is hit. If so, the intermediate cache data is directly called for subsequent calculations. Otherwise, the system responds to the inference request, performs inference processing, and sends the intermediate cache data generated during the inference processing to D node 23 and S node 32. Through D node 23, inference processing is performed based on the received intermediate cache data and tokens to continue calculating and generating all subsequent tokens. The intermediate cache data stored in S node 32 can also be sent to P nodes managed by the intelligent computing center 1. The Prompt prefix refers to the prefix sequence in the complete Prompt input by the user that has been processed by the model. Each token in the Prompt prefix corresponds to a unique "position slot" in the intermediate cache data. Querying based on the Prompt prefix essentially involves "extracting the Key / Value of the corresponding slot based on the position range of the Prompt prefix."

[0083] Currently, the open-source model has significantly lowered the barrier to entry for AI applications. Enterprises widely fine-tune and distill basic models to obtain more powerful industry models, and then perform further inference based on these industry models. This has led to a significant increase in users' post-training needs. However, small and medium-sized enterprises only deploy a small amount of private computing power locally, which is insufficient to meet users' needs for high-concurrency, low-latency inference services. Therefore, a trend towards multi-computing center collaborative distributed inference architectures is evident. During post-training, enterprises also have higher requirements for the security of sensitive data, leading to a surge in demand for private deployments. However, building larger-scale computing clusters themselves is costly and difficult to maintain; directly renting computing power from operators poses a risk of privacy data leakage. In inference deployment based on a PD separation architecture, only intermediate cache data (KVCache) is transferred between Prefill and Decode, effectively protecting the security of the original data. Furthermore, during large model inference, GPU resources (especially memory) are limited, and the intermediate cache data (KVCache) generated during inference consumes a significant amount of GPU memory. This leads to memory contention between KVCache and inference computation, limiting efficient memory usage and inference performance. Especially under high load scenarios, it can easily cause memory overflow or cached duplicate computation issues.

[0084] In one exemplary embodiment, a distributed reasoning system is provided, comprising: Figure 1 The diagram shows a decision scheduler, a pre-fill pool, a decoding pool, and a distributed shared storage pool. The pre-fill pool contains multiple pre-fill nodes. The decoding pool contains multiple decoding nodes. The distributed shared storage pool contains local nodes and multiple storage nodes.

[0085] The decision scheduler is used to execute the steps of the inference request scheduling method as described in any of the above embodiments.

[0086] In this embodiment, as Figure 4 As shown, an inference request scheduling method may include the following steps:

[0087] Step 401: In response to the inference request from the user, if the intermediate cache data corresponding to the inference request exists in the storage location corresponding to the local node of the distributed shared storage pool, then the intermediate cache data is sent to the user.

[0088] For example, in response to a user's inference request, the local node of the distributed shared storage pool is queried based on the Prompt prefix to determine whether the intermediate cache data corresponding to the inference request exists in the storage location corresponding to the local node. If so, the intermediate cache data is sent directly to the user.

[0089] Step 402: If the storage location corresponding to the local node of the distributed shared storage pool does not contain intermediate cache data corresponding to the inference request, and any storage node contained in the distributed shared storage pool contains intermediate cache data corresponding to the inference request, then the address of the storage node is sent to the user.

[0090] The user terminal is used to initiate a download request for intermediate cached data to the storage node based on the received storage node address.

[0091] For example, in response to a user's inference request, the local node of the distributed shared storage pool is first queried to determine whether the intermediate cache data corresponding to the inference request exists at the storage location corresponding to the local node. If not, the storage nodes of the distributed shared storage pool are then queried to determine whether the intermediate cache data corresponding to the inference request exists. If it is determined that any storage node of the distributed shared storage pool contains the intermediate cache data corresponding to the inference request, the address of the storage node is sent to the user. After receiving the address of the storage node, the user directly initiates a download request for the intermediate cache data to the distributed shared storage pool. In response to the download request, the distributed shared storage pool sends the intermediate cache data to the user.

[0092] Step 403: If the local storage of the distributed shared storage pool and the storage nodes of the distributed shared storage pool do not have intermediate cache data corresponding to the inference request, then determine the corresponding inference service requirements based on the inference request.

[0093] Step 404: Determine the demand matching degree corresponding to each inference processing node combination based on the inference service requirements and the communication link attributes between nodes in each inference processing node combination.

[0094] For example, communication link attributes include at least two of the following attributes: user demand service quality, node load of nodes in the inference processing node portfolio, and network bandwidth, latency, and communication cost between nodes in the inference processing node portfolio.

[0095] Step 405: Based on the demand matching degree, schedule the inference request to the pre-filled node contained in the target inference processing node combination in each inference processing node combination.

[0096] The pre-filled node is used to query the corresponding intermediate cache data in the distributed shared storage pool based on the Prompt prefix. If the query is successful, the intermediate cache data is directly called for subsequent calculations. If the query is unsuccessful, the intermediate cache data generated is pushed to the decoding node included in the combination of the distributed shared storage pool and the target inference processing node after the pre-filled node completes the calculation. The decoding node is used to continue the subsequent calculations and returns the generated Token to the user to complete the inference.

[0097] For example, such as Figure 3 As shown in the diagram, this embodiment provides a distributed inference request scheduling schematic for a PDS (Power Distribution System) separation architecture. The PDS separation architecture deploys the Prefill stage of large model inference in intelligent computing center 1, the Decode stage in intelligent computing center 2, and the storage stage in intelligent computing center 3, thereby expanding computing power through the collaboration of multiple DCs (Data Centers). Specifically, intelligent computing center 1 manages multiple P nodes (i.e., prefill nodes), intelligent computing center 2 manages multiple D nodes (i.e., decode nodes), and intelligent computing center 3 manages multiple S nodes (i.e., storage nodes).

[0098] For example, after determining the target inference processing node combination, such as Figure 5 As shown, inference requests initiated by users through the user terminal are scheduled to the Prefill pool. The Prefill pool processes the inference requests and pushes the generated KVCache to the distributed shared storage pool and the Decode pool. The Decode pool continues to process the inference requests and pushes the generated KVCache to the distributed shared storage pool. The Decode pool can also return the inference processing results to the user terminal through the ONT (Optical Network Terminal). The Prefill pool is used for centralized inference, and the Decode pool is used for edge inference.

[0099] In this embodiment, by deploying the Prefill, Decode, and Storage stages on servers in different intelligent computing centers, the computing power is expanded through the collaboration of multiple intelligent computing centers. This protects user privacy data security while meeting users' high-concurrency, low-latency inference service needs, alleviating memory overflow issues under high-load scenarios. Furthermore, the prefill pool, decoding pool, and distributed shared storage pool only transmit intermediate cached data and not the user's original data, ensuring that the user's original data does not leave the campus and guaranteeing the security of privacy data. Considering multiple attribute information affecting inference processing and the inference service requirements corresponding to inference requests, a target inference processing node combination is determined from various inference processing node combinations. User inference requests are scheduled to the target inference processing node combination to maximize the satisfaction of different users' inference service needs and improve user experience.

[0100] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0101] Based on the same inventive concept, this application also provides a reasoning request scheduling apparatus for implementing the reasoning request scheduling method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more of the reasoning request scheduling apparatus embodiments provided below can be found in the limitations of the reasoning request scheduling method described above, and will not be repeated here.

[0102] In one exemplary embodiment, such as Figure 6 As shown, an inference request scheduling device is provided, which can be applied to... Figure 1 The decision scheduler in the system may include:

[0103] The response module 601 is used to respond to the inference request from the user. If the local node and storage node of the distributed shared storage pool do not have intermediate cache data corresponding to the inference request, the corresponding inference service requirement is determined according to the inference request.

[0104] The determination module 602 is used to determine the demand matching degree corresponding to each inference processing node combination based on the inference service requirements and the communication link attributes between nodes in each inference processing node combination.

[0105] The scheduling module 603 is used to schedule inference requests to the pre-filled nodes and decoding nodes included in the target inference processing node combination in each inference processing node combination for inference processing based on the demand matching degree.

[0106] In an exemplary embodiment, the determining module 603 is further configured to obtain the attribute value corresponding to each attribute information in the communication link attribute according to the communication link attribute between the nodes in each inference processing node combination; obtain the weight value corresponding to each attribute information in the communication link attribute according to the inference service requirements; and determine the requirement matching degree corresponding to each inference processing node combination according to the attribute value and the weight value corresponding to each attribute information. The communication link attribute between the nodes in the inference processing node combination includes at least two attribute information from the network bandwidth, latency and communication cost between the pre-filled node and the decoding node in the inference processing node combination, as well as the node load information of the pre-filled node and the decoding node in the inference processing node combination.

[0107] In one exemplary embodiment, the inference request scheduling device may further include a first query sending module. The first query sending module is used to query the local nodes of the distributed shared storage pool; if the storage location corresponding to the local node of the distributed shared storage pool contains intermediate cache data corresponding to the inference request, then the intermediate cache data is sent to the user terminal.

[0108] In one exemplary embodiment, the inference request scheduling device may further include a second query sending module. The second query sending module is configured to: if the storage location corresponding to the local node in the distributed shared storage pool does not contain intermediate cache data corresponding to the inference request, query whether any of the storage nodes in the distributed shared storage pool contains intermediate cache data corresponding to the inference request; if any storage node in the distributed shared storage pool contains intermediate cache data corresponding to the inference request, send the address of the storage node to the user terminal; the user terminal is configured to: initiate a download request for the intermediate cache data to the storage node based on the received address of the storage node.

[0109] In an exemplary embodiment, the scheduling module 603 is further configured to schedule the inference request to a pre-filled node included in the target inference processing node combination based on the demand matching degree; the pre-filled node is configured to respond to the inference request and perform inference processing, and send the intermediate cache data generated during the inference processing to the decoding node and the storage node of the distributed shared storage pool included in the target inference processing node combination; the decoding node is configured to perform inference processing based on the intermediate cache data, and send the tokens generated during the inference processing to the user terminal; the tokens are used by the user terminal to obtain the intermediate cache data corresponding to the inference request.

[0110] Each module in the aforementioned inference request scheduling device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the network device in hardware form or independent of it, or stored in the memory of the network device in software form, so that the processor can call and execute the operations corresponding to each module.

[0111] In one exemplary embodiment, a network device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7 As shown, this network device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data such as communication link attributes between nodes. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a reasoning request scheduling method.

[0112] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the solution of this application and does not constitute a limitation on the network device to which the solution of this application is applied. Specific network devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0113] In one exemplary embodiment, a network device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the inference request scheduling method described above.

[0114] In one exemplary embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the inference request scheduling method described above.

[0115] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the inference request scheduling method described above.

[0116] It should be noted that the user information (including but not limited to user device information, inference service requirements corresponding to user inference requests, etc.) and data (including but not limited to data used for analysis, stored data, and displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant regulations.

[0117] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0118] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0119] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for scheduling inference requests, characterized in that, The method comprises: in response to an inference request of a user terminal, if the local node and the storage node of the distributed shared storage pool do not have intermediate cache data corresponding to the inference request, determining a corresponding inference service requirement according to the inference request; determining a requirement matching degree corresponding to each inference processing node combination according to the inference service requirement and the communication link attribute between the nodes in each inference processing node combination; scheduling the inference request to the pre-filled nodes and the decoding nodes included in the target inference processing node combination in each inference processing node combination for inference processing according to the requirement matching degree.

2. The method of claim 1, wherein, The communication link attribute between the nodes in the inference processing node combination includes at least two attribute information in the network bandwidth, the time delay and the communication cost between the pre-filled nodes and the decoding nodes in the inference processing node combination and the node load information of the pre-filled nodes and the decoding nodes in the inference processing node combination; The determination of the requirement matching degree corresponding to each inference processing node combination according to the inference service requirement and the communication link attribute between the nodes in each inference processing node combination comprises: obtaining an attribute value corresponding to each attribute information in the communication link attribute according to the communication link attribute between the nodes in each inference processing node combination; obtaining a weight value corresponding to each attribute information in the communication link attribute according to the inference service requirement; determining the requirement matching degree corresponding to each inference processing node combination according to the attribute value corresponding to each attribute information and the weight value corresponding to each attribute information.

3. The method of claim 1, wherein, The method further comprises: querying the local node of the distributed shared storage pool; if the intermediate cache data corresponding to the inference request exists in the storage location corresponding to the local node of the distributed shared storage pool, sending the intermediate cache data to the user terminal.

4. The method of claim 3, wherein, The method further comprises: if the intermediate cache data corresponding to the inference request does not exist in the storage location corresponding to the local node of the distributed shared storage pool, querying whether the storage node included in the distributed shared storage pool has the intermediate cache data corresponding to the inference request; if any storage node included in the distributed shared storage pool has the intermediate cache data corresponding to the inference request, sending the address of the storage node to the user terminal; the user terminal is used for initiating a download request for the intermediate cache data to the storage node according to the received address of the storage node.

5. The method according to any one of claims 1 to 4, characterized in that, The scheduling of the inference request to the pre-filled nodes and the decoding nodes included in the target inference processing node combination in each inference processing node combination for inference processing according to the requirement matching degree comprises: According to the demand matching degree, the inference request is dispatched to a pre-filled node included in the target inference processing node combination; the pre-filled node is configured to perform inference processing in response to the inference request, and send intermediate cache data generated in the inference processing to a decoding node included in the target inference processing node combination and a storage node of the distributed shared storage pool; the decoding node is configured to perform inference processing according to the intermediate cache data, and send a word element generated in the inference processing to the user end; the word element is used by the user end to obtain intermediate cache data corresponding to the inference request.

6. A distributed reasoning system, characterized by Comprise: a decision scheduler, a pre-filled pool, a decoding pool and a distributed shared storage pool; the pre-filled pool includes a plurality of pre-filled nodes; the decoding pool includes a plurality of decoding nodes; the distributed shared storage pool includes a local node and a plurality of storage nodes; wherein the decision scheduler is configured to perform the steps of the method of any one of claims 1 to 5.

7. An inference request scheduling apparatus characterized by comprising: The device comprises: a response module configured to, in response to an inference request of a user end, if the local node and the storage node of the distributed shared storage pool do not have intermediate cache data corresponding to the inference request, determine an inference service demand corresponding to the inference request according to the inference request; a determination module configured to determine a demand matching degree corresponding to each inference processing node combination according to the inference service demand and a communication link attribute between nodes in each inference processing node combination; a scheduling module configured to, according to the demand matching degree, dispatch the inference request to a pre-filled node and a decoding node included in a target inference processing node combination of each inference processing node combination for inference processing. 8.A network device, comprising a memory and a processor, wherein the memory stores a computer program, and the network device is configured to perform the method according to any one of claims 1-7. The processor executes the computer program to realize the steps of the method of any one of claims 1 to 5.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the method of any one of claims 1 to 5.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the method of any one of claims 1 to 5.

Images