Processing method and device of reasoning request, storage medium and product

Abstract

The invention relates to the technical field of machine learning, and discloses a reasoning request processing method and device, a storage medium and a product, and the method comprises the steps: obtaining a first reasoning queue corresponding to a first node, the first reasoning queue storing N1 reasoning requests being executed in a node memory of the first node; obtaining available storage resources of the first node and target storage resources required for executing the N1 reasoning requests; if the available storage resources are smaller than the target storage resources, the first reasoning queue is updated to be a second reasoning queue according to the storage resources and the available storage resources required by each reasoning request in the N1 reasoning requests, N2 reasoning requests are stored in the second reasoning queue, and N2 is larger than or equal to 1 and smaller than N1; and sending indication information containing the second reasoning queue to the first node. The problem that the utilization rate of video memory resources is low when the reasoning task is executed based on a parallel pipeline mode can be solved.

Description

Technical Field

[0001] The present invention relates to the field of machine learning technology, and in particular to a method, device, storage medium and product for processing an inference request. Background Art

[0002] With the development of deep learning technology, especially the increasing application of large-scale pre-trained models (such as large language models), the demand for computing resources has also increased dramatically. Large-scale pre-trained models usually have hundreds of millions or even billions of parameters. Their reasoning process requires not only a large amount of computing resources, but also sufficient video memory to store intermediate results, especially for storing key-value pairs (KVCache) of the self-attention mechanism. However, in the actual reasoning process, due to the limitation of video memory resources, video memory capacity often becomes a bottleneck for managing KVCache, especially in scenarios where multiple tenants or multiple tasks coexist.

[0003] At present, usually before the inference starts, the central processing unit (CPU) allocates fixed video memory resources (also called node memory) to each inference task through static memory allocation. However, due to the different requirements of different inference tasks for video memory resources, this method will cause some tasks to occupy a large amount of video memory resources while other tasks cannot get enough video memory resources, which will lead to low utilization of video memory resources. Alternatively, the CPU reuses the video memory space of completed tasks based on memory reuse technology to reduce video memory requirements. However, this method is only applicable to situations where there is a clear time interval between tasks. For task environments with high concurrency, simple memory reuse technology is difficult to meet the needs, resulting in poor flexibility in allocating video memory resources. Summary of the invention

[0004] In view of this, the present invention provides a method, device, storage medium and product for processing an inference request to solve the problem of low utilization of video memory resources when executing inference tasks based on a parallel pipeline approach.

[0005] In a first aspect, the present invention provides a method for processing an inference request, the method comprising: obtaining a first inference queue corresponding to a first node, the first inference queue storing N1 inference requests being executed in a node memory of the first node, N1 ≥ 1 and being a positive integer; obtaining the available storage resources of the first node, and obtaining the target storage resources required for the first node to execute the N1 inference requests; if the available storage resources are less than the target storage resources, updating the first inference queue to a second inference queue based on the storage resources required for each inference request in the N1 inference requests and the available storage resources, the second inference queue storing N2 inference requests, 1 ≤ N2 < N1, and N2 being a positive integer; sending an indication message containing the second inference queue to the first node, the indication message being used to instruct the first node to swap out the N1-N2 inference requests from the node memory to the host memory, and continue to execute the N2 inference requests.

[0006] Based on the method of the first aspect above, a first inference queue corresponding to the first node among L nodes can be obtained, the available storage resources of the first node can be obtained, and the target storage resources required for the first node to execute N1 inference requests can be obtained; it is determined whether the available storage resources are greater than or equal to the target storage resources. If not, the first inference queue is updated to a second inference queue based on the storage resources required for each inference request among the N1 inference requests and the available storage resources, and an indication information containing the second inference queue is sent to the first node.

[0007] When the available storage resources of the first node are greater than or equal to the target storage resources required by the first node to execute N1 inference requests, it indicates that the available storage resources of the first node cannot satisfy the inference requests in the first inference queue executed by the first node. At this time, N1-N2 inference requests can be determined from the first inference queue, and the N1-N2 inference requests can be swapped out from the node memory to the host memory of the processor. Therefore, the problem of insufficient node memory of the first node when executing inference tasks based on a parallel pipeline can be effectively solved. At the same time, based on the different requirements of different inference requests for node memory, the memory resources of the first node can be reasonably allocated to avoid the problem that some tasks occupy a large amount of video memory while other tasks cannot get enough resources, improve the resource utilization of the node memory of the first node, improve the scheduling efficiency of the node memory, simplify the complexity of memory management, and thus improve the overall performance of large model inference.

[0008] In an optional embodiment, the first node is the first node among L nodes, L ≥ 1 and is a positive integer; obtaining the available storage resources of the first node includes: obtaining the free storage resources of each of the L nodes; determining the minimum free resource value among the L free storage resource values, and based on the minimum free resource value, determining a target free storage resource of the minimum free resource value size among the free storage resources of the first node, and determining the target free storage resource as the available storage resource of the first node.

[0009] Based on the above method, when multiple nodes use different memory spaces, the available storage resources allocated to each node will be allocated according to the minimum value of the free storage resources that can be allocated to all nodes, ensuring that data storage on L nodes will not overflow.

[0010] In an optional implementation, obtaining the available storage resources of the first node includes: obtaining all idle storage resources of L nodes; and determining all idle storage resources as the available storage resources of the first node.

[0011] Based on the above method, when multiple nodes share a memory space, the available storage resources of the first node cannot exceed all free storage resources. This approach allows the first node to have more sufficient storage resources when storage resources are needed to execute inference requests.

[0012] In an optional embodiment, obtaining the target storage resources required by the first node to execute N1 inference requests includes: obtaining the length of each inference request corresponding to each inference request in the N1 inference requests; calculating a first ratio between the length of each inference request and a preset data block size, and rounding the first ratio to obtain N1 first values; calculating the sum of each first value and the preset value to obtain N1 second values, each second value being the number of data blocks corresponding to the storage resources required by the first node to execute each inference request; calculating the sum of the N1 second values ​​to obtain a third value, the third value representing the target number of data blocks required by the first node to execute N1 inference requests corresponding to the target storage resource.

[0013] Based on the above method, the number of data blocks corresponding to the storage resources required by the first node to execute each inference request can be determined through the length of each inference request and the preset data block size, and then the target number of data blocks required for the first node to execute N1 inference requests can be determined, which is convenient for subsequent determination of whether it is necessary to swap out the inference request from the first inference queue.

[0014] In an optional embodiment, before updating the first inference queue to the second inference queue based on the storage resources required for each of the N1 inference requests and the available storage resources, it includes: calculating a second ratio between the available storage resources and the preset data block size, and rounding the second ratio to obtain a fourth value, the fourth value representing the number of available data blocks of the first node corresponding to the available storage resources; comparing the third value and the fourth value to determine whether the available storage resources are greater than or equal to the target storage resources.

[0015] Based on the above method, the available storage resources of the first node can be converted into the number of available data blocks of the first node, so as to judge whether the available storage resources are greater than or equal to the target storage resources by comparing the number of available data blocks of the first node and the target number of data blocks required for the first node to execute N1 inference requests.

[0016] In an optional implementation, the first inference queue is updated to the second inference queue according to the storage resources required and available storage resources for each of the N1 inference requests, including: obtaining a first exchange queue corresponding to the first node, the first exchange queue storing M inference requests, the M inference requests being M inference requests swapped out from the node memory to the host memory of the processor, M≥1 and being a positive integer; determining N1-N2 inference requests swapped out from the first inference queue according to the storage resources required and available storage resources for each of the N1 inference requests; swapping out N1-N2 inference requests from the first inference queue to obtain a second inference queue; swapping the N1-N2 inference requests into the first exchange queue to obtain a second exchange queue, the second exchange queue storing M+N1-N2 inference requests.

[0017] Based on the above method, the first exchange queue corresponding to the first node can be obtained, and N1-N2 inference requests can be switched into the first exchange queue to obtain a second exchange queue, so that the first node can subsequently switch the inference request from the second exchange queue into the second inference request.

[0018] In an optional embodiment, based on the storage resources required by each inference request in the N1 inference requests and the available storage resources, N1-N2 inference requests swapped out from the first inference queue are determined, including: comparing each second value corresponding to each inference request in the N1 inference requests with a fourth value in order from large to small, to determine the N1-N2 inference requests swapped out from the first inference queue, each second value corresponding to each inference request in the N1-N2 inference requests being greater than or equal to each second value corresponding to each inference request in the N2 inference requests.

[0019] Based on the above method, the inference request requiring more storage resources to execute the inference request can be determined from the N1 inference requests in the order of each second value from large to small, that is, as few inference requests as possible can be swapped out from the N1 inference requests to avoid the first node frequently performing swap-in and swap-out operations, which affects the node performance of the first node.

[0020] In an optional embodiment, the second exchange queue also stores a mapping table, which records the mapping relationship between each inference request in M+N1-N2 inference requests and the address information of each inference request in the host memory; the method also includes: if the available storage resources are greater than or equal to the target storage resources, then calculating the first difference between the fourth value and the third value, and detecting whether the first exchange queue is empty; if not, detecting whether the first difference is greater than or equal to at least one second value among the M+N1-N2 second values, and the M+N1-N2 second values ​​are the data blocks required for each inference request in the M+N1-N2 inference requests.

[0021] If so, then according to each second value and fourth value corresponding to each inference request in the M+N1-N2 inference requests, determine the R inference requests that need to be swapped into the second inference queue from the M+N1-N2 inference requests, 1≤R<M+N1-N2, and R is a positive integer; according to the mapping table and the R inference requests, swap the R second inference requests from the second exchange queue into the second inference queue to obtain an updated third inference queue; send a first indication message containing the third inference queue to the first node, the first indication message being used to instruct the first node to swap out the R inference requests from the host memory to the node memory.

[0022] Based on the above method, the first difference value can represent the remaining storage resources of the first node except for executing the inference requests in the first inference queue. When the first exchange queue is not empty and the first difference value satisfies the condition of switching from the second exchange queue to the second inference queue, the R inference requests that need to be switched into the second inference queue are determined from the M+N1-N2 inference requests. In addition, the first node can quickly find the host memory data in the corresponding processor based on the mapping table and move the host memory data back to the node memory.

[0023] In an optional embodiment, if so, then based on each second value and fourth value corresponding to each inference request in the M+N1-N2 inference requests, R inference requests that need to be swapped into the second inference queue are determined from the M+N1-N2 inference requests, including: if it is detected that the first difference is greater than or equal to at least one second value among the M+N1-N2 second values, then each swap-in time of each inference request in the M+N1-N2 inference requests being swapped into the first swap queue is obtained; and each second value corresponding to each inference request in the M+N1-N2 inference requests is compared with the fourth value in the time sequence of the M+N1-N2 swap-in times to determine R inference requests, the sum of the R second values ​​corresponding to the R inference requests being less than the fourth value, and the R inference requests being the first R inference requests in the M+N1-N2 inference requests.

[0024] Based on the above method, R inference requests can be determined from M+N1-N2 inference requests according to the swap-in time of the inference requests in the first swap queue, and the inference request that first enters the swap queue among the M+N1-N2 inference requests can be swapped into the inference queue first, thereby reducing the waiting time and response delay of the inference request in the first node.

[0025] In an optional implementation, if so, then according to each second value and fourth value corresponding to each inference request in the M+N1-N2 inference requests, determine the R inference requests that need to be swapped into the second inference queue from the M+N1-N2 inference requests, including: if it is detected that the first difference is greater than or equal to at least one of the M+N1-N2 second values, then obtain each swap-in time of each inference request in the M+N1-N2 inference requests swapped into the first swap queue; calculate the second difference between each swap-in time and the current time, and obtain M+N1-N2. second difference; determine P second difference values ​​from M+N1-N2 second difference values, each of the P second difference values ​​is less than or equal to a preset time period, the P second difference values ​​correspond to P reasoning requests, R≤P≤M+N1-N2, and P is a positive integer; compare the P second values ​​with the fourth value in ascending order of the P second values ​​corresponding to the P reasoning requests to determine R reasoning requests, the sum of the R second values ​​corresponding to the R reasoning requests is less than the fourth value, and the R second values ​​corresponding to the R reasoning requests are the first R second values ​​among the P second values.

[0026] Based on the above method, since each of the P second difference values ​​is less than or equal to the preset time period, all the P inference requests in the first exchange queue whose swap-in time is less than or equal to the preset time period are used as swap-in candidate requests, and R inference requests are determined in ascending order of the second numerical values ​​of the inference requests in the first exchange queue, that is, as many requests as possible are transferred to the first inference queue, thereby improving the resource utilization of the node memory of the first node and the scheduling efficiency of the node memory.

[0027] In an optional embodiment, the method also includes: if it is detected that the first exchange queue is empty, obtaining the first waiting queue corresponding to the first node, and detecting whether W inference requests are stored in the first waiting queue, the W inference requests are the W inference requests waiting to be executed in the node memory of the first node, W ≥ 1 and is a positive integer; if so, detecting whether the first difference is greater than or equal to at least one of the W second values, the W second values ​​are detected as the data blocks required for each of the W inference requests.

[0028] If not, then determine the Q inference requests that need to be replaced into the first inference queue from the W inference requests according to each second value and the fourth value corresponding to each inference request of the W inference requests, 1≤Q<W, and Q is a positive integer; replace the Q inference requests from the first waiting queue into the first inference queue to obtain an updated fourth inference queue; send a second indication message containing the fourth inference queue to the first node, and the second indication message is used to instruct the first node to execute Q additional inference requests.

[0029] Based on the above method, since there are W inference requests stored in the first waiting queue, and whether the first difference is greater than or equal to at least one of the W second values, it means that the first node has sufficient remaining storage resources except for executing the inference requests in the first inference queue. Therefore, Q inference requests can be moved from the first waiting queue to the first inference queue, so that the first node can execute Q additional inference requests.

[0030] In an optional embodiment, the method also includes: if it is detected that the first waiting queue is empty, it is determined that the first node has completed the first reasoning stage of the reasoning service, and a third indication message is sent to the first node, where the third indication message is used to instruct the first node to execute the second reasoning stage of the reasoning service.

[0031] Based on the above method, when the first waiting queue is empty, it means that all tasks in the first reasoning stage have been completed. At this time, the third indication information is immediately sent to let the first node enter the second reasoning stage, so that the first node can switch tasks in time, maximize the use of computing resources, avoid idle time, and ensure the continuity of the reasoning process, reducing the delay caused by waiting for the tasks of the next stage.

[0032] In a second aspect, the present invention provides a device for processing an inference request; the device comprises: A transceiver module, used to obtain a first inference queue corresponding to the first node, where the first inference queue stores N1 inference requests being executed in the node memory of the first node, where N1 ≥ 1 and is a positive integer; The transceiver module is further used to obtain the available storage resources of the first node and to obtain the target storage resources required by the first node to execute N1 inference requests; The processing module is further configured to update the first inference queue to a second inference queue if the available storage resources are less than the target storage resources, according to the storage resources required by each inference request in the N1 inference requests and the available storage resources, wherein the second inference queue stores N2 inference requests, 1≤N2<N1, and N2 is a positive integer; The transceiver module is also used to send indication information containing the second reasoning queue to the first node, where the indication information is used to instruct the first node to swap out N1-N2 reasoning requests from the node memory to the host memory and continue to execute N2 reasoning requests.

[0033] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, computer instructions being stored in the memory, and the processor executing the method for processing an inference request of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0034] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method for processing an inference request of the first aspect or any corresponding embodiment thereof.

[0035] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the method for processing an inference request of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 This is a schematic diagram of executing reasoning tasks based on pipeline; Figure 2is a topological diagram of a system for processing inference requests according to an embodiment of the present invention; Figure 3 is a flowchart of a method for processing an inference request according to an embodiment of the present invention; Figure 4 is a schematic diagram of a second node performing a reasoning task according to an embodiment of the present invention; Figure 5 is a structural block diagram of a device for processing an inference request according to an embodiment of the present invention; Figure 6 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0039] The embodiments of the present invention are applied to the reasoning process of a large-scale pre-trained model, especially in the scenario where the reasoning task is performed based on a parallel pipeline.

[0040] like Figure 1 As shown, Figure 1 This is a schematic diagram of executing inference tasks based on pipeline. Figure 1 In the pipeline parallel method, when performing reasoning, the reasoning task generally starts from the first node (such as Figure 1 The nodes shown in 1) are executed sequentially to the last computing node. The computing node generates intermediate results (KVCache) when executing the reasoning task.

[0041] Among them, the reasoning task can be divided into the prefill stage (Prefill stage) and the decoding stage (Decode stage).

[0042] Each node is designed with n inference groups (consistent with the number of pipeline stages), and each inference group can accommodate a certain number of inference requests at the same time. Each node can only execute tasks in one inference group at a time, and other inference groups are in a waiting state.

[0043] As described in the background art, the CPU has poor flexibility in allocating video memory resources to each node, resulting in low utilization of video memory resources when each node performs reasoning tasks.

[0044] In order to solve the above problems, an embodiment of the present invention provides a method for processing an inference request, which includes obtaining a first inference queue corresponding to the first node among L nodes; obtaining the available storage resources of the first node, and obtaining the target storage resources required by the first node to execute N1 inference requests; judging whether the available storage resources are greater than or equal to the target storage resources; if not, updating the first inference queue to a second inference queue based on the storage resources and available storage resources required for each inference request among the N1 inference requests, wherein the second inference queue stores N2 inference request identifiers, N2<N1; swapping out the N1-N2 inference requests from the node memory to the host memory of the processor, and sending indication information containing the second inference queue to the first node. This is to achieve the effect of improving the utilization rate of video memory resources when each node executes inference tasks based on a parallel pipeline method.

[0045] Below Figure 2 Taking the shown inference request processing system 200 as an example, the method provided in the embodiment of the present application is described. Figure 2 It is only a schematic diagram and does not constitute a limitation on the applicable scenarios of the technical solution provided in this application.

[0046] like Figure 2 As shown, Figure 2 4 is a topological diagram of a system for processing inference requests according to an embodiment of the present invention. Figure 2 In the example, the inference request processing system 200 may include a processor 201, a first node 202, and a second node 203. In addition, the inference request processing system 200 may further include nodes, such as a third node and a fourth node, which is not limited in this embodiment.

[0047] The processor 201 in this embodiment can be any device with computing and communication capabilities. For example, the processor 201 can be a central processing unit. The processor 201 corresponds to the host memory.

[0048] The first node 202 or the second node 203 in this embodiment may be any device deployed with one or more graphics processing units (GPUs). For example, the first node 202 or the second node 203 may be a graphics processor.

[0049] Figure 2 The inference request processing system 200 shown is only used as an example and is not used to limit the technical solution of the present application. Those skilled in the art should understand that in the specific implementation process, the inference request processing system 200 may also include other nodes, and the number of nodes may also be determined according to specific needs without limitation.

[0050] According to an embodiment of the present invention, an embodiment of a method for processing an inference request is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0051] In this embodiment, a method for processing an inference request is provided, which can be used for the above-mentioned processor; the processor and L nodes perform the inference task based on a parallel pipeline mode; Figure 3 is a flowchart of a method for processing an inference request according to an embodiment of the present invention. Figure 3 As shown, the process includes the following steps: S301: Obtain a first inference queue corresponding to a first node.

[0052] The first inference queue stores N1 inference requests. The N1 inference requests are the N1 inference requests being executed in the node memory of the first node. N1≥1 and is a positive integer. The first inference queue is also called run_tasks.

[0053] Each inference request corresponds to a number of tokens, and the length of each inference request corresponding to each inference request is the total length of multiple tokens. The first node executes the several tokens of each inference request and infers the kvcache.

[0054] The N1 inference requests being executed are requests for which the first node has already inferred some tokens and the corresponding kvcache obtained by inference is stored in the node memory of the first node.

[0055] The first inference queue also stores inference data block information (run_blocks). The run_blocks information records the request identifier (request_id) of each inference request being executed, as well as the address information of the corresponding kvcache storage obtained by inference. The address information includes the identifier (block ID) of the data block storing the kvcache and the spatial address of the data block storing the kvcache on the node memory.

[0056] It is understandable that for each inference request, each request inference corresponds to several tokens, which require multiple blocks to store. As the corresponding kvcache obtained by inference increases, more blocks are needed to store kvcache.

[0057] S302: Obtain available storage resources of the first node, and obtain target storage resources required by the first node to execute N1 inference requests.

[0058] The available storage resources are storage resources allocated by the processor to the first node.

[0059] The target storage resource is the storage resource required to store all kvcaches corresponding to all tokens of all reasoning requests executed by the first node.

[0060] In the embodiment of the present invention, storage resources are all in units of data blocks.

[0061] In some optional embodiments, the processor obtains the free storage resource value of each node among L nodes; determines the minimum free storage resource value among the L free storage resource values, and based on the minimum free resource value, determines a target free storage resource of the minimum free resource value size among the free storage resources of the first node, and determines the target free storage resource as the available storage resource of the first node.

[0062] It can be understood that in order to ensure that data storage does not overflow on the L nodes, the available storage resources allocated to each node will be allocated according to the minimum value of the free storage resources that can be allocated to all nodes.

[0063] Example 1: Taking the inference task as being divided into n pieline segments, each block being a preset data block size (block_size), the free storage resources of each node being allo_memi, and the storage resources required for the kvcache corresponding to each token in each pieline segment being pkv_memi as an example, the processor determines that the available storage resources of the first node are min(allo_memi / (block_size×pkv_memi)).

[0064] Among them, 0≤i<n, block_size can be 64Bytes or 128Bytes.

[0065] It is understandable that when multiple nodes use different memory spaces, due to the differences in memory space and weights of different nodes, the storage resources allocated by the processor to each node to store kvcache are different. In order to ensure that data storage does not overflow on L nodes, the available storage resources allocated to each node will be allocated according to the minimum value of the free storage resources that can be allocated to all nodes. At different pipeline stages, the size of storage resources required for the same token corresponding to the kvcache may also be different. In all pipeline segments, for a specific model, the total size of storage resources required for the kvcache corresponding to each token is fixed.

[0066] In some optional implementations, all idle storage resources of L nodes are acquired; and all idle storage resources are determined as available storage resources of the first node.

[0067] Example 2: Taking the inference task as being divided into n pieline segments, each block being a preset data block size (block_size), all free storage resources being pkv_mem, and the storage resources required for the kvcache corresponding to each token in all pieline segments being pkv_mem as an example, the processor determines that the available storage resources of the first node are min(allo_mem / (block_size×pkv_mem)).

[0068] It is understandable that when multiple nodes share a memory space, the available storage resources of the first node cannot exceed all free storage resources, that is, the total amount of shared memory.

[0069] It can be understood that the available storage resources of the first node are the storage resources in the node memory of the first node. The host memory on the processor is much larger than the node memory of the node. Accordingly, the storage resources of the first node on the host memory of the processor are 3 times or more of the node memory. The processor records the data block identifier (block ID) and the corresponding address corresponding to the storage resources of the first node in the host memory of the processor.

[0070] In some optional embodiments, the processor obtains the length of each inference request corresponding to each inference request in N1 inference requests; calculates a first ratio between the length of each inference request and a preset data block size, and rounds the first ratio to obtain N1 first values; calculates the sum of each first value and the preset value to obtain N1 second values; calculates the sum of the N1 second values ​​to obtain a third value.

[0071] Each second value is the number of data blocks corresponding to the storage resource required by the first node to execute each inference request. The third value represents the number of target data blocks required by the first node corresponding to the target storage resource to execute N1 inference requests. The preset value may be 1.

[0072] Example 3, taking the preset data block size as 64Bytes and the preset value as 1 as an example, for example: the processor obtains the inference request length of the first inference request as 127, calculates the first ratio as 1.98, and rounds 1.98 to obtain the first value of 1; calculates the sum of 1 and 1, and obtains the second value of 2, that is, the number of data blocks required for the first node to execute the first inference request is 2.

[0073] For another example: the processor obtains the inference request length of the second inference request as 128, calculates the first ratio as 2, and rounds 2 to obtain a first value of 2; calculates the sum of 2 and 1, and obtains a second value of 3, that is, the number of data blocks required for the first node to execute the second inference request is 3.

[0074] S303: If the available storage resources are less than the target storage resources, the first inference queue is updated to the second inference queue according to the storage resources required by each inference request in the N1 inference requests and the available storage resources.

[0075] The second inference queue stores N2 inference requests, 1≤N2<N1, and N2 is a positive integer.

[0076] In some optional embodiments, the processor calculates a second ratio between the available storage resources and the preset data block size, and rounds the second ratio to obtain a fourth value, where the fourth value represents the number of available data blocks of the first node corresponding to the available storage resources; compares the third value and the fourth value to determine whether the available storage resources are greater than or equal to the target storage resources.

[0077] In one example, if the fourth value is greater than or equal to the third value, the processor determines that the available storage resources are greater than or equal to the target storage resources. If the fourth value is less than the third value, the processor determines that the available storage resources are less than the target storage resources.

[0078] It is understandable that the processor can compare the available storage resources and the target storage resources by converting the preset data block size into the number of data blocks, and can accurately determine whether the available resources of the first node can meet the storage resources required by the first node to execute the N1 inference requests in the first inference queue. In some optional implementations, the processor obtains the first exchange queue corresponding to the first node; determines the N1-N2 inference requests to be swapped out from the first inference queue based on the storage resources and available storage resources required for each inference request in the N1 inference requests; swaps out the N1-N2 inference requests from the first inference queue to obtain a second inference queue; swaps the N1-N2 inference requests into the first exchange queue to obtain a second exchange queue.

[0079] The first exchange queue stores M inference requests and a first mapping table. The M inference requests are M inference requests swapped out from the node memory to the host memory of the processor. The first mapping table records a first mapping relationship between each inference request in the M inference requests and the address information of each inference request in the host memory. The first exchange queue is also called run_tasks.

[0080] The address information is the address information of the corresponding kvcache storage obtained by reasoning for each reasoning request.

[0081] The second exchange queue stores M+N1-N2 inference requests and a mapping table. The mapping table records the mapping relationship between each inference request in the M+N1-N2 inference requests and the address information of each inference request in the host memory.

[0082] In one example, the processor compares each second value corresponding to each inference request in N1 inference requests with the fourth value in order from large to small, and determines N1-N2 inference requests, where each second value corresponding to each inference request in N1-N2 inference requests is greater than or equal to each second value corresponding to each inference request in N2 inference requests.

[0083] For example, taking block_size=64, 4 inference requests are stored in the first exchange queue, and the lengths of the 4 inference requests are [127, 128, 127, 128], and the 4 second values ​​corresponding to the 4 inference requests are [2, 3, 2, 3], if the fourth value is 6, the processor compares each second value with the fourth value in sequence according to the order of each second value from large to small [3, 3, 2, 2], and determines that N1-N2 inference requests are 2 inference requests corresponding to the first 2 second values; if the fourth value is 5, the processor compares each second value with the fourth value in sequence according to the order of each second value from large to small [3, 3, 2, 2], and determines that N1-N2 inference requests are 2 inference requests corresponding to the first 2 second values. ], compares each second value with the fourth value in turn, and determines that N1-N2 inference requests are 2 inference requests corresponding to the first second value and the third second value; if the fourth value is 4, the processor compares each second value with the fourth value in turn according to the order of each second value from large to small [3, 3, 2, 2], and determines that N1-N2 inference requests are 1 inference request corresponding to the first second value; if the fourth value is 1, the processor compares each second value with the fourth value in turn according to the order of each second value from large to small [3, 3, 2, 2], and determines that N1-N2 inference requests are 0 inference requests.

[0084] In another example, the processor compares each second value corresponding to each of the N1 inference requests with the fourth value in turn to determine that the sum of the N1-N2 second values ​​corresponding to the N1-N2 inference requests is less than or equal to the fourth value.

[0085] For example, taking block_size=64, 4 inference requests are stored in the first exchange queue, and the length of the 4 inference requests is [127, 128, 127, 128], and the 4 second numerical values ​​corresponding to the 4 inference requests are [2, 3, 2, 3], if the fourth numerical value is 5 or 6, the processor compares each second numerical value with the fourth numerical value in turn, and determines that N1-N2 inference requests are 2 inference requests corresponding to the first 2 second numerical values; if the fourth numerical value is 4, the processor compares each second numerical value with the fourth numerical value in turn, and determines that N1-N2 inference requests are 1 inference request corresponding to the first second numerical value.

[0086] In another example, the processor obtains each swap-in time of each of N1 inference requests into the first inference queue; calculates the third difference between each swap-in time and the current time to obtain N1 third differences; determines T third differences from the N1 third differences; and compares the T second values ​​with the fourth value in descending order of the T second values ​​corresponding to the T inference requests to determine S inference requests.

[0087] Among them, each third difference value of the T third difference values ​​is less than or equal to the first preset time period, and the T third difference values ​​correspond to T reasoning requests. The sum of the S second values ​​corresponding to the S reasoning requests is less than the fourth value. The S second values ​​corresponding to the S reasoning requests are the first S second values ​​of the T second values.

[0088] The first preset time period can be set according to actual needs and is not limited.

[0089] It can be understood that the processor obtains the time when N1 inference requests are swapped into the first inference queue, and determines T third differences from the N1 third differences, that is, the T inference requests that are swapped into the first inference queue later are screened out from the N1 inference requests. Since the T inference requests are swapped into the first inference queue later, it indicates that the inference progress of the T inference requests is small, and the number of tokens obtained by the inference requests in the T inference requests is small, which facilitates swapping out the T inference requests from the first exchange queue, thereby improving the utilization rate of the node memory (i.e., video memory resources).

[0090] S304: Send indication information including the second inference queue to the first node.

[0091] The indication information is used to swap out N1-N2 inference requests from the node memory to the host memory of the processor, and continue to execute N2 inference requests.

[0092] Optionally, if the available storage resources are greater than or equal to the target storage resources, the processor calculates a first difference between the fourth value and the third value, and detects whether the first exchange queue is empty; if not, it detects whether the first difference is greater than or equal to at least one of the M+N1-N2 second values; if so, based on each second value and the fourth value corresponding to each inference request in the M+N1-N2 inference requests, determine R inference requests that need to be swapped into the second inference queue from the M+N1-N2 inference requests; based on the mapping table and the R inference requests, swap the R second inference requests from the second exchange queue into the second inference queue to obtain an updated third inference queue; and send a first indication message containing the third inference queue to the first node.

[0093] The first indication information is used to indicate that R inference requests are swapped out from the host memory to the node memory. The M+N1-N2 second values ​​are the data blocks required for each inference request in the M+N1-N2 inference requests, and R<M+N1-N2.

[0094] In one example, when the fourth value is less than or equal to the number of inference requests in the second inference queue, the inference request is stopped from being swapped into the second inference queue. This can avoid frequent swapping in or out operations of the first node, thereby improving the overall performance of the first node.

[0095] In one example, if the processor detects that the first difference is greater than or equal to at least one of the M+N1-N2 second values, then each swap-in time of each of the M+N1-N2 inference requests swapped into the first swap queue is obtained; and each second value corresponding to each of the M+N1-N2 inference requests is compared with the fourth value in the time sequence of the M+N1-N2 swap-in times to determine R inference requests.

[0096] Among them, the sum of the R second numerical values ​​corresponding to the R inference requests is less than the fourth numerical value, and the R inference requests corresponding to the R inference requests are the first R inference requests among the M+N1-N2 inference requests.

[0097] In another example, if the processor detects that the first difference is greater than or equal to at least one of the M+N1-N2 second values, then each swap-in time of each of the M+N1-N2 inference requests swapped into the first swap queue is obtained; the second difference between each swap-in time and the current moment is calculated to obtain M+N1-N2 second difference values; P second difference values ​​are determined from the M+N1-N2 second difference values; and the P second values ​​are compared with the fourth value in order from small to large of the P second values ​​corresponding to the P inference requests to determine R inference requests.

[0098] Among them, each second difference value among the P second difference values ​​is less than or equal to a preset time period, and the P second difference values ​​correspond to P reasoning requests. The sum of the R second values ​​corresponding to the R reasoning requests is less than the fourth value. The R second values ​​corresponding to the R reasoning requests are the first R second values ​​among the P second values.

[0099] A possible design is that the preset time period can be designed according to actual needs without restriction.

[0100] In one example, if the processor detects that the first exchange queue is empty, it obtains the first waiting queue corresponding to the first node, and detects whether W inference requests are stored in the first waiting queue, where the W inference requests are the W inference requests waiting to be executed in the node memory of the first node, and W ≥ 1 is a positive integer; if so, it detects whether the first difference is greater than or equal to at least one of the W second values, and the W second values ​​are detected as data blocks required for each of the W inference requests.

[0101] If not, then determine the Q inference requests that need to be replaced into the first inference queue from the W inference requests according to each second value and the fourth value corresponding to each inference request of the W inference requests, 1≤Q<W, and Q is a positive integer; replace the Q inference requests from the first waiting queue into the first inference queue to obtain an updated fourth inference queue; send a second indication message containing the fourth inference queue to the first node, and the second indication message is used to instruct the first node to execute Q additional inference requests.

[0102] In some optional implementations, if the processor detects that the first waiting queue is empty, it determines that the first node has completed the first reasoning stage, and sends third indication information to the first node.

[0103] The third indication information is used to instruct the first node to execute the second reasoning stage.

[0104] The first inference stage is the Prefill stage; the second inference stage is the Decode stage.

[0105] In an embodiment of the present invention, the exchange queue (swapped_tasks), inference queue (run_tasks) and waiting queue (wait_tasks) of each of the L nodes are synchronized with the exchange queue (for example, the first exchange queue, the second exchange queue), inference queue (first inference queue, the second inference queue) and waiting queue (first waiting queue) of the first node.

[0106] For example, Figure 4 As shown, Figure 4is a schematic diagram of a second node performing an inference task according to an embodiment of the present invention. After the second node receives the intermediate result sent by the first node, the second node may further perform the following steps: S401: Detect whether the inference request in swapped_tasks has changed.

[0107] S402: If yes, determine whether the inference requests in swapped_tasks are reduced.

[0108] S403: If the inference requests decrease and the available storage resources of the second node meet the swap-in condition, a swap-in operation is performed, wherein the swap-in operation is to swap the inference requests in run_tasks into swapped_tasks.

[0109] S404: If the inference request increases and all the inference requests in the run_tasks of the second node complete the inference, a swap-out operation is performed, wherein the swap-out operation is to swap the inference requests in swapped_tasks to run_tasks.

[0110] S405: If it is detected that the inference request in swapped_tasks has not changed, then it is detected whether the inference request in wait_tasks has changed.

[0111] S406: If yes, then perform prefill phase reasoning for new requests in wait_tasks (requests moved from wait_tasks to run_tasks).

[0112] S407: If not, execute the decode phase reasoning of the reasoning request in run_tasks.

[0113] S408: Send the exchange queue, the inference queue, the waiting queue and the inference data to the next node.

[0114] It can be understood that when the inference request of the second node in swapped_tasks changes, if the execution speed of the second node is faster than that of the first node, the inference request that can be swapped in can be swapped in first, thereby reducing the waiting time and improving the pipeline efficiency.

[0115] In one example, during the process of executing an inference request, the first node may also send the kvcache data (ie, inference data) corresponding to the token in the inference request to the second node, so that the second node performs an inference task based on the inference data.

[0116] It can be understood that the previous node can send the reasoning data corresponding to the execution reasoning request to the next node; if the second node is the last node, the reasoning data is sent to the first node to perform the next reasoning stage.

[0117] The relationship between the L nodes is reflected in the fact that they sequentially pass data and tasks through the pipeline, ensuring the smooth operation of the entire process from task reception, preliminary processing to final completion. The first node starts the reasoning task and passes the processing results step by step to node L, which is responsible for the final processing steps, working together to achieve efficient task processing and resource management. Node L will pass the calculated tokens value to the first node as input for subsequent reasoning. In addition, the wait_tasks, run_tasks, and swapped_tasks information can be transmitted in sequence using the above-mentioned node transmission scheme, or it can be distributed by the first node to all nodes L.

[0118] Optionally, when an inference group of the second node is executing a task, if the second node has remaining storage resources, the inference requests of other inference groups in swapped_tasks can be scheduled to the currently executing inference group, thereby achieving more full and efficient use of the storage resources of the second node.

[0119] Based on the above Figure 3 According to the method, the processor can obtain a first inference queue corresponding to a first node among L nodes, obtain available storage resources of the first node, and obtain target storage resources required for the first node to execute N1 inference requests; determine whether the available storage resources are greater than or equal to the target storage resources, and if not, update the first inference queue to a second inference queue based on the storage resources required for each inference request among the N1 inference requests and the available storage resources, and send indication information containing the second inference queue to the first node.

[0120] When the available storage resources of the first node are greater than or equal to the target storage resources required by the first node to execute N1 inference requests, it indicates that the available storage resources of the first node cannot satisfy the inference requests in the first inference queue executed by the first node. At this time, N1-N2 inference requests can be determined from the first inference queue, and the N1-N2 inference requests can be swapped out from the node memory to the host memory of the processor. Therefore, the problem of insufficient node memory of the first node when executing inference tasks based on a parallel pipeline can be effectively solved. At the same time, based on the different requirements of different inference requests for node memory, the memory resources of the first node can be reasonably allocated to avoid the problem that some tasks occupy a large amount of video memory while other tasks cannot get enough resources, improve the resource utilization of the node memory of the first node, improve the scheduling efficiency of the node memory, simplify the complexity of memory management, and thus improve the overall performance of large model inference.

[0121] In this embodiment, a processing device for reasoning requests is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0122] This embodiment provides a device for processing an inference request, such as Figure 5 As shown, the device comprises: The transceiver module 501 is used to obtain a first inference queue corresponding to the first node, where the first inference queue stores N1 inference requests being executed in the node memory of the first node, where N1≥1 and is a positive integer.

[0123] The transceiver module 501 is also used to obtain the available storage resources of the first node, and to obtain the target storage resources required by the first node to execute N1 inference requests.

[0124] The processing module 502 is also used to update the first inference queue to the second inference queue if the available storage resources are less than the target storage resources, based on the storage resources required for each inference request in N1 inference requests and the available storage resources. The second inference queue stores N2 inference requests, 1≤N2<N1, and N2 is a positive integer.

[0125] The transceiver module 501 is also used to send indication information containing the second inference queue to the first node, and the indication information is used to instruct the first node to swap out N1-N2 inference requests from the node memory to the host memory of the processor, and continue to execute N2 inference requests.

[0126] In some optional embodiments, the first node is the first node among L nodes, L ≥ 1 and is a positive integer; the transceiver module 501 is specifically used to obtain the free storage resource value of each node among the L nodes; determine the minimum free resource value among the L free storage resource values, and based on the minimum free resource value, determine the target free storage resource of the minimum free resource value size in the free storage resources of the first node, and determine the target free storage resource as the available storage resource of the first node.

[0127] In some optional implementations, the transceiver module 501 is further specifically configured to obtain all idle storage resources of L nodes; and determine all idle storage resources as available storage resources of the first node.

[0128] In some optional embodiments, the transceiver module 501 is further specifically used to obtain the length of each inference request corresponding to each of the N1 inference requests; calculate a first ratio between the length of each inference request and a preset data block size, and round the first ratio to obtain N1 first values; calculate the sum of each first value and the preset value to obtain N1 second values, each second value being the number of data blocks corresponding to the storage resources required by the first node to execute each inference request; calculate the sum of the N1 second values ​​to obtain a third value, and the third value represents the target number of data blocks required for the first node corresponding to the target storage resource to execute N1 inference requests.

[0129] In some optional embodiments, the transceiver module 501 is also specifically used to calculate a second ratio between the available storage resources and the preset data block size, and round the second ratio to obtain a fourth value, where the fourth value represents the number of available data blocks of the first node corresponding to the available storage resources; compare the third value and the fourth value to determine whether the available storage resources are greater than or equal to the target storage resources.

[0130] In some optional embodiments, the processing module 502 is specifically used to obtain a first exchange queue corresponding to the first node, the first exchange queue stores M inference requests, the M inference requests are M inference requests swapped out from the node memory to the host memory of the processor, M≥1 and is a positive integer; according to the storage resources required for each inference request in the N1 inference requests and the available storage resources, determine the N1-N2 inference requests swapped out from the first inference queue; swap out the N1-N2 inference requests from the first inference queue to obtain a second inference queue; swap the N1-N2 inference requests into the first exchange queue to obtain a second exchange queue, and the second exchange queue stores M+N1-N2 inference requests.

[0131] In some optional embodiments, the processing module 502 is further specifically used to compare, in descending order of each second value, each second value corresponding to each inference request in the N1 inference requests with the fourth value, to determine the N1-N2 inference requests swapped out from the first inference queue, and each second value corresponding to each inference request in the N1-N2 inference requests is greater than or equal to each second value corresponding to each inference request in the N2 inference requests.

[0132] In some optional embodiments, the second exchange queue also stores a mapping table, which records the mapping relationship between each inference request in the M+N1-N2 inference requests and the address information of each inference request in the host memory; the processing module 502 is also used to calculate the first difference between the fourth value and the third value if the available storage resources are greater than or equal to the target storage resources, and detect whether the first exchange queue is empty; if not, detect whether the first difference is greater than or equal to at least one second value among the M+N1-N2 second values, and the M+N1-N2 second values ​​are the data blocks required for each inference request in the M+N1-N2 inference requests. ; If so, then according to each second value and fourth value corresponding to each inference request in the M+N1-N2 inference requests, determine the R inference requests that need to be swapped into the second inference queue from the M+N1-N2 inference requests, 1≤R<M+N1-N2, and R is a positive integer; according to the mapping table and the R inference requests, swap the R second inference requests from the second exchange queue into the second inference queue to obtain an updated third inference queue; the transceiver module 501 is also used to send a first indication message containing the third inference queue to the first node, and the first indication message is used to instruct the first node to swap out the R inference requests from the host memory to the node memory.

[0133] In some optional embodiments, the processing module 502 is further specifically used to obtain each swap-in time of each inference request in the M+N1-N2 inference requests into the first swap queue if it is detected that the first difference is greater than or equal to at least one second value among the M+N1-N2 second values; compare each second value corresponding to each inference request in the M+N1-N2 inference requests with the fourth value in the time sequence of the M+N1-N2 swap-in times, determine R inference requests, the sum of the R second values ​​corresponding to the R inference requests is less than the fourth value, and the R inference requests are the first R inference requests in the M+N1-N2 inference requests.

[0134] In some optional embodiments, the processing module 502 is further specifically used to obtain each swap-in time of each inference request in the M+N1-N2 inference requests into the first exchange queue if it is detected that the first difference is greater than or equal to at least one second value among the M+N1-N2 second values; calculate the second difference between each swap-in time and the current moment to obtain M+N1-N2 second differences; determine P second differences from the M+N1-N2 second differences, each of the P second differences is less than or equal to a preset time period, the P second differences correspond to P inference requests, R≤P≤M+N1-N2, and P is a positive integer; compare the P second values ​​with the fourth value in the order of the P second values ​​corresponding to the P inference requests from small to large, determine R inference requests, the sum of the R second values ​​corresponding to the R inference requests is less than the fourth value, and the R second values ​​corresponding to the R inference requests are the first R second values ​​among the P second values.

[0135] In some optional embodiments, the processing module 502 is also used to obtain the first waiting queue corresponding to the first node if it is detected that the first exchange queue is empty, and detect whether W inference requests are stored in the first waiting queue, where the W inference requests are the W inference requests waiting to be executed in the node memory of the first node, and W ≥ 1 and is a positive integer; if so, detect whether the first difference is greater than or equal to at least one of the W second values, and the W second values ​​are detected as the data blocks required for each inference request in the W inference requests; if not, determine Q inference requests that need to be swapped into the first inference queue from the W inference requests according to each second value and the fourth value corresponding to each of the W inference requests, where Q < W; swap the Q inference requests from the first waiting queue into the first inference queue to obtain an updated fourth inference queue; the transceiver module 501 is also used to send a second indication message containing the fourth inference queue to the first node, and the second indication message is used to instruct the first node to execute Q new inference requests.

[0136] In some optional embodiments, the processing module 502 is also used to determine that the first node has completed the first reasoning stage of the reasoning service if it is detected that the first waiting queue is empty, and send a third indication message to the first node, where the third indication message is used to instruct the first node to execute the second reasoning stage of the reasoning service.

[0137] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0138] The processing device for the inference request in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0139] The embodiment of the present invention also provides a computer device having the above Figure 5 An inference request processing device is shown.

[0140] See also Figure 6 , Figure 6 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 6 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 6 A processor 10 is taken as an example.

[0141] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0142] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0143] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0144] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0145] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0146] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0147] Part of the present application may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present application through the operation of the computer. Those skilled in the art should understand that the existence of computer program instructions in computer-readable media includes but is not limited to source files, executable files, installation package files, etc., and accordingly, the way in which computer program instructions are executed by a computer includes but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to the computer.

[0148] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for processing an inference request, characterized in that: The method comprises: Obtain a first inference queue corresponding to the first node, where the first inference queue stores N1 inference requests being executed in the node memory of the first node, where N1≥1 and is a positive integer; Acquire available storage resources of the first node, and acquire target storage resources required by the first node to execute the N1 inference requests; If the available storage resources are less than the target storage resources, the first inference queue is updated to a second inference queue according to the storage resources required by each of the N1 inference requests and the available storage resources, wherein the second inference queue stores N2 inference requests, 1≤N2<N1, and N2 is a positive integer; Sending indication information including the second inference queue to the first node, wherein the indication information is used to instruct the first node to swap out the N1-N2 inference requests from the node memory to the host memory, and continue to execute the N2 inference requests.

2. The method according to claim 1, characterized in that: The first node is the first node among L nodes, where L≥1 and is a positive integer; and obtaining the available storage resources of the first node includes: Obtaining the free storage resource value of each of the L nodes; Determine the minimum free resource value among the L free storage resource values, and based on the minimum free resource value, determine a target free storage resource of the size of the minimum free resource value in the free storage resources of the first node, and determine the target free storage resource as the available storage resource of the first node.

3. The method according to claim 2, characterized in that The obtaining the available storage resources of the first node includes: Obtain all idle storage resources of the L nodes; All the idle storage resources are determined as the available storage resources of the first node.

4. The method according to claim 2 or 3, characterized in that: The obtaining the target storage resources required by the first node to execute the N1 inference requests includes: Obtaining the length of each inference request corresponding to each inference request in the N1 inference requests; Calculating a first ratio between each of the inference request lengths and a preset data block size, and rounding the first ratio to obtain N1 first values; Calculate the sum of each of the first values ​​and a preset value to obtain N1 second values, each of the second values ​​being the number of data blocks corresponding to the storage resources required by the first node to execute each of the inference requests; The sum of the N1 second numerical values ​​is calculated to obtain a third numerical value, where the third numerical value represents the number of target data blocks required for the first node corresponding to the target storage resource to execute the N1 inference requests.

5. The method according to claim 4, characterized in that Before updating the first inference queue to the second inference queue according to the storage resources required by each of the N1 inference requests and the available storage resources, the method includes: Calculating a second ratio between the available storage resource and the preset data block size, and rounding the second ratio to obtain a fourth value, where the fourth value represents the number of available data blocks of the first node corresponding to the available storage resource; The third value and the fourth value are compared to determine whether the available storage resource is greater than or equal to the target storage resource.

6. The method according to claim 5, characterized in that The updating the first inference queue to a second inference queue according to the storage resources required by each of the N1 inference requests and the available storage resources includes: Obtain a first exchange queue corresponding to the first node, where the first exchange queue stores M inference requests, where the M inference requests are M inference requests swapped out from the node memory to the host memory, where M≥1 and is a positive integer; Determine, according to the storage resources required by each of the N1 inference requests and the available storage resources, N1-N2 inference requests to be swapped out from the first inference queue; swapping out the N1-N2 inference requests from the first inference queue to obtain the second inference queue; The N1-N2 inference requests are exchanged into the first exchange queue to obtain a second exchange queue, and the second exchange queue stores M+N1-N2 inference requests.

7. The method according to claim 6, characterized in that The determining, according to the storage resources required by each of the N1 inference requests and the available storage resources, N1-N2 inference requests swapped out from the first inference queue comprises: In order of each of the second values ​​from large to small, compare each of the second values ​​corresponding to each of the N1 inference requests with the fourth value in turn to determine the N1-N2 inference requests swapped out from the first inference queue, and each of the second values ​​corresponding to each of the N1-N2 inference requests is greater than or equal to each of the second values ​​corresponding to each of the N2 inference requests.

8. The method according to claim 7, characterized in that The second exchange queue further stores a mapping table, wherein the mapping table records a mapping relationship between each of the M+N1-N2 inference requests and address information of each of the inference requests in the host memory; The method further comprises: If the available storage resource is greater than or equal to the target storage resource, calculating a first difference between the fourth value and the third value, and detecting whether the first exchange queue is empty; If not, detecting whether the first difference is greater than or equal to at least one of the M+N1-N2 second values, where the M+N1-N2 second values ​​are data blocks required for each inference request in the M+N1-N2 inference requests; If so, then determine R inference requests that need to be swapped into the second inference queue from the M+N1-N2 inference requests according to each of the second numerical values ​​and the fourth numerical values ​​corresponding to each of the M+N1-N2 inference requests, where 1≤R<M+N1-N2, and R is a positive integer; According to the mapping table and the R inference requests, swapping the R second inference requests from the second exchange queue into the second inference queue to obtain an updated third inference queue; Sending first indication information including the third inference queue to the first node, wherein the first indication information is used to instruct the first node to swap out the R inference requests from the host memory to the node memory.

9. The method according to claim 8, characterized in that If so, determining R inference requests that need to be swapped into the second inference queue from the M+N1-N2 inference requests according to each second value corresponding to each inference request in the M+N1-N2 inference requests and the fourth value, including: If it is detected that the first difference is greater than or equal to at least one of the M+N1-N2 second values, then obtaining each swap-in time of each inference request in the M+N1-N2 inference requests being swapped into the first swap queue; According to the time sequence of the M+N1-N2 swap-in times, each second numerical value corresponding to each inference request in the M+N1-N2 inference requests is compared with the fourth numerical value to determine the R inference requests, the sum of the R second numerical values ​​corresponding to the R inference requests is less than the fourth numerical value, and the R inference requests are the first R inference requests in the M+N1-N2 inference requests.

10. The method according to claim 8, characterized in that If so, determining R inference requests that need to be swapped into the second inference queue from the M+N1-N2 inference requests according to each second value corresponding to each inference request in the M+N1-N2 inference requests and the fourth value, including: If it is detected that the first difference is greater than or equal to at least one of the M+N1-N2 second values, then obtaining each swap-in time of each inference request in the M+N1-N2 inference requests being swapped into the first swap queue; Calculate the second difference between each of the swap-in times and the current time to obtain M+N1-N2 second differences; Determine P second differences from the M+N1-N2 second differences, each of the P second differences is less than or equal to a preset time period, the P second differences correspond to P inference requests, R≤P≤M+N1-N2, and P is a positive integer; In the order of the P second values ​​corresponding to the P inference requests from small to large, the P second values ​​are compared with the fourth value to determine the R inference requests, the sum of the R second values ​​corresponding to the R inference requests is less than the fourth value, and the R second values ​​corresponding to the R inference requests are the first R second values ​​among the P second values.

11. The method according to claim 8, characterized in that The method further comprises: If it is detected that the first exchange queue is empty, obtain the first waiting queue corresponding to the first node, and detect whether W inference requests are stored in the first waiting queue, where the W inference requests are W inference requests waiting to be executed in the node memory of the first node, where W≥1 and is a positive integer; If so, detecting whether the first difference is greater than or equal to at least one of the W second values, the W second values ​​being detected as data blocks required for each of the W inference requests; If not, determining Q inference requests that need to be swapped into the first inference queue from the W inference requests according to each of the second values ​​and the fourth values ​​corresponding to each of the W inference requests, where 1≤Q<W, and Q is a positive integer; swapping the Q inference requests from the first waiting queue into the first inference queue to obtain an updated fourth inference queue; Sending second indication information including the fourth inference queue to the first node, where the second indication information is used to instruct the first node to newly execute the Q inference requests.

12. The method according to claim 11, characterized in that The method further comprises: If it is detected that the first waiting queue is empty, it is determined that the first node has completed the first reasoning stage of the reasoning service, and a third indication message is sent to the first node, where the third indication message is used to instruct the first node to execute the second reasoning stage of the reasoning service.

13. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method for processing an inference request according to any one of claims 1 to 12 by executing the computer instructions.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method for processing an inference request according to any one of claims 1 to 12.

15. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the method for processing an inference request according to any one of claims 1 to 12.

Images