Node management method and apparatus

Abstract

A node management method and device are disclosed, and relate to the technical field of computers. The method is applied to an inference cluster, the inference cluster comprising nodes of a first type and nodes of a second type, the nodes of the first type being used to perform pre-filling tasks, and the nodes of the second type being used to perform decoding tasks. In the process of the inference cluster executing an inference request, a target node can be determined from the nodes of the first type and the nodes of the second type, the type of the target node being the type with less load among the first type and the second type, and the type of the target node being configured to be the type with more load among the first type and the second type. In this way, not only can the number of nodes of the type with more load be increased without increasing the total number of nodes of the inference cluster, thereby improving the rate at which the nodes of the type with more load process inference tasks, but also the nodes of the type with less load of the inference cluster can be fully utilized, thereby improving the utilization rate of the nodes of the type with less load.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a node management method and apparatus. Background Technology

[0002] With the development of inference technology, inference services are widely used in daily life. In related technologies, to avoid interference between prefill and decode tasks during inference request processing, the approach is to distribute these tasks across different nodes in the inference cluster. However, this inference method results in low throughput for the inference cluster, severely impacting its performance.

[0003] Therefore, improving the throughput of inference clusters has become a pressing technical problem that needs to be solved. Summary of the Invention

[0004] This application provides a node management method and apparatus that can improve the throughput of an inference cluster, thereby improving the performance of the inference cluster.

[0005] In a first aspect, a node management method is provided for an inference cluster, which includes a first type of node and a second type of node. The first type of node is used to perform pre-filling tasks, and the second type of node is used to perform decoding tasks. The method includes: determining a target node from the first type of node and the second type of node; the target node is of the type with lower load between the first type and the second type; and configuring the type of the target node to be of the type with higher load between the first type and the second type.

[0006] For ease of description, the first type of node will be referred to as the first type node, and the second type of node will be referred to as the second type node.

[0007] Specifically, if the load of the first type of node is less than the load of the second type of node, then the first type is the type with low load, and the second type is the type with high load. If the load of the first type of node is greater than the load of the second type of node, then the first type is the type with high load, and the second type is the type with low load.

[0008] For ease of description, we will use "small load node" to indicate nodes of the low-load type and "large load node" to indicate nodes of the high-load type. For example, if the first type is low-load and the second type is high-load, the small load node indicates the first type of node and the large load node indicates the second type of node. Conversely, if the first type is high-load and the second type is low-load, the large load node indicates the first type of node and the small load node indicates the second type of node.

[0009] In the above scheme, the inference cluster determines the target node from the first type and the second type of nodes, and converts the target node to the type with the highest load among the first and second types. This not only increases the number of high-load nodes without increasing the total number of nodes in the inference cluster, thus improving the processing speed of inference tasks (pre-filling or decoding tasks) on high-load nodes, but also fully utilizes the low-load nodes in the inference cluster, thereby improving their utilization rate. On one hand, increasing the processing speed of inference tasks on high-load nodes increases their throughput, thereby improving the throughput of the inference cluster and ultimately its performance. On the other hand, increasing the processing speed of inference requests on high-load nodes allows for a rapid reduction in their load, thus balancing the load across different types of nodes. This helps prevent high-load nodes from reaching performance bottlenecks, avoiding issues like lag and system crashes, and also helps prevent impacts on the performance of tasks processed by high-load nodes.

[0010] In one possible implementation, the target node type is configured to be the type with higher load between the first type and the second type, including: if the target node type is the first type, the target node type is configured to be the second type.

[0011] In this implementation, when the load on the second type of node is high, the target node in the first type of node is converted into a second type of node, increasing the number of second type of nodes. This not only improves the efficiency of the second type of node in processing decoding tasks, but also improves the utilization rate of the first type of node.

[0012] Another possible implementation involves configuring the target node type as the type with higher load between the first and second types, including: configuring the target node type as the first type when the target node type is the second type.

[0013] In this implementation, when the load on the first type of node is high, the target node in the second type of node is converted into a first type of node, increasing the number of first type of nodes. This not only improves the efficiency of the first type of node in processing decoding tasks, but also improves the utilization rate of the second type of node.

[0014] In another possible implementation, after determining the target node from the first type of node and the second type of node, the method further includes: if the target node processes the first inference request, transferring the first data obtained by the target node in processing the first inference request to the target memory of the inference cluster.

[0015] In this implementation, the first data obtained from processing the first inference request is transferred to the target memory, which allows the target node to stop processing the inference request being processed, and other nodes to continue processing the first inference request based on the first data. This not only improves the efficiency of changing the type of the target node, but also avoids affecting the first inference request being processed, thereby helping to improve the performance of the inference cluster.

[0016] In another possible implementation, the target node includes a first processor and a second processor; transmitting the first data obtained by the target node from processing the first inference request to the target memory of the inference cluster includes: transmitting the first data obtained by the target node from processing the first inference request to the target memory during the communication gap between the first processor and the second processor.

[0017] In this implementation, the first data is transmitted during the communication gap between the two processors on the target node. This allows the first data to be transmitted when the communication resources of the target node are relatively idle. Compared to transmitting the first data during the communication process between the two processors, this helps to avoid preempting the communication resources of the two processors, thereby helping to avoid affecting the rate at which the target node processes inference requests.

[0018] In another possible implementation, the first data obtained by the target node in processing the first inference request is transferred to the target memory of the inference cluster, including: when the target node performs a pre-filling task on the first inference request, the key-value data obtained by each neural network layer on the target node in processing the first inference request is transferred to the target memory.

[0019] In this implementation, since the pre-filling task only performs one round of inference, the key-value data on the target node is exported in units of neural network layers. This helps to ensure the integrity of the key-value data obtained by each neural network layer, thereby helping to improve the accuracy and reliability of the key-value data obtained by each neural network layer in subsequent use.

[0020] In another possible implementation, the first data obtained by the target node in processing the first inference request is transferred to the target memory of the inference cluster, including: when the target node performs a decoding task on the first inference request, the key-value data obtained by the target node in each round of inference on the first inference request is transferred to the target memory.

[0021] In this implementation, since the decoding task requires multiple rounds of inference, the key-value data of the target node is exported in units of each round of inference. This helps to ensure the integrity of the key-value data obtained in each round of inference, thereby helping to improve the accuracy of the key-value data obtained in subsequent rounds of inference.

[0022] In another possible implementation, the first data obtained by the target node in processing the first inference request is transmitted to the target memory of the inference cluster, including: when the target node performs a decoding task on the first inference request, the token obtained by the target node in processing the first inference request is transmitted to the target memory of the inference cluster.

[0023] In this implementation, by exporting the tokens obtained from each round of inference when the target node processes the first inference request, it helps to avoid repeatedly calculating previously obtained tokens, thereby improving inference performance.

[0024] In another possible implementation, the target memory is the memory of the first node, and the type of the first node is the type with less load between the first type and the second type.

[0025] In another possible implementation, the target memory is the memory of the second node, which is a node on the inference cluster other than the first type of node and the second type of node.

[0026] In another possible implementation, the second type of node includes a third node, and the method further includes: if the target node is of the first type, transmitting the second data obtained by the third node in processing the second inference request to the target node; and if the target node is configured to be of the second type, the target node performs the decoding task of the second inference request based on the second data.

[0027] In this implementation, the inference requests being processed by the second type of node are transferred to the target node for processing, which not only helps to improve the processing efficiency of the second inference requests, but also helps to increase the throughput of the second type of node as quickly as possible.

[0028] In another possible implementation, the third node includes a third processor and a fourth processor; transmitting the second data obtained by the third node in processing the second inference request to the target node includes: transmitting the second data obtained by the third node in processing the second inference request to the target node during the communication gap between the third processor and the fourth processor.

[0029] In this implementation, the second data is transmitted during the communication gap between the two processors on the third node. This allows the second data to be transmitted when the communication resources of the third node are relatively idle. Compared to transmitting the second data during the communication between the two processors, this helps to avoid preempting the communication resources of the two processors, thereby helping to avoid affecting the rate at which the third node processes inference requests.

[0030] In another possible implementation, the target node type is configured to be the type with higher load between the first type and the second type, including: configuring the target node's identity identifier as a target identity identifier, which is used to indicate the type with higher load between the first type and the second type.

[0031] In this implementation, the target node's type is changed to a high-load type by modifying the identity identifier stored on the target node with the target identity identifier. In this way, the target node can accurately determine its type based on the identity identifier on its own node, which helps to ensure the accuracy of the tasks executed by the target node.

[0032] In another possible implementation, the target node type is configured to be the type with higher load between the first type and the second type, including: updating node scheduling information, wherein the updated node scheduling information is used to indicate the target node's identity as the target identity; the node scheduling information is used to indicate the identity of the first type of node and the identity of the second type of node.

[0033] In this implementation, by modifying the target node's identity identifier in the node scheduling information to the target identity identifier, the scheduling node can allocate tasks to the target node based on the updated node scheduling information. This allows the scheduling node to directly assign high-load tasks to the target node, thus improving the accuracy and efficiency of task allocation to the target node.

[0034] In another possible implementation, determining the target node from the first type of nodes and the second type of nodes includes: determining the target node based on the first memory utilization rate of the first type of nodes and the second memory utilization rate of the second type of nodes; the type of the target node is the type with lower memory utilization rate between the first type and the second type.

[0035] In this implementation, the type of the node to be converted is determined by memory utilization. This helps to improve the convenience and accuracy of obtaining the parameters for determining the node to be converted, thereby improving the convenience and accuracy of determining the type of the node to be converted.

[0036] In another possible implementation, determining the target node from the first type of nodes and the second type of nodes includes: determining the target node based on the first processing rate of the first type of nodes for the pre-filling task and the second processing rate of the second type of nodes for the decoding task, wherein the type of the target node is the type with the smaller processing rate between the first type and the second type.

[0037] In this implementation, the type of node to be converted is determined by the node's processing efficiency for tasks, which helps to improve the diversity of ways to determine the type of node to be converted.

[0038] In another possible implementation, after determining the target node from the first type of nodes and the second type of nodes, the method further includes: stopping the task of assigning inference requests to the target node.

[0039] In this implementation, after determining the node to be converted, a task is assigned to the node to be converted by setting a node notification. This helps to avoid tasks piling up on the target node and becoming unprocessable, thereby helping to avoid affecting the task processing rate.

[0040] In another possible implementation, when the target node is of type 1, the target node includes at least one of the following: the node with the lowest current memory utilization among the nodes of type 1, the node with current memory utilization less than or equal to a first occupancy threshold among the nodes of type 1, or the node whose current memory utilization is sorted in ascending order. This helps to broaden the selection range of target nodes.

[0041] In another possible implementation, when the target node is of type two, the target node includes at least one of the following: the node with the lowest current memory utilization among the second-type nodes, the node with current memory utilization less than or equal to a first occupancy threshold among the second-type nodes, or the node whose current memory utilization is sorted in ascending order. This helps to broaden the selection range of target nodes.

[0042] In another possible implementation, determining the target node from the first type of nodes and the second type of nodes includes: determining the target node from the first type of nodes and the second type of nodes according to the target period.

[0043] In this implementation, by setting the inference cluster to determine the target node according to the target period, the type of the node can be adjusted periodically, which helps to continuously improve the throughput in the inference process and thus helps to continuously guarantee the inference performance of the inference cluster.

[0044] In another possible implementation, the target node type is the type with less load in the first type and the second type, including: the target node type is the type with less load in the current first type and the second type or the target node type is the type with less load in the first type and the second type during the target time period.

[0045] In this implementation, by setting the time for evaluating the load size to be at least one of the current or target time periods, it helps to increase the probability of switching node types, thereby helping to increase the probability of increasing the throughput of the inference cluster.

[0046] In another possible implementation, the target node's type is configured as the type with higher load among the first type and the second type, including: the target node's type is configured as the type with higher load among the current first type and the second type, or the target node's type is configured as at least one of the types with higher load among the first type and the second type during the target time period.

[0047] In another possible implementation, the method further includes: inputting the current memory utilization rate of the first type of node into the first neural network model to obtain the first memory utilization rate; wherein, the first memory utilization rate is the memory utilization rate of the first type of node in the target time period, and the target time period is the time period after the current moment.

[0048] In this implementation, the utilization rate of the target time period is determined through a neural network model, which helps to improve the determination efficiency.

[0049] In another possible implementation, the method further includes: obtaining the number and length of inference requests processed by the first type of nodes during the target time period; inputting the number and length of inference requests processed by the first type of nodes during the target time period into a second neural network model to obtain a second memory utilization rate. The second memory utilization rate is the memory utilization rate of the second type of nodes during the target time period.

[0050] In another possible implementation, obtaining the number and length of inference requests processed by the first type of node in the target time period includes: using the number and length of inference requests currently processed by the first type of node in the third neural network model to obtain the number and length of inference requests processed by the first type of node in the target time period.

[0051] In another possible implementation, the method further includes: inputting the current memory utilization of the second type of node into the first neural network model to obtain the second memory utilization.

[0052] In another possible implementation, determining the target node based on the first memory utilization rate of the first type of node and the second memory utilization rate of the second type of node includes: determining the target node based on the ratio between the first memory utilization rate and the second memory utilization rate.

[0053] In this implementation, the type of node to be converted is determined by the ratio of the first memory utilization rate to the second memory utilization rate. This helps to determine the proportion of the first memory utilization rate to the second memory utilization rate, which in turn helps to accurately measure the gap between the first memory utilization rate and the second memory utilization rate, thereby improving the accuracy of determining the type of node to be converted.

[0054] In another possible implementation, the target node is determined based on the ratio between the first memory utilization rate and the second memory utilization rate, including: determining a first ratio, which is equal to the ratio of the first memory utilization rate to the second memory utilization rate; if the first ratio is less than or equal to a first ratio threshold, determining the type of the target node as a first type; if the first ratio is greater than or equal to a second ratio threshold, determining the type of the target node as a second type.

[0055] In another possible implementation, determining the target node based on the ratio between the first memory utilization rate and the second memory utilization rate includes: determining a second ratio, the second ratio being equal to the ratio of the second memory utilization rate to the first memory utilization rate; if the second ratio is less than or equal to a threshold value of the first ratio, determining the type of the target node as a second type; if the first ratio is greater than or equal to the threshold value of the second ratio, determining the type of the target node as a first type.

[0056] In another possible implementation, determining the target node based on the first memory utilization rate of the first type of node and the second memory utilization rate of the second type of node includes: determining the target node based on the difference between the first memory utilization rate and the second memory utilization rate.

[0057] In this implementation, the type of node to be converted is determined by the difference between the first memory utilization rate and the second memory utilization rate, which helps to improve the diversity of ways to determine the type of node to be converted.

[0058] In another possible implementation, determining the first node based on the difference between the first memory utilization rate and the second memory utilization rate includes: determining the first difference, which is equal to the first memory utilization rate minus the second memory utilization rate; if the first difference is less than or equal to the first difference threshold, determining the type of the target node as the first type; if the first difference is greater than or equal to the second difference threshold, determining the type of the target node as the second type.

[0059] In another possible implementation, determining the first node based on the difference between the first memory utilization rate and the second memory utilization rate includes: determining a second difference, the second difference being equal to the second memory utilization rate minus the first memory utilization rate; if the second difference is less than or equal to the first difference threshold, determining the type of the target node as the second type; if the second difference is greater than or equal to the second difference threshold, determining the type of the target node as the first type.

[0060] In another possible implementation, the method further includes: inputting the current processing rate of the first type of node for the pre-filling task into the fourth neural network model to obtain a first processing rate; wherein, the first processing rate is the processing rate of the first type of node for the pre-filling task in the target time period, and the target time period is the time period after the current time.

[0061] In another possible implementation, the method further includes: obtaining the number and length of inference requests processed by the first type of nodes during the target time period; inputting the number and length of inference requests processed by the first type of nodes during the target time period into the fifth neural network model to obtain a second processing rate. The second processing rate is the processing rate of the second type of nodes for the pre-filled task during the target time period.

[0062] In another possible implementation, obtaining the number and length of inference requests processed by the first type of node in the target time period includes: using the number and length of inference requests currently processed by the first type of node in the third neural network model to obtain the number and length of inference requests processed by the first type of node in the target time period.

[0063] In another possible implementation, the method further includes: inputting the current processing rate of the second type of node for the pre-filled task into the fourth neural network model to obtain the second processing rate.

[0064] In another possible implementation, the method includes determining the target node based on the ratio of the first processing rate and the second processing rate of the node of the first type to the pre-filling task and the node of the second type to the decoding task.

[0065] In another possible implementation, determining the target node based on the ratio of the first processing rate and the second processing rate includes: determining a first ratio, which is equal to the ratio of the first processing rate to the second processing rate; if the first ratio is less than or equal to a first ratio threshold, determining the type of the target node as a first type; if the first ratio is greater than or equal to a second ratio threshold, determining the type of the target node as a second type.

[0066] In another possible implementation, determining the target node based on the ratio of the first processing rate and the second processing rate includes: determining a second ratio, the second ratio being equal to the ratio of the second processing rate to the first processing rate; if the second ratio is less than or equal to a threshold value of the first ratio, determining the type of the target node as a second type; if the first ratio is greater than or equal to the threshold value of the second ratio, determining the type of the target node as a first type.

[0067] In another possible implementation, based on the first processing rate of the first type of node for the pre-filling task and the second processing rate of the second type of node for the decoding task, the method includes: determining the target node based on the difference between the first processing rate and the second processing rate.

[0068] In another possible implementation, determining the first node based on the difference between the first processing rate and the second processing rate includes: determining the first difference, which is equal to the first processing rate minus the second processing rate; if the first difference is less than or equal to a first difference threshold, determining the type of the target node as a first type; if the first difference is greater than or equal to a second difference threshold, determining the type of the target node as a second type.

[0069] In another possible implementation, determining the first node based on the difference between the first processing rate and the second processing rate includes: determining a second difference, the second difference being equal to the second processing rate minus the first processing rate; if the second difference is less than or equal to the first difference threshold, determining the type of the target node as a second type; if the second difference is greater than or equal to the second difference threshold, determining the type of the target node as a first type.

[0070] In another possible implementation, the method further includes: outputting a target interface; the target interface includes a first control, which indicates whether the type of configuration node is supported. The first control includes a first state or a second state, where the first state indicates the type of configuration node supported, and the second state indicates the type of configuration node not supported. The first control is used to indicate whether the type of configuration node is supported or not.

[0071] In this implementation, by setting the output target node to include the first control, users can determine whether the inference cluster they are using supports the type of configuration node based on the target interface, thereby helping to improve the user experience.

[0072] In another possible implementation, the target interface further includes a second control, which indicates whether the functionality of configuring node types is enabled. This second control may have a third or fourth state; the third state indicates whether the functionality of configuring node types is enabled, and the fourth state indicates whether the functionality of configuring node types is disabled. The second control is used to indicate whether the functionality of configuring node types is enabled or disabled (i.e., not enabled).

[0073] In this implementation, by setting the target interface to include a second control, users can choose whether to enable or disable the function of configuring node types. In this way, users can choose to enable or disable the function of configuring node types according to their own needs, which helps to improve the user experience.

[0074] In a second aspect, a node management device is provided, comprising: a functional unit for performing any of the methods provided in the first aspect, wherein the actions performed by each functional unit are implemented by hardware or by hardware executing corresponding software. For example, the node management device may include a determination module and a configuration module; the determination module is used to determine a target node from nodes of a first type and nodes of a second type; the type of the target node is the type with low load in the first type and the second type; and the configuration module is used to configure the type of the target node to be the type with high load in the first type and the second type.

[0075] Thirdly, a processor is provided that can be used to execute any of the methods provided in the first aspect above.

[0076] Fourthly, a chip is provided, comprising: a processor and a power supply circuit; the power supply circuit can be used to supply power to the chip; the processor can be used to execute any of the methods provided in the first aspect above.

[0077] Fifthly, a management node is provided, comprising: a processor, a memory, and a computer program / instructions stored in the memory; the processor executes the computer program / instructions to cause the management node to perform any of the methods provided in the first aspect above.

[0078] A sixth aspect provides a computing device cluster, comprising: at least one management node, at least one first-type node, and at least one second-type node, wherein the at least one management node includes a processor, a memory, and a computer program / instructions stored in the memory; the processor in the at least one management node executes the computer program / instructions stored in the memory to enable the at least one management node to implement any of the methods provided in the first aspect above, so as to perform computing task switching on the at least one first-type node and / or the at least one second-type node.

[0079] In a seventh aspect, a computer program product is provided, comprising a computer program / instructions that, when executed by a computing device, implement any of the methods provided in the first aspect above.

[0080] Eighthly, a computer-readable storage medium is provided, on which a computer program / instructions are stored, which, when executed by a computing device, implement any of the methods provided in the first aspect above.

[0081] The technical effects of any of the implementation methods in aspects two through eight can be seen in the technical effects of different implementation methods in aspect one above, and will not be repeated here. Attached Figure Description

[0082] Figure 1 One of the schematic diagrams of a system architecture provided in this application;

[0083] Figure 2 A second schematic diagram of a system architecture provided for this application;

[0084] Figure 3 The third schematic diagram of a system architecture provided for this application;

[0085] Figure 4 A flowchart of a node management method provided in this application;

[0086] Figure 5 A schematic diagram of a reasoning process provided for this application;

[0087] Figure 6 A schematic diagram of a management interface for an inference cluster provided in this application;

[0088] Figure 7 A flowchart of another node management method provided in this application;

[0089] Figure 8 One of the schematic diagrams illustrating the node management method provided in this application;

[0090] Figure 9 The second schematic diagram of the node management method provided in this application;

[0091] Figure 10 A flowchart of yet another node management method provided in this application;

[0092] Figure 11 Schematic diagram three of the node management methods provided in this application;

[0093] Figure 12 Schematic diagram four of the node management method provided in this application;

[0094] Figure 13 A schematic diagram of a node management device provided in this application;

[0095] Figure 14 A schematic diagram of a computing device provided in this application;

[0096] Figure 15 A schematic diagram of a computing device cluster provided in this application;

[0097] Figure 16 This is a schematic diagram of the connection of a computing device cluster provided in this application. Detailed Implementation

[0098] The technical solution provided in this application will be described in detail below with reference to the accompanying drawings.

[0099] With the development of inference technology, inference services are widely used in daily life. In related technologies, to avoid interference between pre-filling and decoding tasks during inference request processing, the approach is to distribute these tasks across different nodes in the inference cluster. However, this inference method results in low throughput for the inference cluster, severely impacting its performance.

[0100] In view of this, this application provides a node management method applied to an inference cluster, which includes a first type of node and a second type of node. The first type of node is used to perform pre-filling tasks, and the second type of node is used to perform decoding tasks. During the execution of inference requests by the inference cluster, by determining a first node from the first type of node and the second type of node, and converting the type of the first node to the type with the higher load between the first and second types, it is possible not only to increase the number of high-load nodes without increasing the number of nodes in the inference cluster, thereby improving the processing speed of inference tasks (pre-filling tasks or decoding tasks) by high-load nodes, but also to fully utilize the low-load nodes of the inference cluster, thereby improving the utilization rate of low-load nodes.

[0101] It should be noted that the various implementation methods of the above node management approach will be discussed later. Figures 4 to 12 The examples shown will be described in detail here, but will not be elaborated upon further.

[0102] Next, the system architecture involved in the technical solution provided in this application will be further described with reference to the accompanying drawings.

[0103] This application provides an inference cluster that can apply the above-described node management method. The inference cluster includes a first type of node and a second type of node. The first type of node is used to perform pre-filling tasks, and the second type of node is used to perform decoding tasks. The first type of node and the second type of node can communicate with each other.

[0104] It should be noted that, in this application, the term "first type node" refers to a type of node used to perform pre-filling tasks; that is, any node used to perform pre-filling tasks can be called a first type node. Similarly, the term "second type node" refers to a type of node used to perform decoding tasks; that is, any node used to perform decoding tasks can be called a second type node.

[0105] For example, the first type of node can be used to implement chunked prefill, paged attention, flash attention, sequence parallel (SP), tensor parallel (TP), and continuous batch processing. The second type of node can be used to implement continuous batch processing, tensor parallel (TP), paged attention, flash attention, and sequence parallel (SP).

[0106] In this application, the first type of node includes at least one node in the inference cluster used to perform pre-filling tasks. Each of the at least one node used to perform the pre-filling task is capable of communicating with the second type of node. For example, such as... Figure 1 As shown, the first type of node includes node 10, ..., node 1M, where M is a positive integer greater than 1.

[0107] It should be noted that in this application, "at least one" includes one or more, and "multiple" can be two or more, which will not be elaborated further.

[0108] For example, at least one node for performing the pre-filling task includes a fourth node. Hereinafter, the fourth node is used as an example to illustrate at least one node for performing the pre-filling task.

[0109] Optionally, an inference model is deployed on the fourth node, which performs a pre-filling task through the inference model. The inference model includes multiple neural network layers, which sequentially execute the pre-filling task of the inference request to obtain the first word of the inference request.

[0110] For example, the inference model can be a large language model (LLM).

[0111] It should be noted that this application does not limit the type of inference model; the above is merely an illustrative example.

[0112] Optionally, the fourth node includes a computing device. This computing device can be a terminal device or a network device.

[0113] Network devices can include servers, bare metal servers, etc. A server can be a single physical server, or two or more physical servers that share different responsibilities and work together to achieve the various functions of the server. For example, a server can be a blade server, a high-density server, a rack server, or a tower server.

[0114] It should be noted that this application does not limit the form factor of the network device; the above is merely an illustrative example.

[0115] The terminal devices can be ultra-mobile personal computers (UMPCs), laptops, netbooks, desktop computers, all-in-one computers, etc.

[0116] It should be noted that this application does not limit the form of the computing device; the above is merely an illustrative example.

[0117] Optionally, the fourth node includes at least one processor and memory for each of the at least one processor. Each processor is used to execute a pre-filling task, and the memory of each processor is used to store the data obtained from executing the pre-filling task. For example, such as... Figure 2 As shown, node 10 includes processor 101 and memory 103 of processor 101, processor 102 and memory 104 of processor 102, etc. Processor 101 is used to perform pre-filling tasks, and memory 103 is used to store data obtained by processor 101 in performing pre-filling tasks.

[0118] Optionally, the processor of the fourth node can be a computing unit with computing capabilities, such as a central processing unit (CPU), graphics processing unit (GPU), data processing unit (DPU), neural processing unit (NPU), or tensor processing unit (TPU).

[0119] It should be noted that this application does not limit the specific form of the processor in the fourth node; the above is merely an illustrative example.

[0120] It should be noted that this application does not limit the specific form of the fourth node; the above is merely an illustrative example.

[0121] Optionally, where the first type of node includes multiple nodes for performing pre-filling tasks, and at least some of the nodes for performing pre-filling tasks include at least one processor, the at least some nodes for performing pre-filling tasks can be deployed on the same computing device. Alternatively, the at least some nodes for performing pre-filling tasks can be deployed on multiple computing devices. That is, a portion of the nodes and another portion of the nodes for performing pre-filling tasks can be deployed on different computing devices.

[0122] It should be noted that in this application, at least some nodes may include all or a portion of the nodes. For example, at least some nodes used to perform the pre-filling task may include all nodes of a plurality of nodes used to perform the pre-filling task, or may include a portion of the nodes of a plurality of nodes used to perform the pre-filling task.

[0123] It should be noted that this application does not restrict the deployment method of at least some nodes used to perform the pre-filling task; the above is merely an illustrative example.

[0124] It should be noted that the deployment method of multiple processors in the fourth node can be referenced from the deployment method of at least some nodes used to perform pre-filling tasks, and will not be repeated here.

[0125] It should be noted that the multiple nodes used to perform the pre-filling task can have the same or different forms, and this application does not impose any restrictions on this. For example, the multiple nodes used to perform the pre-filling task can all be computing devices, or the multiple nodes used to perform the pre-filling task can all be processors, or some of the multiple nodes used to perform the pre-filling task can be computing devices and the other part can be processors.

[0126] In this application, the second type of node includes at least one node in the inference cluster used to perform decoding tasks. Each node in the execution node used to perform the decoding task is capable of communicating with the first type of node. For example, such as... Figure 1 As shown, the second type of node includes node 20, ..., node 2N, where N is a positive integer greater than 1.

[0127] For example, the second type of node includes a fifth node. Hereinafter, the fifth node will be used as an example to illustrate at least one node used to perform the decoding task.

[0128] In this application, the form of the fifth node may be the same as that of the fourth node, or it may be different from the form of the fourth node; this application does not impose any restrictions on this.

[0129] It should be noted that other related explanations for the fifth node can be found in the explanations for the fourth node above, and will not be repeated here.

[0130] Optionally, the inference cluster may also include scheduling nodes for executing scheduling tasks. These scheduling tasks include pre-filling tasks for scheduling inference requests, decoding tasks for scheduling inference requests, etc. The pre-filling tasks for scheduling inference requests include pre-filling task allocation nodes, and the decoding tasks for scheduling inference requests include allocation nodes for decoding tasks.

[0131] Optionally, the scheduling node includes at least one processor and at least one processor's memory. The processor of the scheduling node is used to execute scheduling tasks, and the memory of the scheduling node is used to store node type information, etc. The node type information is used to indicate the identity of each node among the multiple nodes in the inference cluster. The multiple nodes include first-type nodes and second-type nodes, and the identity of each node is used to indicate the type of each node, which includes either the first type or the second type.

[0132] It should be noted that the relevant descriptions of the processors of the scheduling nodes can be found in the descriptions of the processors of the fourth node, and will not be repeated here.

[0133] For example, an inference cluster may not include a scheduling node. In the case where the inference cluster does not include a scheduling node, the scheduling task may be executed by a first type of node, or by a second type of node.

[0134] Optionally, the inference cluster may also include a management node, which is used to execute the node management method provided in this application.

[0135] For example, the management node can be a node selected from the scheduling node, the first type of node, and the second type of node, or it can be a node other than the scheduling node, the first type of node, and the second type of node.

[0136] It should be noted that, in the following description, the management node is an example other than the scheduling node, the first type of node, and the second type of node, to illustrate this application.

[0137] It should be noted that the description of the form of the management node can be found in the description of the fourth node above, and will not be repeated here.

[0138] Optionally, the inference cluster can communicate with electronic devices. For example, such as... Figure 1 As shown, the scheduling node of the inference cluster communicates with electronic devices via a network. Electronic devices can send inference requests to the inference cluster and receive inference results returned by the inference cluster.

[0139] In one example, a user could send an inference request to the scheduling node of the inference cluster via an electronic device. In another example, the electronic device could automatically send an inference request to the scheduling node of the inference cluster while running a service.

[0140] Alternatively, the electronic device may be a mobile phone, tablet computer, handheld computer, personal computer (PC), personal digital assistant (PDA), ultra-mobile personal computer (UMPC), laptop computer, netbook, desktop computer, or all-in-one computer, etc.

[0141] Alternatively, the electronic device may be a network device.

[0142] It should be noted that this application does not limit the form of the electronic device; the above is merely an illustrative example.

[0143] This application also provides target software that can be used to implement the node management method provided in this application. For example, the target software is installed on an inference cluster, and the inference cluster executes the node management method provided in this application through the target software.

[0144] It should be noted that this application does not limit the name of the target software. For example, the target software can be called inference software, split inference software, etc.

[0145] For example, such as Figure 3 As shown, the inference cluster includes a scheduling module and target software. The target software includes a prediction module, a decision-making module, and a transformation module. The scheduling module is deployed on scheduling nodes, which can execute scheduling tasks through the scheduling module.

[0146] For example, the prediction module is used to obtain the load of the first type of node and the load of the second type of node, etc. The decision module is used to determine the target node of the type to be converted, etc. The conversion module is used to convert the type of the target node, for example, converting the target node from the first type of node to the second type of node, or converting the target node from the second type of node to the first type of node, etc.

[0147] It should be noted that this application does not restrict the name of the scheduling module; for example, the scheduling module can also be called a scheduler, service request scheduler, etc. Furthermore, this application does not restrict the names of the prediction module, decision module, and conversion module. For example, the prediction module can also be called a load predictor, load prediction module, etc.; the decision module can also be called an identity decision maker, identity decision module, etc.; and the conversion module can also be called a PD identity converter, PD identity conversion module, etc. Here, P represents prefill, and D represents decode.

[0148] It should be noted that the above division of the target software is only an illustrative example. For example, the target software may be divided into more or fewer modules, or the target software may be implemented as a whole to implement the node management method provided in this application.

[0149] For example, the target software is installed on the management node, and the management node can execute the node management method provided in this application through the target software.

[0150] Optionally, the inference cluster of this application can be a cluster provided by a cloud service platform. The cloud computing provided by the cloud service platform may include the node management method provided in this application. For example, the cloud service platform may be a public cloud, a private cloud, a hybrid cloud, etc.

[0151] For example, a user can communicate with an inference cluster on a cloud service platform via an electronic device to instruct the inference cluster on the cloud service platform to execute an inference request. During the execution of the inference request by the inference cluster on the cloud service platform, the node management method provided in this application can be used to convert the type of a first type of node or the type of a second type of node.

[0152] It needs to be explained that, Figures 1 to 3 The system architecture shown does not constitute a limitation on the system architecture for implementing the node management method provided in this application.

[0153] It should be noted that the system architecture and application scenarios described in this application are for the purpose of more clearly illustrating the technical solutions of this application, and do not constitute a limitation on the technical solutions provided in this application. As those skilled in the art will know, with the evolution of system architecture and the emergence of new application scenarios, the technical solutions provided in this application are also applicable to similar technical problems.

[0154] For ease of understanding, the node management method provided in this application will be described below with reference to the above system architecture and accompanying drawings.

[0155] Figure 4A flowchart of a node management method provided in this application. Exemplarily, the node management method includes steps 401-402.

[0156] It should be noted that the "step" in this application can be abbreviated as "S", and will not be elaborated further hereafter.

[0157] The following, combined with Figures 1 to 3 The system architecture shown is for Figure 4 The node management method shown is illustrated by example.

[0158] In this application, the inference cluster includes a first type of node and a second type of node. The first type of node performs pre-filling tasks, and the second type of node performs decoding tasks. The pre-filling task involves inferring the first token based on the prompt word indicated in the inference request. The decoding task involves inferring the second token based on the first token, the third token based on the second token, and so on, until the k-th token is used to obtain the inference result of the inference request. In other words, the pre-filling task consists of one round of inference, and the decoding task consists of multiple rounds of inference.

[0159] The following, combined with Figure 1 This section provides an example of how the inference cluster executes inference requests.

[0160] For example, after receiving inference request 1 from user 1, the scheduling node of the inference cluster determines node 10 from the first type of nodes to perform the pre-filling task of inference request 1, and determines node 20 from the second type of nodes to perform the decoding task of inference request 1. Then, node 10 performs inference based on the prompt word 1 indicated by inference request 1, obtaining lexical units. Node 10 sends the lexical units to node 20, and node 20 performs the decoding task based on the lexical units, obtaining the inference result 1 of inference request 1. Finally, the scheduling node returns the inference result 1 to user 1.

[0161] For example, such as Figure 5 As shown, a user sends Q inference requests, where Q is a positive integer greater than or equal to 1. The inference cluster receives the Q inference requests from the user. Then, the inference cluster schedules the Q inference requests to achieve distributed processing of the Q inference requests and obtains the inference results for the Q inference requests. During the execution of the Q inference requests, the inference cluster can execute the node management method provided in this application.

[0162] The following sections, S401-S402, provide an exemplary description of the process by which the inference cluster executes the node management method provided in this application.

[0163] S401: The management node determines the target node from the first type of node and the second type of node.

[0164] The target node is of type one or two, with a lower load.

[0165] It should be noted that the target node in this application is used to characterize the node of the type to be converted. The target node may include at least one node.

[0166] For ease of description, the node whose type is to be converted will be referred to as the node to be converted, and will not be described again in detail below.

[0167] For example, such as Figure 5 As shown, during the execution of Q inference requests in the inference cluster, the management node can obtain the load of the first type of node and the load of the second type of node through the prediction module of the target software. The load of the first type of node indicates the workload of the first type of node in handling pre-filling tasks, and the load of the second type of node indicates the workload of the second type of node in handling decoding tasks. Afterwards, the management node can determine the target node as the type to be converted through the decision module of the target software.

[0168] Example 1a: If the load of the first type of node is less than the load of the second type of node (i.e., the first type has a lower load and the second type has a higher load), then the target node is of type first. The target node is the node used to perform the pre-filling task.

[0169] For example, such as Figure 4 As shown, the first type is the type with low load. The management node determines the node to be converted to type 1i (i.e. the node of the first type) from nodes 10, ..., nodes 1M, nodes 20, ..., nodes 2N. Here, i is a positive integer greater than 1 and less than M.

[0170] Example 1b: If the load of the second type of node is less than the load of the second type of node (i.e., the first type has a high load and the second type has a low load), then the target node is of type second. The target node is the node used to perform the decoding task.

[0171] For example, such as Figure 4 As shown, the first type is the type with low load. The management node determines the node to be converted to type 2j (i.e. the node of the second type) from nodes 10, ..., nodes 1M, nodes 20, ..., nodes 2N. Here, j is a positive integer greater than 1 and less than N.

[0172] In this application, the target node type is the type with low load in the first type and the second type, which includes multiple cases. Hereinafter, cases 1 and 2 are used as examples.

[0173] Case 1: The target node type is the type with the lower load among the first and second types within the target time period. The target time period is the time period following the current moment.

[0174] Optionally, the target time period can be any value between 1 minute and 10 minutes.

[0175] In one example, the target time period is 1 minute. For example, if the current time is 10:00, then the target time period is 10:00-10:01.

[0176] In this example, by setting the target time period to 1 minute, the target time period is closer to the current time. This results in higher accuracy when predicting the load for the target time period based on the current load, which helps to improve the accuracy of identifying the type of target node.

[0177] In another example, the target time period is 10 minutes. For example, if the current time is 10:00, then the target time period is 10:00-10:10.

[0178] In this example, by setting the target time period to 10 minutes, it helps to reduce the frequency of adjusting node types, thereby helping to reduce the computing resources used for adjusting node types.

[0179] In yet another example, the target time period is 5 minutes. For instance, if the current time is 10:00, then the target time period is 10:00-10:05.

[0180] In this example, setting the target time period to 5 minutes not only helps to ensure the accuracy of the determined target node type, but also helps to control the computing resources used to adjust the node type.

[0181] It should be noted that this application does not impose any restrictions on the specific duration of the target time period; the above is merely an illustrative example.

[0182] In this scenario, determining the type of the target node based on the load size over a future time period allows for advance adjustment of the node type, thereby improving the timeliness of node type adjustments.

[0183] Case 2: The target node is the type with the lower load among the current first and second types.

[0184] In one example, "current" includes the current moment. For instance, the inference cluster obtains the load of a first-type node and the load of a second-type node at the current moment, where "current moment" includes the time when the loads of the first-type and second-type nodes are obtained. Then, based on the loads of the first-type and second-type nodes at the current moment, the type of the target node is determined. The target node is the type with the lower load between the first and second types at the current moment.

[0185] In another example, "current" includes the current time period. The current time period refers to the time period preceding the current moment. Because it references the load over a period of time, this helps improve the reliability of adjusting node types.

[0186] It should be noted that for information regarding the current time period, please refer to the description of the target time period mentioned above; it will not be repeated here.

[0187] In this situation, determining the type of the target node based on the current load helps improve the accuracy of adjusting the node type.

[0188] It should be noted that this application does not impose any restrictions on the time period for the target node to be of the type with less load in the first or second type; the above is merely an illustrative example.

[0189] For example, the management node can first determine the type of the node to be converted (i.e., the type of the target node) from a first type and a second type, where the type of the node to be converted includes the type with the highest load from the first type and the second type. Then, the management node determines the target node from the type with the highest load, that is, it determines the target node from the nodes of the first type or from the nodes of the second type.

[0190] The following examples illustrate various methods for determining the type of target node using Method 1 and Method 2.

[0191] The following describes Method 1 by way of example through S1-S2.

[0192] S1: The management node obtains the first memory utilization rate of the first type of node and the second memory utilization rate of the second type of node.

[0193] The first memory utilization rate of the first type of node indicates the ratio of the amount of data stored in the memory of the first type of node to the total storage space of the memory of the first type of node. For example, if the total storage space of the memory of the first type of node is 20GB and the amount of data stored in the memory of the first type of node is 5GB, then the memory utilization rate of the first type of node is 25%. The memory utilization rate of the second type of node indicates the ratio of the amount of data stored in the memory of the second type of node to the total storage space of the memory of the second type of node.

[0194] For example, a first memory utilization rate is used to characterize the load of a first type of node, and the first memory utilization rate of the first type of node is positively correlated with the load of the first type of node. A second memory utilization rate is used to characterize the load of a second type of node, and the second memory utilization rate of the second type of node is positively correlated with the load of the second type of node. When the first memory utilization rate is greater than the second memory utilization rate, the load of the first type of node is greater than the load of the second type of node. When the first memory utilization rate is less than the second memory utilization rate, the load of the first type of node is less than the load of the second memory utilization rate. When the first memory utilization rate is equal to the second memory utilization rate, the load of the first type of node is equal to the load of the second type of node.

[0195] The following example, using the first type of node comprising M nodes, illustrates how to obtain the first memory utilization rate.

[0196] Example A, combined with situation 2 above, the management node obtains the current memory utilization rate of the first type of node and gets the first memory utilization rate of the first type of node.

[0197] Example A1: The first memory utilization rate is the average memory utilization rate of M nodes. The management node obtains the current memory utilization rate of each of the M nodes and calculates the average of the current memory utilization rates of each node to obtain the current average memory utilization rate of the M nodes.

[0198] For example, the M nodes include node 1, node 2, and node 3. The current memory utilization rate of node 1 is A1, the current memory utilization rate of node 2 is A2, and the current memory utilization rate of node 3 is A3. Then the first memory utilization rate is (A1+A2+A3) / 3.

[0199] In this example, setting the first memory utilization rate as the average memory utilization rate of the first type of node helps to improve the accuracy of the first memory utilization rate.

[0200] Example A2: The first memory utilization rate is the highest memory utilization rate among the memory utilization rates of M nodes. The management node obtains the current memory utilization rate of each of the M nodes and determines the highest memory utilization rate among the current memory utilization rates of each node as the first memory utilization rate.

[0201] Exemplarily, if A1 < A2 < A3, then the first memory utilization rate is A3.

[0202] In this example, by setting the first memory utilization rate as the maximum memory utilization rate of the first type of nodes, this helps to improve the diversity of the first memory utilization rate.

[0203] Example A3, the first memory utilization rate is the minimum memory utilization rate among the memory utilization rates of M nodes. The management node obtains the current memory utilization rate of each of the M nodes and determines the minimum utilization rate among the current memory utilization rates of each node as the first memory utilization rate.

[0204] Exemplarily, if A1 < A2 < A3, then the first memory utilization rate is A1.

[0205] In this example, by setting the first memory utilization rate as the minimum memory utilization rate of the first type of nodes, this helps to improve the diversity of the first memory utilization rate.

[0206] It should be noted that the present application does not limit the type of the first memory utilization rate, and the above is only an exemplary illustration. Hereinafter, taking the first memory utilization rate as the average utilization rate of M nodes as an example, the present application will be exemplarily illustrated.

[0207] Example B, in combination with the above situation 1, the management node obtains the current memory utilization rate of the first type of nodes and obtains the first memory utilization rate according to the current memory utilization rate of the first type of nodes and the first neural network model.

[0208] Exemplarily, the first neural network module can be a long short term memory (LSTM), etc.

[0209] It should be noted that the present application does not limit the type of the first network model, and the above is only an exemplary illustration.

[0210] In one example, the management node obtains the current memory utilization rate of each of the M nodes, calculates the average value of the current utilization rate of each node, and obtains the current average memory utilization rate of the M nodes. Then, the management node inputs the current average utilization rate of the M nodes into the first neural network model and obtains the first memory utilization rate output by the first neural network model. This first memory utilization rate is the memory utilization rate of the first type of nodes in the target time period.

[0211] In this example, by inputting the current average memory utilization rate of the first type of nodes into the first neural network model to obtain the first memory utilization rate, it helps to reduce the computational amount of the first neural network model, and thus helps to reduce the occupied computing resources.

[0212] In another example, the management node obtains the current memory utilization rate of each of the M nodes and inputs this rate into the first neural network model to obtain the memory utilization rate of each node for the target time period, as output by the first neural network model. Then, the management node calculates the average memory utilization rate of each node for the target time period, obtaining the average utilization rate of the first type of nodes for the target time period, i.e., the first memory utilization rate.

[0213] For further explanation of Example B, please refer to the explanation of Example A; it will not be repeated here.

[0214] In this example, the current memory utilization of each node is input into the first neural network model to obtain the memory utilization of each node in the target time period, thereby obtaining the first memory utilization. This helps to provide diversity in the way the first memory utilization is determined.

[0215] The following example, using N nodes of the second type of node, illustrates how to obtain the first memory utilization rate.

[0216] Example C, combined with situation 2 above, the management node obtains the current memory utilization rate of the second type of node, and thus obtains the second memory utilization rate of the second type of node.

[0217] It should be noted that the relevant explanations for Example C can be found in the explanations for Example A, and will not be repeated here.

[0218] Example D, combined with situation 1 above, the management node obtains the current memory utilization of the second type of node, and obtains the second memory utilization based on the current utilization of the second type of node and the first neural network model.

[0219] It should be noted that the relevant explanations for Example D can be found in the explanations for Examples A and C above, and will not be repeated here.

[0220] Example E, combined with Case 1 above, the management node obtains the number and length of pre-filled tasks processed by the first type of nodes during the target time period, and obtains the second memory utilization rate based on the number and length of pre-filled tasks processed by the first type of nodes during the target time period and the second neural network model. The length of the pre-filled task includes the length of the data obtained from executing the pre-filled task, and the data obtained from executing the pre-filled task includes processed key-value data, etc.

[0221] It should be noted that the explanation of the second neural network model can be found in the explanation of the first neural network model, and will not be repeated here.

[0222] For example, the management node can obtain the number and length of pre-filled tasks currently being processed by the first type of node, and input the number and length of pre-filled tasks currently being processed by the first type of node into the third neural network model, thereby obtaining the number and length of pre-filled tasks processed by the first type of node in the target time period output by the third neural network model.

[0223] It should be noted that the explanation of the third neural network model can be found in the explanation of the first neural network model, and will not be repeated here.

[0224] For example, N nodes include node 4, node 5, and node 6. Node 1 processes pre-filling tasks and assigns the resulting words to node 4. Node 2 processes pre-filling tasks and assigns the resulting words to node 5. Node 3 processes pre-filling tasks and assigns the resulting words to node 6. After obtaining the number and length of pre-filling tasks processed by node 1 in the target data segment, the management node inputs these numbers into the third neural network model to obtain the memory utilization rate of node 4 in the target time segment, output by the third neural network model. Based on the same principle, the management node can obtain the memory utilization rate of node 5 and node 6 in the target time segment.

[0225] Then, the management node determines the memory utilization rate of the second type of node in the target time period, i.e., the second memory utilization rate, based on the memory utilization rates of node 4, node 5, and node 6 in the target time period.

[0226] S2: The management node determines the type of the target node based on the first memory utilization rate and the second memory utilization rate. The target node type is the one with lower memory utilization.

[0227] Hereinafter, various methods of S2 will be illustrated by way of method 1A and method 1B.

[0228] In Method 1A, the management node determines the type of the target node based on the ratio of the first memory utilization rate to the second memory utilization rate.

[0229] The first implementation of method 1A will be illustrated below through S2a-S2c.

[0230] S2a: The management node determines the first ratio, which is equal to the ratio of the first memory utilization to the second memory utilization.

[0231] For example, if the first memory utilization is X and the second memory utilization is Y, then the first ratio is equal to X / Y.

[0232] S2b: If the management node determines the target node's type as type one when the first ratio is less than or equal to a first ratio threshold, the first ratio threshold is less than 1.

[0233] For example, if the value of X / Y is less than or equal to a first ratio threshold, the type of the node to be converted is determined to be the first type, that is, the type of the target node is the first type.

[0234] Optionally, the first ratio threshold is any value between 0.3 and 0.7.

[0235] In one example, the first ratio threshold is 0.3. For instance, if X is 20% and Y is 80%, then the value of X / Y is 0.25. Based on the fact that 0.25 is less than the first ratio threshold, the type of the node to be converted is determined to be the first type.

[0236] In this example, by setting the first ratio threshold to 0.3, the node type can be switched only when the load of the first type of node differs significantly from that of the second type of node, thus helping to avoid affecting the pre-filled tasks processed by the first type of node.

[0237] In another example, the first ratio threshold is 0.7. For instance, if X is 56% and Y is 80%, then the value of X / Y is 0.7. Based on the fact that 0.7 equals the first ratio threshold, the type of the node to be converted is determined to be the first type.

[0238] In this example, by setting the first ratio threshold to 0.7, the node type can be switched even when the load of the first type of node and the load of the second type of node are not significantly different, thereby helping to increase the probability of switching node types.

[0239] In another example, the first ratio threshold is 0.5. For instance, if X is 40% and Y is 80%, then the value of X / Y is 0.5. Based on the fact that 0.5 equals the first ratio threshold, the type of the node to be converted is determined to be the first type.

[0240] In this example, by setting the first ratio threshold to 0.5, we can not only control the load difference between the first type of node and the second type of node, but also take into account the probability of switching node types.

[0241] It should be noted that this application does not limit the specific value of the first ratio threshold; the above is merely an illustrative example.

[0242] S2c: If the management node determines the target node's type as the second type when the first ratio is greater than or equal to the second ratio threshold, the second ratio threshold is greater than 1.

[0243] For example, if the value of X / Y is greater than or equal to the second ratio threshold, the type of the node to be converted is determined to be the second type, that is, the type of the target node is the second type.

[0244] Optionally, the second ratio threshold is any value between 1.5 and 2.5.

[0245] In one example, the second ratio threshold is 1.5. For instance, if X is 80% and Y is 50%, then the value of X / Y is 1.6. Based on the fact that 1.6 is greater than the second ratio threshold, the type of the node to be converted is determined to be the second type.

[0246] In this example, by setting the second ratio threshold to 1.5, the node type can be switched even when the load of the first type of node and the load of the second type of node are not significantly different, thereby helping to increase the probability of switching the node type.

[0247] In another example, the second ratio threshold is 2.5. For instance, if X is 80% and Y is 25%, then the value of X / Y is 3.2. Based on the fact that 3.2 is greater than the second ratio threshold, the type of the node to be converted is determined to be the second type.

[0248] In this example, by setting the first ratio threshold to 2.5, the node type can be switched only when the load of the first type of node differs significantly from that of the second type of node, thus helping to avoid affecting the pre-filled tasks processed by the first type of node.

[0249] In another example, the second ratio threshold is 2. For instance, if X is 80% and Y is 40%, then the value of X / Y is 2. Based on the fact that 2 equals the second ratio threshold, the type of the node to be converted is determined to be the second type.

[0250] In this example, by setting the second ratio threshold to 2, we can not only control the load difference between the first type of node and the second type of node, but also take into account the probability of switching node types.

[0251] It should be noted that this application does not impose any restrictions on the specific value of the second ratio threshold; the above is merely an illustrative example.

[0252] The second implementation of method 1A will be illustrated below using S2d-S2f.

[0253] S2d: The management node determines the second ratio, which is equal to the second memory utilization rate divided by the first memory utilization rate;

[0254] S2e: If the management node determines the second type as the type of the target node when the second ratio is less than or equal to the first ratio threshold;

[0255] S2f: If the management node determines the first type as the type of the target node when the first ratio is greater than or equal to the second ratio threshold.

[0256] It should be noted that the relevant explanations for S2d-S2f can be found in the explanations for S2a-S2c above, and will not be repeated here.

[0257] In this method, the type of node to be converted is determined by the ratio of the first memory utilization rate to the second memory utilization rate. This helps to determine the proportion of the first memory utilization rate to the second memory utilization rate, which in turn helps to accurately measure the gap between the first memory utilization rate and the second memory utilization rate, thereby improving the accuracy of determining the type of node to be converted.

[0258] Method 1B determines the type of the target node based on the difference between the first memory utilization rate and the second memory utilization rate.

[0259] The first implementation of method 1b will be illustrated below using S2g-S2i.

[0260] S2g: The management node determines the first difference, which is equal to the first memory utilization rate minus the second memory utilization rate.

[0261] For example, if the first memory utilization is X and the second memory utilization is Y, then the first difference is equal to XY.

[0262] S2h: If the management node determines the target node's type as type one if the first difference is less than or equal to the first difference threshold, where the first difference threshold is less than 0.

[0263] For example, if the value of XY is less than or equal to a first difference threshold, the type of the node to be converted is determined to be the first type.

[0264] Optionally, the first difference threshold is any value between -0.6 and -0.2.

[0265] In one example, the first difference threshold is -0.6. For instance, if X is 20% and Y is 80%, then the value of XY is -0.6. Based on the fact that -0.6 equals the first difference threshold, the type of the node to be converted is determined to be the first type.

[0266] In this example, by setting the first difference threshold to -0.6, the node type can be switched only when the load of the first type of node differs significantly from that of the second type of node, thus helping to avoid affecting the pre-filled tasks processed by the first type of node.

[0267] In another example, the first difference threshold is -0.2. For instance, if X is 56% and Y is 80%, then the value of XY is -0.24. Based on the fact that -0.24 is less than the first ratio threshold, the type of the node to be converted is determined to be the first type.

[0268] In this example, by setting the first difference threshold to -0.2, the node type can be switched even when the load of the first type of node and the load of the second type of node are not significantly different, thereby helping to increase the probability of switching the node type.

[0269] In another example, the first difference threshold is -0.4. For instance, if X is 40% and Y is 80%, then the value of XY is -0.4. Based on the fact that -0.4 equals the first difference threshold, the type of the node to be converted is determined to be the first type.

[0270] In this example, by setting the first difference threshold to -0.4, we can not only control the load difference between the first type of node and the second type of node, but also take into account the probability of switching node types.

[0271] It should be noted that this application does not limit the specific value of the first difference threshold; the above is merely an illustrative example.

[0272] S2i: If the management node determines the target node's type as the second type when the first difference is greater than or equal to the second difference threshold, the second difference threshold is greater than 0.

[0273] For example, if the value of XY is greater than or equal to the second difference threshold, the type of the node to be converted is determined to be the second type.

[0274] Optionally, the second difference threshold is any value between 0.2 and 0.6.

[0275] In one example, the second difference threshold is 0.2. For instance, if X is 80% and Y is 50%, then the value of XY is 0.3. Based on the fact that 0.3 is greater than the second ratio threshold, the type of the node to be converted is determined to be the second type.

[0276] In this example, by setting the second difference threshold to 0.2, the node type can be switched even when the load of the first type of node and the load of the second type of node are not significantly different, thereby helping to increase the probability of switching the node type.

[0277] In another example, the second difference threshold is 0.6. For instance, if X is 80% and Y is 20%, then the value of XY is 0.6. Based on the fact that 0.6 equals the second ratio threshold, the type of the node to be converted is determined to be the second type.

[0278] In this example, by setting the second difference threshold to 0.6, the node type can be switched only when the load of the first type of node differs significantly from that of the second type of node, thus helping to avoid affecting the pre-filled tasks processed by the first type of node.

[0279] In another example, the second difference threshold is 0.4. For instance, if X is 80% and Y is 40%, then the value of XY is 0.4. Based on the fact that 0.4 equals the second ratio threshold, the type of the node to be converted is determined to be the second type.

[0280] In this example, by setting a second difference threshold of 0.4, we can not only control the load difference between the first type of node and the second type of node, but also take into account the probability of switching node types.

[0281] It should be noted that this application does not impose any restrictions on the specific value of the second difference threshold; the above is merely an illustrative example.

[0282] The second implementation of method 1b will be illustrated below through S2j-S2l.

[0283] S2j: The management node determines the second difference, which is equal to the second memory utilization rate minus the first memory utilization rate;

[0284] S2k: If the management node determines the target node to be of type two when the second difference is less than or equal to the first difference threshold;

[0285] S2l: If the management node determines the target node's type as the first type when the second difference is greater than or equal to the second difference threshold.

[0286] It should be noted that the relevant explanations for S2j-S2l can be found in the explanations for S2g-S2i above, and will not be repeated here.

[0287] In this approach, the type of node to be converted is determined by the difference between the first memory utilization rate and the second memory utilization rate, which helps to improve the diversity of ways to determine the type of node to be converted.

[0288] In the above method, the type of node to be converted is determined by memory utilization. This helps to improve the convenience and accuracy of obtaining the parameters for determining the node to be converted, thereby improving the convenience and accuracy of determining the type of node to be converted.

[0289] The following describes method 2 by way of example through S3-S4.

[0290] S3, the management node obtains the first processing efficiency of the first type of node for the pre-filling task and the second processing efficiency of the second type of node for the decoding task.

[0291] The first processing efficiency of the first type of node for pre-filling tasks indicates the number of pre-filling tasks processed by the first type of node per unit time. The second processing efficiency of the second type of node for decoding tasks indicates the number of decoding tasks processed by the second type of node per unit time.

[0292] For example, the unit of time can be milliseconds, seconds, etc.

[0293] It should be noted that this application does not limit the granularity of unit time; the above is merely an illustrative example.

[0294] For other related explanations of S3 that need to be explained, please refer to the explanation of S1 above, which will not be repeated here.

[0295] For example, a first processing efficiency is used to characterize the load of a first type of node, and the first processing efficiency of the first type of node is positively correlated with the load of the first type of node. A second processing efficiency is used to characterize the load of a second type of node, and the second processing efficiency of the second type of node is positively correlated with the load of the second type of node. When the first processing efficiency is greater than the second processing efficiency, the load of the first type of node is greater than the load of the second type of node. When the first processing efficiency is less than the second processing efficiency, the load of the first type of node is less than the load of the second processing efficiency. When the first processing efficiency is equal to the second processing efficiency, the load of the first type of node is equal to the load of the second type of node.

[0296] S4: The management node determines the type of the target node based on the first processing efficiency and the second processing efficiency.

[0297] It should be noted that the relevant explanations for S4 can be found in the relevant explanations for S2 above, and will not be repeated here.

[0298] In the above method, the type of node to be converted is determined by the processing efficiency of the task, which helps to improve the diversity of ways to determine the type of node to be converted.

[0299] It should be noted that this application does not limit the method of determining the type of target node; the above is merely an illustrative example.

[0300] Optionally, S401 includes: the management node determining the target node from the first type of nodes and the second type of nodes according to the target period.

[0301] Optionally, the target period can be any value between 1 minute and 3 minutes.

[0302] In one example, the target period is 1 minute. This helps improve the timeliness of adjusting node types.

[0303] In another example, the target period is 3 minutes. This helps reduce the frequency of node type adjustments, thereby reducing the computational resources used for node type adjustments.

[0304] In another example, the target period is 2 minutes. This not only helps to balance the timeliness of adjusting node types, but also helps to avoid adjusting node types too frequently.

[0305] It should be noted that this application does not limit the specific duration of the target period; the above is merely an illustrative example.

[0306] In this embodiment, by setting the inference cluster to determine the target node according to the target period, the type of node can be adjusted periodically, which helps to continuously improve the throughput in the inference process and thus helps to continuously guarantee the inference performance of the inference cluster.

[0307] Optionally, the duration of the target period can be shorter than the duration of the target time period. For example, the target period is 3 minutes and the target time period is 5 minutes.

[0308] In this embodiment, by setting the duration of the target period to be shorter than the duration of the target time period, sufficient time can be reserved for determining the target node and converting the type of the target node. This allows the node type conversion to be completed before the target time period is reached, thereby helping to improve the performance of the inference cluster.

[0309] Optionally, after S401, the node management method further includes: scheduling the node to stop assigning inference requests to the target node.

[0310] For example, after the management node determines that the node to be converted is the target node, it sends a first scheduling notification to the scheduling node. The first scheduling notification instructs the node to stop assigning tasks to the target node. In response to the received first scheduling notification, the scheduling node stops assigning tasks to the target node.

[0311] In this embodiment, after determining the node of the type to be converted, a task is assigned to the node of the type to be converted by setting a node call notification.

[0312] This helps prevent tasks from piling up on the target node and becoming unprocessable, thus helping to avoid impacting the task processing rate.

[0313] S402: The management node configures the target node's type to the type with higher load between the first and second types.

[0314] For example, such as Figure 5 As shown, the management node can convert the type of the target node through the conversion module of the target software.

[0315] Example 2a, combined with Example 1a above, the first type is the type with low load, the second type is the type with high load, and the target node type is the first type, then the target node type is configured to the second type.

[0316] For example, such as Figure 4 As shown, the second type is the type with high load, and the management node configures node 1i as the second type.

[0317] Example 2b, combined with Example 1b above, the first type is the type with high load, the second type is the type with low load, and the target node type is the second type, then the target node type is configured as the first type.

[0318] For example, such as Figure 4 As shown, the first type is the type with high load, and the management node configures node 2j as the first type.

[0319] Optionally, if the target node's identity identifier is stored on the target node, S402 includes: the management node configuring the target node's identity identifier as the target identity identifier. The target identity identifier is used to indicate the type with higher load between the first type and the second type. The target node's identity identifier is used to indicate the type of the target node.

[0320] For example, the identity identifier of the first type of node is a first identity identifier, which is used to indicate at least one of the first type or to indicate the execution of a pre-filling task. The identity identifier of the second type of node is a second identity identifier, which is used to indicate at least one of the second type or to indicate the execution of a pre-filling task.

[0321] Since the first type is the type of node used to perform the pre-filling task, the first identity identifier is used to indicate the first type, which is equivalent to indicating the execution of the pre-filling task.

[0322] For example, if the target node's identity is stored on the target node, the management node can configure the identity stored on the target node as the target identity, thereby converting the target node's type to the type with the higher load between the first and second types.

[0323] Example 3a, combining Examples 1a and 2a above, shows that the first type is the low-load type, the second type is the high-load type, and the target node's type is the first type. Therefore, the target node's identity identifier is used to indicate the first type. Based on this, the target identity identifier includes a second identity identifier. Configuring the target node's identity identifier as the second identity identifier achieves the goal of configuring the target node's type as the second type, that is, configuring the target node's type as the high-load type.

[0324] Example 3b, combining Examples 1b and 2b above, shows that the first type is the high-load type, the second type is the low-load type, and the target node's type is the second type. Therefore, the target node's identity identifier is used to indicate the second type. Based on this, the target identity identifier includes a first identity identifier. Configuring the target node's identity identifier as the first identity identifier achieves the goal of configuring the target node's type as the first type, that is, configuring the target node's type as the high-load type.

[0325] Optionally, if the target node stores its own identifier, after receiving a task assigned by the scheduling node, the target node determines whether the task type indicated by its identifier is the same as the task type assigned by the scheduling node. If the task type indicated by its identifier is the same as the task type assigned by the scheduling node, the target node executes the task assigned by the scheduling node. If the task type indicated by its identifier is different from the task type assigned by the scheduling node, the target node does not execute the task assigned by the scheduling node.

[0326] For example, if the task type indicated by the identity of this node is different from the task type assigned by the scheduling node, the target node can return a notification message to the scheduler. The notification message is used to indicate that the type of the assigned task is different from the type of task executed by the target node.

[0327] In the above embodiments, by modifying the identity identifier stored on the target node to the target identity identifier, the type of the target node is changed to a type with high load. In this way, the target node can accurately determine its type based on the identity identifier on its own node, thereby helping to ensure the accuracy of the tasks performed by the target node.

[0328] Optionally, S402 includes: the management node updating node scheduling information, wherein the updated node scheduling information is used to indicate that the target node's identity is the target identity. The node scheduling information is used to indicate the identity of the first type of node and the identity of the second type of node.

[0329] For example, the scheduling node stores node scheduling information, and assigns tasks to first-type nodes and second-type nodes based on this information. Based on this, the management node modifies the target node's identity identifier in the node scheduling information to the target's identity identifier, thereby updating the node scheduling information.

[0330] In the above embodiments, by modifying the target node's identity identifier in the node scheduling information to the target identity identifier, the scheduling node can allocate tasks to the target node according to the updated node scheduling information. This allows the scheduling node to directly allocate heavy tasks to the target node, which helps to improve the accuracy and efficiency of task allocation to the target node.

[0331] Optionally, after the management node executes S401, the node management method may also include the following S5.

[0332] S5: When the target node processes the first inference request, the first data obtained by the target node from processing the first inference request is transferred to the target memory of the inference cluster.

[0333] For example, after the management node determines that the node to be converted is the target node, while the target node is processing the first inference request, the management node sends a conversion instruction to the target node, whereby the conversion instruction indicates the type of the node to be converted. In response to the received conversion instruction, the target node, while processing the first inference request, transfers the first data obtained from processing the first inference request to the target memory.

[0334] In the above embodiments, the first data obtained from processing the first inference request is transmitted to the target memory, which allows the target node to stop processing the inference request being processed, and other nodes to continue processing the first inference request based on the first data. This not only improves the efficiency of changing the type of the target node, but also avoids affecting the first inference request being processed, thereby helping to improve the performance of the inference cluster.

[0335] Optionally, the target memory is the memory of the first node. The type of the first node is the type with less load between the first type and the second type. That is, the type of the first node is the same as the type of the target node.

[0336] For example, if the size of the remaining memory space of the first node is greater than or equal to the size of the first data, the target node can transfer the first data to the memory of the first node.

[0337] In one example, combining with Example 1a above, the first type is the type with low load, the second type is the type with high load, the target node type is the first type, then the type of the first node is the first type.

[0338] In another example, combining with Example 1b above, if the first type is the type with high load and the second type is the type with low load, and the target node is of the second type, then the first node is of the first type.

[0339] The following example, taking the target node as the first type, illustrates the process of determining the first node.

[0340] For example, the first type of node includes the target node and M1 other nodes. After the management node determines that the node to be converted is of the first type, it can obtain the size of the first data from the target node and the size of the remaining memory space of each of the M1 nodes. Then, the management node determines M2 nodes from the M1 nodes whose remaining memory space is greater than or equal to the size of the first data, wherein the M2 nodes include the first node.

[0341] In one example, the first node is the node with the largest remaining memory space among the M2 nodes. In another example, the first node is any one of the M nodes.

[0342] It should be noted that this application does not restrict the method of determining the first node from M nodes; the above is merely an illustrative example.

[0343] It should be noted that this application does not restrict the executing entity that determines the first node; the above is merely an illustrative example. For instance, the first node can also be determined by the target node.

[0344] In the above embodiments, by transmitting the first data obtained from processing the first inference request to the memory of the first node, since the type of the first node is the same as the type of the target node, that is, the type of task executed by the first node is the same as the type of task executed by the target node, the first node can continue to execute the first inference request based on the first data after receiving the first data, thereby helping to improve the processing efficiency of the first inference request and thus helping to avoid the process of converting the node type from affecting the inference request being processed by the inference cluster.

[0345] Optionally, the target memory is the memory of the second node, which is a node on the inference cluster other than the first type of node and the second type of node.

[0346] For example, if the remaining memory space of each node in the first type of node is less than the size of the first data, the target node stores the first data in the memory of the second node.

[0347] For example, the second node can be a scheduling node, a management node, etc.

[0348] It should be noted that this application does not limit the type of the second node; the above is merely an illustrative example.

[0349] In the above embodiments, by storing the first data in the memory of the second node, when the remaining memory space of the first type node is insufficient, the first data is temporarily stored in the second node. This not only helps to ensure the efficiency of converting the type of the target node, but also helps to avoid the loss of the first data, thereby helping to avoid affecting the first inference request being processed by the inference cluster.

[0350] Optionally, after storing the first data in the memory of the second node, the node management method further includes: storing the second data stored in the memory of the first node in the memory of the fifth node in the memory of the fifth node if the remaining memory space of the fifth node is greater than or equal to the size of the first data.

[0351] It should be noted that the explanation of the fifth node can be found in the explanation of the first node above, and will not be repeated here.

[0352] In the above embodiments, by storing the second data in the memory of the first node into the memory of the fifth node, the first node can continue to process the first inference request based on the first data, thereby helping to avoid affecting the execution efficiency of the first inference request.

[0353] Optionally, if the target node includes multiple processors, S5 may include: during the communication gap between the first processor and the second processor, transferring the first data obtained by the target node from processing the first inference request to the target memory.

[0354] The first processor and the second processor can be any two processors on the target node that process the first inference request.

[0355] For example, during the parallel processing of the first inference request by the first processor and the second processor, the target node may transfer the first data obtained from processing the first inference request to the target memory during the communication gap between the first processor and the second processor.

[0356] In the above embodiments, by transmitting the first data during the communication gap between the two processors of the target node, the first data can be transmitted when the communication resources of the target node are relatively idle. Compared with transmitting the first data during the communication process between the two processors, this helps to avoid preempting the communication resources of the two processors, thereby helping to avoid affecting the rate at which the target node processes inference requests.

[0357] Optionally, transferring the first data obtained by the target node in processing the first inference request to the target memory includes: transferring the key-value data obtained by the target node in processing the first inference request to the target memory.

[0358] In the above embodiments, by exporting the key-value (KV) data obtained from processing the first inference request by the target node, these key-value data can be reused when subsequent inference requests are executed, thereby helping to avoid redundant calculations and improve inference performance.

[0359] Optionally, the node management method also includes: the inference cluster output target interface.

[0360] The target interface includes a first control, which indicates whether the type of configuration node is supported, that is, whether the type of configuration node is supported or not. The first control includes a first state or a second state; the first state indicates the type of configuration node is supported, and the second state indicates the type of configuration node is not supported.

[0361] For example, a user can display the target interface output by the inference cluster via an electronic device. Figure 6 As shown in (a) above, the first control is in the first state, indicating that it supports configuring the type of node. Therefore, the inference cluster to which the target interface belongs supports configuring the type of node through the node management method provided in this application. Figure 6 As shown in (b), the first control is in the second state, indicating that the type of configuration node is not supported.

[0362] It should be noted that this application does not limit how the first control indicates the first state or the second state; the above is merely an illustrative example.

[0363] In this embodiment, by setting the output target interface to include a first control, the user can determine whether the inference cluster being used supports the type of configuration node based on the target interface, thereby helping to improve the user experience.

[0364] Optionally, the target interface also includes a second control, which is used to indicate whether the function of configuring the type of node is enabled, that is, to indicate whether the function of configuring the type of node is enabled or disabled (i.e., not enabled).

[0365] The second control includes a third state or a fourth state. The third state indicates the function of enabling the configuration node type, and the fourth state indicates the function of disabling the configuration node type.

[0366] For example, a user can display the target interface output by the inference cluster via an electronic device. Figure 6 As shown in (c), the second control is in the third state, indicating that the function of configuring the node type has been enabled. Therefore, the inference cluster to which the target interface belongs can configure the node type through the node management method provided in this application. Figure 6As shown in (d), the second control is in the fourth state, indicating that the function of configuring the type of node has been turned off.

[0367] It should be noted that this application does not limit how the second control indicates the third or fourth state; the above is merely an illustrative example.

[0368] It should be noted that this application does not restrict whether the second control and the first control are located on the same interface; the above is merely an illustrative example. For instance, the second control and the first control can be located on different interfaces output by the inference cluster. For example, the first control may be located on the first interface output by the inference cluster, and the second control may be located on the second control output by the inference cluster, where the first interface and the second interface are different interfaces.

[0369] In this embodiment, by setting the target interface to include a second control, users can choose whether to enable or disable the function of configuring node types. In this way, users can choose to enable or disable the function of configuring node types according to their needs, thereby helping to improve the user experience.

[0370] In the above scheme, the inference cluster determines the target node from the first type and the second type of nodes, and converts the target node to the type with the highest load among the first and second types. This not only increases the number of high-load nodes without increasing the total number of nodes in the inference cluster, thereby improving the processing speed of inference tasks (pre-filling or decoding tasks) on high-load nodes, but also fully utilizes the low-load nodes in the inference cluster, thus improving their utilization rate. Furthermore, increasing the processing speed of inference tasks on high-load nodes increases their throughput, thereby improving the throughput of the inference cluster and ultimately its performance. In addition, increasing the processing speed of inference requests on high-load nodes allows for a rapid reduction in their load, thus balancing the load across different types of nodes. This helps prevent high-load nodes from reaching performance bottlenecks, thus avoiding issues such as lag and system crashes, and also helps prevent impacts on the performance of tasks processed by high-load nodes.

[0371] The above scheme introduces the node management method provided in this application. The following, in conjunction with... Figures 7 to 9 The solution for the first type of target node is described in detail.

[0372] It should be noted that the following will only elaborate on the differences between the two schemes; the similarities will not be repeated.

[0373] For example, the first type of node includes the fourth node. Hereinafter, this application will provide an illustrative example using the fourth node as the target node.

[0374] It should be noted that this application does not limit the number of nodes included in the target node.

[0375] Figure 7 A flowchart of another node management method provided in this application. Exemplarily, the node management method includes steps S701-S702.

[0376] S701: The management node determines the fourth node from the first type of nodes and the second type of nodes, and the fourth node is of the first type.

[0377] In this application, after determining that the type of the node to be converted is the first type, the management node can determine the fourth node from the first type of nodes.

[0378] For example, such as Figure 7 As shown, the first type is the type with low load. The management node determines the node to be converted to the fourth type from nodes 10, ..., 1M, 20, ..., 2N.

[0379] Optionally, the fourth node includes at least one of the following: the node with the lowest current memory utilization among the first type of nodes, the node with current memory utilization less than or equal to a first occupancy threshold among the first type of nodes, or the node whose current memory utilization is sorted by a sorting threshold less than or equal to a first sorting threshold among the first type of nodes, wherein sorting refers to sorting from smallest to largest.

[0380] Optionally, if the current memory utilization of each node in the first type of nodes is sorted by size, the fourth node includes: nodes whose current memory utilization in the first type of nodes is greater than or equal to a second sorting threshold.

[0381] Optionally, the fourth node can be any node from the first type of nodes.

[0382] It should be noted that this application does not restrict the method of selecting the fourth node as the management node; the above is merely an illustrative example.

[0383] It should be noted that other related explanations for S701 can be found in the explanations for S401 above, and will not be repeated here.

[0384] S702: The management node configures the type of the fourth node to the second type.

[0385] For example, such as Figure 7 As shown, the second type is the type with high load, and the management node configures the fourth node as the second type.

[0386] Example 1: The management node can modify the identity identifier stored on the fourth node to the second identity identifier, thereby changing the type of the fourth node to the second type.

[0387] Example 2: The management node can modify the identity identifier of the fourth node in the node scheduling information to the second identity identifier, obtain the updated node scheduling information, and thus change the type of the fourth node to the second type.

[0388] For example, such as Figure 8 As shown, the fourth node is a first-type node from time T1 to T5. During this period, the fourth node performs a pre-filling task and transmits the resulting tokens to second-type nodes. The second-type nodes then perform decoding based on the pre-filled tokens to obtain the inference result. Based on this, the prediction module obtains the load of the first-type nodes and the load of the second-type nodes. The decision module, based on the loads of the first-type nodes and the second-type nodes, determines that the fourth node is among the nodes to be converted. At time T6, the conversion module converts the fourth node to a second-type node. That is, after time T6, the fourth node is of type second and is used to perform the decoding task.

[0389] For example, such as Figure 8 As shown, the first type of node also includes a sixth node, which is a first type node from time T1 to time T2. The sixth node is used to perform pre-filling tasks during its time as a first type node; when no pre-filling task is assigned to the sixth node, it is idle. The decision module can also determine that the sixth node is among the nodes to be converted. At time T3, the conversion module converts the sixth node's type to the second type; that is, after time T3, the sixth node is used to perform decoding tasks.

[0390] It should be noted that this application does not limit the number of times a node can be converted to a different type. For example, after the type of the fourth node is converted from type 1 to type 2, it can be converted back from type 2 to type 1.

[0391] Optionally, after configuring the fourth node's type to the second type, the management node sends a second scheduling notification to the scheduling node, which instructs that a task be assigned to the fourth node. In response to the received second scheduling notification, the scheduling node assigns a decoding task to the fourth node based on its converted type being the second type.

[0392] In this example, after the fourth node changes type, a scheduling notification is sent to the scheduling node to instruct the scheduling node to assign tasks to the fourth node after the type change, which helps to improve the scheduling node's ability to assign tasks to the fourth node in a timely manner after the type change.

[0393] Optionally, transferring the first data to the target memory includes: transferring the key-value data obtained by each neural network layer on the fourth node from processing the first inference request to the target memory.

[0394] For example, the neural network model on the fourth node includes multiple neural network layers, which are used to sequentially execute inference requests. When the fourth node receives the conversion instruction sent by the management node, W neural network layers have already completed the pre-filling task. The key-value data obtained by each of the W neural network layers from executing the first inference request is stored in the fourth node's cache. Based on this, the fourth node can transfer the key-value data obtained by each neural network layer to the target memory through the request layer of each neural network layer, thereby achieving the layer-by-layer transfer of key-value data from each neural network layer to the target memory.

[0395] For example, such as Figure 9 As shown, request layer 1 of neural network layer 1 transmits the key-value data obtained by neural network layer 1 to the target memory, request layer 2 of neural network layer 2 transmits the key-value data obtained by neural network layer 2 to the target memory, request layer 3 of neural network layer 3 transmits the key-value data obtained by neural network layer 3 to the target memory, and so on.

[0396] In this embodiment, the key-value data on the fourth node is exported in units of neural network layers. This helps to ensure the integrity of the key-value data obtained from each neural network layer, thereby helping to improve the accuracy and reliability of the key-value data obtained from each neural network layer in subsequent use.

[0397] Optionally, the second type of node includes a third node, and the node management method also includes the following S6 and S7.

[0398] S6: Transfer the second data obtained from the second inference request processed by the third node to the fourth node.

[0399] The second data includes key-value data and lexical units obtained by the third node in processing the second inference request.

[0400] For example, after the management node determines that the node to be converted is of type 1, it can determine a third node from the second type of nodes, so as to transfer a portion of the inference requests processed by the third node to the fourth node. The management node can send a data transfer instruction to the third node, and the third node, in response to the transfer instruction, transfers the second data obtained from the second inference request it is processing to the fourth node.

[0401] Optionally, the third node includes a third processor and a fourth processor. S6 may include: during the communication gap between the third processor and the fourth processor, transferring the second data obtained by the fourth node from processing the second inference request to the fourth node.

[0402] It should be noted that the relevant explanation of "transferring the second data obtained by the fourth node in processing the second inference request to the fourth node during the communication gap between the third processor and the fourth processor" can be referred to the above explanation of "transferring the first data obtained by the target node in processing the first inference request to the target memory during the communication gap between the first processor and the second processor", and will not be repeated here.

[0403] In this embodiment, by transmitting the second data during the communication gap between the two processors of the third node, the second data can be transmitted when the communication resources of the third node are relatively idle. Compared with transmitting the second data during the communication between the two processors, this helps to avoid preempting the communication resources of the two processors, thereby helping to avoid affecting the rate at which the third node processes inference requests.

[0404] S7: After the type of the fourth node is configured as the second type, the fourth node performs the decoding task of the second inference request based on the second data.

[0405] For example, the fourth node can be converted to a second type node to perform a pre-filling task. Based on this, the fourth node continues to perform the decoding task of the second inference request according to the second data from the third node.

[0406] In this embodiment, transferring the inference request being processed by the second type of node to the fourth node not only helps improve the processing efficiency of the second inference request, but also helps to increase the throughput of the second type of node as quickly as possible.

[0407] In the above embodiments, when the load of the first type of nodes is less than the load of the second type of nodes, the inference cluster converts the fourth node in the first type of nodes into a second type of node. This allows the fourth node to switch from performing pre-filling tasks to performing decoding tasks, thereby increasing the number of nodes performing decoding tasks. This improves the processing efficiency of decoding tasks without increasing the total number of nodes in the inference cluster, i.e., without increasing hardware costs, thereby increasing the throughput of the second type of nodes and ultimately the overall throughput of the inference cluster. Furthermore, it fully utilizes the computing resources of the first type of nodes, preventing idle nodes. Additionally, by improving the processing efficiency of decoding tasks, the load on the second type of nodes can be reduced rapidly, allowing them to receive the results of the pre-filling tasks as quickly as possible, thus reducing the load on the first type of nodes. This not only balances the load of different types of nodes but also helps prevent the second type of nodes from reaching performance bottlenecks, thus preventing issues such as lag and system crashes, and ultimately avoiding impacts on the performance of the second type of nodes in processing decoding tasks.

[0408] The above embodiments provide a detailed description of the scheme where the target node is of the first type. The following, in conjunction with... Figures 10 to 12 The solution for the second type of target node is described in detail.

[0409] It should be noted that the following will only elaborate on the differences between the two schemes; the similarities will not be repeated.

[0410] For example, the second type of node includes the fifth node. In the case where the target node is of the second type, the target node is the fifth node.

[0411] Figure 10 A flowchart of another node management method provided in this application. Exemplarily, the node management method includes S1001-S1002.

[0412] S1001: The management node determines the fifth node from the first type of node and the second type of node, and the fifth node is of the second type.

[0413] In this application, after determining that the type of the node to be converted is the first type, the management node can determine the fifth node from the second type of nodes.

[0414] For example, such as Figure 10 As shown, the second type is the type with low load. The management node determines the node to be converted to the fifth type from nodes 10, ..., 1M, 20, ..., 2N.

[0415] Optionally, the fifth node includes at least one of the following: the node with the lowest current memory utilization among the second type of nodes, the node with current memory utilization less than or equal to the first occupancy threshold among the second type of nodes, or the node whose current memory utilization is sorted by a number less than or equal to the first sorting threshold among the second type of nodes, wherein sorting refers to sorting from smallest to largest.

[0416] Optionally, if the current memory utilization of each node in the first type of nodes is sorted by size, the fifth node includes: nodes whose current memory utilization in the first type of nodes is greater than or equal to a second sorting threshold.

[0417] Optionally, the fifth node can be any node from the first type of nodes.

[0418] It should be noted that this application does not restrict the method of selecting the fifth node as the management node; the above is merely an illustrative example.

[0419] It should be noted that other related explanations for S1001 can be found in the explanations for S401 above, and will not be repeated here.

[0420] S1002: The management node configures the type of the fifth node to the first type.

[0421] For example, such as Figure 10 As shown, the first type is the type with high load, and the management node configures the fifth node as the first type.

[0422] Example 1: The management node can modify the identity identifier stored on the fifth node to the first identity identifier, thereby changing the type of the fifth node to the first type.

[0423] Example 2: The management node can modify the identity identifier of the fifth node in the node scheduling information to the first identity identifier to obtain the updated node scheduling information, thereby changing the type of the fifth node to the first type.

[0424] For example, such as Figure 11 As shown, the fifth node is a second-type node from time T7 to T11. During this period, the fifth node, as a second-type node, performs a decoding task based on the tokens from the first-type nodes and obtains the inference result. Based on this, the prediction module obtains the load of the first-type nodes and the load of the second-type nodes, and the decision module determines that the nodes to be converted include the fifth node based on the loads of the first-type nodes and the second-type nodes. At time T12, the conversion module converts the type of the fifth node to the first type; that is, after time T12, the type of the fifth node is the first type, and the fifth node is used to perform the pre-filling task.

[0425] For example, such as Figure 11 As shown, the second type of node also includes a seventh node, which is a second type node from time T7 to T8. During the time period when the seventh node is a second type node, it is used to perform a decoding task based on the tokens from the first type of node and obtain the inference result. When the seventh node is not assigned a decoding task, it is in an idle state. The decision module can also determine that the node to be converted includes the seventh node. At time T9, the conversion module converts the type of the seventh node to the first type; that is, after time T9, the type of the seventh node is the first type, and the seventh node is used to perform the pre-filling task.

[0426] Optionally, after configuring the fifth node's type to the first type, the management node sends a third scheduling notification to the scheduling node. The third scheduling notification instructs the scheduling node to assign a task to the fifth node. In response to the received third scheduling notification, the scheduling node assigns a decoding task to the fifth node based on the fifth node's converted type being the first type.

[0427] In this example, after the fifth node changes type, a scheduling notification is sent to the scheduling node to instruct the scheduling node to assign tasks to the fifth node after the type change, which helps to improve the scheduling node's ability to assign tasks to the fifth node in a timely manner after the type change.

[0428] Optionally, transferring the first data to the target memory includes: transferring the key-value data obtained by the fifth node in each round of inference in response to the first inference request to the target memory.

[0429] For example, the fifth node's decoding task includes performing multiple rounds of inference. Each round of inference yields one block of key-value data and its corresponding word. After receiving the conversion instruction, the fifth node has completed R rounds of inference, obtaining R blocks of key-value data and their corresponding word. Based on this, the fifth node transfers the R blocks of key-value data to the target memory in blocks. The R blocks of key-value data can be transferred in multiple transfers, with each transfer transmitting at least one key-value data block.

[0430] For example, such as Figure 12 As shown, the fifth node can transfer the key-value data obtained from the first round of reasoning, the key-value data obtained from the second round of reasoning, the key-value data obtained from the third round of reasoning, and so on, to the target memory.

[0431] In this embodiment, by exporting the key-value data of the fifth node in units of each round of reasoning, the integrity of the key-value data obtained in each round of reasoning is ensured, thereby improving the accuracy of the key-value data obtained in subsequent rounds of reasoning.

[0432] Optionally, transmitting the first data obtained by the target node in processing the first inference request to the target memory includes: transmitting the token obtained by the fifth node in processing the first inference request to the target memory of the inference cluster.

[0433] For example, after receiving the conversion instruction, the fifth node transmits the tokens corresponding to the obtained R block key value data to the target memory.

[0434] It should be noted that this application does not restrict the order of transmitting key-value data and tokens.

[0435] It should be noted that this application does not impose a time limit on when the fifth node stops executing the first inference request. For example, it can stop processing the first inference request upon receiving a conversion instruction. Alternatively, it can stop processing the first inference request after transmitting the R block key-value data.

[0436] In this embodiment, by deriving the lexical units obtained from each round of reasoning when the fifth node processes the first reasoning request, it helps to avoid repeatedly calculating previously obtained lexical units, thereby helping to improve reasoning performance.

[0437] In the above embodiments, when the load of the second type of nodes is less than that of the first type of nodes, the inference cluster converts the fifth node in the second type of nodes into a first type of node. This allows the fifth node to switch from performing decoding tasks to performing pre-filling tasks, thereby increasing the number of nodes performing pre-filling tasks. This improves the processing efficiency of pre-filling tasks without increasing the total number of nodes in the inference cluster, i.e., without increasing hardware costs, thereby increasing the throughput of the first type of nodes and ultimately the overall throughput of the inference cluster. Furthermore, it fully utilizes the computing resources of the second type of nodes, preventing idle nodes. Additionally, by improving the processing efficiency of pre-filling tasks, the load on the first type of nodes can be reduced rapidly, allowing them to obtain the results of the pre-filling tasks as quickly as possible and transfer them to the second type of nodes so they can continue performing decoding tasks. This not only balances the load of different types of nodes but also helps prevent the first type of nodes from reaching performance bottlenecks, thus preventing issues such as lag and system crashes, and ultimately avoiding impacts on the performance of the first type of nodes in processing pre-filling tasks.

[0438] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, the node management device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0439] This application embodiment can, according to the above method, exemplarily divide the node management device into functional modules. For example, the node management device may include functional modules corresponding to each functional division, or two or more functions may be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division; in actual implementation, there may be other division methods.

[0440] For example, Figure 13 A possible schematic diagram of the node management device (i.e., node management device 1300) involved in the above embodiments is shown. The actions performed by the node management device 1300 can be implemented by the nodes of the inference cluster or by the nodes of the inference cluster executing corresponding software. The node management device 1300 may include: a determination module 1301 and a configuration module 1302. The determination module 1301 is used to determine a target node from nodes of a first type and nodes of a second type; the type of the target node is the type with low load between the first type and the second type. For example, Figure 4 S401 shown Figure 7 The S701 shown and Figure 10 S1001 is shown. Configuration module 1302 is used to configure the type of the target node as the type with higher load between the first type and the second type. For example, Figure 4 S402 shown Figure 7 The S702 shown and Figure 10 S1002 is shown.

[0441] Optionally, the configuration module 1302 is specifically used to: configure the type of the target node to the second type when the type of the target node is the first type.

[0442] Optionally, the configuration module 1302 is specifically used to: configure the type of the target node to the first type when the type of the target node is the second type.

[0443] Optionally, the node management device 1300 further includes a transmission module 1303. After determining the target node from the first type of node and the second type of node, the transmission module 1303 is configured to: transmit the first data obtained by the target node in processing the first inference request to the target memory of the inference cluster when the target node processes the first inference request.

[0444] Optionally, the target node includes a first processor and a second processor; the transmission module 1303 is specifically used to: during the communication gap between the first processor and the second processor, transmit the first data obtained by the target node in processing the first inference request to the target memory.

[0445] Optionally, the transmission module 1303 is specifically used to: transmit the key-value data obtained by each neural network layer on the target node from processing the first inference request to the target memory when the target node performs a pre-filling task for the first inference request.

[0446] Optionally, the transmission module 1303 is specifically used to: transmit the key-value data obtained by the target node in each round of inference of the first inference request to the target memory when the target node performs a decoding task on the first inference request.

[0447] Optionally, the transmission module 1303 is specifically used to: transmit the token obtained by the target node in processing the first inference request to the target memory of the inference cluster when the target node performs a decoding task on the first inference request.

[0448] Optionally, the target memory is the memory of the first node, and the type of the first node is the type with less load between the first type and the second type.

[0449] Optionally, the target memory is the memory of the second node, which is a node on the inference cluster other than the first type of node and the second type of node.

[0450] Optionally, the node management device 1300 further includes an execution module 1304; the second type of node includes a third node; the transmission module 1303 is further configured to: transmit the second data obtained by the third node in processing the second inference request to the target node when the target node is of the first type; the execution module 1304 is configured to: after the target node is configured to be of the second type, execute the decoding task of the second inference request according to the second data.

[0451] Optionally, the third node includes a third processor and a fourth processor; the transmission module 1303 is specifically used to: transmit the second data obtained by the third node in processing the second inference request to the target node during the communication gap between the third processor and the fourth processor.

[0452] Optionally, the configuration module 1302 is specifically used to: configure the identity identifier of the target node as the target identity identifier, which is used to indicate the type with higher load in the first type and the second type.

[0453] Optionally, the configuration module 1302 is specifically used to: update node scheduling information, wherein the updated node scheduling information is used to indicate that the identity of the target node is the target identity; and the node scheduling information is used to indicate the identity of the first type of node and the identity of the second type of node.

[0454] Optionally, the determining module 1301 is specifically used to: determine a target node based on the first memory utilization rate of the first type of node and the second memory utilization rate of the second type of node; the type of the target node is the type with lower memory utilization rate among the first type and the second type.

[0455] Optionally, the determining module 1301 is specifically used to: determine a target node based on the first processing rate of the first type of node for the pre-filling task and the second processing rate of the second type of node for the decoding task, wherein the type of the target node is the type with the smaller processing rate between the first type and the second type.

[0456] Optionally, the determining module 1301 is specifically used to: determine a second difference, the second difference being equal to a second memory utilization rate minus a first memory utilization rate; if the second difference is less than or equal to a first difference threshold, determine the type of the target node as a second type; if the second difference is greater than or equal to a second difference threshold, determine the type of the target node as a first type.

[0457] Optionally, when the target node is of type 1, the target node includes at least one of the following: the node with the lowest current memory utilization among the nodes of type 1, the node with current memory utilization less than or equal to a first occupancy threshold among the nodes of type 1, or the node whose current memory utilization is sorted by a sorting threshold less than or equal to a first sorting threshold among the nodes of type 1, wherein sorting refers to sorting from smallest to largest.

[0458] Optionally, when the target node is of type 2, the target node includes at least one of the following: the node with the lowest current memory utilization among the nodes of type 2, the node with current memory utilization less than or equal to a first occupancy threshold among the nodes of type 2, or the node whose current memory utilization is sorted less than or equal to a first sorting threshold among the nodes of type 2, wherein sorting refers to sorting from smallest to largest.

[0459] Optionally, the determining module 1301 is specifically used to: determine a target node from the first type of nodes and the second type of nodes according to the target period.

[0460] Optionally, the target node type is the type with less load in the first type and the second type, including: the target node type is the type with less load in the current first type and the second type or the target node type is the type with less load in the first type and the second type during the target time period.

[0461] Optionally, the target node type is configured as the type with higher load among the first type and the second type, including: the target node type is configured as the type with higher load among the current first type and the second type, or the target node type is configured as at least one of the types with higher load among the first type and the second type during the target time period.

[0462] Optionally, the node management device 1300 further includes an allocation module 1305; after determining the target node from the first type of nodes and the second type of nodes, the allocation module 1305 is used to: stop the task of allocating inference requests to the target node.

[0463] Optionally, the determining module 1301 is further configured to: input the current memory utilization rate of the first type of node into the first neural network model to obtain the first memory utilization rate; wherein, the first memory utilization rate is the memory utilization rate of the first type of node in the target time period, and the target time period is the time period after the current time.

[0464] Optionally, the determining module 1301 is further configured to: obtain the number and length of inference requests processed by the first type of node during the target time period; input the number and length of inference requests processed by the first type of node during the target time period into the second neural network model to obtain the second memory utilization rate. The second memory utilization rate is the memory utilization rate of the second type of node during the target time period.

[0465] Optionally, the determining module 1301 is further configured to: use the number and length of inference requests currently being processed by the first type of node in the third neural network model to obtain the number and length of inference requests processed by the first type of node in the target time period.

[0466] Optionally, the determining module 1301 is further configured to: input the current memory utilization rate of the second type of node into the first neural network model to obtain the second memory utilization rate.

[0467] Optionally, the determining module 1301 is specifically used to: determine the target node based on the ratio between the first memory utilization rate and the second memory utilization rate.

[0468] Optionally, the determining module 1301 is specifically used to: determine a first ratio, the first ratio being equal to a first memory utilization rate compared to a second memory utilization rate; if the first ratio is less than or equal to a first ratio threshold, determine the type of the target node as a first type; if the first ratio is greater than or equal to a second ratio threshold, determine the type of the target node as a second type.

[0469] Optionally, the determining module 1301 is specifically used to: determine a second ratio, the second ratio being equal to the ratio of a second memory utilization rate to a first memory utilization rate; if the second ratio is less than or equal to a first ratio threshold, determine that the type of the target node is a second type; if the first ratio is greater than or equal to the second ratio threshold, determine that the type of the target node is a first type.

[0470] Optionally, the determining module 1301 is specifically used to: determine the target node based on the difference between the first memory utilization rate and the second memory utilization rate.

[0471] Optionally, the determining module 1301 is specifically used to: determine a first difference, the first difference being equal to a first memory utilization rate minus a second memory utilization rate; if the first difference is less than or equal to a first difference threshold, determine the type of the target node as a first type; if the first difference is greater than or equal to a second difference threshold, determine the type of the target node as a second type.

[0472] Optionally, the determining module 1301 is further configured to: input the current processing rate of the first type of node for the pre-filling task into the fourth neural network model to obtain a first processing rate; wherein, the first processing rate is the processing rate of the first type of node for the pre-filling task within the target time period, and the target time period is the time period after the current time.

[0473] Optionally, the determining module 1301 is further configured to: obtain the number and length of inference requests processed by the first type of node during the target time period; input the number and length of inference requests processed by the first type of node during the target time period into the fifth neural network model to obtain a second processing rate. The second processing rate is the processing rate of the second type of node for the pre-filled task during the target time period.

[0474] Optionally, the determining module 1301 is further configured to: use the number and length of inference requests currently being processed by the first type of node in the third neural network model to obtain the number and length of inference requests processed by the first type of node in the target time period.

[0475] Optionally, the determining module 1301 is further configured to: input the current processing rate of the second type of node for the pre-filled task into the fourth neural network model to obtain the second processing rate.

[0476] Optionally, the determining module 1301 is specifically used to: determine the target node based on the ratio of the first processing rate and the second processing rate.

[0477] Optionally, the determining module 1301 is specifically used to: determine a first ratio, the first ratio being equal to the ratio of a first processing rate to a second processing rate; if the first ratio is less than or equal to a first ratio threshold, determine that the type of the target node is a first type; if the first ratio is greater than or equal to a second ratio threshold, determine that the type of the target node is a second type.

[0478] Optionally, the determining module 1301 is specifically used to: determine a second ratio, the second ratio being equal to the ratio of a second processing rate to a first processing rate; if the second ratio is less than or equal to a first ratio threshold, determine that the type of the target node is a second type; if the first ratio is greater than or equal to the second ratio threshold, determine that the type of the target node is a first type.

[0479] Optionally, the determining module 1301 is specifically used to: determine the target node based on the difference between the first processing rate and the second processing rate.

[0480] Optionally, the determining module 1301 is specifically used to: determine a first difference, the first difference being equal to a first processing rate minus a second processing rate; if the first difference is less than or equal to a first difference threshold, determine the type of the target node as a first type; if the first difference is greater than or equal to a second difference threshold, determine the type of the target node as a second type.

[0481] Optionally, the determining module 1301 is specifically used to: determine a second difference, the second difference being equal to a second processing rate minus a first processing rate; if the second difference is less than or equal to a first difference threshold, determine the type of the target node as a second type; if the second difference is greater than or equal to the second difference threshold, determine the type of the target node as a first type.

[0482] Optionally, the node management device 1300 further includes an output module 1306, which is used to: output a target interface; the target interface includes a first control, which is used to indicate whether the type of configuration node is supported. The first control includes a first state or a second state, where the first state indicates the type of configuration node is supported, and the second state indicates the type of configuration node is not supported.

[0483] Optionally, the target interface also includes a second control, which indicates whether the function of configuring the node type is enabled. The second control may include a third state or a fourth state; the third state indicates whether the function of configuring the node type is enabled, and the fourth state indicates whether the function of configuring the node type is disabled.

[0484] For a detailed description of the above-mentioned optional methods, please refer to the foregoing method embodiments, which will not be repeated here. Furthermore, the explanation of any of the node management devices 1300 provided above, as well as the description of their beneficial effects, can be found in the corresponding method embodiments described above, and will not be repeated here.

[0485] In this application, the determination module 1301, configuration module 1302, transmission module 1303, execution module 1304, allocation module 1305, and output module 1306 can all be implemented in software or in hardware. For example, the implementation of the determination module 1301 will be described below. Similarly, the implementation of the configuration module 1302, transmission module 1303, execution module 1304, allocation module 1305, and output module 1306 can refer to the implementation of the determination module 1301.

[0486] As an example of a software functional unit, module 1301 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the aforementioned computing instance may be one or more. For example, module 1301 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0487] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0488] As an example of a hardware functional unit, module 1301 may include at least one computing device, such as a server. Alternatively, module 1301 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0489] The multiple computing devices included in module 1301 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in module 1301 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in module 1301 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0490] It should be noted that, in other embodiments, the determining module 1301 can be used to execute any step in the node management method, the configuring module 1302 can be used to execute any step in the node management method, the transmitting module 1303 can be used to execute any step in the node management method, the executing module 1304 can be used to execute any step in the node management method, the allocating module 1305 can be used to execute any step in the node management method, and the output module 1306 can be used to execute any step in the node management method. The steps implemented by the determining module 1301, the configuring module 1302, the transmitting module 1303, the executing module 1304, the allocating module 1305, and the output module 1306 can be specified as needed. By implementing different steps in the node management method through the determining module 1301, the configuring module 1302, and the transmitting module 1303, all functions of the data storage device can be realized.

[0491] This application also provides a computing device 1400. For example... Figure 14As shown, the computing device 1400 includes a bus 1402, a processor 1404, a memory 1406, and a communication interface 1408. The processor 1404, the memory 1406, and the communication interface 1408 communicate with each other via the bus 1402. The computing device 1400 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1400.

[0492] Bus 1402 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 14 The bus 1402 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1402 may include a path for transmitting information between various components of the computing device 1400 (e.g., memory 1406, processor 1404, communication interface 1408).

[0493] The processor 1404 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0494] The memory 1406 may include volatile memory, such as random access memory (RAM). The processor 1404 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0495] The memory 1406 stores executable program code, and the processor 1404 executes this executable program code to implement the functions of the aforementioned determining module 1301, configuration module 1302, transmission module 1303, execution module 1304, allocation module 1305, and output module 1306, thereby realizing the node management method. That is, the memory 1406 stores instructions for executing the node management method.

[0496] The communication interface 1408 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 1400 and other devices or communication networks.

[0497] For example, the computing device 1400 described above may be Figure 1 The diagram shows management nodes, first-type nodes, second-type nodes, etc.

[0498] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0499] like Figure 15 As shown, the computing device cluster 1500 includes at least one computing device 1400. The memory 1406 of one or more computing devices 1400 in the computing device cluster 1500 may store the same instructions for executing node management methods.

[0500] In some possible implementations, the memory 1406 of one or more computing devices 1400 in the computing device cluster 1500 may also store partial instructions for executing node management methods. In other words, a combination of one or more computing devices 1400 can jointly execute instructions for executing node management methods.

[0501] It should be noted that the memory 1406 in the different computing devices 1400 in the computing device cluster 1500 can store different instructions, which are used to execute some functions of the node management device. That is, the instructions stored in the memory 1406 in the different computing devices 1400 can realize the functions of one or more modules among the determination module 1301, configuration module 1302, transmission module 1303, execution module 1304, allocation module 1305 and output module 1306.

[0502] For example, the computing device cluster described above may include at least one management node, at least one first-type node, and at least one second-type node. The at least one management node includes a processor, memory, and computer programs / instructions stored in the memory; the processor in the at least one management node executes the computer programs / instructions stored in the memory to enable the at least one management node to implement the node management method in the above embodiments, thereby switching computing tasks between at least one first-type node and / or at least one second-type node.

[0503] In some possible implementations, one or more computing devices in the computing device cluster 1500 can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 16 One possible implementation is shown. For example... Figure 16 As shown, the two computing devices 1400A and 1400B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device.

[0504] In this type of possible implementation, the memory 1406 in computing device 1400A stores instructions for executing the functions of determination module 1301, configuration module 1302, and transmission module 1303. Meanwhile, the memory 1406 in computing device 1400B stores instructions for executing the functions of execution module 1304, allocation module 1305, and output module 1306.

[0505] Figure 16 The connection method between the computing device clusters shown can be such that, considering the node management method provided in this application requires a large amount of computation, the functions of the execution module 1304, the allocation module 1305 and the output module 1306 are delegated to the computing device 1400B.

[0506] It should be understood that Figure 16 The functions of computing device 1400A shown can also be performed by multiple computing devices 1400. Similarly, the functions of computing device 1400B can also be performed by multiple computing devices 1400.

[0507] It needs to be explained that, Figure 16 For further details regarding the computing cluster shown, please refer to [link / reference]. Figure 15 The description of the computing cluster shown will not be repeated here.

[0508] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 15 and Figure 16 The connection method of the computing device cluster shown is different in that the memory 1406 of one or more computing devices 1400 in this computing device cluster can store the same instructions for executing node management methods.

[0509] In some possible implementations, the memory 1406 of one or more computing devices 1400 in the computing device cluster may also store partial instructions for executing node management methods. In other words, a combination of one or more computing devices 1400 can jointly execute instructions for executing node management methods.

[0510] This application also provides a processor that can be used to perform the above-described methods.

[0511] This application also provides a chip, including: a processor and a power supply circuit; the power supply circuit can be used to supply power to the processor; the processor can be used to perform the above-described methods.

[0512] This application also provides a computing device cluster, which includes at least one computing device; each computing device includes a processor, a memory, and computer programs / instructions stored in the memory, wherein the processor of each computing device executes the computer programs / instructions stored in the memory of each computing device to enable each computing device to implement the above-described method.

[0513] This application also provides a computer program product. The computer program product includes a computer program / instructions, which are software or program products capable of running on a computing device or stored on any usable medium. When the computer program / instructions are executed on at least one computing device, the at least one computing device can perform the methods described above.

[0514] This application also provides a computer-readable storage medium, which can be any available medium capable of being stored by a computing device or a data storage device such as a data center containing one or more available media. The computer-readable storage medium stores a computer program / instructions that, when executed on at least one computing device, enable the at least one computing device to perform the described method.

[0515] For example, the available media may be magnetic media (e.g., floppy disks, magnetic disks, magnetic tapes), optical media (e.g., digital video discs (DVDs)), or semiconductor media (e.g., solid state drives (SSDs)).

[0516] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A node management method, characterized in that, The method is applied to an inference cluster, which includes a first type of node and a second type of node, wherein the first type of node is used to perform a pre-filling task and the second type of node is used to perform a decoding task; the method includes: A target node is determined from the first type of nodes and the second type of nodes; the type of the target node is the type with low load between the first type and the second type; Configure the target node type to the type with higher load between the first type and the second type.

2. The method according to claim 1, characterized in that, Configure the target node type to the type with higher load between the first type and the second type, including: If the target node is of the first type, configure the target node type to the second type.

3. The method according to claim 1, characterized in that, Configure the target node type to the type with higher load between the first type and the second type, including: If the target node is of the second type, configure the target node type to the first type.

4. The method according to any one of claims 1-3, characterized in that, After determining the target node from the first type of nodes and the second type of nodes, the process includes: When the target node processes the first inference request, the first data obtained by the target node from processing the first inference request is transmitted to the target memory of the inference cluster.

5. The method according to claim 4, characterized in that, The target node includes a first processor and a second processor; the step of transferring the first data obtained by the target node from processing the first inference request to the target memory of the inference cluster includes: During the communication gap between the first processor and the second processor, the first data obtained by the target node from processing the first inference request is transmitted to the target memory.

6. The method according to claim 4 or 5, characterized in that, The step of transmitting the first data obtained by the target node from processing the first inference request to the target memory of the inference cluster includes: When the target node performs the pre-filling task in response to the first inference request, the key-value data obtained by each neural network layer on the target node from processing the first inference request is transferred to the target memory; or... When the target node performs the decoding task on the first inference request, the key-value data obtained by the target node in each round of inference on the first inference request is transferred to the target memory.

7. The method according to any one of claims 4-6, characterized in that, The step of transmitting the first data obtained by the target node from processing the first inference request to the target memory of the inference cluster includes: When the target node performs the decoding task on the first inference request, the token obtained by the target node from processing the first inference request is transmitted to the target memory of the inference cluster.

8. The method according to any one of claims 4-7, characterized in that, The target memory is the memory of the first node, and the type of the first node is the type with lower load between the first type and the second type; or, The target memory is the memory of the second node, which is a node on the inference cluster other than the first type of node and the second type of node.

9. The method according to any one of claims 4-8, characterized in that, The second type of node includes a third node, and the method further includes: If the target node is of the first type, the second data obtained by the third node in processing the second inference request will be transmitted to the target node; After the target node is configured to the second type, the target node performs the decoding task of the second inference request based on the second data.

10. The method according to claim 9, characterized in that, The third node includes a third processor and a fourth processor. The step of transmitting the second data obtained from processing the second inference request by the third node to the target node includes: During the communication gap between the third processor and the fourth processor, the second data obtained by the third node in processing the second inference request is transmitted to the target node.

11. The method according to any one of claims 4-10, characterized in that, Configure the target node type to the type with higher load between the first type and the second type, including: Configure the target node's identity identifier as a target identity identifier, which is used to indicate the type with higher load between the first type and the second type; and / or, Update node scheduling information, the updated node scheduling information is used to indicate that the identity of the target node is the target identity; the node scheduling information is used to indicate the identity of the first type of node and the identity of the second type of node.

12. The method according to any one of claims 1-11, characterized in that, Determining the target node from the nodes of the first type and the nodes of the second type includes: The target node is determined based on the first memory utilization rate of the first type of node and the second memory utilization rate of the second type of node; the type of the target node is the type with the lower memory utilization rate between the first type and the second type; or... The target node is determined based on the first processing rate of the first type of node for the pre-filling task and the second processing rate of the second type of node for the decoding task, wherein the type of the target node is the type with the smaller processing rate between the first type and the second type.

13. A node management device, characterized in that, The device is applied to an inference cluster, which includes a first type of node and a second type of node, wherein the first type of node is used to perform a pre-filling task and the second type of node is used to perform a decoding task; the device includes: A determination module is used to determine a target node from the first type of nodes and the second type of nodes; the type of the target node is the type with lower load between the first type and the second type; The configuration module is used to configure the type of the target node as the type with higher load between the first type and the second type.

14. The apparatus according to claim 13, characterized in that, The configuration module is specifically used for: If the target node is of the first type, configure the target node type to the second type.

15. The apparatus according to claim 13, characterized in that, The configuration module is specifically used for: If the target node is of the second type, configure the target node type to the first type.

16. The apparatus according to any one of claims 13-15, characterized in that, The device further includes a transmission module; after determining the target node from the first type of node and the second type of node, the transmission module is used to: When the target node processes the first inference request, the first data obtained by the target node from processing the first inference request is transmitted to the target memory of the inference cluster.

17. The apparatus according to claim 16, characterized in that, The target node includes a first processor and a second processor; the transmission module is specifically used for: During the communication gap between the first processor and the second processor, the first data obtained by the target node from processing the first inference request is transmitted to the target memory.

18. The apparatus according to claim 16 or 17, characterized in that, The transmission module is specifically used for: When the target node performs the pre-filling task in response to the first inference request, the key-value data obtained by each neural network layer on the target node from processing the first inference request is transferred to the target memory; or, When the target node performs the decoding task on the first inference request, the key-value data obtained by the target node in each round of inference on the first inference request is transferred to the target memory.

19. The apparatus according to any one of claims 16-18, characterized in that, The transmission module is specifically used for: When the target node performs the decoding task on the first inference request, the token obtained by the target node from processing the first inference request is transmitted to the target memory of the inference cluster.

20. The apparatus according to any one of claims 16-19, characterized in that, The target memory is the memory of the first node, and the type of the first node is the type with lower load between the first type and the second type; or, The target memory is the memory of the second node, which is a node on the inference cluster other than the first type of node and the second type of node.

21. The apparatus according to any one of claims 16-20, characterized in that, The device further includes an execution module, and the second type of node includes a third node; The transmission module is further configured to: transmit the second data obtained by the third node in processing the second inference request to the target node when the target node is of the first type; The execution module is used to: after the target node is configured to be of the second type, the target node executes the decoding task of the second inference request based on the second data.

22. The apparatus according to claim 21, characterized in that, The third node includes a third processor and a fourth processor; the transmission module is specifically used for: During the communication gap between the third processor and the fourth processor, the second data obtained by the third node in processing the second inference request is transmitted to the target node.

23. The apparatus according to any one of claims 16-22, characterized in that, The configuration module is specifically used for: Configure the target node's identity identifier as a target identity identifier, which is used to indicate the type with higher load between the first type and the second type; And / or, Update node scheduling information, the updated node scheduling information is used to indicate that the identity of the target node is the target identity; the node scheduling information is used to indicate the identity of the first type of node and the identity of the second type of node.

24. The apparatus according to any one of claims 13-23, characterized in that, The determining module is specifically used for: The target node is determined based on the first memory utilization rate of the first type of node and the second memory utilization rate of the second type of node; the type of the target node is the type with the lower memory utilization rate between the first type and the second type; or... The target node is determined based on the first processing rate of the first type of node for the pre-filling task and the second processing rate of the second type of node for the decoding task, wherein the type of the target node is the type with the smaller processing rate between the first type and the second type.

25. A management node, characterized in that, The management node includes a processor, a memory, and a computer program / instruction stored in the memory; the processor executes the computer program / instruction stored in the memory to enable the management node to implement the method as described in any one of claims 1-12.

26. A computing device cluster, characterized in that, The computing device cluster includes at least one management node, at least one first-type node, and at least one second-type node; The at least one management node includes a processor, a memory, and a computer program / instruction stored in the memory; the processor in the at least one management node executes the computer program / instruction stored in the memory to enable the at least one management node to implement the method as described in any one of claims 1-12, so as to switch computing tasks between the at least one first type of node and / or the at least one second type of node.

27. A computer program product, characterized in that, The computer program product includes a computer program / instruction, which, when executed by a computing device, implements the method as described in any one of claims 1-12.

28. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program / instruction, which, when executed by a computing device, implements the method as described in any one of claims 1-12.

Images