A processing method and electronic device
By acquiring task parameters and server resource status, a suitable server is selected to execute the target task, solving the resource contention problem in GPU server sharing scenarios, reducing communication latency, and improving task execution efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- LENOVO (BEIJING) LTD
- Filing Date
- 2026-03-31
- Publication Date
- 2026-07-21
AI Technical Summary
In shared GPU server scenarios, concurrent tasks such as model loading and model application lead to resource contention, resulting in increased latency during task execution.
By obtaining the task parameters of the target task and the resource status of the server, the resource status of the server for executing the target task is determined based on the resource status and task parameters. Servers that meet the target conditions are selected to execute the target task, and resource allocation is optimized to reduce communication latency.
It effectively alleviates resource contention issues in shared GPU server scenarios, reduces communication latency, and improves task execution efficiency.
Smart Images
Figure CN122431817A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of artificial intelligence technology, and more specifically, to a processing method and an electronic device. Background Technology
[0002] In shared GPU server scenarios, concurrent tasks such as model loading and model application lead to resource contention for the server, resulting in increased latency during task execution. Summary of the Invention
[0003] In view of this, the present disclosure provides a processing method and an electronic device.
[0004] The first aspect of this disclosure provides a processing method, comprising:
[0005] In response to the target task input, the task parameters of the target task are obtained. The task parameters represent the server resources required to execute the target task.
[0006] The first resource status of multiple servers is obtained. These servers are used to load the target model and call the target model to execute inference tasks. The first resource status represents the resource status of each server before executing the target task.
[0007] Based on the first resource state and task parameters, the second resource state of each server is determined when executing the target task. The second resource state represents the resource state of each server during the execution of the target task.
[0008] Servers whose second resource status meets the target conditions are identified as target servers for executing target tasks, which include loading the target model and / or calling the target model to execute inference tasks.
[0009] According to embodiments of this disclosure, determining a second resource state for each server to execute a target task based on a first resource state and task parameters includes:
[0010] Based on the initial resource status and task parameters, determine the resource utilization rate of each server in executing the target task;
[0011] Based on resource utilization, the latency parameters of each server are determined. The latency parameters characterize the degree of increase in communication latency when the corresponding server executes the target task.
[0012] Based on latency parameters and task parameters, the communication latency of each server when executing the target task is determined.
[0013] According to embodiments of this disclosure, the target task includes at least one of a first target task and a second target task;
[0014] Obtain the task parameters for the target task, including:
[0015] Obtain the communication resource parameters of the first target task. The communication resource parameters represent the communication resource requirements of the server during the loading of the target model.
[0016] Obtain the computational resource parameters for the second target task. These parameters represent the computational resource requirements of the server during the execution of the inference task using the target model.
[0017] According to embodiments of this disclosure, a second resource state is determined for each server when executing a target task based on a first resource state and task parameters, including at least one of the following:
[0018] Based on the first resource status and communication resource parameters, determine the communication resource status of each server when executing the first target task;
[0019] Based on the first resource status and computing resource parameters, determine the computing resource status of each server when executing the second target task.
[0020] According to embodiments of this disclosure, the communication latency of each server when executing a target task is determined based on latency parameters and task parameters, including:
[0021] Get the remaining execution time of other tasks currently being executed on each server;
[0022] Based on the task parameters, determine the target execution time for each server to execute the target task;
[0023] Based on latency parameters, remaining execution time, and target execution time, the communication latency of each server when executing the target task is determined.
[0024] According to embodiments of this disclosure, the latency parameters of each server are determined based on resource utilization, including:
[0025] If the resource utilization rate does not exceed the target utilization rate, the delay parameter is determined as the first delay parameter;
[0026] When the resource utilization rate exceeds the target utilization rate, the delay parameter is determined as the second delay parameter, which increases as the resource utilization rate increases.
[0027] According to embodiments of this disclosure, determining a server whose second resource state meets the target conditions as the target server for executing the target task includes:
[0028] Based on the second resource status of each server, determine the resource usage of each server;
[0029] Based on the resource usage of each server and the task parameters of the target task, determine the predicted communication latency for each server in executing the target task;
[0030] The predicted communication latency of each server is sorted from low to high, and the server ranked first is identified as the target server.
[0031] According to embodiments of this disclosure, a server whose second resource state satisfies the target condition is identified as the target server for executing the target task, including at least one of the following:
[0032] If the second resource status of multiple servers meets the target conditions, the target server for executing the target task is determined based on the task type of other tasks being executed on the multiple servers; wherein, the other tasks being executed on the target server are of the same task type as the target task.
[0033] If the second resource status of multiple servers meets the target conditions, the idle resources of the multiple servers when executing the target task are determined based on the second resource status of the multiple servers; based on the idle resources, the target server for executing the target task is determined; wherein, the idle resources of the target server are less than those of the other servers among the multiple servers.
[0034] According to embodiments of this disclosure, obtaining the first resource status of multiple servers further includes:
[0035] In response to the completion of at least one task on the server, update the first resource state of the server;
[0036] And / or, in response to the target task being assigned to the target server, update the first resource state of the target server.
[0037] According to embodiments of this disclosure, the method further includes:
[0038] If the second resource status of each server does not meet the target conditions, the target task will be configured in the waiting queue.
[0039] In response to an update in the first resource state of each server, the second resource state of each server is redefined.
[0040] A second aspect of this disclosure provides an electronic device, comprising:
[0041] At least one processor;
[0042] and memory connected to at least one processor; wherein,
[0043] The memory stores instructions that can be executed by at least one processor, and the instructions, when executed by at least one processor, enable the at least one processor to perform the following operations:
[0044] In response to the target task input, the task parameters of the target task are obtained. The task parameters represent the server resources required to execute the target task.
[0045] The first resource status of multiple servers is obtained. These servers are used to load the target model and call the target model to execute inference tasks. The first resource status represents the resource status of each server before executing the target task.
[0046] Based on the first resource status and task parameters, determine the second resource status of each server in executing the target task. The second resource status includes at least the resource status of the corresponding server in executing the target task.
[0047] Servers whose second resource status meets the target conditions are identified as target servers for executing target tasks, which include loading the target model and / or calling the target model to execute inference tasks.
[0048] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description
[0049] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:
[0050] Figure 1 The flowchart illustrating one of the processing methods according to an embodiment of the present disclosure is shown schematically.
[0051] Figure 2 A second flowchart illustrating a processing method according to an embodiment of this disclosure is shown schematically.
[0052] Figure 3 A flowchart of a processing method according to an embodiment of the present disclosure is illustrated schematically as follows:
[0053] Figure 4 A block diagram of an electronic device according to an embodiment of the present disclosure is shown schematically. Detailed Implementation
[0054] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.
[0055] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.
[0056] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.
[0057] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).
[0058] Figure 1 One of the flowcharts illustrating a processing method according to an embodiment of the present disclosure is shown schematically.
[0059] like Figure 1 As shown, the first aspect of this disclosure provides a processing method, including operations S110 to S140.
[0060] Operation S110, in response to the target task input, obtains the task parameters of the target task, which represent the server resources required to execute the target task.
[0061] The processing method in this embodiment can be applied to, but is not limited to, the management device of the server cluster; wherein, the management device can be selected from the server cluster or configured independently of the server cluster.
[0062] In the embodiments of this disclosure, the target task is a type of operation that is input into the server group and can be executed by the server. For example, in a server that deploys a large language model, the target task can load the configuration file of the relevant model into the server. Or, the target task can be an instruction for the server to call the local model of the server to perform an inference task.
[0063] In the embodiments of this disclosure, the task parameters are parameters that characterize the type and quantity of server resources required to execute the target task, such as communication resource requirements, computing resource requirements, etc.
[0064] Operation S120 obtains the first resource status of multiple servers. These servers are used to load the target model and call the target model to execute inference tasks. The first resource status represents the resource status of each server before executing the target task.
[0065] In the embodiments of this disclosure, a server group consisting of multiple servers can be used as a set of devices to load the target model and call the target model to perform inference tasks. The hardware configuration and resource allocation capabilities of each server may be different or partially the same, but they all have a certain ability to process the target task.
[0066] In the embodiments of this disclosure, the first resource state characterizes the resource state of each server before executing the target task, including but not limited to the server's memory usage, bandwidth usage, GPU utilization, etc., reflecting the server's initial resource usage and availability.
[0067] Operation S130: Based on the first resource state and task parameters, determine the second resource state of each server in executing the target task. The second resource state represents the resource state of each server in the process of executing the target task.
[0068] Specifically, the resource status of each server during the execution of the target task is calculated based on the initial resource status and task parameters. This resource status includes the usage and changes of resources such as memory, bandwidth, and GPU on each server during task execution. The resource status is used to predict the resource status of each server during the execution of the target task based on the initial resource status and task parameters, thereby determining the server performance of each server during the execution of the target task.
[0069] Operation S140: The server whose second resource status meets the target conditions is identified as the target server for executing the target task. The target task includes loading the target model and / or calling the target model to execute the inference task.
[0070] Specifically, after predicting the resource status of each server during the execution of the target task, the server with the best task processing performance is determined from multiple servers based on the relationship between various resource indicators in the resource status and the target conditions.
[0071] In the embodiments of this disclosure, the target task includes loading a target model and / or calling the target model to perform an inference task. Wherein, when the target task includes both loading a target model and calling the target model to perform an inference task, the target model to be loaded is not the same model as the model involved in performing the inference task.
[0072] By employing the above method, the task parameters of the target task and the first resource status of multiple servers are obtained. Based on these two, the second resource status of the servers executing the target task is determined, and target servers that meet the target conditions are selected to execute the target task, which includes loading the target model and / or calling the target model to perform inference tasks. This solves the resource contention problem caused by multi-task concurrency in GPU server sharing scenarios. By reasonably evaluating the matching degree between server resources and task requirements, communication latency can be reduced and task execution efficiency can be improved.
[0073] According to an embodiment of this disclosure, in operation S130, based on the first resource state and task parameters, the second resource state of each server executing the target task is determined, including operations S1310 to S1330.
[0074] Operation S1310 determines the resource utilization rate of each server executing the target task based on the first resource status and task parameters.
[0075] In the embodiments of this disclosure, the resource utilization rate of each server performing the target task includes the communication resource utilization rate and the computing resource utilization rate.
[0076] Operation S1320 determines the latency parameters of each server based on resource utilization. The latency parameters characterize the degree of increase in communication latency when the corresponding server executes the target task.
[0077] Operation S1330 determines the communication latency of each server when executing the target task based on latency parameters and task parameters.
[0078] Specifically, when resource utilization is below a certain threshold, server resources are not saturated, and there is no significant resource competition among multiple target tasks. When resource utilization exceeds a certain threshold, server resources are relatively saturated, and significant resource competition occurs among multiple target tasks, thus significantly increasing the latency of the server when processing multiple target tasks. Therefore, different latency parameters are configured for different resource utilization rates. When resource utilization is low, the increase in communication latency due to latency parameters is small; when resource utilization is high, the increase in communication latency due to latency parameters is rapidly increased.
[0079] By adopting the above method, the resource utilization rate is determined based on the first resource status and task parameters, the latency parameter is determined based on the resource utilization rate, and finally the communication latency is determined by combining the latency parameter and task parameters. This improves the evaluation results of communication latency when the server executes tasks, helps to more accurately select servers suitable for executing target tasks, thereby optimizing resource allocation in GPU server sharing scenarios and further reducing communication latency.
[0080] According to an embodiment of this disclosure, the target task includes at least one of a first target task and a second target task; in operation S110, the task parameters of the target task are obtained, including operations S1110 to S1120.
[0081] Operation S1110 obtains the communication resource parameters of the first target task. The communication resource parameters represent the communication resource requirements during the server loading the target model.
[0082] Specifically, communication resource consumption mainly occurs in target tasks such as model loading. Model loading requires the transmission of static weights, resulting in significant bandwidth consumption. The communication resource utilization rate is defined as the data transmission time of the first target task divided by the end-to-end latency, where the end-to-end latency is the latency from the client sending the first target task to the server.
[0083] In the embodiments of this disclosure, the initial value of the communication resource utilization rate can be obtained in advance through theoretical calculation. Specifically, for the target task of model loading, the theoretical transmission time is the number of model parameters divided by the network bandwidth; therefore, the initial value of the communication utilization rate is the theoretical transmission time divided by the end-to-end delay. During model loading, the communication resource utilization rate is dynamically updated based on the real-time transmission rate.
[0084] Operation S1120 obtains the computational resource parameters of the second target task. The computational resource parameters represent the amount of computational resources required by the server during the process of calling the target model to execute the inference task.
[0085] Specifically, computational resource consumption mainly occurs in target tasks such as invoking the target model for inference. When executing this task, the server's GPU primarily executes kernel functions. Therefore, the computational resource utilization rate can be considered equivalent to the average GPU utilization rate during the target model inference task. The initial value of the computational resource utilization rate can be obtained through theoretical calculations. First, the theoretical computation time is defined as the number of target model parameters multiplied by the computational complexity divided by the server's GPU computing power. Then, the initial value of the computational resource utilization rate is the theoretical computation time divided by the end-to-end inference latency. During the execution of the target model inference task, the computational resource utilization rate is dynamically updated based on the real-time GPU utilization rate.
[0086] By adopting the above method, corresponding communication resource parameters and computing resource parameters are obtained for different target task types. This enables more reasonable allocation of server resources in GPU server sharing scenarios based on the specific resource requirements of different types of target tasks, thereby reducing resource contention and communication latency.
[0087] According to an embodiment of this disclosure, in operation S130, a second resource state is determined based on a first resource state and task parameters when each server executes a target task, including at least one of operations S1340 to S1350.
[0088] Operation S1340 determines the communication resource status of each server when executing the first target task based on the first resource status and communication resource parameters.
[0089] Specifically, for the model loading task, based on the communication resource parameters required for the model loading task and the first resource state of each server before executing the model loading task, the communication resource state of each server when executing the model loading task is predicted, thus obtaining the communication resource state of each server when executing the model loading task.
[0090] Operation S1350 determines the computing resource status of each server when executing the second target task, based on the first resource status and computing resource parameters.
[0091] Specifically, for the inference task of calling the target model, based on the computing resource parameters required to call the target model and the first resource state of each server before executing the model loading task, the computing resource state of each server when executing the inference task is predicted, so as to obtain the computing resource state of each server when executing the model loading task.
[0092] By employing the above method, the communication resource status is determined for the first target task and the computing resource status is determined for the second target task, which can more accurately grasp the resource status of the server when executing different tasks. Then, based on the second resource status of each server when executing the target task, the server is rationally selected to execute the target task, effectively alleviating the communication delay problem caused by resource contention.
[0093] like Figure 2 As shown, according to an embodiment of this disclosure, in operation S1330, the communication latency of each server when executing the target task is determined based on the latency parameter and the task parameter, including operations S210 to S230.
[0094] Operate S210 to obtain the remaining execution time of other tasks currently being executed on each server.
[0095] In this embodiment of the disclosure, the other tasks are the first target task and the second target task that are being executed on the server in response to the input of the target task.
[0096] Specifically, for other tasks loading the target model, the total number of bytes of the target model weight is fixed. The remaining execution time ratio is determined by dividing the number of bytes already transferred by the total number of bytes. The remaining execution time is: (100% - the ratio of the number of bytes already transferred to the total number of bytes) × the loading latency under no contention conditions.
[0097] Specifically, for other tasks that call the target model to perform inference tasks, the process of performing the inference task is broken down into the generation stage of the first word and the generation stage of subsequent words. The inference latency in the absence of competition is correspondingly divided into... (The delay in the generation of the first lexical unit) and (Delay in subsequent word generation stage).
[0098] Specifically, regarding the delay in the generation stage of the first word element, the progress of the inference task is determined by the proportion of the number of synchronization calls already made to the total number of synchronization calls. The remaining time is (1 - number of synchronization calls already made / total number of synchronization calls) × .
[0099] Specifically, the remaining time for the delay in the subsequent lexical generation stage is (1 - number of synchronization calls / total number of synchronizations) × ×N, where N is the number of remaining lexical units.
[0100] In the embodiments of this disclosure, the number of times synchronization has been called refers to the cumulative number of times the server actively calls the CUDA synchronization class API, that is, the CUDA device synchronization operation, which is the instruction interaction synchronization between the CPU and the GPU.
[0101] For example, the communication latency of the target model in the no-contention case is decomposed into =0.5s and =0.01s. If the total number of tokens is 100 and 10 tokens have been generated, then the remaining execution time is 0.5 + 0.01 × 99 – 0.5 – 0.01 × 10 = 0.01 × 89 = 0.89s.
[0102] Operation S220 determines the target execution time for each server to perform the target task based on the task parameters.
[0103] In the embodiments of this disclosure, the target execution time of each server executing the target task can be obtained by referring to the method of obtaining the remaining execution time of other tasks.
[0104] Specifically, for the first objective task, the target execution time is: 100% × loading latency under no-contention conditions. For the second objective task, the target execution time is also divided into the first word generation stage and the subsequent word generation stage; in the first word generation stage, the remaining time is (1 - number of synchronization calls / total number of synchronizations) × In the subsequent lexical generation stage, the remaining time is (1 - number of synchronization calls / total number of synchronizations) × ×N, where N is the number of remaining tokens; since the second target task has not yet been executed, the number of synchronization calls / total number of synchronization calls is 0. Therefore, the target execution time of the second target task is + ×N.
[0105] Operation S230 determines the communication latency of each server when executing the target task based on the latency parameter, remaining execution time, and target execution time.
[0106] Specifically, the communication latency of each server when executing the target task is determined based on the following formula:
[0107] ,
[0108] in, It is the remaining execution time of the i-th task. It is the resource utilization rate of the i-th task. It is a delay parameter. This is the total number of tasks;
[0109] The target task is considered as a task that has just started executing on the server, and its target execution time is calculated as equal to the remaining execution time of other tasks being executed on the server.
[0110] In some scenarios, tasks on the server are sorted from longest to shortest remaining execution time to ensure that tasks with the longest remaining execution time are processed first, as they have a greater impact on overall server latency.
[0111] By adopting the above method, the delay parameter is determined based on the comparison between the resource utilization rate and the target utilization rate. When the resource utilization rate does not exceed the target utilization rate, it is determined as the first delay parameter, and when it exceeds the target utilization rate, it is determined as the second delay parameter that increases with the increase of the resource utilization rate. The impact of the resource utilization rate on the communication delay is dynamically adjusted according to the actual situation of the server resource utilization, thereby improving the prediction result of the communication delay when the server executes the target task.
[0112] According to an embodiment of this disclosure, in operation S1320, based on resource utilization, the latency parameter of each server is determined, including: if the resource utilization does not exceed the target utilization, the latency parameter is determined as a first latency parameter; if the resource utilization exceeds the target utilization, the latency parameter is determined as a second latency parameter, wherein the second latency parameter increases as the resource utilization increases.
[0113] Specifically, when resource utilization is low, i.e., not exceeding the target utilization rate, resource utilization has no significant impact on server communication latency. In this case, the latency parameter is configured as the first latency parameter, indicating that when resource utilization changes within a low range, the server's communication latency does not change with changes in resource utilization. When resource utilization is high, i.e., exceeding the target utilization rate, resource utilization has a significant impact on server communication latency. In this case, the latency parameter is configured as the second latency parameter, indicating that when resource utilization changes within a high range, the server's communication latency increases with increasing resource utilization.
[0114] For example, when concurrent target tasks are running, if the resource utilization rate does not exceed 1, the resources are not saturated, there is no significant competition, and the latency parameter is 1. When the total utilization rate exceeds 1, resource competition intensifies, and the latency increases linearly with the total utilization rate; the latency parameter then equals the total utilization rate. Configuring the latency parameter can prevent excessive competition between concurrent tasks from causing a surge in latency. For instance, when three target tasks with a resource utilization rate of 30% run concurrently, the total utilization rate is 0.9, and there is no additional increase in latency; when four tasks run concurrently, the resource utilization rate is 1.2, and the latency increases by 20%.
[0115] By adopting the above method, the delay parameter is determined based on the comparison between the resource utilization rate and the target utilization rate. Under the condition of low resource utilization rate, the impact of changes in resource utilization rate on communication delay is ignored, and under the condition of high resource utilization rate, the evaluation result of communication delay is linearly adjusted based on the change in the size of resource utilization rate, thereby improving the accuracy of communication delay evaluation results.
[0116] like Figure 3 As shown, according to an embodiment of this disclosure, in operation S140, the server whose second resource state meets the target condition is determined as the target server for executing the target task, including operations S310 to S330.
[0117] Operation S310 determines the resource usage of each server based on the second resource status of each server.
[0118] Operation S320 determines the predicted communication latency for each server in executing the target task based on the resource usage of each server and the task parameters of the target task.
[0119] Operate S330 to sort the predicted communication latency values of each server from low to high, and determine the server ranked first as the target server.
[0120] Specifically, when selecting a target server, the resource usage of each server is determined through its second resource status. Then, combined with the task parameters of the target task (such as required GPU memory size and estimated computation time), the predicted communication latency for each server to execute the task is calculated. Finally, the predicted communication latency values of each server are sorted from low to high, and the server with the highest predicted latency is selected as the target server. For example, if the predicted communication latency values for servers A, B, and C are 200ms, 300ms, and 150ms respectively, then server C is selected as the target server to execute the target task.
[0121] By employing the above method, resource occupancy is determined based on the server's second resource status, and communication latency prediction is determined in conjunction with task parameters. These predictions are then sorted from low to high, and the server with the highest latency is selected as the target server. This prioritizes the server with the lowest communication latency for executing the target task, effectively reducing communication latency and improving task execution efficiency and quality.
[0122] According to an embodiment of this disclosure, in operation S140, determining the server whose second resource state meets the target condition as the target server for performing the target task includes at least one of operation S340 and operation S350.
[0123] Operation S340: If the second resource status of multiple servers meets the target condition, determine the target server for executing the target task based on the task type of other tasks being executed on the multiple servers; wherein, the other tasks being executed on the target server are of the same task type as the target task.
[0124] Specifically, when the second resource states of multiple servers all meet the target conditions, the task types of other tasks being executed on these servers are further considered. Since tasks of the same type may share the same model weights or intermediate results, thereby improving task execution efficiency and cache hit rate, if a server is executing a task of the same type as the target task, for example, both being natural language processing tasks, then that server is selected as the target server.
[0125] Operation S350: If the second resource status of multiple servers meets the target conditions, determine the idle resources of multiple servers when executing the target task based on the second resource status of multiple servers; determine the target server for executing the target task based on the idle resources; wherein the idle resources of the target server are less than those of the other servers among the multiple servers.
[0126] Specifically, assuming multiple servers meet the requirements, the amount of idle resources (such as remaining GPU memory, available bandwidth, etc.) of each server when executing the target task is determined based on the second resource status of each server. Then, the server with the least amount of idle resources is selected as the target server (while still meeting the task execution requirements) to achieve more efficient and balanced resource utilization.
[0127] For example, if the target task requires less than 2GB of memory, and servers A, B, and C all meet the conditions for executing the target task, but server A has the least remaining GPU memory, only 2GB, while servers B and C have more remaining memory, 10GB and 20GB respectively, then server A is selected as the target server. This avoids resource waste while saving servers with more remaining resources for subsequent target tasks with high computing power requirements.
[0128] By employing the above methods, two ways to determine the target server are provided when the second resource states of multiple servers all meet the target conditions. The first method selects the target server based on the fact that other tasks being executed by the server are of the same type as the target task, leveraging the server's advantage in handling similar tasks to improve execution efficiency. The second method selects the target server based on the server's idle resources while executing the target task, choosing servers with fewer idle resources than other servers, which helps to utilize server resources more rationally. Furthermore, the above methods further optimize the server selection strategy in complex situations, better addressing resource contention and reducing communication latency in complex server cluster scenarios.
[0129] According to embodiments of this disclosure, in operation S120, obtaining the first resource status of a plurality of servers, the method further includes: updating the first resource status of the server in response to the completion of at least one task in the server; and / or, updating the first resource status of the target server in response to the assignment of a target task to a target server.
[0130] Specifically, the system continuously monitors server status to obtain the initial resource status of each server. When a target task completes, the resource status of that server is updated, for example, identifying released memory and computing resources, and reducing GPU utilization and bandwidth usage. Similarly, when a target task is assigned to a server, the resource status information of that server is also updated synchronously to reflect the latest resource usage. For example, if a target task is assigned to server C and uses 24GB of memory and increases GPU utilization to 40%, the initial resource status information of server C is updated accordingly.
[0131] By adopting the above-mentioned resource status update mechanism, changes in server resources can be reflected in a timely manner, providing data support for a more accurate assessment of the server's ability to perform target tasks, managing server resources more effectively, and reducing problems such as unreasonable resource allocation and communication delays caused by untimely resource status updates.
[0132] According to embodiments of this disclosure, the method further includes operations S150 to S160.
[0133] Operation S150 configures the target task in the waiting queue if the second resource status of each server does not meet the target conditions.
[0134] Operation S160, in response to the update of the first resource state of each server, redetermines the second resource state of each server.
[0135] Specifically, when the second resource status of each server does not meet the target conditions, the target task is placed in a waiting queue for processing. Simultaneously, the server status monitoring module continuously monitors changes in the first resource status of each server. Once the first resource status of a server is updated—for example, when the target task completes and releases resources, or when new resources are added—the server's second resource status is re-established to determine if any server meets the target conditions to execute the target task in the waiting queue. For instance, if server A completes a task and releases 16GB of memory, leaving sufficient memory to execute the target task in the waiting queue (requiring 12GB of memory), the target task is removed from the waiting queue and assigned to server A for execution.
[0136] By employing the above method, when the second resource state of each server does not meet the target conditions, the target task is configured in a waiting queue, and the second resource state is re-determined when the first resource state of the server is updated. This ensures that the target task can wait in an orderly manner when the current server resources are insufficient, and is re-evaluated and allocated after the resource state changes, avoiding resource waste or further exacerbation of communication delays caused by the temporary inability to execute tasks.
[0137] The following provides details of the methods of this disclosure in some application scenarios.
[0138] In some scenarios, the target task includes inference tasks and model loading tasks.
[0139] Task parameters for inference tasks include: total number of synchronizations, contention-free inference latency. Memory requirements.
[0140] The theoretical initial value of the computational resource utilization rate for inference tasks is calculated as follows:
[0141] Determine the target execution time of the reasoning task :
[0142] = (Number of inference model parameters × Computational complexity of a single parameter) / Server GPU computing power;
[0143] Determine the initial value of computational resource utilization. : = / .
[0144] The task parameters for model loading include: total number of bytes transferred (i.e., the amount of weight data corresponding to the number of model parameters) and contention-free loading latency. Memory requirements.
[0145] Theoretical initial value of communication resource utilization for model loading task The calculation method is as follows:
[0146] Determine the execution duration of the theoretical target : = Total bytes transmitted by the target model / Network bandwidth;
[0147] Determine the initial value of communication resource utilization. : = / .
[0148] For example, the configuration of each server in the server cluster is shown in the table below.
[0149] Table 1
[0150]
[0151] For example, for the four target tasks submitted by the client (two inference tasks and two model loading tasks), a status monitoring daemon thread is started to report the resource status of each server every 1 second. The reported information includes real-time GPU utilization (used to update computational utilization), real-time transmission rate (used to update communication utilization), memory usage, and bandwidth usage. At the same time, a status receiving thread is started to store the reported information in the server status table. Simultaneously, a latency predictor and scheduler are started, the initial values of the above computational utilization and communication utilization are loaded, and the scheduling logic is set to be called every 5 seconds to initialize the suspended queue and the assigned task table to empty.
[0152] Furthermore, the four target tasks are divided into inference groups (T1, T2) and loading groups (T3, T4), and the task parameters and memory requirements of each target task are determined as shown in the table below.
[0153] Table 2
[0154]
[0155] Furthermore, available servers were selected from the server cluster. For inference tasks, all four servers (ID0~ID3) had ≥6GB of memory and were available. For loading tasks, ID2 (RTX4070, only 12GB, after allocating 10GB, the remaining 2GB could not accommodate subsequent inference) was excluded, and the available servers were ID0 (20GB), ID1 (24GB), and ID3 (48GB).
[0156] Furthermore, the predicted delay is calculated, where the initial theoretical value is used for the first scheduling and the dynamically updated value reported by the server monitoring thread is used for subsequent scheduling.
[0157] The following example uses server ID2 to calculate the single-task prediction latency of inference task T1 on each server.
[0158] The server ID2 status includes: no active tasks, 0% GPU utilization, and 12GB of free memory.
[0159] Calculate the remaining time =(1−0 / 10)×3=3s
[0160] Calculate the latency parameter 'a'. The current total resource utilization of server ID2 is 0% + 30% = 30% ≤ 100%; therefore, latency parameter 'a' = 1.
[0161] Single-task prediction latency = a × =1×3=3s.
[0162] The following example uses ID2 to calculate the group prediction delay for inference task T1+T2.
[0163] Among these requirements, the number of groups G must be sufficient to ensure that the total utilization rate of computing resources is ≤100% and that the server's memory resources are sufficient to run inference tasks T1+T2.
[0164] When the single-task computational resource utilization rate is 30% and G=2, the total computational resource utilization rate is 30%×2=60%≤100%, and the delay parameter a=1.
[0165] The memory requirement is 6GB x 2 = 12GB, which perfectly matches the 12GB memory of ID2.
[0166] Therefore, reasoning tasks T1 and T2 can be grouped together, with no contention delay.
[0167] The following calculates the communication delay after inference tasks T1 and T2 are grouped together:
[0168] Step 1: Task sorting (T1 and T2 both have 3 seconds remaining, the order has no effect);
[0169] Step 2: Divide the time interval (only 1 interval: [0, 3s], inference tasks T1 and T2 are concurrent throughout);
[0170] Step 3: Calculate the interval delay = a × interval duration = 1 × (3 - 0) = 3s;
[0171] The communication delay after grouping is 3 seconds (when two tasks are executed simultaneously, the total time is equal to the time of a single task, with no additional delay).
[0172] Furthermore, the packet latency of inference tasks T1 and T2 on other servers is compared as follows:
[0173] ID0: When G=2, the total utilization rate is 60%, a=1, and the delay is 3s, but the memory is 20GB (8GB is left after allocating 12GB, and the utilization rate is lower than ID2).
[0174] ID1: The delay is 3 seconds when G=2, but ID1 has higher processor computing power and is more suitable for subsequent high-priority inference tasks.
[0175] The final result shows that ID2 is the optimal server for inference tasks T1+T2 (3s latency, fully compatible with memory).
[0176] In the embodiments of this disclosure, the determination of task grouping is based on the number of target tasks and the size of task occupancy. Target tasks of the same type, as well as multiple target tasks with a total resource occupancy of ≤1, are preferentially assigned to the same task group; in some scenarios, the GPU memory requirements of the target tasks (sorted in descending order) are also given priority, and the predicted latency of the server is also considered in conjunction with the determination.
[0177] The following calculation uses server ID1 as an example to calculate the prediction latency of model loading task T3.
[0178] Load task parameters for task T3: =69%, =41.5s, total number of bytes transmitted Q=0.
[0179] Server ID1 status: No active tasks, 24GB of memory free, current bandwidth usage 0Mbps;
[0180] Global bandwidth utilization + loading task T3 =0%+69%=69%<100%, meaning that the bandwidth of server ID1 is not saturated at this time, and the target task can be allocated.
[0181] Calculate the remaining time =(1−0 / 10)×41.5=41.5s;
[0182] Calculate the total latency (loading + subsequent inference);
[0183] Loading delay: a = 1 (no bandwidth contention) × 41.5s = 41.5s;
[0184] Prediction delay = 41.5s.
[0185] The following calculation uses server ID0 as an example to calculate the prediction latency of model loading task T4.
[0186] Global bandwidth utilization + loading task T4 =69%+69%=138%>1;
[0187] Prediction delay = 1.38 × 41.5s = 57.3s;
[0188] Total delay = 41.5s.
[0189] Therefore, the model loading task T4 does not meet the principle of controllable delay and is placed in the suspension queue.
[0190] Among the above latency prediction results, the server with the lowest total latency is selected. To further consider the inference efficiency after loading the model, the performance of different servers for this task can also be statistically analyzed. The numerical value is preferred. The smallest server.
[0191] It should be noted that the resource configuration of each server in the above server group and the type configuration of the target task are illustrative examples and are not intended to limit the processing method disclosed herein.
[0192] A second aspect of this disclosure provides an electronic device, comprising:
[0193] At least one processor;
[0194] and memory connected to at least one processor; wherein,
[0195] The memory stores instructions that can be executed by at least one processor, and the instructions, when executed by at least one processor, enable the at least one processor to perform the following operations:
[0196] In response to the target task input, the task parameters of the target task are obtained. The task parameters represent the server resources required to execute the target task.
[0197] The first resource status of multiple servers is obtained. These servers are used to load the target model and call the target model to execute inference tasks. The first resource status represents the resource status of each server before executing the target task.
[0198] Based on the first resource status and task parameters, determine the second resource status of each server in executing the target task. The second resource status includes at least the resource status of the corresponding server in executing the target task.
[0199] Servers whose second resource status meets the target conditions are identified as target servers for executing target tasks, which include loading the target model and / or calling the target model to execute inference tasks.
[0200] According to embodiments of this disclosure, at least one processor is also capable of performing the following operations:
[0201] Based on the initial resource status and task parameters, determine the resource utilization rate of each server in executing the target task;
[0202] Based on resource utilization, the latency parameters of each server are determined. The latency parameters characterize the degree of increase in communication latency when the corresponding server executes the target task.
[0203] Based on latency parameters and task parameters, the communication latency of each server when executing the target task is determined.
[0204] According to embodiments of this disclosure, at least one processor is also capable of performing the following operations:
[0205] Obtain the task parameters of the target task, including: obtaining the communication resource parameters of the first target task, which represent the communication resource requirements of the server during the process of loading the target model; and obtaining the computational resource parameters of the second target task, which represent the computational resource requirements of the server during the process of calling the target model to execute the inference task.
[0206] According to embodiments of this disclosure, at least one processor is also capable of performing the following operations:
[0207] Based on the first resource status and communication resource parameters, determine the communication resource status of each server when executing the first target task;
[0208] Based on the first resource status and computing resource parameters, determine the computing resource status of each server when executing the second target task.
[0209] According to embodiments of this disclosure, at least one processor is also capable of performing the following operations:
[0210] Get the remaining execution time of other tasks currently being executed on each server;
[0211] Based on the task parameters, determine the target execution time for each server to execute the target task;
[0212] Based on latency parameters, remaining execution time, and target execution time, the communication latency of each server when executing the target task is determined.
[0213] According to embodiments of this disclosure, at least one processor is also capable of performing the following operations:
[0214] If the resource utilization rate does not exceed the target utilization rate, the delay parameter is determined as the first delay parameter;
[0215] When the resource utilization rate exceeds the target utilization rate, the delay parameter is determined as the second delay parameter, which increases as the resource utilization rate increases.
[0216] According to embodiments of this disclosure, at least one processor is also capable of performing the following operations:
[0217] Based on the second resource status of each server, determine the resource usage of each server;
[0218] Based on the resource usage of each server and the task parameters of the target task, determine the predicted communication latency for each server in executing the target task;
[0219] The predicted communication latency of each server is sorted from low to high, and the server ranked first is identified as the target server.
[0220] According to embodiments of this disclosure, at least one processor is also capable of performing the following operations:
[0221] If the second resource status of multiple servers meets the target conditions, the target server for executing the target task is determined based on the task type of other tasks being executed on the multiple servers; wherein, the other tasks being executed on the target server are of the same task type as the target task.
[0222] If the second resource status of multiple servers meets the target conditions, the idle resources of the multiple servers when executing the target task are determined based on the second resource status of the multiple servers; based on the idle resources, the target server for executing the target task is determined; wherein, the idle resources of the target server are less than those of the other servers among the multiple servers.
[0223] According to embodiments of this disclosure, at least one processor is also capable of performing the following operations:
[0224] In response to the completion of at least one task on the server, update the first resource state of the server;
[0225] And / or, in response to the target task being assigned to the target server, update the first resource state of the target server.
[0226] According to embodiments of this disclosure, at least one processor is also capable of performing the following operations:
[0227] If the second resource status of each server does not meet the target conditions, the target task will be configured in the waiting queue.
[0228] In response to an update in the first resource state of each server, the second resource state of each server is redefined.
[0229] Figure 4 A block diagram of an electronic device suitable for implementing the data processing method described above, according to an embodiment of the present disclosure, is illustrated schematically. Figure 4 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.
[0230] like Figure 4 As shown, an electronic device 400 according to an embodiment of the present disclosure includes a processor 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage portion 408 into a random access memory (RAM) 403. The processor 401 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 401 may also include onboard memory configured for caching purposes. The processor 401 may include a single processing unit or multiple processing units configured to perform different actions of the method flow according to an embodiment of the present disclosure.
[0231] RAM 403 stores various programs and data required for the operation of electronic device 400. Processor 401, ROM 402, and RAM 403 are interconnected via bus 404. Processor 401 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 402 and / or RAM 403. It should be noted that the programs may also be stored in one or more memories other than ROM 402 and RAM 403. Processor 401 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.
[0232] According to embodiments of this disclosure, the electronic device 400 may further include an input / output (I / O) interface 405, which is also connected to a bus 404. The electronic device 400 may also include one or more of the following components connected to the input / output (I / O) interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the input / output (I / O) interface 405 as needed. A removable medium 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 410 as needed so that computer programs read from it can be installed into the storage section 408 as needed.
[0233] According to embodiments of this disclosure, the method flow according to embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code configured to perform the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by processor 401, it performs the functions defined in the system of embodiments of this disclosure. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.
[0234] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.
[0235] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
[0236] For example, according to embodiments of this disclosure, a computer-readable storage medium may include the ROM 402 and / or RAM 403 described above and / or one or more memories other than ROM 402 and RAM 403.
[0237] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code configured to perform the methods provided in the embodiments of this disclosure. When the computer program product is run on an electronic device, the program code is configured to cause the electronic device to implement the processing methods provided in the embodiments of this disclosure.
[0238] When the computer program is executed by the processor 401, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.
[0239] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via communication section 409, and / or installed from removable medium 411. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.
[0240] According to embodiments of this disclosure, program code configured to execute the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages include, but are not limited to, languages such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).
[0241] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions configured to perform a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features described in the various embodiments of the present disclosure can be combined and / or combined in various ways, even if such combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.
[0242] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.
Claims
1. A processing method, comprising: In response to the target task input, the task parameters of the target task are obtained, wherein the task parameters represent the server resources required to execute the target task; Obtain the first resource status of multiple servers, which are used to load the target model and call the target model to perform inference tasks; The first resource state represents the resource state of each server before executing the target task; Based on the first resource state and the task parameters, a second resource state is determined for each of the servers executing the target task. The second resource state characterizes the resource state of each server during the execution of the target task. The server whose second resource status meets the target condition is determined as the target server for executing the target task, which includes loading the target model and / or calling the target model to execute the inference task.
2. The method according to claim 1, wherein determining a second resource state for each of the servers executing the target task based on the first resource state and the task parameters includes: Based on the first resource status and the task parameters, determine the resource utilization rate of each server in executing the target task; Based on the resource utilization rate, the latency parameters of each server are determined, and the latency parameters represent the degree of increase in communication latency when the corresponding server executes the target task; Based on the latency parameter and the task parameter, the communication latency of each of the servers when executing the target task is determined.
3. The method according to claim 1 or 2, wherein the target task includes at least one of a first target task and a second target task; Obtaining the task parameters of the target task includes: Obtain the communication resource parameters of the first target task, wherein the communication resource parameters represent the communication resource requirements of the server during the loading of the target model; Obtain the computational resource parameters of the second target task, wherein the computational resource parameters represent the amount of computational resources required by the server during the process of calling the target model to execute the inference task.
4. The method according to claim 3, wherein determining the second resource state of each of the servers when executing the target task based on the first resource state and the task parameters includes at least one of the following: Based on the first resource status and the communication resource parameters, the communication resource status of each server when executing the first target task is determined; Based on the first resource status and the computing resource parameters, the computing resource status of each server when executing the second target task is determined.
5. The method according to claim 2, wherein determining the communication latency of each of the servers when executing the target task based on the latency parameter and the task parameter includes: Obtain the remaining execution time of other tasks currently being executed by each of the aforementioned servers; Based on the task parameters, the target execution time for each of the servers to execute the target task is determined; Based on the latency parameter, the remaining execution time, and the target execution time, the communication latency of each of the servers when executing the target task is determined.
6. The method according to claim 2, wherein determining the latency parameters of each of the servers based on the resource utilization rate includes: If the resource utilization rate does not exceed the target utilization rate, the delay parameter is determined as the first delay parameter; If the resource utilization rate exceeds the target utilization rate, the delay parameter is determined as a second delay parameter, wherein the second delay parameter increases as the resource utilization rate increases.
7. The method according to claim 1, wherein determining the server whose second resource state meets the target condition as the target server for executing the target task comprises: Based on the second resource status of each server, determine the resource usage of each server; Based on the resource usage of each server and the task parameters of the target task, determine the predicted communication latency for each server in executing the target task; The predicted communication latency values of each server are sorted from low to high, and the server ranked first is determined as the target server.
8. The method according to claim 1 or 7, wherein determining the server whose second resource state satisfies the target condition as the target server for executing the target task includes at least one of the following: If the second resource status of multiple servers meets the target condition, the target server for executing the target task is determined based on the task types of other tasks being executed on the multiple servers; wherein... The other tasks being executed by the target server are of the same type as the target task. If the second resource status of multiple servers meets the target condition, the idle resources of the multiple servers when executing the target task are determined based on the second resource status of the multiple servers. Based on the available resources, a target server for executing the target task is determined; wherein the available resources of the target server are less than those of the other servers among the plurality of servers.
9. The method according to claim 1, further comprising obtaining the first resource status of multiple servers: In response to the completion of at least one task in the server, update the first resource state of the server; And / or, in response to the target task being assigned to the target server, update the first resource state of the target server.
10. An electronic device, comprising: At least one processor; and a memory connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform the following operations: In response to the target task input, the task parameters of the target task are obtained, wherein the task parameters represent the server resources required to execute the target task; The first resource status of multiple servers is obtained. These servers are used to load the target model and call the target model to perform inference tasks. The first resource status represents the resource status of each server before executing the target task. Based on the first resource status and the task parameters, a second resource status is determined for each of the servers executing the target task. The second resource status includes at least the resource status of the corresponding server in executing the target task. The server whose second resource status meets the target condition is determined as the target server for executing the target task, which includes loading the target model and / or calling the target model to execute the inference task.