Load balancing method and device for GPU (Graphics Processing Unit) resources and computer equipment

Abstract

The invention relates to a load balancing method and device for GPU resources and computer equipment, and relates to the technical field of cloud computing. The method comprises the following steps: in response to a GPU instance calling request initiated by a GPU resource user, obtaining an instance address of each GPU instance from a pre-configured GPU cluster; according to the instance address of each GPU instance, querying to obtain metadata of each GPU instance, and detecting to obtain load condition description data of each GPU instance; and according to the metadata and the load condition description data, screening from the GPU cluster to obtain a GPU instance meeting the condition as a target instance, and sending an instance address of the target instance as a response of the GPU instance call request to a GPU resource user, so that the GPU resource user calls computing resources of the target instance. By adopting the method, the load balancing effect of GPU instance resources can be improved.

Description

Technical Field

[0001] The present application relates to the field of cloud computing technology, and in particular to a method, apparatus, computer equipment, computer-readable storage medium, and computer program product for load balancing GPU resources. Background Art

[0002] In the field of cloud computing, current load balancing technologies include random, polling, and consistent hashing. The random method refers to the method of randomly obtaining a certain instance IP under the service for request; the polling method refers to each request requesting the A, B, and C instances under the service in sequence; consistent hashing refers to evenly distributing access requests to service instances according to different hash factors, and maintaining even distribution when the number of instances changes. Requests with the same hash factor calculation results will be dispatched to the same instance machine.

[0003] However, the random method is only applicable to CPU instance resources and cannot perceive the GPU load situation. It can only randomly and blindly request any GPU instance. The polling method uses CPU instance resources that can process thousands or tens of thousands of user requests at the same time if the machine load permits. There is no control over the number of requests to each instance at the same time. In the consistent hashing method, CPU instances do not naturally have affinity, and do not need to load a large amount of special resources when executing tasks. The efficiency of general request tasks distributed to any CPU instance is basically the same.

[0004] In summary, the current method of load balancing for GPU instance resources is blind in the instance allocation process, lacks consideration of multiple requests at the same time, and lacks the option of achieving machine affinity, resulting in poor load balancing effect for GPU instance resources. Summary of the invention

[0005] Based on this, it is necessary to provide a GPU resource load balancing method, apparatus, computer equipment, computer-readable storage medium and computer program product that can improve the effect of load balancing of GPU instance resources in response to the above technical problems.

[0006] In a first aspect, an embodiment of the present application provides a method for load balancing of GPU resources. The method comprises:

[0007] In response to a GPU instance call request initiated by a GPU resource user, obtaining an instance address of each of the GPU instances from a preconfigured GPU cluster;

[0008] According to the instance address of each GPU instance, query and obtain metadata of each GPU instance, and detect and obtain load description data of each GPU instance; the metadata is used to reflect the task processing status of each GPU instance;

[0009] According to the metadata and the load description data, the GPU instance that meets the conditions is screened from the GPU cluster as the target instance, and the instance address of the target instance is sent to the GPU resource user as a response to the GPU instance call request, so that the GPU resource user can call the computing resources of the target instance.

[0010] In one embodiment, the metadata includes historically processed tasks and the number of tasks currently being processed simultaneously, and the filtering, based on the metadata and the load description data, to obtain the GPU instance that meets the conditions from the GPU cluster as the target instance includes:

[0011] Obtain the task type of the current task corresponding to the GPU instance call request;

[0012] For any GPU instance in the GPU cluster, if a target condition is met, use the any GPU instance as the target instance;

[0013] Among them, the target condition is that the task type of the historically processed task is consistent with the task type of the current task, and the number of tasks currently being processed simultaneously and each load data in the load situation description data meet the upper limit requirements.

[0014] In one embodiment, the metadata includes an average response time, and the method further includes:

[0015] For any GPU instance in the GPU cluster, when the target condition is not met, the priority score of each GPU instance is calculated according to the value of each load data in the load situation description data and the average response time;

[0016] Any GPU instance with the highest priority score is used as the target instance.

[0017] In one embodiment, the querying obtains metadata of each GPU instance, including:

[0018] Initiate a query request to a distributed storage system according to the address of the GPU instance; the distributed storage system stores metadata of each of the GPU instances in the GPU cluster;

[0019] Receive metadata of each of the GPU instances in response to the query request from each of the distributed storage systems.

[0020] In one embodiment, the detecting and obtaining the load description data of each GPU instance includes:

[0021] Obtain an interface corresponding to each item of load data in the load condition description data;

[0022] According to the interface corresponding to each item of load data, read the each item of load data from the driver program in running state corresponding to each of the GPU instances;

[0023] The load data includes GPU usage, machine real-time video memory usage, physical temperature, and frame rate.

[0024] In one embodiment, the method further comprises:

[0025] Initiating a resource release request for the target instance when the GPU resource user has finished using the computing resources of the target instance; the resource release request is used to release the computing resources of the target instance;

[0026] When the computing resources of the target instance have been released, the metadata corresponding to the target instance is recalculated, and the recalculated metadata is updated to the distributed storage system.

[0027] In one embodiment, the method further comprises:

[0028] Establishing a service of the GPU cluster in a management component of the GPU cluster, adding the GPU instance to be added to the service of the GPU cluster, and obtaining the preconfigured GPU cluster;

[0029] The obtaining the instance address of each GPU instance includes:

[0030] The service name corresponding to the service of the GPU cluster is used as an input parameter to obtain the instance address of each GPU instance.

[0031] In one embodiment, the method further comprises:

[0032] Calculate the average response time of each GPU cluster in response to the call request based on the service function of the operation and maintenance side;

[0033] When the average response time is greater than a preset threshold, GPU instances are added to the GPU cluster to automatically expand the capacity of the GPU cluster; and when the average response time is less than or equal to the preset threshold, GPU instances are reduced in the GPU cluster to automatically shrink the capacity of the GPU cluster.

[0034] In a second aspect, the present application also provides a GPU resource load balancing device. The device includes:

[0035] An instance address acquisition module, configured to obtain the instance address of each GPU instance from a preconfigured GPU cluster in response to a GPU instance call request initiated by a GPU resource user;

[0036] A data query and detection module, used to query and obtain metadata of each GPU instance according to the instance address of each GPU instance, and detect and obtain load description data of each GPU instance; the metadata is used to reflect the task processing status of each GPU instance;

[0037] The target instance determination module is used to screen the GPU instance that meets the conditions from the GPU cluster as the target instance according to the metadata and the load description data, and send the instance address of the target instance as a response to the GPU instance call request to the GPU resource user, so that the GPU resource user can call the computing resources of the target instance.

[0038] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0039] In response to a GPU instance call request initiated by a GPU resource user, obtaining an instance address of each of the GPU instances from a preconfigured GPU cluster;

[0040] According to the instance address of each GPU instance, query and obtain metadata of each GPU instance, and detect and obtain load description data of each GPU instance; the metadata is used to reflect the task processing status of each GPU instance;

[0041] According to the metadata and the load description data, the GPU instance that meets the conditions is screened from the GPU cluster as the target instance, and the instance address of the target instance is sent to the GPU resource user as a response to the GPU instance call request, so that the GPU resource user can call the computing resources of the target instance.

[0042] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0043] In response to a GPU instance call request initiated by a GPU resource user, obtaining an instance address of each of the GPU instances from a preconfigured GPU cluster;

[0044] According to the instance address of each GPU instance, query and obtain metadata of each GPU instance, and detect and obtain load description data of each GPU instance; the metadata is used to reflect the task processing status of each GPU instance;

[0045] According to the metadata and the load description data, the GPU instance that meets the conditions is screened from the GPU cluster as the target instance, and the instance address of the target instance is sent to the GPU resource user as a response to the GPU instance call request, so that the GPU resource user can call the computing resources of the target instance.

[0046] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0047] In response to a GPU instance call request initiated by a GPU resource user, obtaining an instance address of each of the GPU instances from a preconfigured GPU cluster;

[0048] According to the instance address of each GPU instance, query and obtain metadata of each GPU instance, and detect and obtain load description data of each GPU instance; the metadata is used to reflect the task processing status of each GPU instance;

[0049] According to the metadata and the load description data, the GPU instance that meets the conditions is screened from the GPU cluster as the target instance, and the instance address of the target instance is sent to the GPU resource user as a response to the GPU instance call request, so that the GPU resource user can call the computing resources of the target instance.

[0050] The above-mentioned GPU resource load balancing method, device, computer equipment, storage medium and computer program product first obtain the instance address of each GPU instance from a preconfigured GPU cluster, then query and obtain the metadata of each GPU instance, detect and obtain the load description data of each GPU instance, and then use the above data to screen out qualified GPU instances from the GPU cluster as target instances, and send the instance address as a response to the GPU instance call request to the GPU resource user for the GPU resource user to call the resource; allocate computing resources by building a GPU cluster, and use the actual load condition of the GPU and the task processing condition in the metadata as conditions for instance screening in the GPU cluster, and consider the real-time processing capability of each GPU instance and the influence of the affinity of the machine on load balancing, thereby improving the scientificity and efficiency of the GPU instance allocation process, and maximizing the use of the available GPU machine computing power on the basis of ensuring the traffic availability in different task scenarios, thereby improving the effect of load balancing for GPU instance resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0052] Figure 1 An application environment diagram of a method for load balancing GPU resources in one embodiment;

[0053] Figure 2 A flowchart of a method for load balancing GPU resources in one embodiment;

[0054] Figure 3 A schematic diagram of a flow chart of a method for load balancing GPU resources in another embodiment;

[0055] Figure 4 It is an overall architecture diagram in one embodiment;

[0056] Figure 5 A computing service timing diagram of a GPU instance acquisition process in one embodiment;

[0057] Figure 6 A computing service timing diagram of a GPU instance release process in one embodiment;

[0058] Figure 7 is a structural block diagram of a load balancing device for GPU resources in one embodiment;

[0059] Figure 8 is an internal structure diagram of a computer device in one embodiment;

[0060] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0062] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0063] The GPU resource load balancing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 through a network.

[0064] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0065] It should be noted that in the computer field, an instance refers to a specific object of a class, which is usually a concept used in object-oriented programming. The instance involved in this application can represent the specific IP and port address of a GPU.

[0066] In one embodiment, Figure 2 As shown, a GPU resource load balancing method is provided, and the method is applied to Figure 1 The terminal 102 in the example is used as an example to illustrate, and the following steps are included:

[0067] S201 , in response to a GPU instance call request initiated by a GPU resource user, obtaining instance addresses of each GPU instance from a preconfigured GPU cluster.

[0068] The GPU resource user refers to an object that has a GPU resource call requirement. For example, when a CPU needs to use the GPU's computing resources for image processing tasks, the CPU is the GPU resource user. Correspondingly, a GPU instance call request is a data request that needs to call the GPU instance's resources.

[0069] Among them, before the load balancing method of GPU resources is implemented, the GPU cluster needs to be pre-configured. The configuration process includes determining the service name of the GPU cluster in the unified name service component and setting the instance address of each GPU instance resource in the GPU cluster, that is, the IP address of each GPU instance, to facilitate subsequent data acquisition and instance resource selection based on the IP address.

[0070] S202: According to the instance address of each GPU instance, query and obtain metadata of each GPU instance, and detect and obtain load description data of each GPU instance.

[0071] The metadata is used to reflect the task processing status of each GPU instance. For example, the metadata may include historically processed tasks (e.g., the tasks processed last time or recently), the number of tasks currently being processed simultaneously, and the average response time (referring to the average response time of the GPU machine). The load situation description data is data used to describe the load situation, such as GPU utilization, machine real-time video memory utilization, physical temperature, frame rate, etc.

[0072] S203, according to the metadata and load description data, screen out a qualified GPU instance from the GPU cluster as a target instance, and send the instance address of the target instance as a response to the GPU instance call request to the GPU resource user, so that the GPU resource user can call the computing resources of the target instance.

[0073] The conditions for selecting the target instance from the GPU cluster may be: the task type of the historically processed task is consistent with the task type of the current task, and the number of tasks currently being processed simultaneously and the various load data in the load situation description data meet the upper limit requirements.

[0074] In the above-mentioned GPU resource load balancing method, the instance address of each GPU instance is first obtained from the preconfigured GPU cluster, and then the metadata of each GPU instance is queried to obtain the load description data of each GPU instance, and then the above-mentioned data is used to screen out qualified GPU instances from the GPU cluster as target instances, and the instance address is sent to the GPU resource user as a response to the GPU instance call request, so that the GPU resource user can make resource calls; computing resources are allocated by building a GPU cluster, and the actual load situation of the GPU and the task processing situation in the metadata are used as conditions for instance screening in the GPU cluster, and the real-time processing capability of each GPU instance and the affinity of the machine are considered to affect load balancing, thereby improving the scientificity and efficiency of the GPU instance allocation process, and maximizing the use of available GPU machine computing power on the basis of ensuring traffic availability in different task scenarios, thereby improving the effect of load balancing for GPU instance resources.

[0075] In one embodiment, according to the metadata and the load description data, a GPU instance that meets the conditions is screened from the GPU cluster as a target instance, including: obtaining the task type of the current task corresponding to the GPU instance call request; for any GPU instance in the GPU cluster, if the target condition is met, any GPU instance is used as the target instance;

[0076] The metadata includes historically processed tasks and the number of tasks currently being processed simultaneously. The target condition is that the task type of the historically processed tasks is consistent with the current task, and the number of tasks currently being processed simultaneously and the various load data in the load situation description data meet the upper limit requirements. The various load data in the load situation description data include GPU usage, machine real-time video memory usage, physical temperature, frame rate, etc.

[0077] Specifically, a specific criterion for determining whether the task type of a historically processed task is consistent with that of the current task may be that the current task and the historical task have the same task identifier.

[0078] Exemplarily, based on the returned metadata, determine whether there is a task identical to this request in the historical tasks. If so, make an identification record, and then determine whether the number of tasks processed simultaneously by the machine has reached an upper limit and whether the load has reached an upper limit. If neither upper limit has been reached, use any GPU instance corresponding to the metadata as the target instance.

[0079] In this embodiment, the GPU instance to be called is selected by comparing the task types of the current task with those of historical tasks, and the upper limit of the current number of tasks and load data is taken into consideration. On the premise of ensuring the availability of the GPU instance, instances are allocated according to the similarity of the tasks, thereby improving the execution efficiency of the tasks.

[0080] In one embodiment, the method also includes: for any GPU instance in the GPU cluster, when the target condition is not met, calculating the priority score of each GPU instance based on the value of each load data in the load situation description data and the average response time; and taking any GPU instance with the highest priority score as the target instance.

[0081] Among them, the metadata also includes the average response time. The target condition is not met, which means that the task type of the historically processed task is inconsistent with the current task, or the number of tasks currently being processed simultaneously and the load data in the load situation description data do not meet the upper limit requirements.

[0082] For example, the values ​​of various load data and the average response time can be used as indicators, corresponding coefficients can be set for each indicator, and weighted sum calculations can be performed based on the data size and coefficients of each indicator to determine the priority score corresponding to each GPU instance. Generally, the higher the score, the higher the priority of the GPU instance in being selected as the target instance.

[0083] For example, the GPU physical monitoring index is obtained, and the priority score of each GPU instance is calculated cyclically based on the average response time and the physical monitoring index. If there is no task identical to the current request in the historical tasks, or the number of tasks processed simultaneously reaches the upper limit, or the load reaches the upper limit, then the highest instance is selected based on the calculated priority score, and the IP address of the instance is returned.

[0084] In this embodiment, when no similar tasks are found in the task type judgment process, or when similar tasks exist but indicators such as the load of the corresponding GPU instance have reached the upper limit, GPU instances are allocated based on the size of the priority score, thereby ensuring the stability of GPU instance allocation in the GPU cluster.

[0085] In one embodiment, querying to obtain metadata of each GPU instance includes: initiating a query request to a distributed storage system according to an address of the GPU instance; and receiving metadata of each GPU instance in response to the query request from each distributed storage system.

[0086] Among them, the distributed storage system stores the metadata of each GPU instance in the GPU cluster. It can be a self-developed distributed Key-Value storage system, or it can be implemented in other distributed components, such as zookeepr, memcached, etc., and you can choose whether to prefer high-availability or high-performance components based on the business scenario.

[0087] In this embodiment, a distributed storage system is set up for the GPU computing service process to access and manage the metadata of the GPU instance usage process, facilitate the reading and updating of metadata, ensure the availability of data and the timeliness of data reading and updating, thereby ensuring the timeliness and accuracy of the instance resource allocation process.

[0088] In one embodiment, the load condition description data of each GPU instance is detected, including: obtaining an interface corresponding to each load data item in the load condition description data; and reading each load data item from a running driver corresponding to each GPU instance according to the interface corresponding to each load data item.

[0089] The load data includes GPU usage, machine real-time video memory usage, physical temperature, and frame rate. The interface corresponding to each load data item may be an API interface for obtaining each data item or a function related to data collection.

[0090] For example, information about each data acquisition interface can be obtained through the driver library corresponding to the computing resource provider, such as gonvml, and each GPU machine is configured with a driver during the installation process. Therefore, during the operation of each GPU instance, the data acquisition interface can be used to obtain various load data related to the GPU machine through the running driver.

[0091] In this embodiment, the interface provided in the driver library corresponding to the computing resource provider is utilized to read various load data from the running driver program, thereby realizing real-time detection of load condition description data, improving the timeliness and accuracy of data acquisition, and thus ensuring the timeliness and accuracy of the instance resource allocation process.

[0092] In one embodiment, the method also includes: when the GPU resource user has finished using the computing resources of the target instance, initiating a resource release request for the target instance; when the computing resources of the target instance have been released, recalculating the metadata corresponding to the target instance, and updating the recalculated metadata to the distributed storage system.

[0093] Among them, the end of computing resource usage means that the target instance has completed the processing of the planned task after calling the instance. The resource release request is used to instruct the computer device to release the computing resources of the target instance. The objects that need to be recalculated in the metadata include historically processed tasks, the current number of tasks being processed simultaneously, and the average response time.

[0094] For example, after the user completes the task, the user returns the GPU instance IP_A and requests the solution to release it. IP_A is used as the key to obtain the corresponding metadata from the distributed storage system to recalculate the average response time, update the unique identifier of the cached processing task, set the last processing task as the current task, and reduce the number of processing tasks by one. Finally, the metadata is updated to the distributed storage system.

[0095] In this embodiment, when the computing resources of the target instance are used up, the resources are released, and the data related to the task processing in the metadata is updated, thereby ensuring the accuracy of the metadata and thus ensuring the accuracy of the instance resource allocation process.

[0096] In one embodiment, the method also includes: establishing a service for the GPU cluster in a management component of the GPU cluster, adding the GPU instance to be added to the service of the GPU cluster, and obtaining a preconfigured GPU cluster; obtaining the instance address of each GPU instance, including: using the service name corresponding to the service of the GPU cluster as an input parameter to obtain the instance address of each GPU instance.

[0097] The management component of the GPU cluster is the unified name service component, and the service of the GPU cluster is a service used to set and allocate instances in the cluster.

[0098] Exemplarily, the user needs to first create a new svr_1 service in the unified name service component and put all GPU instance machines under svr_1; the user uses svr_1 as an input parameter to call the instance acquisition interface of this solution; and obtains all configured GPU instance addresses in the unified name service component based on svr_1.

[0099] In this embodiment, by pre-configuring the service of the GPU cluster in the management component and using the service name as the basis for calling the instance address in the instance cluster, the calling accuracy in the allocation process of the GPU instance is improved.

[0100] In one embodiment, the method also includes: calculating the average response time of each GPU cluster in response to the call request based on the service function of the operation and maintenance side; when the average response time is greater than a preset threshold, adding GPU instances to the GPU cluster to automatically expand the capacity of the GPU cluster; and, when the average response time is less than or equal to the preset threshold, reducing GPU instances in the GPU cluster to automatically shrink the GPU cluster.

[0101] The average response time of each GPU cluster in responding to a call request may refer to the time from the initiation of the call request to the completion of task processing, which is used to characterize the speed of response of each GPU cluster, for example, the time of service addressing.

[0102] Exemplarily, the time of the svr_1 service addressing is asynchronously reported, and the operation and maintenance side expands or reduces the capacity of the GPU instance cluster under svr_1 according to specific rules.

[0103] It should be noted that the expansion on the operation and maintenance side is automatic expansion, and the reduction on the operation and maintenance side is automatic reduction. Specifically, the capacity can be expanded smoothly and automatically when there is a sudden flow, and the sudden flow can be dealt with in time without manual access; correspondingly, the capacity can also be automatically reduced when the flow is small, saving a lot of machine costs.

[0104] In this embodiment, the average response time of each GPU cluster in response to the call request is calculated, and the cluster is expanded or reduced in capacity according to the average response time, and the number of instances in the cluster is updated in real time. While improving the response efficiency of the instances in the cluster, the cost of using instance resources is reduced, thereby improving the load balancing effect of the GPU instances.

[0105] In another embodiment, Figure 3 As shown, a method for load balancing of GPU resources is provided, comprising the following steps:

[0106] S301, obtaining the task type of the current task corresponding to the GPU instance call request.

[0107] S302 , for any GPU instance in the GPU cluster, if a target condition is met, use any GPU instance as a target instance.

[0108] S303, for any GPU instance in the GPU cluster, when the target condition is not met, the priority score of each GPU instance is calculated according to the value of each load data in the load situation description data and the average response time.

[0109] S304: Use any GPU instance with the highest priority score as the target instance.

[0110] S305 , when the GPU resource user finishes using the computing resources of the target instance, a resource release request is initiated for the target instance.

[0111] S306: When the computing resources of the target instance have been released, recalculate the metadata corresponding to the target instance, and update the recalculated metadata to the distributed storage system.

[0112] It should be noted that the specific limitations of the above steps can refer to the specific limitations of a GPU resource load balancing method mentioned above, which will not be repeated here.

[0113] The current service load balancing technology includes the solution of implementing load balancing on the client side of microservices, which is specifically implemented by Ribbon components, etc. For example, the current implementation solution provides that the service called by the service will have three machine instances A, B, and C, which will be registered with the service registration center; the calling client will obtain the three available machine resources of the Service from the service registration center; the calling client will decide to send the request to a certain instance (one of A, B, and C) for processing based on the load balancing algorithm.

[0114] The current mainstream load balancing algorithms are: (1) Random: randomly obtain an instance IP under the service for request; (2) Polling: each request requests the A, B, and C instances under the service in sequence; (3) Consistent Hashing: the access requests are evenly distributed to the service instances according to different hash factors, and the even distribution is maintained when the number of instances changes. Requests with the same hash factor calculation results will be scheduled to the same instance machine. In addition, dynamic load balancing methods also include the minimum connection method, the shortest response time method, and the instance resource load-based method. These are scheduling methods that are dynamically calculated based on the actual processing conditions and performance indicators of the machine.

[0115] The shortcomings of the above scheme are: (1) it is only applicable to CPU instance resources, cannot sense the GPU load, and can only randomly and blindly request any GPU instance; (2) CPU instance resources can process thousands or tens of thousands of user requests at the same time if the machine load allows, and there is no control over the number of requests for each instance at the same time; (3) CPU instances do not naturally have affinity, and do not need to load a large amount of special resources when executing tasks. The efficiency of general request tasks distributed to any CPU instance is basically the same. Therefore, the above scheme does not implement machine affinity selection.

[0116] Based on this, an embodiment of the present application provides a load balancing method for GPU resources. Through the hardware driver gonvml officially provided by the chip manufacturer, the load status of each GPU instance can be obtained in real time, including: GPU utilization, video memory utilization, temperature, etc.; because the GPU machine instance can only process one or several tasks (not more than 20) at the same time, if the task limit carried by the GPU is exceeded, the machine will crash abnormally. Therefore, the distributed cache component CKV+ is used to record the number of tasks processed by each GPU at the same time, and the number of tasks of each instance is strictly controlled; due to the specificity of GPU resource processing, the efficiency of processing each different task is not the same. For repeated requests for the same task, this solution implements the best effort to forward the request to the previously processed GPU instance when the machine load permits, saving time for loading additional material resources; at the same time, the average time for GPU instance to process tasks can be counted, and the GPU machine with the best efficiency can be selected to process the request to achieve machine affinity.

[0117] Reference below Figures 4 to 6 , a specific embodiment is used to describe in detail a method for load balancing GPU resources. It is worth noting that the following description is only an exemplary description and not a specific limitation on the application.

[0118] The GPU resource load balancing method provided in this application is also called a GPU load balancing solution for CPU services. With the development of AIGC (Artificial Intelligence Generated Content) applications, the demand for GPU resources has also increased dramatically. The GPU resource load balancing method provided in this application can be used to load balance multiple GPU machine resources under the same algorithm service, so that the demand side can maximize the use of GPU machine computing power and can adapt to different business scenarios to ensure high availability, and implement affinity calculation for the machine to obtain the most efficient GPU machine.

[0119] The key technologies involved in the GPU resource load balancing method provided in this application include: (1) Using the components of the distributed storage system to manage multiple GPU machine resources. Each machine address is a key that stores the machine's metadata. Each operation uses the mget command to calculate all addresses under the same algorithm service, and the final result is obtained based on indicators such as machine occupancy (such as GPU utilization), recent task efficiency (such as average response time), machine real-time video memory utilization (such as video memory) and physical temperature. (2) Dynamically manage the GPU cluster under the same algorithm service, and realize automatic expansion and contraction of GPU machines by monitoring the time consumption of algorithm service load balancing to cope with sudden traffic of demand-side business and save the cost of redundant machines during low traffic. (3) Use priority scheduling algorithm to achieve load balancing.

[0120] The GPU resource load balancing method provided in this application performs load balancing on a GPU machine instance cluster for the first time, and supports dynamic expansion and contraction of a GPU machine instance cluster to ensure high availability and low cost. In addition, in terms of affinity implementation, the same task is bound to the same machine or the optimal machine is calculated based on the actual load and previous execution efficiency, thereby maximizing the execution efficiency of the task.

[0121] On the product side, the load balancing method of GPU resources provided in this application mainly serves the CPU services on the algorithm side, such as NLP (Natural Language Processing), TTS (Text To Speech), CV (Computer Vision), video production, etc.; the user usually creates a new service name in the unified name service component (a component used to configure the GPU cluster) in advance to configure one or more GPU machine instance resources; before actually requesting the GPU to execute the task, the load balancing of this solution will be used to obtain the available GPU address, without having to care about the status of all GPU machines, and a direct request will be made; after the task processing is completed, the load balancing will be requested again for release.

[0122] The GPU resource load balancing method provided in this application guarantees the availability and execution efficiency of the GPU to the greatest extent, because the architecture of the GPU and the particularity of the execution tasks determine that each GPU can only process one or more (generally within 20) tasks at the same time. Once the tasks exceed the GPU's tolerance, an abnormal crash will occur. The GPU resource load balancing method provided in this application can be applied to music retrieval and playback software, karaoke software, audio and video recording and editing software, and generative artificial intelligence software.

[0123] On the technical side, the overall architecture diagram of the GPU resource load balancing method provided in this application is as follows: Figure 4 As shown, the overall architecture includes users, unified name service components, distributed storage systems, computing services, GPU machine drivers, data warehouse tools, operation and maintenance services, user CPU services, and GPU machines.

[0124] Among them, the unified name service component is used to configure the GPU cluster. For example, it can be a unified name service component jointly built by internal open source. The name service is used to solve the service registration discovery, dynamic routing, load balancing and fault tolerance of remote calls (RPC). It should be noted that the platform for configuring GPU instances can be replaced with any service registration center. It only needs to realize the mapping of one service name to multiple GPU instance IP addresses.

[0125] The implementation of the load balancing method for GPU resources provided in this application requires the pre-configuration of the unified name service component. The configuration process may include: creating a new service in the service list, for example, a new service name, and configuring its corresponding environment type, department, application business and person in charge, and then configuring the instance IP address corresponding to each instance in the GPU cluster under the service name.

[0126] Among them, the distributed storage system is used to store and use metadata. It can be a self-developed distributed Key-Value storage system that is compatible with open source protocols such as Memcached and Redis. It has the characteristics of high performance, high availability, and high reliability, and can meet business scenarios such as high concurrency and low latency. It should be noted that the metadata storage of GPUs can be implemented in other distributed components, such as zookeepr, memcached, etc., depending on the business scenario, choose whether to prefer high availability or high performance, and select components based on the operation and maintenance direction.

[0127] The driver of the GPU machine may be a program in a driver library officially provided by a vendor providing computing services and used to obtain GPU graphics card performance monitoring indicators, such as a driver in a gonvml driver library.

[0128] Among them, data warehouse tools are used by the operation and maintenance side to access and process data such as service time. For example, Hive table is a Hadoop-based data warehouse tool used for data extraction, transformation and loading (ETL) operations.

[0129] The computing service timing diagram of the GPU instance acquisition process is as follows Figure 5 As shown in the figure, the computing service timing diagram of the GPU instance release process is as follows Figure 6 shown.

[0130] like Figure 5 As shown in the figure, the overall process of obtaining an instance includes:

[0131] (1) The user needs to first create a new svr_1 service in the unified naming service component and place all GPU instance machines under svr_1.

[0132] (2) The user uses svr_1 in step (1) as an input parameter to call the instance acquisition interface of this solution.

[0133] (3) First, obtain the addresses (IP addresses) of all configured GPU instances from the unified name service component based on svr_1.

[0134] (4) Loop through the metadata of all GPU addresses in the distributed storage system and obtain the metadata value using the IP as the key (the metadata may include the last / recently processed tasks, the number of tasks processed simultaneously, the average response time of the machine, etc.).

[0135] (5) Based on the returned metadata, determine whether there is a task that is the same as this request (input image, audio URL, etc.) (the most recent few items will be cached in the metadata). If so, make an identification record.

[0136] (6) Use the gonvml driver library to obtain GPU physical monitoring indicators (e.g., GPU usage, real-time machine memory usage, physical temperature, frame rate, etc.) from the driver of the vendor providing computing services, and cyclically calculate the priority score of each GPU instance based on the average response time and the actual machine load (also known as monitoring load or physical monitoring indicators). It should be noted that in terms of the selection of the priority selection algorithm, after calculating the priorities of all GPUs in the computing service, how to select the optimal GPU can be implemented according to different task scheduling algorithms or different factor weights, and this application does not limit the application of specific algorithms.

[0137] (7) In step (5), if there are identical task identifiers, determine whether the number of tasks that the machine can process simultaneously has reached the upper limit and whether the load has reached the upper limit. If none of these have reached the upper limit, it indicates that the machine is available and the IP is returned directly to end the process.

[0138] (8) If one of the conditions in step (7) is not met, the highest IP is selected based on the score in step (6) and returned.

[0139] (9) Update the metadata of the returned GPU instance IP and end the process.

[0140] Finally, the time of the svr_1 service addressing is asynchronously reported, and the operation and maintenance side expands or reduces the capacity of the GPU instance cluster under svr_1 according to specific rules.

[0141] like Figure 6 As shown in the figure, the overall process of releasing an instance includes:

[0142] (1) After the user completes the task, it returns the GPU instance IP_A and requests this solution to release it.

[0143] (2) Use IP_A as the key to obtain the corresponding metadata from the distributed storage system.

[0144] (3) Recalculate the average response time, update the unique identifier of the cached processing task (set the last processing task as the current task), and reduce the number of simultaneous processing tasks by 1.

[0145] (4) Update metadata to the distributed storage system.

[0146] (5) End the process.

[0147] The advantages of the GPU resource load balancing method provided in this application are: (1) It allows users (staff members conducting algorithm research) to focus on the effects of the algorithm model without having to worry about the use of multiple GPU machine resources or service crashes caused by GPU overload. (2) It maximizes the role of GPU machines. Currently, GPU resources such as A100, A10, and V100 are very scarce and expensive, and the demand for GPUs from various businesses is very large. The performance and efficiency of a single GPU can be maximized, which can relieve business users of their shortcomings. (3) Cost control: Linking the operation and maintenance side, monitoring the time consumption of the entire GPU cluster, and adjusting the cluster size in real time to cope with the traffic of different business scenarios, ensuring the normal operation of the business without wasting GPU machine resources. In actual online production environments, capacity expansion requirements can be completed in two minutes.

[0148] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0149] Based on the same inventive concept, the embodiment of the present application also provides a GPU resource load balancing device for implementing the GPU resource load balancing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiment of the load balancing device for one or more GPU resources provided below can refer to the limitations of the GPU resource load balancing method above, and will not be repeated here.

[0150] In one embodiment, Figure 7 As shown, a GPU resource load balancing device is provided, including: an instance address acquisition module 701, a data query and detection module 702, and a target instance determination module 703, wherein:

[0151] The instance address acquisition module 701 is used to obtain the instance address of each GPU instance from the pre-configured GPU cluster in response to the GPU instance call request initiated by the GPU resource user;

[0152] The data query and detection module 702 is used to query and obtain metadata of each GPU instance according to the instance address of each GPU instance, and detect and obtain load description data of each GPU instance; the metadata is used to reflect the task processing status of each GPU instance;

[0153] The target instance determination module 703 is used to describe the data according to the metadata and load conditions, screen out qualified GPU instances from the GPU cluster as target instances, and send the instance address of the target instance as a response to the GPU instance call request to the GPU resource user so that the GPU resource user can call the computing resources of the target instance.

[0154] In one embodiment, the device is used to: obtain the task type of the current task corresponding to the GPU instance call request; for any GPU instance in the GPU cluster, if the target condition is met, use any GPU instance as the target instance; wherein the target condition is that the task type of the historically processed task is consistent with the task type of the current task, and the number of tasks currently being processed simultaneously and the various load data in the load situation description data meet the upper limit requirements.

[0155] In one embodiment, the device is used to: for any GPU instance in a GPU cluster, when the target condition is not met, calculate the priority score of each GPU instance based on the values ​​of each load data in the load situation description data and the average response time; and use any GPU instance with the highest priority score as the target instance.

[0156] In one embodiment, the device is used to: initiate a query request to a distributed storage system according to the address of a GPU instance; the distributed storage system stores metadata of each GPU instance in a GPU cluster; and receive metadata of each GPU instance in response to the query request from each distributed storage system.

[0157] In one embodiment, the device is used to: obtain interfaces corresponding to various load data in the load condition description data; read various load data from the running drivers corresponding to each GPU instance according to the interfaces corresponding to the various load data; wherein the load data includes GPU utilization rate, machine real-time video memory utilization rate, physical temperature, and frame rate.

[0158] In one embodiment, the device is used to: initiate a resource release request for the target instance when the GPU resource user has finished using the computing resources of the target instance; the resource release request is used to release the computing resources of the target instance; when the computing resources of the target instance have been released, recalculate the metadata corresponding to the target instance, and update the recalculated metadata to the distributed storage system.

[0159] In one embodiment, the device is used to: establish a service for a GPU cluster in a management component of the GPU cluster, add the GPU instance to be added to the service of the GPU cluster, and obtain a preconfigured GPU cluster; obtain the instance address of each GPU instance, including: using the service name corresponding to the service of the GPU cluster as an input parameter to obtain the instance address of each GPU instance.

[0160] In one embodiment, the device is used to: calculate the average response time of each GPU cluster in response to a call request based on the service function of the operation and maintenance side; when the average response time is greater than a preset threshold, add GPU instances to the GPU cluster to automatically expand the capacity of the GPU cluster; and, when the average response time is less than or equal to the preset threshold, reduce GPU instances in the GPU cluster to automatically shrink the GPU cluster.

[0161] Each module in the above-mentioned GPU resource load balancing device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0162] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a load balancing method for GPU resources is implemented.

[0163] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a load balancing method for GPU resources is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0164] Those skilled in the art will understand that Figure 8 and Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0165] In one embodiment, a computer device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the steps in the above method embodiments are implemented.

[0166] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0167] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0168] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0169] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0170] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A GPU resource load balancing method, characterized in that: The method comprises: In response to a GPU instance call request initiated by a GPU resource user, obtaining an instance address of each of the GPU instances from a preconfigured GPU cluster; According to the instance address of each GPU instance, query and obtain metadata of each GPU instance, and detect and obtain load description data of each GPU instance; the metadata is used to reflect the task processing status of each GPU instance; According to the metadata and the load description data, the GPU instance that meets the conditions is screened from the GPU cluster as the target instance, and the instance address of the target instance is sent to the GPU resource user as a response to the GPU instance call request, so that the GPU resource user can call the computing resources of the target instance.

2. The method according to claim 1, characterized in that The metadata includes historically processed tasks and the number of tasks currently being processed simultaneously. The filtering, based on the metadata and the load description data, to obtain the GPU instance that meets the conditions from the GPU cluster as the target instance includes: Obtain the task type of the current task corresponding to the GPU instance call request; For any GPU instance in the GPU cluster, if a target condition is met, use the any GPU instance as the target instance; Among them, the target condition is that the task type of the historically processed task is consistent with the task type of the current task, and the number of tasks currently being processed simultaneously and each load data in the load situation description data meet the upper limit requirements.

3. The method according to claim 2, characterized in that The metadata includes an average response time, and the method further includes: For any GPU instance in the GPU cluster, when the target condition is not met, the priority score of each GPU instance is calculated according to the value of each load data in the load situation description data and the average response time; Any GPU instance with the highest priority score is used as the target instance.

4. The method according to claim 1, characterized in that: The query obtains metadata of each GPU instance, including: Initiate a query request to a distributed storage system according to the address of the GPU instance; the distributed storage system stores metadata of each of the GPU instances in the GPU cluster; Receive metadata of each of the GPU instances in response to the query request from each of the distributed storage systems.

5. The method according to claim 1, characterized in that The detection obtains load description data of each GPU instance, including: Obtain an interface corresponding to each item of load data in the load condition description data; According to the interface corresponding to each item of load data, read the each item of load data from the driver program in running state corresponding to each of the GPU instances; The load data includes GPU usage, machine real-time video memory usage, physical temperature, and frame rate.

6. The method according to claim 1, characterized in that The method further comprises: Initiating a resource release request for the target instance when the GPU resource user has finished using the computing resources of the target instance; the resource release request is used to release the computing resources of the target instance; When the computing resources of the target instance have been released, the metadata corresponding to the target instance is recalculated, and the recalculated metadata is updated to the distributed storage system.

7. The method according to claim 1, characterized in that The method further comprises: Establishing a service of the GPU cluster in a management component of the GPU cluster, adding the GPU instance to be added to the service of the GPU cluster, and obtaining the preconfigured GPU cluster; The obtaining the instance address of each GPU instance includes: The service name corresponding to the service of the GPU cluster is used as an input parameter to obtain the instance address of each GPU instance.

8. The method according to claim 1, characterized in that The method further comprises: Calculate the average response time of each GPU cluster in response to the call request based on the service function of the operation and maintenance side; When the average response time is greater than a preset threshold, GPU instances are added to the GPU cluster to automatically expand the capacity of the GPU cluster; and when the average response time is less than or equal to the preset threshold, GPU instances are reduced in the GPU cluster to automatically shrink the capacity of the GPU cluster.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images