GPU resource scheduling method and device

Abstract

The invention relates to the technical field of container resource management, and discloses a GPU resource scheduling method and device.The GPU resource scheduling method comprises the steps that GPU resources are used for starting at least one instance in each container application; detecting a request queue state and a GPU resource idle state when it is determined that a target request for the target container application arrives or a calculation task is ended; according to the current GPU resource idle state and the request queue state, resource balance state control and request response are carried out, and the resource balance state is that at least one instance of each container application is in an idle or running state. According to the method, GPU resources can be distributed more reasonably, and request response delay caused by the fact that the instance of the container application needs to be started when the request arrives is reduced, so that the request time is shortened, and the condition of service interruption is avoided as far as possible.

Description

Technical Field

[0001] The present disclosure relates to the technical field of container resource management, for example, to a GPU resource scheduling method and device. Background Art

[0002] With the development of AI (Artificial Intelligence), the demand for GPU (Graphics Processing Unit) resources is increasing. Although high-performance GPUs are powerful, they are expensive, which increases the overall cost of AI applications. Therefore, how to reduce the cost of GPU use and improve its utilization has become a key issue.

[0003] Virtual GPU technology divides the computing power and video memory of a physical GPU into multiple logical small GPUs, allowing multiple applications to share the same GPU resources. This improves GPU utilization and reduces costs to a certain extent. For example, a GPU card with large video memory can be virtualized into multiple GPUs with small video memory, thus achieving a higher cost-effectiveness.

[0004] In some related technologies, powerful container orchestration and management capabilities can be provided through cluster services (such as Kubernetes). By integrating virtual GPU technology, containers can declare the required GPU resources when deployed instead of monopolizing the entire GPU resources. This fine-grained resource allocation method further optimizes the use of GPU resources, reduces costs and improves utilization. Other related technologies use traffic-based automatic scaling technology. When there is no traffic request, the container will automatically end and release GPU resources after a period of time; when it is determined that there is a new traffic request, the container will be restarted. This method avoids resource waste during periods of no traffic and enables the released resources to be used by other containers. If the GPU resources required when a new container is started exceed the remaining resources of the host machine, the container cannot be started until the resources are available.

[0005] However, when the container is deployed, it declares the required GPU resources. Even if the application does not always need the GPU, these resources will continue to be occupied, resulting in waste. The automatic scaling technology based on traffic is too rigid and does not take into account the time required for container startup, which may lead to response delays and service interruptions. This technology also does not take into account the dynamic balance of the overall resource status, which may lead to uneven resource allocation and affect system performance and stability. It can be seen that although the relevant technologies have a certain effect in improving GPU resource utilization and reducing costs, there has been progress in improving GPU resource utilization and reducing costs, and there are still problems such as unreasonable static resource allocation, lack of flexibility, response delays and service interruptions.

[0006] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0007] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical components or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

[0008] The embodiments of the present disclosure provide a GPU resource scheduling method and device, which can more reasonably allocate GPU resources and reduce request response delay.

[0009] According to a first aspect of the present disclosure, a GPU resource scheduling method is provided, comprising:

[0010] Launch at least one instance of each container application using GPU resources;

[0011] Determine whether a target request for the target container application has arrived or a computing task has ended, and detect the request queue status and GPU resource idle status;

[0012] According to the current GPU resource idle state and request queue state, resource balancing state control and request response are performed, wherein the resource balancing state is that at least one instance of each container application is in an idle or running state.

[0013] In some embodiments, when it is determined that a target request for a target container application arrives or a computing task ends, detecting a request queue state and a GPU resource idle state includes:

[0014] When it is determined that a target request for the target container arrives, the request queue status is checked;

[0015] When the request queue is not empty, add the target request to the request queue state;

[0016] When the request queue is empty, detect the idle state of GPU resources.

[0017] In some embodiments, resource balancing state control and request response are performed according to the current GPU resource idle state and request queue state, including:

[0018] When idle GPU resources are detected, the idle GPU resources are used to run the instances in the target container application, where the idle GPU resources are GPU resources occupied by the idle instances;

[0019] Respond to the target request based on the instance in the target container application.

[0020] In some embodiments, when idle GPU resources are detected, using the idle GPU resources to run an instance in the target container application includes:

[0021] When idle GPU resources are detected and the idle GPU resources are occupied by instances in the target container application, the idle GPU resources are used to run the instances of the target container application;

[0022] When idle GPU resources are detected and the idle GPU resources are occupied by instances in other container applications other than the target container application, the instances in other container applications that occupy the idle GPU resources are closed, and the idle GPU resources are used to run the instance of the target container application.

[0023] In some embodiments, the method further includes: when no idle GPU resources are detected, adding the target request to a request queue state and continuing to detect idle GPU resources.

[0024] In some embodiments, when it is determined that a target request for a target container application arrives or a computing task ends, detecting a request queue state and a GPU resource idle state includes:

[0025] When a computing task is determined to be finished, the idle state of GPU resources is detected;

[0026] When idle GPU resources are detected, the request queue status is checked.

[0027] In some embodiments, resource balancing state control and request response are performed according to the current GPU resource idle state and request queue state, including:

[0028] When the request queue is not empty, use the idle GPU resources to run the instance in the container application corresponding to the request, and respond to the request based on the running instance;

[0029] When the request queue is empty, it is detected whether it is currently in a resource balance state. If it is not in a resource balance state, the idle GPU resources are used to control the resource balance state.

[0030] In some embodiments, when the request queue is not empty, using idle GPU resources to run an instance in the container application corresponding to the request, and responding to the request based on the running instance, including:

[0031] When the request queue is not empty, based on the computing task time consumption information of the container application corresponding to each request and the waiting order information of each request, the first request with the highest priority is determined from the request queue;

[0032] Use idle GPU resources to run the instance of the container application corresponding to the first request, and respond to the request based on the running instance.

[0033] In some embodiments, when the request queue is empty, detecting whether the resource is currently in a balanced state, and when the resource is not in a balanced state, using idle GPU resources to control the resource balanced state includes:

[0034] When the request queue is empty, check whether it is currently in a resource balancing state;

[0035] When the resource is not in a balanced state, at least one instance of the unbalanced container application is started using idle GPU resources, wherein the unbalanced container application is a container application that does not have instances in an idle or running state.

[0036] According to a second aspect of the present disclosure, a GPU resource scheduling device is provided, comprising a processor and a memory storing program instructions, wherein the processor is configured to execute the GPU resource scheduling method provided by the first aspect of the present disclosure when running the program instructions.

[0037] The GPU resource scheduling method and device provided by the embodiments of the present disclosure can achieve the following technical effects: dynamically determine the allocation of resources according to the current request queue status and GPU resource idle state, so that it can be flexibly adjusted according to actual needs, avoiding the problem of GPU resource waste or shortage caused by static resource allocation. Whenever a new request arrives or a computing task is completed, a check of the request queue status and GPU resource status will be triggered. This mechanism ensures that resource allocation can respond quickly to changing request conditions and improves the flexibility of the system. In particular, by trying to maintain the resource balance state of "each container application has at least one instance in an idle or running state", even if a container application does not respond to the request temporarily, it still reserves a certain share of GPU resources to maintain at least one instance in a started state, so as to reduce the request response delay caused by the need to start the instance of the container application when the request arrives, thereby shortening the request time and avoiding service interruptions as much as possible.

[0038] The foregoing general description and the following description are exemplary and explanatory only and are not intended to limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] One or more embodiments are exemplarily described by corresponding drawings, which do not limit the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements, and the drawings do not constitute a scale limitation, and wherein:

[0040] Figure 1 is a schematic diagram of a software architecture deployed in a GPU resource scheduling device provided in an embodiment of the present disclosure;

[0041] Figure 2It is a flowchart of a GPU resource scheduling method provided by an embodiment of the present disclosure;

[0042] Figure 3 is a flowchart of another GPU resource scheduling method provided by an embodiment of the present disclosure;

[0043] Figure 4 is a flowchart of another GPU resource scheduling method provided by an embodiment of the present disclosure;

[0044] Figure 5 is a flowchart of another GPU resource scheduling method provided by an embodiment of the present disclosure;

[0045] Figure 6 It is a structural diagram of a GPU resource scheduling device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0046] In order to be able to understand the features and technical contents of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present disclosure. In the following technical description, for the convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.

[0047] The terms "first", "second", etc. in the specification and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the embodiments of the embodiments of the present disclosure described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions.

[0048] Unless otherwise stated, the term "plurality" means two or more.

[0049] In the embodiment of the present disclosure, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B indicates: A or B.

[0050] The term "and / or" is a description of the association relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or, A and B.

[0051] The term "correspondence" may refer to an association relationship or a binding relationship. The correspondence between A and B means that there is an association relationship or a binding relationship between A and B.

[0052] The embodiment of the present disclosure provides a GPU resource scheduling device, which can implement the GPU resource scheduling method provided by the embodiment of the present disclosure. The GPU resource scheduling device can be a computer, terminal, server or other device with computing capabilities. Figure 1 As shown, the GPU resource scheduling device can communicate with the server to receive requests for each container application. The GPU resource scheduling device has a processor and is deployed with a software architecture for executing the GPU resource scheduling method, and the processor can execute the GPU resource scheduling method based on the software architecture. The software architecture includes a request queue, a container group, and GPU resources. The container group contains multiple container applications, for example, the container group contains container application A, container application B, and container application C. Here, each container application can apply for GPU resources or release occupied GPU resources.

[0053] In some embodiments, the software architecture includes a performance statistics module, which can count the average task execution time of each container application. The average task execution time refers to the average time required for each task during multiple executions of the requested computing tasks by the container application. The average task execution time is obtained by statistically calculating multiple running records of the container application in the past period of time.

[0054] In some embodiments, the software architecture includes a container management module. When a container is idle or a new request arrives, in order to ensure full utilization of GPU resources, it is necessary to be able to determine how to allocate GPU resources. Generally speaking, starting a container application will take a certain amount of time, and the startup order of the container application needs to be reasonably arranged, so the container management module needs to be referenced to maintain a balanced resource state.

[0055] In combination with the GPU resource scheduling device provided in the embodiment of the present disclosure, the embodiment of the present disclosure provides a GPU resource scheduling method, the execution subject of the method is a GPU resource scheduling device (hereinafter referred to as the device), such as Figure 2 As shown, the GPU resource scheduling method includes:

[0056] S201, the device uses GPU resources to start at least one instance in each container application.

[0057] In the embodiment of the present disclosure, after the device uses GPU resources to initially start at least one instance in each container application, subsequent S202 to S204 may be executed in a loop.

[0058] In the embodiments of the present disclosure, an instance refers to an independently running copy of a container application. Each instance is a separate execution unit of the container application, and can use the GPU resources allocated to it to process computing tasks.

[0059] S202, when the device determines that a target request for a target container application has arrived or a computing task has ended, it detects the request queue status and the GPU resource idle status.

[0060] In the disclosed embodiment, the request queue status reflects whether the request queue is empty, and the GPU resource idle status reflects whether there are idle GPU resources, where idle GPU resources refer to GPU resources occupied by idle instances. When the request queue is empty, it indicates that there may be idle GPU resources at present; when the request queue is not empty, it indicates that there are no idle GPU resources at present.

[0061] S203, the device performs resource balance state control and request response according to the current GPU resource idle state and request queue state.

[0062] In the disclosed embodiment, if it is determined that the current GPU resources can satisfy the request response but cannot satisfy the resource balance state according to the current GPU resource idle state and request queue state, the request response is given priority, the computing task of the request is executed, and then the resource balance state control is performed. If it is determined that the current GPU resources can satisfy both the request response and the resource balance state according to the current GPU resource idle state and request queue state, the request response (i.e., the computing task of the request is executed) and the resource balance state control can be performed simultaneously. If it is determined that there is no request that needs to be responded to according to the current GPU resource idle state and request queue state, the resource balance state control is performed directly.

[0063] In the embodiment of the present disclosure, the resource balancing state is that at least one instance of each container application is in an idle or running state. When an instance of a container application is in an idle or running state, the instance is in a startup state so as to be able to respond to and process new requests for the container application in a timely manner.

[0064] The GPU resource scheduling method provided by the embodiment of the present disclosure dynamically determines the allocation of resources according to the current request queue status and the idle state of GPU resources, so that it can be flexibly adjusted according to actual needs, avoiding the problem of GPU resource waste or shortage caused by static resource allocation. Whenever a new request arrives or a computing task is completed, a check of the request queue status and the GPU resource status will be triggered. This mechanism ensures that resource allocation can respond quickly to changing request conditions and improves the flexibility of the system. In particular, by trying to maintain the resource balance state of "each container application has at least one instance in an idle or running state", even if a container application does not respond to the request temporarily, it still reserves a certain share of GPU resources to maintain at least one instance in a started state, so as to reduce the request response delay caused by the need to start the instance of the container application when the request arrives, thereby shortening the request time and avoiding service interruptions as much as possible.

[0065] In some embodiments, when it is determined that a target request for a target container application has arrived or a computing task has ended, a request queue state and a GPU resource idle state are detected, including: when it is determined that a target request for a target container has arrived, the request queue state is detected; when the request queue is not empty, the target request is added to the request queue state; when the request queue is empty, the GPU resource idle state is detected.

[0066] In some embodiments, resource balancing state control and request response are performed according to the current GPU resource idle state and request queue state, including: when idle GPU resources are detected, using the idle GPU resources to run instances in the target container application, wherein the idle GPU resources are GPU resources occupied by idle instances; and responding to target requests based on the instances in the target container application.

[0067] Combination Figure 3 As shown, the embodiment of the present disclosure provides another GPU resource scheduling method, and the GPU resource scheduling method includes:

[0068] S301, the device uses GPU resources to start at least one instance in each container application.

[0069] In the embodiment of the present disclosure, after the device uses GPU resources to initially start at least one instance in each container application, subsequent S302 to S306 may be executed in a loop.

[0070] S302: When the device determines that a target request for a target container has arrived, the device detects a request queue status.

[0071] In the embodiment of the present disclosure, for ease of understanding and description, a newly received request is defined as a target request, and a container application that requires a corresponding target request is defined as a target container application. When the device determines that a target request has arrived, it triggers a check on the request queue status to determine whether the request queue is empty.

[0072] S303: When the request queue is not empty, the device adds the target request to the request queue state.

[0073] It is understandable that when the request queue is not empty, it indicates that there are currently no idle GPU resources, so the target request is added to the request queue state and will be responded to when the GPU resources are sufficient.

[0074] S304: When the request queue is empty, the device detects the idle state of the GPU resources.

[0075] It is understandable that when the request queue is empty, it indicates that there may be idle GPU resources at present, so it is necessary to detect the idle state of the GPU resources to determine whether there are idle GPU resources.

[0076] S305: When the device detects idle GPU resources, it uses the idle GPU resources to run the instance in the target container application.

[0077] S306: The device responds to the target request based on the instance in the target container application.

[0078] In an embodiment of the present disclosure, if idle GPU resources are detected, the idle GPU resources can be used to run an instance in the target container application so that the instance maintains the capability of the corresponding request, so as to respond to the target request based on the instance in the target container application and execute the computing task of the target request.

[0079] In some embodiments, when a target request for a target container arrives, if idle GPU resources are detected, the idle GPU resources may be occupied by an instance in the target container application, or may be occupied by an instance in another container application other than the target container application. When idle GPU resources are detected, the instance in the target container application is run using the idle GPU resources, including: when idle GPU resources are detected and the idle GPU resources are occupied by an instance in the target container application, the instance in the target container application is run using the idle GPU resources; when idle GPU resources are detected and the idle GPU resources are occupied by an instance in another container application other than the target container application, the instance in the other container application that occupies the idle GPU resources is closed, and the instance in the target container application is run using the idle GPU resources.

[0080] Combination Figure 4 As shown, the embodiment of the present disclosure provides another GPU resource scheduling method, and the GPU resource scheduling method includes:

[0081] S401: The device uses GPU resources to start at least one instance in each container application.

[0082] S402: When the device determines that a target request for a target container has arrived, the device detects a request queue status.

[0083] S403: When the request queue is not empty, the device adds the target request to the request queue state.

[0084] S404: When the request queue is empty, the device detects the idle state of the GPU resources.

[0085] S405: When the device detects idle GPU resources and the idle GPU resources are occupied by an instance in the target container application, the device uses the idle GPU resources to run the instance of the target container application.

[0086] S406: When the device detects idle GPU resources and the idle GPU resources are occupied by instances in other container applications other than the target container application, the device closes the instances in other container applications that occupy the idle GPU resources and uses the idle GPU resources to run the instance of the target container application.

[0087] S407, the device responds to the target request based on the instance in the target container application.

[0088] In an embodiment of the present disclosure, after responding to a target request based on an instance in a target container application, the process also includes: after completing a computing task of the target request, shutting down an instance of the computing task that completes the target request to release GPU resources, and using the released GPU resources to restart the instance in the container application that previously borrowed the idle GPU resources, so as to be able to return to a resource balanced state.

[0089] In some embodiments, if the request queue is empty and no free GPU resources are detected, the target request is added to the request queue state, and free GPU resources may be detected again.

[0090] For example, if the total amount of GPU resources is 32 GB, 6 GB is required to start an instance of container application A, 6 GB is required to start an instance of container application B, and 12 GB is required to start an instance of container application C. In the initial stage, you can start one instance of container application A, container application B, and container application C respectively, and 8 GB of GPU resources will be left. In other words, the free GPU resources in the initial stage are 8 GB.

[0091] Assume that at a certain moment, a request A for container application A arrives, and the device detects the status of the request queue. If the request queue is not empty, request A is added to the request queue. If the request queue is empty, the device detects the idle state of the GPU resources, and 8GB of idle GPU resources may be detected at this time. Since starting an instance of container application A only requires 6GB of GPU resources, 8GB of idle GPU resources can satisfy both the request response and resource balance status. In this case, 6GB of idle GPU resources are used to start and restart an instance of container application A, and an instance of container application A is used to respond to request A, and another instance of container application A is in an idle or running state so that it can respond to the next request A that may arrive in time.

[0092] Assume that at a certain moment, two requests C for container application C arrive, and the device detects the status of the request queue. If the request queue is not empty, request C is added to the request queue. If the request queue is empty, the device detects the idle state of the GPU resources. At this time, container application A, container application B, and container application C each have an instance in an idle state. In this case, the device will detect 8GB of idle GPU resources not occupied by any instance, and 24GB of idle GPU resources occupied by the idle instances of container application A, container application B, and container application C. Container application C requires two instances to respond to the two requests C, so another instance of container application C needs to be started. Starting an instance of container application C requires 12GB of GPU resources. At this time, an instance of container application A can be closed to release 6GB of idle GPU resources, and the available idle GPU resources will reach 14GB. Then, use 12GB of idle GPU resources to start and restart an instance of container application C, and use two instances of container application C to respond to the two requests C. After completing the computing task of the target request, an instance of container application C is shut down to release 12 GB of GPU resources, and an instance of container application A is restarted using the released 6 GB of GPU resources, thereby returning to a resource balance state.

[0093] When it is determined that the current GPU resources can satisfy the request response but cannot satisfy the resource balance state, the request response is prioritized, the computing task of the request is executed, and then the resource balance state control is performed. If it is determined based on the current GPU resource idle state and the request queue state that the current GPU resources can satisfy both the request response and the resource balance state, the request response (i.e., the computing task of the request is executed) and the resource balance state control can be performed simultaneously. If it is determined based on the current GPU resource idle state and the request queue state that there is no request that needs to be responded to, the resource balance state control is performed directly.

[0094] In some embodiments, when it is determined that a target request for a target container application arrives or a computing task ends, the request queue status and the GPU resource idle status are detected, including: when it is determined that a computing task ends, the GPU resource idle status is detected; when idle GPU resources are detected, the request queue status is detected.

[0095] In some embodiments, resource balancing state control and request response are performed according to the current GPU resource idle state and request queue state, including: when the request queue is not empty, using idle GPU resources to run the instance in the container application corresponding to the request, and responding to the request based on the running instance; when the request queue is empty, detecting whether it is currently in a resource balancing state, and when it is not in a resource balancing state, using idle GPU resources to control the resource balancing state.

[0096] Combination Figure 5 As shown, the embodiment of the present disclosure provides another GPU resource scheduling method, and the GPU resource scheduling method includes:

[0097] S501: The device uses GPU resources to start at least one instance in each container application.

[0098] S502: When the device determines that a computing task has ended, it detects the idle state of GPU resources.

[0099] In the embodiment of the present disclosure, when the device determines that a computing task has ended, it indicates that an instance of the container application has entered an idle state. At this time, a check of the request queue status will be triggered to detect the idle state of the GPU resources and determine whether there are GPU resources occupied by instances in the idle state.

[0100] S503: When the device detects idle GPU resources, it detects the request queue status.

[0101] In the embodiment of the present disclosure, when an instance in an idle state is detected, the idle GPU resources occupied by the instance in the idle state are identified as idle GPU resources. The request queue state can then be detected to determine whether the request queue is empty.

[0102] S504: When the request queue is not empty, the device uses idle GPU resources to run an instance in the container application corresponding to the request, and responds to the request based on the running instance.

[0103] It is understandable that when the request queue is not empty, it indicates that there is at least one request waiting to be responded to in the request queue. At this time, the idle GPU resources can be used to run the instance in the container application corresponding to the request waiting to be responded to, and the computing task of the request is executed based on the response request of the running instance.

[0104] S505: When the request queue is empty, the device detects whether it is currently in a resource balance state.

[0105] It is understandable that when the request queue is empty, it indicates that there is no request that needs to be responded to in the request queue. At this time, it can be detected whether the current resource is in a balanced state, that is, whether each container application has at least one instance in an idle or running state.

[0106] S506: When the device determines that it is not in a resource balancing state, it uses idle GPU resources to control the resource balancing state.

[0107] In the embodiment of the present disclosure, if it is detected that a container application has no instance in an idle or running state, it can be determined that it is not in a resource balanced state. Here, a container application that does not have an instance in an idle or running state can be defined as an unbalanced container application. At this time, it is necessary to use idle GPU resources to start at least one instance of the unbalanced container application to achieve a resource balanced state.

[0108] In some embodiments, when the request queue is not empty, using idle GPU resources to run an instance in a container application corresponding to the request, and responding to the request based on the instance being run, including: when the request queue is not empty, based on computing task time information of the container application corresponding to each request and waiting order information of each request, determining the first request with the highest priority from the request queue; using idle GPU resources to run an instance in the container application corresponding to the first request, and responding to the request based on the instance being run.

[0109] In an embodiment of the present disclosure, the computing task time-consuming information of the container application may include startup time, average task execution time, current task execution time and estimated remaining task time, wherein the estimated remaining task time is the difference between the average task execution time and the current task execution time.

[0110] Startup time refers to the time required to start a container application.

[0111] The average task execution time refers to the average time required for each task during multiple executions of the requested computing tasks by the container application. The average task execution time is calculated by statistically calculating multiple running records of the container application over a period of time.

[0112] The current task execution time refers to the length of time that the requested computing task being executed by the container application has been in progress. It can be understood that the current task execution time changes dynamically and increases as the computing task proceeds.

[0113] The estimated remaining task time refers to how much time the current computing task of the container application is expected to take to complete. The estimated remaining task time is calculated by subtracting the current task execution time from the average task execution time.

[0114] In the embodiment of the present disclosure, the waiting order information of the request includes the ranking of the same type of requests in the request queue and the waiting time of the request. The request queue may contain one or more requests corresponding to the same container application. The embodiment of the present disclosure regards the set of requests corresponding to the same container application as the same type of request set, and regards the ranking of each request in its same type of request set as the same type of request ranking of the request. It can be understood that the earlier the request is added to the request queue, the higher the ranking of the same type of requests of the request.

[0115] In an embodiment of the present disclosure, for each request in a request queue, a priority score of the request is calculated based on the startup time, average task execution time, current task execution time, and estimated remaining task time of the container application corresponding to the request, as well as the position of the request in the request queue and the waiting time of the request; the request with the highest priority score is used as the first request with the highest priority.

[0116] In the disclosed embodiment, the system can more accurately determine the priority requests by comprehensively considering the startup time, average task execution time, current task execution time, and estimated remaining task time of the container application corresponding to the request, as well as the waiting time of the request. Prioritize requests with short estimated remaining task time and long waiting time, significantly shorten the user waiting time and improve the user experience. Consider the waiting time of the request to ensure that requests with long waiting time can also be reasonably processed, and maintain fairness among all requests.

[0117] Optionally, the estimated remaining task time is calculated based on the average task execution time of the container application corresponding to the request and the current task execution time; the execution waiting time of the request is calculated based on the estimated remaining task time of the request, the ranking of similar requests and the average task execution time of the corresponding container application. Here, the execution waiting time of the request refers to the time required for the request to be responded to by the corresponding container application.

[0118] Optionally, the priority score of the request is determined based on a comparison result of the execution waiting time of the request and the startup time of the container application corresponding to the request, as well as the waiting time of the request.

[0119] The execution waiting time of the request is calculated based on the following formula: T = (P + (N-1) * M). In the above formula, T represents the execution waiting time of the request, P represents the estimated remaining task time, N represents the ranking of similar requests of the request, and M represents the average task execution time of the container application corresponding to the request.

[0120] Optionally, when the execution waiting time of the request is less than the startup time of the container application corresponding to the request, the priority score of the request is determined to be 0; when the execution waiting time of the request is not less than the startup time of the container application corresponding to the request, the time difference between the execution waiting time of the request and the startup time of the container application corresponding to the request is calculated; and the fusion value of the time difference between the execution waiting time of the request and the startup time of the container application corresponding to the request and the waiting time of the request is calculated as the priority score of the request.

[0121] In some embodiments, the fused value of the time difference between the execution waiting time of the request and the startup time of the container application corresponding to the request and the waiting time of the request is the sum of the time difference between the execution waiting time of the request and the startup time of the container application corresponding to the request and the execution waiting time of the request.

[0122] Optionally, when the fusion value of the time difference between the execution waiting time of the request and the startup time of the container application corresponding to the request and the waiting time of the request is the sum of the time difference between the execution waiting time of the request and the startup time of the container application corresponding to the request and the waiting time of the request, the priority score of the request can be calculated by the following formula: S=W+T0. Wherein, S is the priority score of the request, W is the waiting time of the request, and T0 is the time difference between the execution waiting time of the request and the startup time of the container application corresponding to the request.

[0123] In some embodiments, the fusion value of the time difference between the execution waiting time of the request and the startup time of the container application corresponding to the request and the waiting time of the request is the weighted sum of the time difference between the execution waiting time of the request and the startup time of the container application corresponding to the request and the waiting time of the request. The weighting coefficient of the waiting time of the request and the weighting coefficient of the time difference between the execution waiting time of the request and the startup time of the container application corresponding to the request can be determined according to actual design requirements.

[0124] Optionally, when the fusion value of the time difference between the execution waiting time of the request and the start time of the container application corresponding to the request and the waiting time of the request is the weighted sum of the time difference between the execution waiting time of the request and the start time of the container application corresponding to the request and the waiting time of the request, the priority score of the request can be calculated by the following formula: S = α*W+β*T0. Among them, S is the priority score of the request, W is the waiting time of the request, T0 is the time difference between the execution waiting time of the request and the start time of the container application corresponding to the request, α is the weighting coefficient of the sum of the waiting time of the request, and β is the weighting coefficient of the time difference between the execution waiting time of the request and the start time of the container application corresponding to the request.

[0125] In the disclosed embodiment, α and β can be set according to the request response speed requirement, fairness requirement and system load. If more attention is paid to the response speed, the value of β can be appropriately increased and the value of α can be reduced. If more attention is paid to fairness, the value of α can be appropriately increased and the value of β can be reduced. Under high load conditions, the value of β can be appropriately increased to ensure that short tasks are processed quickly and the queue backlog is reduced. Under low load conditions, the value of α can be appropriately increased to ensure that requests that have been waiting for a long time are processed.

[0126] In some embodiments, when the request queue is empty, it is detected whether it is currently in a resource balanced state, and when it is not in a resource balanced state, idle GPU resources are used to control the resource balanced state, including: when the request queue is empty, it is detected whether it is currently in a resource balanced state; when it is not in a resource balanced state, at least one instance of an unbalanced container application is started using idle GPU resources, wherein the unbalanced container application is a container application that does not have instances in an idle or running state.

[0127] In an embodiment of the present disclosure, it is detected whether the current state is resource balanced, that is, whether each container application has at least one instance in an idle or running state. If it is detected that a container application has no instance in an idle or running state, it can be determined that it is not in a resource balanced state. This container application that does not have an instance in an idle or running state is an unbalanced container application. At this time, it is necessary to use idle GPU resources to start at least one instance of the unbalanced container application to achieve a resource balanced state. Specifically, the instance that is in an idle state can be closed to release idle GPU resources, and the released idle GPU resources can be used to start at least one instance of the unbalanced container application to achieve a resource balanced state.

[0128] Combination Figure 6 As shown, the embodiment of the present disclosure provides a GPU resource scheduling device 600, which includes a processor 601 and a memory 602. Optionally, the GPU resource scheduling device 600 may also include a communication interface 603 and a bus 604. The processor 601, the communication interface 603, and the memory 602 may communicate with each other through the bus 604. The communication interface 603 may be used for information transmission. The processor 601 may call the logic instructions in the memory 602 to execute the GPU resource scheduling method of the above embodiment.

[0129] In addition, the logic instructions in the memory 602 described above may be implemented in the form of software functional units and when sold or used as independent products, may be stored in a computer-readable storage medium.

[0130] The memory 602 is a computer-readable storage medium that can be used to store software programs and computer executable programs, such as program instructions / modules corresponding to the method in the embodiment of the present disclosure. The processor 601 executes the functional application and data processing by running the program instructions / modules stored in the memory 602, that is, the GPU resource scheduling method in the above embodiment is implemented.

[0131] The memory 602 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 602 may include a high-speed random access memory and may also include a non-volatile memory.

[0132] An embodiment of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to execute the above-mentioned GPU resource scheduling method.

[0133] The technical solution of the embodiment of the present disclosure can be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiment of the present disclosure. The aforementioned storage medium may be a non-transient storage medium, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes.

[0134] The above description and the accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent only possible changes. Unless explicitly required, separate components and functions are optional, and the order of operation may vary. The parts and features of some embodiments may be included in or replace the parts and features of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and the claims, unless the context clearly indicates, the singular forms of "a", "an" and "the" are intended to include plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of listings containing one or more associated ones. In addition, when used in the present application, the term "comprise" and its variants "comprises" and / or comprising refer to the presence of stated features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the presence of other identical elements in the process, method or device comprising the elements. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the various embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can refer to the description of the method part.

[0135] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. The technicians may use different methods for each specific application to implement the described functions, but such implementations should not be considered to exceed the scope of the embodiments of the present disclosure. The technicians may clearly understand that, for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above may refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0136] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units can be only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to implement this embodiment. In addition, each functional unit in the embodiment of the present disclosure may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit.

[0137] The flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to the embodiment of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowchart and the block diagram in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in a different order from the order disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. A GPU resource scheduling method, characterized in that: include: Launch at least one instance of each container application using GPU resources; Determine whether a target request for the target container application has arrived or a computing task has ended, and detect the request queue status and GPU resource idle status; According to the current GPU resource idle state and request queue state, resource balancing state control and request response are performed, wherein the resource balancing state is that at least one instance of each container application is in an idle or running state.

2. The GPU resource scheduling method according to claim 1, characterized in that: When a target request for the target container application arrives or a computing task ends, the request queue status and GPU resource idle status are detected, including: When it is determined that a target request for the target container arrives, the request queue status is checked; When the request queue is not empty, add the target request to the request queue state; When the request queue is empty, detect the idle state of GPU resources.

3. The GPU resource scheduling method according to claim 2, characterized in that: Based on the current GPU resource idle state and request queue state, resource balancing state control and request response are performed, including: When idle GPU resources are detected, the idle GPU resources are used to run the instances in the target container application, where the idle GPU resources are GPU resources occupied by the idle instances; Respond to the target request based on the instance in the target container application.

4. The GPU resource scheduling method according to claim 3, characterized in that: When idle GPU resources are detected, the idle GPU resources are used to run instances in the target container application, including: When idle GPU resources are detected and the idle GPU resources are occupied by instances in the target container application, the idle GPU resources are used to run the instances of the target container application; When idle GPU resources are detected and the idle GPU resources are occupied by instances in other container applications other than the target container application, the instances in other container applications that occupy the idle GPU resources are closed, and the idle GPU resources are used to run the instance of the target container application.

5. The GPU resource scheduling method according to claim 3, characterized in that: The method is characterized in that it also includes: when no idle GPU resources are detected, adding the target request to the request queue state and continuing to detect idle GPU resources.

6. The GPU resource scheduling method according to claim 1, characterized in that: When a target request for the target container application arrives or a computing task ends, the request queue status and GPU resource idle status are detected, including: When a computing task is determined to be finished, the idle state of GPU resources is detected; When idle GPU resources are detected, the request queue status is checked.

7. The GPU resource scheduling method according to claim 6, characterized in that: Based on the current GPU resource idle state and request queue state, resource balancing state control and request response are performed, including: When the request queue is not empty, use the idle GPU resources to run the instance in the container application corresponding to the request, and respond to the request based on the running instance; When the request queue is empty, it is detected whether it is currently in a resource balance state. If it is not in a resource balance state, the idle GPU resources are used to control the resource balance state.

8. The GPU resource scheduling method according to claim 7, characterized in that: When the request queue is not empty, use idle GPU resources to run the instance of the container application corresponding to the request, and respond to the request based on the running instance, including: When the request queue is not empty, based on the computing task time consumption information of the container application corresponding to each request and the waiting order information of each request, the first request with the highest priority is determined from the request queue; Use idle GPU resources to run the instance of the container application corresponding to the first request, and respond to the request based on the running instance.

9. The GPU resource scheduling method according to claim 7, characterized in that: When the request queue is empty, check whether it is currently in a resource balance state. If it is not in a resource balance state, use idle GPU resources to control the resource balance state, including: When the request queue is empty, check whether it is currently in a resource balancing state; When the resource is not in a balanced state, at least one instance of the unbalanced container application is started using idle GPU resources, wherein the unbalanced container application is a container application that does not have instances in an idle or running state.

10. A GPU resource scheduling device, comprising a processor and a memory storing program instructions, characterized in that: The processor is configured to execute the GPU resource scheduling method according to any one of claims 1 to 9 when running program instructions.

Images