Method for scheduling of acceleration devices, server cluster, device, medium and product

Abstract

The application discloses a scheduling method of an acceleration device, a server cluster, a device, a medium and a product, relates to the technical field of computers, and is based on at least one first thread to initiate a calling request for the acceleration device, to pre-establish a second thread corresponding to the at least one first thread in an interface forwarding server, to form a mapping relationship between the first thread and the second thread, to establish the mapping relationship between the first thread and the second thread, to directly execute access on the corresponding second thread through the mapping relationship, to improve the order and completeness of the calling request, to avoid invalid thread exit, and to ensure normal calling operation of the acceleration device. Multiple threads are configured with exclusive interface forwarding clients when an application process is configured, and the mapping of the second thread is pre-configured when transmission to the interface forwarding server, so that the order from the source to the final execution end is ensured, invalid thread exit is avoided, and the technical effect of ensuring normal calling of the acceleration device is achieved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a scheduling method for acceleration devices, server clusters, devices, media, and products. Background Technology

[0002] In typical container-based (Pod) calls to accelerated devices, an Application Programming Interface (API) is used to access the scheduled accelerated device. Switching between different threads on the current accelerated device context is uniformly sent to the API forwarding server for execution. With a large number of threads, incorrect valid accelerated device addresses may be sent to other accelerated devices. For example, if thread one calls the API to switch to accelerated device A, and thread two calls the same API to switch to accelerated device B, both APIs will be executed on the same API forwarding server. This will cause the valid address for switching to accelerated device A to be set to accelerated device B, leading to thread errors, invalid thread termination, and affecting normal accelerated device calls.

[0003] Therefore, how to achieve thread ordering to ensure normal operation of acceleration devices is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] This application provides a scheduling method for acceleration devices, a server cluster, devices, media, and products to at least solve the problem in related technologies where thread disorder and invalid thread termination affect the normal operation of acceleration devices.

[0005] This application provides a scheduling method for acceleration devices, including:

[0006] A request to call the acceleration device is initiated by at least one first thread within the application process;

[0007] Configure a corresponding interface forwarding client for at least one first thread to send the call request to the interface forwarding server corresponding to the application process;

[0008] Within the interface forwarding server, the acceleration device is accessed through the interface of the acceleration device requested by the call request via at least one second thread corresponding to the first thread, so as to complete the scheduling of the acceleration device.

[0009] This application also provides a cluster, the server cluster including a management server and multiple computing servers; wherein the multiple computing servers are connected to the management server;

[0010] The management server is used to execute the steps of the above-described scheduling method for the acceleration device in order to complete the scheduling of the acceleration device.

[0011] This application also provides an electronic device, including:

[0012] Memory, used to store computer programs;

[0013] A processor is used to implement the scheduling method of any of the above-mentioned acceleration devices when executing a computer program.

[0014] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the scheduling method of any of the above-described acceleration devices.

[0015] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described acceleration device scheduling methods.

[0016] This application addresses several key aspects. First, it addresses the issue of at least one first thread within an application process initiating a call request to an acceleration device. It pre-configures a corresponding interface forwarding client for each first thread, essentially assigning a unique identifier to each thread. This establishes a many-to-one mapping between the interface forwarding client and the corresponding interface forwarding server within the application process. Compared to conventional solutions where no interface forwarding client is configured for any of the first threads within the application process, or where only one interface forwarding client exists, leading to chaotic call request sources, this application ensures the orderliness of call request transmission at the source by assigning a dedicated interface forwarding client to each first thread. On the other hand, at least one second thread corresponding to the first thread is pre-established in the interface forwarding server, and a mapping relationship is formed between the first thread and the second thread. Compared with the conventional technical solution, which does not configure a thread mechanism in the interface forwarding server, resulting in thread chaos after transmission to the interface forwarding server and the use of the wrong acceleration device address, the mapping relationship between the first thread and the second thread allows access requests transmitted from the first thread to be directly executed on the corresponding second thread through this mapping relationship. This improves the orderliness and integrity of the call requests, avoids invalid thread exits, and ensures the normal call operation of the acceleration device.

[0017] Therefore, this method can solve the technical problem of invalid thread exits caused by the confusion between the source and final execution end when multiple threads correspond to the execution of an interface forwarding server, which in turn affects the normal call of the acceleration device. It achieves the technical effect of configuring a dedicated interface forwarding client for multiple threads during the application process and pre-configuring the mapping of the second thread when transmitting to the interface forwarding server, so as to ensure the orderly execution of the second thread from the source to the final execution end, avoiding invalid thread exits and ensuring the normal call of the acceleration device. Attached Figure Description

[0018] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating a scheduling method for an acceleration device provided in this application embodiment;

[0020] Figure 2 A schematic diagram illustrating a thread transmission processing method for an application process provided in an embodiment of this application;

[0021] Figure 3 A schematic diagram illustrating a scenario where remote invocation is used to schedule acceleration devices for a routine business operation.

[0022] Figure 4 This is a schematic diagram of a pooling relationship provided in an embodiment of this application;

[0023] Figure 5 A schematic diagram illustrating the transmission between a computing server and a management server provided in an embodiment of this application;

[0024] Figure 6 This application provides a schematic diagram of the process between a management server and a computing server in an embodiment of the present application.

[0025] Figure 7 This is a structural diagram of a scheduling device for an acceleration device provided in an embodiment of this application. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0027] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0028] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] The specific application environment architecture or specific hardware architecture on which the execution of the scheduling method of the acceleration device depends is described here.

[0030] In a server cluster, compute servers and management servers can work collaboratively. Taking a high-performance computing (HPC) cluster as an example, HPCs are typically used to handle complex scientific computing, engineering simulations, and data analysis tasks. These clusters are characterized by the need for substantial computing resources and efficient parallel processing capabilities. Compute servers are equipped with high-performance central processing units (CPUs), graphics processing units (GPUs), or field-programmable gate arrays (FPGAs) to perform computationally intensive tasks. Management servers are responsible for cluster resource management, job scheduling, monitoring, and maintenance. Similarly, cloud computing clusters are used to provide various cloud services, such as virtual machines, containers, and storage services. These clusters are characterized by dynamic resource allocation and elastic scaling. Compute servers provide computing resources to support the operation of virtual machines or containers. They are typically equipped with high-performance CPUs, memory, and acceleration devices. Management servers are responsible for resource virtualization, scheduling, monitoring, and management. Management servers run cloud management platforms (such as OpenStack, Kubernetes, etc.) and distributed storage systems store user data. Taking Artificial Intelligence (AI) and Machine Learning (ML) clusters as an example, these are specifically designed for training and deploying machine learning models. Such clusters require significant computing resources, especially GPUs or dedicated acceleration devices. Computation servers: Equipped with high-performance GPUs or FPGAs for training and inference tasks. Management servers: Responsible for task scheduling, resource management, and monitoring and optimization of model training. Taking edge computing clusters as another example, these are used to process data closer to the data source, reducing data transmission latency. Such clusters are typically deployed at the edge, close to the user or device. Computation servers: Miniaturized, high-performance computing nodes for real-time data processing and analysis. Management servers: Responsible for resource management, task scheduling, and security monitoring of edge nodes. The method described in this application is applicable to all types of clusters.

[0031] In practical applications, an interface is provided within the application process to switch the context of the acceleration device, controlling which device the interface in a thread executes on. In conventional solutions based on remote calls to acceleration devices, different threads switching their current device context are all uniformly sent to the interface forwarding server for execution, which can lead to the interface executing on the wrong device. For example, if thread one of the application first calls an interface to switch to acceleration device A, and then thread two subsequently calls the same interface to switch to acceleration device B, since both interfaces are executed in the same interface forwarding server, the actual current acceleration device on the computing server is set to B. Subsequent interface requests forwarded by thread one will also be executed on device B. This causes thread misalignment during the entire scheduling process, affecting the scheduling of the corresponding acceleration device. The acceleration device scheduling method provided in this application can solve this technical problem.

[0032] Figure 1 A flowchart of a scheduling method for an acceleration device provided in an embodiment of this application is shown below. Figure 1 As shown, the method includes:

[0033] S11: Initiate a call request to the acceleration device based on at least one first thread within the application process;

[0034] S12: Configure a corresponding interface forwarding client for at least one first thread to send the call request to the interface forwarding server corresponding to the application process;

[0035] S13: Within the interface forwarding server, the interface of the acceleration device that made the call request is accessed through at least one second thread corresponding to the first thread to complete the scheduling of the acceleration device.

[0036] Specifically, in step S11, the initiation of the acceleration device call request is achieved by receiving the acceleration device call request issued by the computing server where the Pod is located through the monitoring process when the Pod is pre-established. The resource status information of each computing server is collected through monitoring, so that the available acceleration devices are transmitted to the computing server through the creation of an API forwarding server.

[0037] The calling process here can directly call the local computing server, or it can use remote calling technology to use acceleration device resources across machines. Alternatively, the local computing server and the remote computing server can be called through pooling management. There are no restrictions here, and you can set it according to the actual situation.

[0038] Regarding the number of acceleration devices to be invoked, it can be set through the requests issued by the Pod. It can query whether the Pod has an acceleration device in use. If it does, the acceleration device is directly assigned to the process. If it does not, it needs to be scheduled. It can be scheduled by combining acceleration devices or by filtering them one by one, etc. There are no restrictions here.

[0039] The process involves at least one thread within the application process initiating a request to the acceleration device. This means the scheduling process is performed on a thread-by-thread basis; an application process may contain one or more threads. Thread creation can be achieved by pre-allocating several threads based on a thread queue. These threads then initiate their respective requests to the acceleration devices corresponding to the number of devices in the task. If there are enough threads, or if there are surplus threads, the remaining idle threads can be cleared.

[0040] The initiation of the call request here corresponds to which first thread initiates which acceleration device. This can be determined in advance based on the current load of different first threads, or the time order of different subtasks under the task, or a comprehensive judgment. There is no limitation here, and it can be set according to the actual situation.

[0041] In step S12, configuring a corresponding interface forwarding client for at least one first thread involves interface forwarding, where an application or service receives API requests from a client, forwards these requests to another backend service or API interface, and returns the backend service's response to the client. This technique is commonly used to build middleware services, enabling functions such as service decoupling, load balancing, and enhanced security.

[0042] In a multi-threaded environment, configuring each thread with a dedicated API forwarding client allows it to send and receive subsequent server-side requests. This avoids thread safety issues that arise when multiple threads share the same API forwarding client, such as data races and state corruption caused by concurrent access. Furthermore, each thread can independently manage its own API forwarding client's connection, request queue, and timeout settings. It also allows for the identification of the originating thread of the request and which subsequent server-side threads it corresponds to.

[0043] The call request is sent to the interface forwarding server corresponding to the application process. It's worth noting that this can be done using a conventional approach where multiple application processes correspond to the same interface forwarding server, or one application process can correspond to a dedicated interface forwarding server. The former is chosen because multiple application processes need to call different backend services with different interfaces, leading to complex and difficult-to-maintain frontend code for these calls. Therefore, a unified interface management system is used, abstracting the interfaces of multiple backend services into a standardized interface. The frontend application process only needs to interact with this unified interface, while the interface forwarding server is responsible for routing the request to the specific backend service. For the latter, conventional approaches consider that the interface forwarding server can only be cleaned up after all application processes have finished. To ensure timely resource cleanup, each application process is assigned a dedicated interface forwarding server, guaranteeing that the resources of that interface forwarding server can be cleaned up promptly after the application process terminates.

[0044] In step S13, the interface forwarding server accesses the acceleration device's interface through at least one second thread corresponding to the first thread to access the acceleration device. In conventional solutions, the creation of a second thread is not present. In this application, a one-to-one mapping relationship is pre-established between the first and second threads to transmit the first thread, which is of interest to the source, to the second thread through the interface forwarding server for ordered calls, avoiding thread chaos from beginning to end.

[0045] Through the embodiments of this application, on the one hand, based on at least one first thread in the application process initiating a call request to the acceleration device, a corresponding interface forwarding client is pre-configured for at least one first thread. In fact, each first thread is configured with an exclusive interface forwarding client, corresponding to the thread's exclusive identifier, so as to match the interface forwarding client with the interface forwarding server corresponding to the application process, realizing a many-to-one mapping relationship. Compared with conventional technical solutions where all first threads in the application process are not configured with interface forwarding clients or only one interface forwarding client exists, resulting in a chaotic source of call requests, this application ensures the orderliness of call request transmission at the source by configuring an exclusive interface forwarding client for each first thread. On the other hand, at least one second thread corresponding to the first thread is pre-established in the interface forwarding server, and a mapping relationship is formed between the first thread and the second thread. Compared with the conventional technical solution, which does not configure a thread mechanism in the interface forwarding server, resulting in thread chaos after transmission to the interface forwarding server and the use of the wrong acceleration device address, the mapping relationship between the first thread and the second thread allows access requests transmitted from the first thread to be directly executed on the corresponding second thread through this mapping relationship. This improves the orderliness and integrity of the call requests, avoids invalid thread exits, and ensures the normal call operation of the acceleration device.

[0046] Therefore, this method can solve the technical problem of invalid thread exits caused by the confusion between the source and final execution end when multiple threads correspond to the execution of an interface forwarding server, which in turn affects the normal call of the acceleration device. It achieves the technical effect of configuring a dedicated interface forwarding client for multiple threads during the application process and pre-configuring the mapping of the second thread when transmitting to the interface forwarding server, so as to ensure the orderly execution of the second thread from the source to the final execution end, avoiding invalid thread exits and ensuring the normal call of the acceleration device.

[0047] In some embodiments, a user's application may request video memory when using an accelerator device, but it may not actively release all the requested video memory. This could be due to an internal mechanism of the application, or it could be because the application crashed and failed to release the memory in time. Therefore, normally, the accelerator device driver will actively release the occupied video memory after detecting a process exiting. However, during a remote call to the accelerator device, the interface forwarding server does not exit, and the video memory remains occupied. Therefore, sending the call request to the interface forwarding server corresponding to the application process includes:

[0048] A first mapping relationship is pre-established, corresponding to one application process and one interface forwarding server.

[0049] The call request is sent to the corresponding interface forwarding server according to the first mapping relationship.

[0050] Specifically, a dedicated interface forwarding server is created for each application process. Through this initial mapping relationship, call requests are sent to the corresponding interface forwarding server. For example, one application process corresponds to one training task. A training task may require calls to multiple acceleration devices. One thread manages one acceleration device. During the execution of the training task, the call request for that training task is sent to the interface forwarding server based on the initial mapping relationship to execute subsequent threads.

[0051] This embodiment provides a dedicated interface forwarding server for each application process. This allows applications to send data directly to the dedicated interface forwarding server without considering load balancing factors, and the thread processing of the interface forwarding server corresponding to that application process can be performed directly. Compared to all application processes corresponding to one interface forwarding server, this saves interface management time and scheduling time, improves scheduling efficiency, and the task of the dedicated interface forwarding server is also completed after the application process finishes execution, so the memory of the dedicated interface forwarding server can be cleaned up directly.

[0052] In some embodiments, when the application process's heartbeat signal is lost, the method further includes:

[0053] Terminate the interface forwarding server corresponding to the application process;

[0054] Clean up the resource information of the acceleration device corresponding to the application process and the resource information of the process occupied by the corresponding interface forwarding server.

[0055] Figure 2 This is a schematic diagram illustrating a thread transmission processing method for an application process, as provided in an embodiment of this application. Figure 2 As shown, when the application process loses its heartbeat signal, it indicates that the current application process has terminated, and the dedicated interface forwarding server it is connected to has also terminated. The underlying acceleration device driver will automatically clean up the system resources and acceleration device resources occupied by the process corresponding to the forwarding server.

[0056] This embodiment provides a dedicated interface forwarding server created based on an application process. When the application process loses its heartbeat signal, the dedicated interface forwarding server can promptly clear the video memory occupied by the dedicated interface forwarding server, saving video memory storage space and computing power.

[0057] In some embodiments, before terminating the interface forwarding server corresponding to the application process, the method further includes:

[0058] Configure a new interface forwarding client for the first thread after termination;

[0059] Establish a new network between the interface forwarding client and the interface forwarding server;

[0060] Control the application process to send heartbeat signals so that the interface forwarding server can obtain the first identifier string of the new interface forwarding client;

[0061] Retrieve the second identifier string corresponding to the interface forwarding client;

[0062] If the first identifier string and the second identifier string match, then the network connection is considered successful.

[0063] If the first identifier string and the second identifier string are inconsistent, the network connection is determined to have failed, and the process proceeds to the step of terminating the interface forwarding server corresponding to the application process to terminate the new interface forwarding server.

[0064] Specifically, in combination Figure 2 As can be seen, since the application process periodically sends heartbeat signals, these signals contain a unique identifier string for the application process. This string can be represented as the hostname of the application process plus a 32-bit Universally Unique Identifier (UUID), which is generated when the application process starts. The application process's heartbeat signal loss can occur in two ways: either the application process is still running, or it has completed or stopped midway. Based on the first scenario, this application details the requirement to configure a new interface forwarding client for the terminated first thread to establish a network connection between the new client and the server for reconnection. Upon successful network connection, the application process sends a heartbeat signal so the server can compare the new client's first identifier string with the previous client's second identifier string. If the two strings match, the network connection is considered successful, and subsequent acceleration device scheduling continues.

[0065] If the two identifier strings are inconsistent, the network connection is determined to have failed. In this case, the first process needs to be terminated, and the process proceeds to the step of terminating the interface forwarding server corresponding to the state where the application process has completed or stopped midway. This corresponds to the termination of the interface forwarding server in the second case, i.e., the steps in the above embodiment.

[0066] In this embodiment, when the heartbeat signal is lost but the application process is still running, considering that the loss of the heartbeat signal is caused by network fluctuations, an attempt is made to re-establish the network connection as successfully as possible to ensure the reliability of the network connection.

[0067] In some embodiments, accessing the acceleration device via the interface of the acceleration device requesting the call through at least one second thread corresponding to a first thread within the interface forwarding server includes:

[0068] Pre-establish a second mapping relationship between the first thread and the second thread;

[0069] The corresponding second thread is determined based on the second mapping relationship to receive the call request;

[0070] The call request is parsed and processed to obtain the corresponding interface function of the acceleration device;

[0071] Based on the interface function mapping, the actual function address of the process occupied by the interface forwarding server;

[0072] The actual function address is passed to the second thread to wake up the second thread to execute access to the accelerated device.

[0073] like Figure 2 As shown, a second mapping relationship is pre-established between the first thread and the second thread to determine the second thread corresponding to the current first thread. This second thread is used to receive call requests, record the second thread's cache, and then enter a blocked state. The first thread, corresponding to the receiving / sending call request, receives the acceleration device call request sent by the interface forwarding client, parses out the specific interface function and parameters of the acceleration device to be executed, and maps this interface function to the actual function address in the interface forwarding server process. Then, the receiving / sending thread passes the function address and parameter address to the designated second thread, wakes up, and delegates the execution to the second thread.

[0074] Furthermore, if the parsed interface is a computation execution or memory allocation type interface, techniques such as time-sharing multiplexing and memory slicing can be used to strictly limit the resource requests of the Pod, thereby enabling a single acceleration device to be allocated to multiple Pods. After the second thread finishes execution, it returns the result to the receiving / sending thread and enters a blocked state again, waiting for the next wake-up. The receiving / sending thread then returns the result to the interface forwarding client via the original path.

[0075] Based on the creation of the second thread provided in this embodiment, the second mapping relationship between the second thread and the first thread is used to receive the call request to call the acceleration device. This avoids the thread disorder caused by the absence of such a mapping relationship during execution, ensuring thread order while preventing the reception of erroneous valid addresses of the acceleration device, thus guaranteeing the normal operation of the thread.

[0076] In some embodiments, initiating a call request to the acceleration device based on at least one first thread within the application process includes:

[0077] Get the current thread load of the first thread;

[0078] The priority order of the first thread is determined by sorting them according to their current thread load.

[0079] Get the arrival time sequence of tasks corresponding to the application process;

[0080] The target first thread corresponding to the acceleration device is determined based on the priority order and the order of task arrival time, so as to initiate the corresponding call request.

[0081] like Figure 2 As shown, a dedicated interface forwarding server is created for each application process. Connection information is obtained and recorded by the dynamic link library's initialization program. Whenever an accelerated device call request is modified, the dynamic link library checks the client cache for a dedicated interface forwarding client based on the thread number executing the request. If not found, a new interface forwarding client is created, connects to the corresponding interface forwarding server, and is recorded in the client cache. Then, this accelerated device call request is sent to the interface forwarding server process through the dedicated interface forwarding client, and the return result is received.

[0082] When refining a single interface forwarding client into multiple interface forwarding clients and assigning them to their respective first threads within the application process, the dynamic allocation of multiple first threads is considered. A thread pool is established to manage the current thread load and determine its priority order. Simultaneously, the arrival time order of each subtask corresponding to an acceleration device within the application process is obtained. Then, based on the priority order and task arrival time order, the first thread corresponding to each acceleration device is determined, and a corresponding call request is initiated for that first thread.

[0083] Based on the examples of the above embodiments, in a training task, the arrival time order of multiple subtasks corresponding to calling the acceleration device is obtained. In the priority order, the first thread E with the first priority is considered first. The acceleration device corresponding to the task whose arrival time order is earlier is assigned to the first thread E, and so on.

[0084] The process provided in this embodiment, which involves at least one first thread within the application process initiating a call request to the acceleration device, can effectively reduce the overhead of thread creation and destruction. It employs priority order and task arrival time order to ensure the orderliness of the first thread's initiation, thereby improving the efficiency of subsequent calls.

[0085] In some embodiments, determining the target first thread corresponding to the acceleration device based on priority order and task arrival time order includes:

[0086] Obtain the priority level and first weight coefficient corresponding to the priority order;

[0087] Obtain the task arrival time and second weight coefficient corresponding to the task arrival time sequence;

[0088] The order in which the first thread initiates the call request is determined based on the priority level, the first weighting coefficient, the task arrival time, and the second weighting coefficient.

[0089] The target first thread corresponding to the acceleration device is determined based on the order in which the first thread is initiated.

[0090] Specifically, in this embodiment, the priority of which first thread initiates the call request can be based on the task arrival time order. In most cases, after the task arrives, whichever first thread is idle will directly initiate the call request. However, considering the load of the thread corresponding to the acceleration device when the task arrives, if the idle first thread has a high load, it may disconnect during the call due to high load, resulting in failure to successfully call the acceleration device. Therefore, it is necessary to comprehensively consider the thread load and the task arrival time.

[0091] The priority level, first weight coefficient, task arrival time, and second weight are summed using parameter weighting to obtain the first thread initiation order. Based on this first thread initiation order, the corresponding target first thread is determined.

[0092] The embodiment provides a comprehensive consideration of the priority level of thread load and the weight parameters corresponding to the task arrival time, which can determine which threads can initiate call requests first, so as to ensure the orderliness of the first thread and improve the load balancing of the first thread.

[0093] In some embodiments, Figure 3 A schematic diagram illustrating a scenario where remote calls are used to schedule acceleration devices for a routine business operation, such as... Figure 3 As shown, considering that AI services are assigned to general and special services, general services are assigned to hosts without acceleration devices, while special services are assigned to hosts with acceleration devices. On hosts with acceleration devices, special services use the Kubernetes device plugin mechanism to allocate acceleration devices and mount them into containers for use; on hosts without acceleration devices, general services access the acceleration devices on hosts with acceleration devices via remote calls.

[0094] Figure 3The technical solution illustrated in the diagram prioritizes business functions, but when these functions access the local GPU (acceleration device), they are still limited by Kubernetes' device plugin mechanism. Acceleration device resources are allocated when containers are created, which is inflexible and limits overall utilization. Furthermore, remote calls to acceleration devices suffer from multi-threaded device access conflicts and GPU memory leaks. Containers can only use acceleration devices owned by their host machine, and acceleration device permissions are assigned during container creation. This allocation method cannot detect real-time acceleration device usage, easily leading to resource contention or idle resources, resulting in uneven resource utilization.

[0095] Therefore, accelerating the process of generating device invocation requests includes:

[0096] Obtain device information for at least one initial acceleration device;

[0097] Accelerate the construction of a resource pool of equipment based on equipment information;

[0098] Obtain scheduling information for acceleration devices in the container acceleration device resource pool of the compute server;

[0099] The scheduling information is used to generate a call request so as to receive call requests for calling acceleration devices within the acceleration device resource pool.

[0100] Figure 4 This is a schematic diagram of a pooling relationship provided in an embodiment of this application, such as... Figure 4 As shown, the initial acceleration device information is obtained, primarily through continuous collection of physical acceleration device information, such as: acceleration device model, acceleration device memory size, recent average utilization rate, acceleration device health status, and a list of Pods currently using this acceleration device. Based on this device information, an acceleration device resource pool is constructed. Here, this device information is reported to the scheduling module to form the acceleration device resource pool. It is also necessary to collect recent average CPU utilization information and some other optional information. "Recent" is a specific value, such as 1 minute. The dynamic link library is responsible for intercepting acceleration device API calls initiated by processes in the container. These calls include acceleration device driver APIs, acceleration device runtime APIs, and acceleration library APIs. After intercepting the acceleration device API calls, they are sent to the API forwarding server through the API forwarding client. The API forwarding server then accesses the acceleration device through the actual API interface and returns the result to the API forwarding client.

[0101] Figure 5 A schematic diagram illustrating the transmission between a computing server and a management server provided in this application embodiment, such as... Figure 5As shown, the management server includes a scheduling module, which comprises an acceleration device scheduling unit, a monitoring aggregation unit, and a container scheduling unit. The compute server includes an acceleration device pooling management module, which comprises an interface forwarding server management unit and a monitoring unit.

[0102] The Kubernetes service is deployed on both the management server and the compute server for unified management of server resources. The scheduling module is deployed on the management server and includes an acceleration device scheduling unit, a monitoring aggregation unit, and a Pod scheduling unit. The acceleration device pooling management module is deployed on the compute server and includes an interface forwarding server management unit and a monitoring unit. The acceleration device pooling management module and the scheduling module establish a connection for exchanging information.

[0103] In the acceleration device scheduling module, the Pod scheduling unit is responsible for scheduling Pods to a compute server, ensuring sufficient CPU and memory resources for operation. The monitoring aggregation unit is responsible for aggregating and collecting monitoring information reported by the acceleration device pooling management module to form a global resource status view. When the acceleration device scheduling unit receives an acceleration device invocation request, it allocates an acceleration device to the Pod from the acceleration device resource pool.

[0104] In the acceleration device pooling management module, the monitoring unit is responsible for collecting local resource status information and reporting it to the scheduling module. Upon receiving an acceleration device call request from an interface forwarding client, the interface forwarding server management unit creates and maintains the interface forwarding server status.

[0105] At the beginning of the process, the container on the computing server uses the scheduling information from the acceleration device to generate a call request. Figure 6 This application provides a schematic diagram of the process between a management server and a computing server, as illustrated in the embodiments of this application. Figure 6 As shown, the acceleration device pooling management module continuously collects physical acceleration device information (initial acceleration device device information), and reports this collected information to the scheduling module. The scheduling module then schedules the created containers into the compute server to initiate acceleration device call requests, i.e., generates call requests.

[0106] The method provided in this embodiment generates call requests for acceleration devices after pooling resources. By establishing pooled resources, resource idleness or contention can be avoided, thereby improving the efficiency of the scheduling process and the utilization rate of acceleration devices.

[0107] In some embodiments, obtaining scheduling information for the use of acceleration devices by containers on a computing server includes:

[0108] When the number of acceleration devices currently in use by a container is zero, the corresponding target acceleration device is determined within the acceleration device resource pool based on the amount of acceleration device resources that are greater than or equal to the pre-requested acceleration device resources; wherein, the target acceleration device for the same container is located on the same compute server;

[0109] Use the device information of the target acceleration device as scheduling information.

[0110] Specifically, in combination Figure 2 When creating a Pod, dynamic link libraries and API forwarding clients need to be mounted. The number of acceleration devices requested by the user, their utilization rate, and their GPU memory are written into the Pod annotation, but do not need to be added to the Pod's resource request field. After the Pod is created, the scheduling module listens for the Pod creation event from Kubernetes and then schedules it based on the Pod's resource requests. These resource requests can be other resource types besides acceleration devices, such as CPU resources and memory resources. When multiple compute servers can meet the Pod's resource requests, the compute server with the lowest recent average CPU utilization is selected. After scheduling, the Pod will start on this compute server, but if it does not initiate an acceleration device call request, it will not occupy any acceleration devices.

[0111] Each Pod can contain multiple containers, and each container can contain multiple application processes. Each application process has its own independent address space. When each application process in a Pod initiates a request to access an accelerated device, it triggers the loading of a dynamic link library. An initializer is embedded in the dynamic link library, and it is executed immediately upon the library's loading. The initializer sends the accelerated device request to the scheduling module. The scheduling module first checks if the Pod already has an accelerated device in use; if so, it directly allocates that device to the application process.

[0112] If the number of acceleration devices currently being used by this Pod (container) is 0, then a specified number of healthy acceleration devices are selected from the acceleration device resource pool. This specified number is the number of acceleration devices pre-requested by the container, i.e., the number of acceleration devices requested in the container annotations. The target acceleration devices are determined based on a quantity greater than or equal to the pre-requested acceleration device resources; that is, the number of acceleration devices to be invoked from the resource pool is determined, and the device information of this target acceleration device is used as scheduling information. Here, the target acceleration devices for the same container are located on the same compute server to facilitate subsequent invocation of acceleration devices from other compute servers and improve the orderliness of the invocation process. The acceleration device resources here refer to two parameters: idle utilization and remaining GPU memory.

[0113] If no acceleration device is available in the resource pool, the scheduling module needs to return information to the initialization program. The initialization program will then output logs to remind the program and retry requests based on time intervals to promptly identify available acceleration devices.

[0114] The process of obtaining scheduling information for the acceleration devices used by containers on the computing server provided in this embodiment requires filtering when a container does not have an acceleration device in use and needs to call an acceleration device in the resource pool. This is done to request acceleration devices based on the number of pre-requested devices, thereby avoiding idle resources in the resource pool and ensuring orderly calling.

[0115] In some embodiments, the process of determining the target acceleration device includes:

[0116] Obtain the first combination of acceleration devices and the corresponding number of first acceleration devices in the acceleration device resource pool;

[0117] If the number of first acceleration devices is greater than or equal to the pre-requested acceleration device resources, then the first acceleration device combination that is greater than or equal to the pre-requested acceleration device resources shall be used as the second acceleration device combination.

[0118] Obtain the utilization rate of the first acceleration device and the first video memory usage rate of the second acceleration device combination;

[0119] Constraints are determined based on the relationship between the acceleration equipment and the container in the second acceleration equipment assembly;

[0120] The first combination score of the second acceleration device combination is determined based on the utilization rate of the first acceleration device, the first video memory usage rate, and the constraint terms.

[0121] The highest combined score is selected from the first combined scores, and the acceleration device corresponding to the highest combined score is selected as the target acceleration device.

[0122] Specifically, the process of determining the target acceleration device can involve using a single acceleration device or a combination of multiple acceleration devices, which can be set according to the actual situation. In this application, to illustrate an embodiment of multiple acceleration device combinations, the number of first acceleration devices corresponding to each first acceleration device combination can be determined. When the number of first acceleration devices is greater than or equal to the number of pre-requested acceleration devices, such device combinations are selected as second acceleration device combinations. Based on the utilization rate of the first acceleration devices and the first memory occupancy rate of the second acceleration device combination, the idle utilization rate and remaining memory rate of the acceleration devices are obtained. Constraints are determined through the relationship between the acceleration devices and containers in the second acceleration device combination to ensure that acceleration devices on the same host are preferentially allocated to the Pod.

[0123] After obtaining the first combination score of all the second acceleration device combinations, the highest combination score is selected, and the acceleration device of the corresponding acceleration combination is used as the target acceleration device.

[0124] The target acceleration device determination process provided in this embodiment can calculate and determine the current acceleration device in real time when there is no acceleration device currently in use in the Pod. This improves real-time performance while constraining the priority allocation of acceleration devices on the same host, so as to improve scheduling efficiency in the future.

[0125] In some embodiments, determining a first combination score for the second acceleration device combination based on a first acceleration device utilization rate, a first video memory utilization rate, and a constraint term includes:

[0126] The corresponding first idle utilization rate is determined based on the utilization rate of the first acceleration device.

[0127] Determine the corresponding first video memory remaining rate based on the first video memory utilization rate;

[0128] Obtain the first idle utilization rate, the first remaining video memory rate, and the weight coefficients corresponding to the constraint terms, respectively.

[0129] The first combined score is determined based on the first idle utilization rate, the first remaining video memory rate, the constraint terms, and the corresponding weight coefficients.

[0130] Specifically, the formula for calculating the first combination fraction is as follows:

[0131] Score1=w1 (1 - Utilization rate of the first acceleration device) + w2 (1 - First memory usage) + w3 ;

[0132] Where w1 + w2 + w3 = 1, w1 corresponds to the weight coefficient of the first idle utilization rate (1 - first acceleration device utilization rate), w2 corresponds to the weight coefficient of the first video memory remaining rate (1 - first video memory occupancy rate), and w3 corresponds to the weight coefficient of the constraint term. Regarding the constraint term... If it is a local acceleration device, that is, an acceleration device within the computing server to which the container belongs, then it is 1; otherwise, it is 0.

[0133] The first indicator means that the lower the recent average utilization of the acceleration device, the higher the score. The second indicator means that the lower the memory usage of the acceleration device, the higher the score. The third indicator is a constraint, which ensures that acceleration devices on the same host are prioritized for allocation to the Pod. Finally, the scheduling module selects a specified number of acceleration devices to allocate to the Pod based on the scores, from highest to lowest.

[0134] The method for determining the first combined score provided in this embodiment improves the accuracy of the method by setting different weight coefficients and combining the parameter settings of the first idle utilization rate, the first remaining video memory rate and the constraint terms.

[0135] In other embodiments, the process of determining the target acceleration device includes:

[0136] Determine the combination of third acceleration devices corresponding to the computing server and the number of corresponding second acceleration devices;

[0137] Obtain the utilization rate of the second acceleration device and the second video memory usage rate of the third acceleration device combination;

[0138] The second combination score of the third acceleration device combination is determined based on the utilization rate of the second acceleration device and the utilization rate of the second video memory.

[0139] If the number of second acceleration devices is greater than or equal to the pre-requested acceleration device resources, then the third acceleration device combination that is greater than or equal to the pre-requested acceleration device resources will be used as the fourth acceleration device combination. The highest combination score will be selected from the second combination scores corresponding to the fourth acceleration device combination, and the acceleration device of the acceleration device combination corresponding to the highest combination score will be used as the target acceleration device.

[0140] If no acceleration device resources greater than or equal to the pre-requested acceleration device resources are found in the second number of acceleration devices, then the highest combination score is selected from the second combination score, and the acceleration device corresponding to the highest combination score is used as the target acceleration device.

[0141] It should be noted that the process for determining the fourth acceleration device combination can be referred to the above embodiments, and will not be repeated here. The process for determining the second combination score takes into account that the target acceleration device can be pre-locked when the current Pod is invoked, so as to facilitate direct invocation.

[0142] In some embodiments, determining a second combination score for the third acceleration device combination based on the second acceleration device utilization and the second video memory usage includes:

[0143] The corresponding second idle utilization rate is determined based on the utilization rate of the second acceleration device.

[0144] Determine the corresponding remaining rate of the second video memory based on the utilization rate of the second video memory;

[0145] Obtain the weighting coefficients corresponding to the second idle utilization rate and the second video memory remaining rate respectively;

[0146] The second combined score is determined based on the second idle utilization rate, the second remaining video memory rate, and the corresponding weighting coefficients.

[0147] The second combination score of the third acceleration device combination is determined based on the utilization rate of the second acceleration device and the utilization rate of the second video memory. The calculation formula is as follows:

[0148] Score2=w (1 - utilization rate of the second acceleration device) + (1 - w) (1 - Second video memory usage rate);

[0149] Where w corresponds to the weighting coefficient of the second idle utilization rate (1 - second acceleration device utilization rate), and 1 - w corresponds to the weighting coefficient of the second video memory remaining rate (1 - second video memory occupancy rate).

[0150] The second combination score determination process provided in this embodiment is achieved by pre-determining only two parameters during the locking process, thereby improving the ease of use.

[0151] After scoring, servers with higher acceleration device utilization and remaining GPU memory than the number of acceleration device resources requested in the Pod annotation are selected (i.e., the number of acceleration devices requested in the Pod annotation must be greater than the number of acceleration devices requested by the Pod). If no compute server meets this condition, the Pod is scheduled to the compute server with the highest total acceleration device score. If multiple compute servers meet this condition, the Top K compute servers with the highest total acceleration device scores are selected, where Top K represents the K acceleration devices with the highest scores, and K is the same as the number of acceleration devices requested in the Pod annotation.

[0152] These acceleration devices are directly allocated to this Pod. When a process in the Pod makes a request, it will directly retrieve this allocation result and will no longer participate in the original acceleration device allocation process. In this way, local acceleration devices can be locked for use by real-time tasks, ensuring performance requirements are met.

[0153] The alternative process for determining the target acceleration device provided in this embodiment is based on the parameter settings of idle utilization and video memory utilization combined with different weight coefficients, so as to reserve the local computing server for real-time tasks in advance to ensure performance requirements.

[0154] In some embodiments, sending the call request to the interface forwarding server corresponding to the application process includes:

[0155] When it is determined from the call request that the target container and the target acceleration device are located on the same computing server, a local interface forwarding server corresponding to the application process is created.

[0156] The call request is sent to the local interface forwarding server by using a shared memory handle.

[0157] Specifically, once the allocated acceleration device has been determined based on the call request, the connection address of the acceleration device pooling management module to which the acceleration device belongs is returned to the dynamic link library initialization program that initiated the acceleration device call request. The scheduling module also sends this allocation information to the acceleration device pooling management module to record the association between Pods, containers, processes, and acceleration devices. Then, the initialization program connects to the corresponding acceleration device pooling management module and requests the interface forwarding server.

[0158] The acceleration device pooling management module creates a new process. This new process determines whether the Pod and the acceleration device are on the same computing server based on the information allocated by the scheduling module. If so, it creates a local interface forwarding server and sends the call request to the local interface forwarding server through a shared memory handle.

[0159] This embodiment provides a way to simplify the sending process and improve sending efficiency when the target container and the target acceleration device are located on the same computing server, by forwarding the server-side call through the local interface.

[0160] In some embodiments, sending the call request to the interface forwarding server corresponding to the application process includes:

[0161] When it is determined from the call request that the target container and the target acceleration device are not located on the same computing server, a remote interface forwarding server corresponding to the application process is created.

[0162] The call request is sent to the remote interface forwarding server via network communication.

[0163] If they are not located on the same computing server, then a remote call is required. The call request is sent to the remote interface forwarding server via network communication. Specifically, this network communication uses the connection address and port of Transmission Control Protocol / Internet Protocol (TCP / IP) and Remote Direct Memory Access (RDMA). Figure 3 As can be seen from the corresponding diagram, this embodiment has both local and remote call capabilities.

[0164] The dynamic link library initializer records the connection information of the interface forwarding server for subsequent connection establishment. This new process also needs to specify an acceleration device visibility environment variable during creation. This environment variable restricts the acceleration device instances that the interface forwarding server can see; these acceleration device instances must be consistent with the acceleration devices allocated by the scheduling module.

[0165] In this embodiment, when the target container and the target acceleration device are not located on the same computing server, a remote computing server is invoked, demonstrating the process of pooling resources. This allows for the selection of an appropriate invocation method based on actual needs, thereby improving the flexibility of invocation.

[0166] In some embodiments, after the scheduling of the acceleration device is completed, the method further includes:

[0167] Terminate the application process;

[0168] Terminate the heartbeat signal of the acceleration device that calls the acceleration device resource pool;

[0169] Terminate the second thread corresponding to the interface forwarding server;

[0170] Update the container list of the compute server's containers that use acceleration devices to remove the target container corresponding to the target acceleration device from the container list.

[0171] Specifically, the initialization program continuously reports heartbeats to the acceleration device pooling management module. The heartbeat cycle is set according to actual conditions, such as 2 seconds. A termination program is also embedded in the dynamic link library. When the application process terminates, the dynamic link library is unloaded, triggering the termination program's execution. The termination program actively sends a heartbeat termination signal to the acceleration device pooling management module and then disconnects all connections. Upon receiving a heartbeat termination signal or when the heartbeat connection fails, the acceleration device pooling management module terminates the corresponding API forwarding server process. Determining a heartbeat connection failure can be achieved by setting a failure threshold, for example, setting the failure threshold to 5 heartbeat cycles. If no heartbeat is received within the specified threshold, it is considered a heartbeat failure. The acceleration device pooling management module also monitors the acceleration devices associated with processes within the same Pod. When all corresponding API forwarding server processes terminate, the acceleration device pooling management module updates the list of Pods currently using acceleration devices reported by the monitoring unit. The scheduling module sees the Pod removed from the list and considers the acceleration device allocated to that Pod to be released.

[0172] In other words, after scheduling the acceleration device, it is necessary to terminate the current application process, stop the heartbeat signal, terminate the use of the second thread, and update the list of containers currently using the acceleration device to ensure that the acceleration device allocated to the target container is released and made available for other Pods to use for their respective call requests.

[0173] This embodiment provides a way to release the acceleration device corresponding to the target container by ending the application process and terminating the second thread after the acceleration device scheduling is completed, thereby updating the container list and not delaying other call requests from calling the target acceleration device.

[0174] Furthermore, this application also provides a server cluster, such as Figure 3As shown, the server cluster includes a management server and multiple compute servers; the multiple compute servers are connected to the management server.

[0175] The management server executes the steps of the above-described acceleration device scheduling method to complete the scheduling of the acceleration devices.

[0176] For a description of the features in the embodiment corresponding to the server cluster, please refer to the relevant description in the embodiment corresponding to the scheduling method of the acceleration device, which will not be repeated here.

[0177] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0178] Embodiments of this application also provide a scheduling device for accelerating equipment. Figure 7 A structural diagram of a scheduling device for an acceleration device provided in an embodiment of this application is shown below. Figure 7 As shown, the device includes:

[0179] The receiving module 11 is used to initiate a call request to the acceleration device based on at least one first thread within the application process;

[0180] The sending module 12 is used to configure a corresponding interface forwarding client for at least one first thread so as to send the call request to the interface forwarding server corresponding to the application process;

[0181] Access module 13 is used to access the interface of the acceleration device by at least one second thread corresponding to the first thread within the interface forwarding server, so as to complete the scheduling of the acceleration device.

[0182] For a description of the features in the embodiment corresponding to the scheduling device of the acceleration device, please refer to the relevant description of the embodiment corresponding to the scheduling method of the acceleration device, which will not be repeated here.

[0183] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above embodiments of the scheduling method for the acceleration device.

[0184] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above embodiments of the scheduling method for acceleration devices when it is run.

[0185] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0186] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described embodiments of the scheduling method for acceleration devices.

[0187] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described embodiments of the scheduling method for acceleration devices.

[0188] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0189] The foregoing has provided a detailed description of the scheduling method, server cluster, device, medium, and product for an acceleration device provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A scheduling method for an acceleration device, characterized in that, include: A request to call the acceleration device is initiated by at least one first thread within the application process; At least one first thread is configured with a corresponding interface forwarding client to send the call request to the interface forwarding server corresponding to the application process, thereby avoiding thread safety issues caused by multiple threads sharing the same interface forwarding client. Each first thread has its own dedicated interface forwarding client, corresponding to a unique thread identifier. This refines one interface forwarding client into multiple clients, each assigned to a first thread within the application process. For dynamic allocation of multiple first threads, a thread pool is established to sort the current thread load and determine priority. Each thread independently manages its own interface forwarding client's connection, request queue, and timeout settings. When the application process initiates a call request, an initialization program embedded in the dynamic link library is triggered, allowing the scheduling module deployed on the management server to query whether the container corresponding to the application process has an acceleration device in use. If it does, the application process is directly allocated the currently used acceleration device; otherwise, an acceleration device is selected from the acceleration device resource pool. Within the interface forwarding server, at least one second thread corresponding to the first thread accesses the interface of the acceleration device requesting the call to complete the scheduling of the acceleration device. Specifically, a dedicated interface forwarding server is created for each application process to ensure timely cleanup of its resources after the application process terminates. The interface forwarding server is created and maintained by the interface forwarding server management unit of the acceleration device pooling management module within the computing server upon receiving an acceleration device call request from an interface forwarding client. The dynamic link library initialization program records the connection information of the interface forwarding server. Whenever an acceleration device call request is modified, the dynamic link library checks the client cache for a dedicated interface forwarding client based on the thread number executing the request. If no dedicated interface forwarding client is found, a new interface forwarding client is created, connects to the corresponding interface forwarding server, and is recorded in the client cache. The acceleration device call request is sent to the interface forwarding server process through the dedicated interface forwarding client, and the return result is received. A one-to-one mapping relationship is pre-established between the first thread and the second thread. Correspondingly, within the interface forwarding server, accessing the acceleration device through the interface of the acceleration device requesting the call via at least one second thread corresponding to the first thread includes: A second mapping relationship is pre-established between the first thread and the second thread; The corresponding second thread is determined based on the second mapping relationship to receive the call request; The call request is parsed and processed to obtain the corresponding interface function of the acceleration device; The interface function is mapped to the actual function address of the process occupied by the interface forwarding server; The actual function address is passed to the second thread to wake up the second thread to access the acceleration device; Correspondingly, the process of accelerating the generation of device invocation requests includes: Obtain device information for at least one initial acceleration device; An acceleration device resource pool is constructed based on the aforementioned device information; Obtain scheduling information of acceleration devices in the acceleration device resource pool used by containers on the computing server; specifically including: when the number of acceleration devices currently used by a container is zero, determining a corresponding target acceleration device within the acceleration device resource pool based on a quantity greater than or equal to the pre-requested acceleration device resources; using the device information of the target acceleration device as the scheduling information; wherein, the target acceleration devices corresponding to the same container are located on the same computing server; The scheduling information is used to generate the invocation request.

2. The scheduling method for the acceleration device according to claim 1, characterized in that, Sending the call request to the interface forwarding server corresponding to the application process includes: A first mapping relationship is pre-established, corresponding to one application process and one interface forwarding server. The call request is sent to the corresponding interface forwarding server according to the first mapping relationship.

3. The scheduling method for the acceleration device according to claim 1, characterized in that, A request to call the acceleration device is initiated by at least one first thread within the application process, including: Get the current thread load of the first thread; The priority order of the first thread is determined by sorting according to the current thread load; Obtain the arrival time sequence of the tasks corresponding to the application process; Based on the priority order and task arrival time order, the target first thread corresponding to the acceleration device is determined to initiate the corresponding call request.

4. The scheduling method for the acceleration device according to claim 3, characterized in that, The target first thread corresponding to the acceleration device is determined based on the priority order and task arrival time order, including: Obtain the priority level and first weight coefficient corresponding to the priority order; Obtain the task arrival time and second weight coefficient corresponding to the task arrival time sequence; The order in which the first thread initiates the call request is determined based on the priority level, the first weighting coefficient, the task arrival time, and the second weighting coefficient. The target first thread corresponding to the acceleration device is determined based on the order in which the first thread is initiated.

5. The scheduling method for the acceleration device according to claim 2, characterized in that, If the heartbeat signal of the application process is lost, the method further includes: Terminate the interface forwarding server corresponding to the application process; Clean up the resource information of the acceleration device corresponding to the application process and the resource information of the process occupied by the corresponding interface forwarding server.

6. The scheduling method for the acceleration device according to claim 5, characterized in that, Before terminating the interface forwarding server corresponding to the application process, the method further includes: Configure a new interface forwarding client for the first thread after termination; Establish a network between the new interface forwarding client and the interface forwarding server; The application process is controlled to send a heartbeat signal so that the interface forwarding server can obtain the first identifier string of the new interface forwarding client; Retrieve the second identifier string corresponding to the interface forwarding client; If the first identifier string and the second identifier string are the same, then the network connection is considered successful. If the first identifier string and the second identifier string are inconsistent, the network connection is determined to have failed, and the process proceeds to the step of terminating the interface forwarding server corresponding to the application process to terminate the new interface forwarding server.

7. The scheduling method for the acceleration device according to claim 1, characterized in that, The process of determining the target acceleration device includes: Obtain the first combination of acceleration devices and the corresponding number of first acceleration devices in the acceleration device resource pool; If the number of the first acceleration devices is greater than or equal to the pre-requested acceleration device resources, then the first acceleration device combination that is greater than or equal to the pre-requested acceleration device resources shall be used as the second acceleration device combination. Obtain the first acceleration device utilization rate and the first video memory usage rate of the second acceleration device combination; Constraints are determined based on the relationship between the acceleration devices in the second acceleration device assembly and the container; The first combination score of the second acceleration device combination is determined based on the utilization rate of the first acceleration device, the first video memory usage rate, and the constraint terms. The highest combined score is selected from the first combined scores, and the acceleration device corresponding to the highest combined score is used as the target acceleration device.

8. The scheduling method for the acceleration device according to claim 7, characterized in that, The first combination score of the second acceleration device combination is determined based on the utilization rate of the first acceleration device, the utilization rate of the first video memory, and the constraint terms, including: The corresponding first idle utilization rate is determined based on the utilization rate of the first acceleration device. Determine the corresponding first video memory remaining rate based on the first video memory utilization rate; Obtain the first idle utilization rate, the first remaining video memory rate, and the weight coefficients corresponding to the constraint terms, respectively. The first combined score is determined based on the first idle utilization rate, the first remaining video memory rate, the constraint terms, and the corresponding weight coefficients.

9. The scheduling method for the acceleration device according to claim 1, characterized in that, The process of determining the target acceleration device includes: Determine the combination of third acceleration devices corresponding to the computing server and the number of corresponding second acceleration devices; Obtain the utilization rate of the second acceleration device and the second video memory usage rate of the third acceleration device combination; The second combination score of the third acceleration device combination is determined based on the utilization rate of the second acceleration device and the utilization rate of the second video memory. If the number of the second acceleration devices is greater than or equal to the pre-requested acceleration device resources, then the third acceleration device combination that is greater than or equal to the pre-requested acceleration device resources is taken as the fourth acceleration device combination. The highest combination score is selected from the second combination scores corresponding to the fourth acceleration device combination, and the acceleration device of the acceleration device combination corresponding to the highest combination score is taken as the target acceleration device. If no acceleration device resources greater than or equal to the pre-requested acceleration device resources are found in the second number of acceleration devices, then the highest combination score is selected from the second combination score, and the acceleration device corresponding to the highest combination score is used as the target acceleration device.

10. The scheduling method for the acceleration device according to claim 9, characterized in that, The second combination score of the third acceleration device combination is determined based on the utilization rate of the second acceleration device and the utilization rate of the second video memory, including: The corresponding second idle utilization rate is determined based on the utilization rate of the second acceleration device. Determine the corresponding remaining rate of the second video memory based on the utilization rate of the second video memory; Obtain the weighting coefficients corresponding to the second idle utilization rate and the second video memory remaining rate respectively; The second combined score is determined based on the second idle utilization rate, the second remaining video memory rate, and the corresponding weighting coefficients.

11. The scheduling method for the acceleration device according to claim 1, characterized in that, Sending the call request to the interface forwarding server corresponding to the application process includes: When it is determined from the call request that the target container and the target acceleration device are located on the same computing server, a local interface forwarding server corresponding to the application process is created. The call request is sent to the local interface forwarding server using a shared memory handle.

12. The scheduling method for the acceleration device according to claim 1, characterized in that, Sending the call request to the interface forwarding server corresponding to the application process includes: When it is determined from the call request that the target container and the target acceleration device are not located on the same computing server, a remote interface forwarding server corresponding to the application process is created. The call request is sent to the remote interface forwarding server via network communication.

13. The scheduling method for the acceleration device according to claim 1, characterized in that, After the scheduling of the acceleration device is completed, the following steps are also included: Terminate the application process; Terminate the heartbeat signal of the acceleration device that calls the acceleration device resource pool; Terminate the second thread corresponding to the interface forwarding server; Update the container list of the computing server that uses acceleration devices to remove the target container corresponding to the target acceleration device from the container list.

14. A server cluster, characterized in that, The server cluster includes a management server and multiple computing servers; wherein the multiple computing servers are connected to the management server. The management server is configured to execute the steps of the scheduling method for the acceleration device according to any one of claims 1 to 13, so as to complete the scheduling of the acceleration device.

15. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the scheduling method of the acceleration device as described in any one of claims 1 to 13 when executing the computer program.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the scheduling method for the acceleration device as described in any one of claims 1 to 13.

17. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the scheduling method for the acceleration device as described in any one of claims 1 to 13.

Images