Multi-center heterogeneous computing power intelligent scheduling method, platform and equipment and storage medium

Abstract

The invention relates to the technical field of computing power resource scheduling, and relates to intelligent scheduling of multi-center heterogeneous computing power, comprising: acquiring training task demand information of a model training task; analyzing the training task demand information to obtain computing power demand information capable of meeting the training task demand; pooling and integrating the multi-center heterogeneous computing resources, and selecting current available computing power resources to obtain computing power resource reserve information; determining a computing power resource allocation scheme based on the computing power demand information and the computing power resource reserve information; and generating a control instruction based on the computing power resource allocation scheme, and controlling the computing node to execute the computing power task. The invention further provides a multi-center heterogeneous computing power intelligent scheduling system and device and a storage medium. According to the method, the utilization efficiency of the multi-center heterogeneous computing power resources can be improved, and the waste of the computing power resources is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of computing resource scheduling, and in particular to an intelligent scheduling method, platform, device and storage medium for multi-center heterogeneous computing power. Background Art

[0002] With the development of the digital economy, natural language processing models, speech models, and image processing models have been widely used in the field of artificial intelligence. In order to improve the efficiency of model training, a multi-center heterogeneous computing power platform is usually used to uniformly manage and schedule the computing power resources (such as CPU, GPU, FPGA, ASIC, etc.) of the multi-center cluster to ensure that various computing power resources can be reasonably allocated and optimized according to the task requirements of model training. At present, when training a neural network model, the scheduling of computing power resources mainly includes the following steps:

[0003] S1. Obtaining performance indicators of nodes and paths: First, it is necessary to obtain the performance indicators of each node, including processing speed, memory capacity, etc. Then, based on the performance indicators of these nodes, the performance indicators of the forward path are calculated.

[0004] S2. Select the optimal forward path: According to business requirements, select the forward path with the best performance indicators. This can be achieved by comparing the processing speed, delay and other indicators of different paths.

[0005] S3. Dynamically adjust resource allocation: During the training process, dynamically adjust resource allocation according to the actual operation situation. For example, if a node is overloaded, part of its tasks can be transferred to other nodes with lower loads to balance the overall load.

[0006] S4. Monitoring and optimization: Continuously monitor the operation status of the computing network, discover and solve potential problems in a timely manner, and improve resource utilization efficiency and reduce resource waste by optimizing algorithms and strategies.

[0007] Traditional computing power scheduling algorithms have many limitations, such as lack of global information, single scheduling strategy, unoptimized energy consumption, lack of dynamic incentive mechanism, waste of resources and lack of cluster collaboration. These problems limit the performance and efficiency of the algorithm in modern dynamic computing environments. More advanced scheduling strategies and technologies are urgently needed to overcome these challenges. Summary of the invention

[0008] In order to improve the utilization efficiency of multi-center heterogeneous computing resources and reduce the waste of computing resources, an intelligent scheduling method, platform, device and storage medium for multi-center heterogeneous computing resources are provided.

[0009] In order to achieve the above-mentioned object of the present invention, the present invention provides an intelligent scheduling method for multi-center heterogeneous computing power, including:

[0010] Obtain training task requirement information for model training tasks;

[0011] Analyze the training task requirement information to obtain the computing power requirement information that can meet the training task requirements;

[0012] Pool and integrate multi-center heterogeneous computing resources, select currently available computing resources to obtain computing resource reserve information;

[0013] Determine the computing power resource allocation plan based on computing power demand information and computing power resource reserve information;

[0014] Generate control instructions based on the computing resource allocation plan to control computing nodes to perform computing tasks.

[0015] Optionally, the computing power requirement information includes computing capacity requirement, memory requirement and task execution strategy requirement;

[0016] Determining the computing power resource allocation scheme based on the computing power demand information and the computing power resource reserve information includes:

[0017] Construct constraints for computing resource scheduling based on computing and memory requirements;

[0018] Select computing resources that meet the constraint conditions from currently idle computing resources to obtain a list of candidate computing resource allocation solutions;

[0019] Based on the task execution strategy requirements, the candidate computing power resource allocation schemes in the candidate computing power resource allocation scheme list are evaluated to determine the final computing power resource allocation scheme.

[0020] Optionally, the total video memory requirement S of the computing task is determined according to the computing demand and the memory demand. The total video memory requirement of the computing task is: S = Qb(2+p o )+S c +p m p cl p hl bp bs ;

[0021] Among them, Q is the number of model parameters of the model to be trained in the model training task, b is the number of bytes of the model accuracy of the model to be trained in the model training task, and p o represents the gradient function and / or optimizer parameters of the model to be trained, S c Represents the parameter occupancy value when the processor executes the calculation process; p m is the layer dimension size of the middle layer of the model to be trained, p cl is the layer dimension size of the hidden layer of the model to be trained, p hl is the maximum text length of the model to be trained, p bsis the number of hidden layers of the model to be trained;

[0022] According to the total video memory requirement S of the computing task, the computing resources that can meet the total video memory requirement of the computing task are selected from the currently idle computing resources to obtain a computing resource allocation plan.

[0023] Optionally, the task execution strategy includes one of a minimum cost strategy, a shortest execution time strategy, and an optimal energy consumption strategy; the method further includes:

[0024] Determine the cost function of intelligent scheduling based on the task execution strategy requirements;

[0025] Determine the computing power resource allocation plan from the candidate computing power allocation plans based on the cost function.

[0026] Optionally, the cost function of the minimum cost strategy is

[0027] Cost function of the minimum cost strategy represents the execution cost of the nth stage of a computing task in a cycle, where ds i Indicates the amount of data processed, Indicates the unit data cost of computing unit k that performs the computing task, where k is the index of the computing unit in the resource pool. is the execution time, is the cost of computing power unit k unit time;

[0028] The cost function of the strategy with the shortest execution time is

[0029] Cost function of the strategy with the shortest execution time Indicates the execution time of the nth stage of the computing task in a cycle, wt i Indicates the task waiting time, Represents the computational task t i The task execution time;

[0030] The cost function of the optimal energy consumption strategy is

[0031] The cost function of the optimal energy consumption strategy is It represents the energy consumption cost of the nth stage of the computing task in a cycle. represents the computational energy consumption of computing hardware resources, It represents the energy consumption per unit time. Indicates non-computing energy consumption.

[0032] Optionally, the evaluating the candidate computing resource allocation schemes in the candidate computing resource allocation scheme list based on the task execution strategy requirement to determine the final computing resource allocation scheme includes:

[0033] Use the Bayesian optimization method to evaluate the candidate computing resource allocation schemes in the candidate computing resource allocation scheme list, find the optimal solution under the task execution strategy selected by the user, and obtain the final computing resource allocation scheme;

[0034] Among them, the objective function and proxy model of Bayesian optimization are determined according to the model training task;

[0035] Determine the sampling function of Bayesian optimization according to the task execution strategy selected by the user;

[0036] The expression of the sampling function is A(X)=A0(X)-ωp(X),

[0037] A0(X) is the original acquisition function, ω is the weight parameter, and p(X) represents the penalty function;

[0038] A0(X)=E[max{0,f(X + )-f(X)}]

[0039] Among them, f(X) represents the cost function, X + To fit the current optimal solution in the solution process, E[.] represents the expectation.

[0040] Optionally, the method further comprises:

[0041] Monitor computing resource usage in real time and update computing resource reserve information based on computing resource usage;

[0042] Obtain the computing task execution status information of the model training task. The computing execution status information is used to characterize the computing task execution process of the computing power unit of each computing node;

[0043] The computing resource allocation plan is dynamically adjusted based on the updated computing resource reserve information and computing task execution status information with a period of time T.

[0044] In order to solve the above problems, the present invention also provides a multi-center heterogeneous computing power intelligent scheduling platform, including:

[0045] A computing power center, which includes a number of computing power resources of different types from different regions;

[0046] A task decomposition module, which is used to parse the training task requirement information to obtain computing power requirement information;

[0047] A resource pooling module, which is used to pool and integrate multi-center heterogeneous computing resources;

[0048] A scheduling module, which is used to execute the above-mentioned intelligent scheduling method for multi-center heterogeneous computing power, and control the computing power resources of the computing power center to execute the computing power resource allocation plan.

[0049] In order to solve the above problem, the present invention further provides an electronic device, the electronic device comprising:

[0050] at least one processor; and,

[0051] a memory communicatively connected to the at least one processor; wherein,

[0052] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the intelligent scheduling method of multi-center heterogeneous computing power described above.

[0053] In order to solve the above problems, the present invention also provides a computer-readable storage medium, in which at least one computer program is stored. The at least one computer program is executed by a processor in an electronic device to implement the above-mentioned intelligent scheduling method of multi-center heterogeneous computing power.

[0054] In summary, this application includes the following beneficial technical effects:

[0055] Pool and integrate heterogeneous computing resources distributed in different geographical locations and computing centers, and centrally manage and schedule them on the platform, effectively improving resource utilization and scheduling flexibility;

[0056] Determine the constraints of the computing resource scheduling problem based on computing power demand information, so that the system can select computing power resources that match the computing power demand from idle computing power resources based on computing power resource reserve information; use the Bayesian optimization method to determine the best computing power scheduling strategy. Compared with the traditional heuristic algorithm (minimum cost priority algorithm), the Bayesian optimization algorithm can more accurately determine the optimal strategy for computing power scheduling based on the constraints of computing power demand information, greatly reducing resource waste and improving utilization efficiency according to the strategy;

[0057] During the model training process, the system will continuously monitor resource usage to ensure that the task can use resources efficiently and flexibly. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A schematic diagram of a flow chart of an intelligent scheduling method for multi-center heterogeneous computing power provided by an embodiment of the present invention;

[0059] Figure 2 A schematic diagram of computing resource pooling provided by an embodiment of the present invention;

[0060] Figure 3 A schematic diagram of a computing power scheduling mechanism provided by an embodiment of the present invention;

[0061] Figure 4 A schematic diagram of a computing power scheduling algorithm provided by an embodiment of the present invention;

[0062] Figure 5 A processing flow chart of a Bayesian computing resource scheduling algorithm according to an embodiment of the present invention;

[0063] Figure 6 A schematic diagram of a flow chart of an intelligent scheduling method for multi-center heterogeneous computing power provided by another embodiment of the present invention;

[0064] Figure 7 An architecture diagram of a multi-center heterogeneous computing power scheduling platform provided in one embodiment of the present invention;

[0065] Figure 8 A schematic diagram of the structure of an electronic device for implementing the intelligent scheduling method of multi-center heterogeneous computing power provided in one embodiment of the present invention.

[0066] Reference numerals: 10, processor; 11, memory; 12, communication bus; 13, communication interface.

[0067] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0068] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0069] In the description of the present invention, it is necessary to understand that the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0070] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0071] Reference Figure 1 FIG. 1 is a flow chart of a method for intelligently scheduling multi-center heterogeneous computing power provided by an embodiment of the present invention. In this embodiment, the method for intelligently scheduling multi-center heterogeneous computing power includes:

[0072] S1. Obtain training task requirement information of a model training task.

[0073] The training task requirement information includes the type of model to be trained, the function of the model to be trained and the model parameters of the model to be trained; the models to be trained include image processing models, natural language processing models, speech recognition models and many other models, and the functions of the models to be trained include reading comprehension, arithmetic calculation, instruction compliance, data analysis or recognition classification, etc.

[0074] S2. Analyze the training task requirement information to obtain computing power requirement information that can meet the training task requirements.

[0075] Specifically, the training task requirement information is analyzed, including the computing type, memory requirements, expected running time, etc. required for model training. The computing amount required for the model training task can be determined based on the training task requirement information, so that the computing power scheduling platform can select a processor with computing power that matches the model training task to process the model training task based on the actual needs of the model training task.

[0076] Specifically, computing power requirement information includes computing power requirements, memory requirements, task execution strategy requirements, and processor type requirements.

[0077] S3. Pool and integrate multi-center heterogeneous computing resources, select currently available computing resources to obtain computing resource reserve information.

[0078] Reference Figure 2 In this embodiment, computing power resource pooling technology is used to perform time-sharing scheduling and management of AI chips. By defining AI computing power through software, heterogeneous computing resources distributed in different geographical locations and different computing power centers are pooled and integrated, and centrally managed and scheduled on the platform, effectively improving resource utilization and scheduling flexibility.

[0079] Resource pooling can rely on cloud computing technology to integrate heterogeneous computing resources such as CPU, GPU, ASIC, FPGA, etc., so as to achieve cross-center scheduling, breaking down the whole into parts, breaking down the parts into the whole, and dynamic scaling of computing resources.

[0080] In this embodiment, computing power resource pooling is implemented based on Kubernetes. The computing power resource pooling logically consists of three parts: resource management, resource proxy, and resource pooling runtime:

[0081] Resource management is based on Kubernetes native management service components and scheduling services. It transfers computing resource application requirements and schedules them according to the number of resources and idle resources. It is responsible for the scheduling and allocation of overall resources and is the core of resource management.

[0082] Reference Figure 2 The resource agent is responsible for redirecting resource access and cross-machine scheduling, ensuring the transparency and efficiency of resource access. It is responsible for creating PODs during the computing power scheduling process. In actual use, a virtual computing power card is added to the POD through the Kubernetes device plug-in, allowing the POD to use pooled computing power resources, realize cross-node resource calls, and support more types and models of hardware and new runtime functions.

[0083] The full name of POD in Kubernetes is "Plain Old Docker", but it is more widely understood as the abbreviation of "Process One Deployment", which means it is a simple unit for deploying a process. In Kubernetes, POD is the smallest deployment unit and can contain one or more containers; these containers share network and storage resources, and they are packaged together so that they can be managed and scheduled as a whole. The containers in POD can communicate with each other and use the same IP address. POD is the smallest unit for application deployment and expansion in Kubernetes.

[0084] The resource runtime is responsible for intercepting and forwarding resource access, achieving fine-grained management and optimization of resources, performing secondary packaging based on the native runtime of the AI ​​chip, and transferring computing tasks to the resource pool agent for execution. It should be noted that the "native runtime" is an execution environment tailored for artificial intelligence applications. It ensures that these applications can achieve fast data processing, model training, and automatic updates based on cloud computing by providing the necessary resources and optimization measures.

[0085] Resource pooling runtime refers to the secondary encapsulation and abstraction of the pooled computing resources on the basis of computing resource pooling to obtain computing resource reserve information to provide more efficient and flexible resource management and scheduling capabilities. It allows applications to apply for, use and manage the required computing resources through a unified interface without having to care about the specific details of the underlying resources, thereby improving the convenience of multi-center heterogeneous computing resource management.

[0086] S4. Determine a computing power resource allocation plan based on computing power demand information and computing power resource reserve information.

[0087] First, the constraints of the computing power scheduling problem were constructed according to the needs of users. Secondly, in view of the differences in users' computing task requirements, the execution strategies of various tasks were set based on the multi-path characteristics of the computing hardware cluster. At the same time, cost functions and reward functions were formulated according to the characteristics of different strategies, so that the system can form a computing power pool that meets the users' model training needs and provide them with use.

[0088] Reference Figure 1 and Figure 3 Specifically, step S4 includes:

[0089] S41. Construct constraints for computing resource scheduling problems based on computing demand and memory demand.

[0090] S42. Select computing resources that meet the constraint conditions from currently idle computing resources to obtain a list of candidate computing resource allocation solutions.

[0091] In the multi-center heterogeneous computing resource platform architecture, computing resources include multiple different types, such as GPU, CPU, ASIC or FPGA, etc. Different types of computing resources are divided into computing resource units of different types and sizes.

[0092] The collection of the total computer clusters of the multi-center heterogeneous computing power scheduling platform can be expressed as:

[0093] C={C g ,C c ,...,C n}

[0094] C g ,C c ,C n The computing clusters representing different types of computing resources can adjust the content of the total computer cluster data set according to the computing resources actually included in the multi-center heterogeneous computing scheduling platform, which is not limited in this embodiment;

[0095] In this embodiment, C g Refers to GPU-type computing clusters, C cRefers to CPU type computing cluster, C n Refers to NPU type computing cluster;

[0096] Further,

[0097] The set of computing clusters of GPU type computing clusters can be recorded as:

[0098]

[0099] They represent the point clouds of computing resource units of different sizes in the GPU type computing cluster, k is the index of the computing resource unit in the GPU type computing cluster, and each cluster contains a limited number of containers n k The number of containers and the computing resources of each container are automatically allocated by the dynamic balancing algorithm of the operation and maintenance end. A single cluster of GPU-type computing clusters The computing resources of the container The following constraints must be met:

[0100]

[0101] The set of computing clusters of CPU type computing clusters can be recorded as:

[0102]

[0103] They represent the point clouds of computing resource units of different sizes in the CPU type computing cluster, k is the index of the computing resource unit in the CPU type computing cluster, and each cluster contains a limited number of containers n k The number of containers and the computing resources of each container are automatically allocated by the dynamic balancing algorithm of the operation and maintenance end. A single cluster of CPU-type computing clusters The computing resources of the container The following constraints must be met:

[0104]

[0105] The set of computing clusters of NPU type computing clusters can be recorded as:

[0106]

[0107] They represent the point clouds of computing resource units of different sizes in the NPU type computing cluster, k is the index of the computing resource unit in the NPU type computing cluster, and each cluster contains a limited number of containers n k The number of containers and the computing power resources of each container are automatically allocated by the dynamic balancing algorithm of the operation and maintenance end. A single cluster of NPU type computing cluster The computing resources of the container The following constraints must be met:

[0108] S<∑S n ;

[0109] S is the total video memory requirement of a single task, and its value varies with the model parameters and training parameters. In this embodiment, the total video memory requirement S of the computing task is determined according to the computing demand and memory demand. The formula for the total video memory requirement of the computing task is: S = Qb(2+p o )+S c +p m p cl p hl bp bs ;

[0110] Where Q is the number of model parameters of the model to be trained in the model training task, b is the number of bytes of the model accuracy of the model to be trained in the model training task, (2+p o ) represents the memory value required by the model itself, p o represents the gradient function and / or optimizer parameters of the model to be trained; S c Represents the parameter occupancy value when the processor executes the calculation process; p m is the layer dimension size of the middle layer of the model to be trained, p cl is the layer dimension size of the hidden layer of the model to be trained, p hl is the maximum text length of the model to be trained, p bs is the number of hidden layers of the model to be trained.

[0111] It should be explained that the memory required by the model itself includes three parts: model parameters, gradients, and optimizer parameters. The amount of memory required for model parameters is equal to the number of value parameters multiplied by the memory required for each parameter. The memory required for gradients and model parameters is equal to the number of parameters multiplied by the memory required for each gradient parameter. Different optimizers store different amounts of parameters. For the commonly used adamW, twice the model parameters need to be stored (to store first-order and second-order momentum).

[0112] According to the total video memory requirement S of the computing task, the computing resources that can meet the total video memory requirement of the computing task are selected from the currently idle computing resources to obtain a computing resource allocation plan.

[0113] S43. Evaluate the candidate computing power resource allocation schemes in the candidate computing power resource allocation scheme list based on the task execution strategy requirements and determine the final computing power resource allocation scheme.

[0114] Determine the cost function of intelligent scheduling based on the task execution strategy requirements;

[0115] Determine the computing power resource allocation plan from the candidate computing power allocation plans based on the cost function.

[0116] In this embodiment, the task execution strategy includes one of the minimum cost strategy, the shortest execution time strategy and the optimal energy consumption strategy; for each strategy, its reward function needs to be defined separately. In this embodiment, we use the inverse of the cost function as the reward function, that is, under a given strategy, the lower the cost, the closer to the optimal solution.

[0117] Due to business requirements, the same task will not be processed by clusters with different hardware, so the subsequent computing clusters are represented by the total computer cluster C. For the tasks to be executed, we define the task set within the time period N as:

[0118] T={t0,t1,...,t n}

[0119] For any user computing task, at least one unit needs to have available container to execute the deployment. i , then the task duration can be recorded as the following set:

[0120]

[0121] Among them, at i It is the task i Arrival time; bt i It is the task i When executing collaborative tasks with multiple computing units, the additional time for data transmission; wt i It is the task i Waiting time of dl i It is the task i The execution period of the business logic determines that the period cannot be indefinite; ds i It is the data that needs to be processed by the task; is the set of containers on the kth cluster required by the task. i The execution time can be recorded as:

[0122]

[0123] Under the same hardware conditions, the algorithm operation efficiency of different containers is considered to be approximately the same. In the above formula, Denoted as the algorithm for data ds i The total processing rate.

[0124] The cost function of the minimum cost strategy is

[0125] Cost function of the minimum cost strategy represents the execution cost of the nth stage of a computing task in a cycle, where ds i Indicates the amount of data processed, Indicates the unit data cost of computing unit k that performs the computing task, where k is the index of the computing unit in the resource pool. is the execution time, is the cost of computing power unit k unit time;

[0126] The cost function of the strategy with the shortest execution time is

[0127] Cost function of the strategy with the shortest execution time Indicates the execution time of the nth stage of the computing task in a cycle, wt i Indicates the task waiting time, Represents the computational task t i The task execution time;

[0128] The cost function of the optimal energy consumption strategy is

[0129] The cost function of the optimal energy consumption strategy is It represents the energy consumption cost of the nth stage of the computing task in a cycle. represents the computational energy consumption of computing hardware resources, It represents the energy consumption per unit time. Indicates non-computing energy consumption, which refers to the energy consumption of the computer room, cooling, lighting, etc. It should be noted here that when different types of computing resources are selected as computing units, the energy consumption per unit time is Non-computational time The numerical value of is different.

[0130] Reference Figure 4 In the preferred implementation of this embodiment, the Bayesian optimization method is used to evaluate the candidate computing power resource allocation schemes in the candidate computing power resource allocation scheme list to find the optimal and final computing power resource allocation scheme under the task execution strategy selected by the user.

[0131] Among them, the objective function and proxy model of Bayesian optimization are determined according to the model training task;

[0132] The sampling function of Bayesian optimization is determined according to the task execution strategy selected by the user.

[0133] Specifically, when solving the optimal strategy using Bayesian optimization, it is necessary to incorporate constraints into the design of the acquisition function. Let the cost function be f(X), and the constraint function for executing the video memory requirement be g(X), where:

[0134] X=(x1,x2,...,x d ), is a d-dimensional vector of decision variables for computing power allocation, x1,x2,...,x d Represents the numerical values ​​of variables of different dimensions, and g(X)≤0.

[0135] Using Gaussian process as the proxy model, its mean function is: μ(X) = E|f(X)|, and the covariance function can be expressed as: k(X,X') = E|(f(X)-μ(X))(f(X')-μ(X'))|, where X' is the transposed vector of X. The Gaussian process is denoted as: f(X)~gp(μ(X),k(X,X')).

[0136] For the new point X * , the predicted mean and variance:

[0137]

[0138] Where X is the matrix of observed points, Y is the corresponding function value vector, K(X,X') is the covariance matrix between observed points, and k(X * ,X' * ) is the covariance matrix between the new observation points, k(X * ,X) is the covariance matrix between the new observation point and the observation point, is the noise variance, and I is the unit matrix. The covariance function can be regarded as the kernel function of the process. By adjusting the parameters of the kernel function, the Gaussian process can be fitted and predicted.

[0139] The expression of the acquisition function is:

[0140] The expression of the sampling function is A(X)=A0(X)-ωp(X),

[0141] A0(X) is the original acquisition function, ω is the weight parameter of the penalty function, and p(X) represents the penalty function;

[0142] In order to deal with nonlinear constraints, it is necessary to introduce a penalty function into the acquisition function. The penalty term in the penalty function can be one or more. In this algorithm, since there are only inequality constraints g(X), the commonly used preset penalty function p(X) is set as follows:

[0143]

[0144] Among them, α is the penalty factor, α is usually 2, m is the number of penalty terms in the penalty function, and each penalty term usually corresponds to a constraint condition, g i (X) represents the i-th constraint function.

[0145] In this embodiment, in the proxy model, the expected improvement is used as the original acquisition function, and the original acquisition function is expressed as:

[0146] A0(X)=E[max{0,f(X + )-f(X)}]

[0147] Among them, F(X) represents the cost function, X + To fit the current optimal solution in the solution process, E[.] represents the expectation.

[0148] By observing the acquisition function during the iteration process, the optimization strategy under the current strategy can be solved. For the specific algorithm flow, please refer to Algorithm 1. For Algorithm 1, please refer to Figure 5 .

[0149] After obtaining the optimal computing power allocation results, the platform will send a request to the cloud server group. Based on the results, the system will allocate different types of computing power cards in varying quantities to form a computing power pool that meets user needs. Compared with traditional heuristic algorithms (minimum cost priority algorithms), the Bayesian optimization algorithm can achieve a better solution, greatly reduce resource waste, and improve usage efficiency according to strategies.

[0150] S5. Generate control instructions based on the computing resource allocation plan to control the computing nodes to perform computing tasks.

[0151] Reference Figure 6 In another embodiment of the present application, the intelligent scheduling method of multi-center heterogeneous computing power also includes:

[0152] S6. Monitor the usage of computing resources in real time and update the computing resource reserve information based on the usage of computing resources.

[0153] S7. Obtain computing task execution status information of the model training task.

[0154] The computing execution status information is used to characterize the computing task execution process of the computing power unit of each computing node.

[0155] S8. Dynamically adjust the computing resource allocation plan based on the updated computing resource reserve information and computing task execution status information with a period of time T.

[0156] The user can set the value of the time T according to actual needs, and this embodiment does not impose any limitation.

[0157] During the model training process, the system will continuously monitor resource usage to ensure that the task can use resources efficiently and flexibly; during task execution, application monitor technology is used to maintain the application status and virtualize the interaction with the system to ensure consistency when executing in multiple locations; after the task is completed, the resources will be recycled and put back into the resource pool for use by other tasks, realizing the recycling of resources.

[0158] Reference Figure 7 Based on the same inventive concept, an embodiment of the present invention provides a multi-center heterogeneous computing power intelligent scheduling platform.

[0159] The computing center includes a number of different types of computing resources from different regions;

[0160] Task decomposition module: The task decomposition module is used to analyze the training task requirement information and obtain the computing power requirement information;

[0161] Resource pooling module, which is used to pool and integrate multi-center heterogeneous computing resources;

[0162] The scheduling module is used to execute the intelligent scheduling method of multi-center heterogeneous computing power and control the computing power resources of the computing power center to execute the computing power resource allocation plan.

[0163] The module described in the present invention may also be referred to as a unit, which refers to a series of computer program segments that can be executed by a processor of an electronic device and can complete fixed functions, and is stored in a memory of the electronic device.

[0164] The various variations and specific examples of the intelligent scheduling method for multi-center heterogeneous computing power provided in the above-mentioned embodiment are also applicable to a multi-center heterogeneous computing power intelligent scheduling platform in the present embodiment. Through the above-mentioned detailed description of the intelligent scheduling method for multi-center heterogeneous computing power, those skilled in the art can clearly understand the implementation method of the multi-center heterogeneous computing power intelligent scheduling platform in the present embodiment. For the sake of brevity of the specification, it will not be described in detail here.

[0165] The present application also discloses an electronic device, such as Figure 8 , which is a schematic diagram of the structure of an electronic device for an intelligent scheduling method for multi-center heterogeneous computing power provided by an embodiment of the present invention. The electronic device may include at least one processor 10, a memory 11 connected to at least one processor for communication, a communication bus 12, and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a method program for intelligent scheduling of multi-center heterogeneous computing power.

[0166] Among them, the processor 10 can be composed of an integrated circuit in some embodiments, for example, it can be composed of a single packaged integrated circuit, or it can be composed of multiple integrated circuits with the same function or different functions, including one or more central processing units (CPU), microprocessors, digital processing chips, graphics processors and various control chips. The processor 10 is the control core (Control Unit) of the electronic device, which uses various interfaces and lines to connect the various components of the entire electronic device, and executes or executes the programs or modules stored in the memory 11 (such as the method of executing the intelligent scheduling of multi-center heterogeneous computing power, etc.), and calls the data stored in the memory 11 to execute various functions of the electronic device and process data.

[0167] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 may also be an external storage device of an electronic device, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Furthermore, the memory 11 may also include both an internal storage unit of the electronic device and an external storage device. The memory 11 can not only be used to store application software and various types of data installed in the electronic device, such as the code of the intelligent scheduling method program of multi-center heterogeneous computing power, but also can be used to temporarily store data that has been output or is to be output.

[0168] The communication bus 12 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory 11 and at least one processor 10, etc.

[0169] The communication interface 13 is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally, the user interface may also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface.

[0170] Figure 8 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 8 The structure shown does not constitute a limitation on the electronic device, and may include fewer or more components than shown, or combine certain components, or arrange the components differently. For example, although not shown, the electronic device may also include a power supply (such as a battery) for supplying power to each component. Preferably, the power supply may be logically connected to at least one processor 10 through a power management device, so that functions such as charging management, discharging management, and power consumption management are implemented through the power management device. The power supply may also include any components such as one or more DC or AC power supplies, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, etc. The electronic device may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0171] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited by this structure.

[0172] Furthermore, if the module / unit integrated in the electronic device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile.

[0173] The embodiment of the present application provides a computer-readable storage medium, for example, including: any entity or device, recording medium, USB flash drive, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM) that can carry the computer program code. The computer-readable storage medium stores a computer program that can be loaded by a processor and execute the intelligent scheduling method of multi-center heterogeneous computing power of the above embodiment.

[0174] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", "an implementation", "a preferred implementation" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0175] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. An intelligent scheduling method for multi-center heterogeneous computing power, characterized in that: The method comprises: Obtain training task requirement information for model training tasks; Analyze the training task requirement information to obtain the computing power requirement information that can meet the training task requirements; Pool and integrate multi-center heterogeneous computing resources, select currently available computing resources to obtain computing resource reserve information; Determine the computing power resource allocation plan based on computing power demand information and computing power resource reserve information; Generate control instructions based on the computing resource allocation plan to control computing nodes to perform computing tasks.

2. The intelligent scheduling method for multi-center heterogeneous computing power according to claim 1, characterized in that: Computing power demand information includes computing capacity demand, memory demand, and task execution strategy demand; Determining the computing power resource allocation scheme based on the computing power demand information and the computing power resource reserve information includes: Construct constraints for computing resource scheduling based on computing and memory requirements; Select computing resources that meet the constraint conditions from currently idle computing resources to obtain a list of candidate computing resource allocation solutions; Based on the task execution strategy requirements, the candidate computing power resource allocation schemes in the candidate computing power resource allocation scheme list are evaluated to determine the final computing power resource allocation scheme.

3. The intelligent scheduling method for multi-center heterogeneous computing power according to claim 2, characterized in that: The total video memory requirement S of the computing task is determined based on the computing demand and memory demand. The formula for the total video memory requirement of the computing task is: S = Qb(2+p o )+S c +p m p cl p hl bp bs ; Among them, Q is the number of model parameters of the model to be trained in the model training task, b is the number of bytes of the model accuracy of the model to be trained in the model training task, and p o represents the gradient function and / or optimizer parameters of the model to be trained, S c Represents the parameter occupancy value when the processor executes the calculation process; p m is the layer dimension size of the middle layer of the model to be trained, p cl is the layer dimension size of the hidden layer of the model to be trained, p hl is the maximum text length of the model to be trained, p bs is the number of hidden layers of the model to be trained; According to the total video memory requirement S of the computing task, the computing resources that can meet the total video memory requirement of the computing task are selected from the currently idle computing resources to obtain a computing resource allocation plan.

4. The intelligent scheduling method for multi-center heterogeneous computing power according to claim 3, characterized in that: The task execution strategy includes one of a minimum cost strategy, a shortest execution time strategy and an optimal energy consumption strategy; the method further includes: Determine the cost function of intelligent scheduling based on the task execution strategy requirements; Determine the computing power resource allocation plan from the candidate computing power allocation plans based on the cost function.

5. The intelligent scheduling method for multi-center heterogeneous computing power according to claim 4, characterized in that: The cost function of the minimum cost strategy is Cost function of the minimum cost strategy represents the execution cost of the nth stage of a computing task in a cycle, where ds i Indicates the amount of data processed, Indicates the unit data cost of computing unit k that performs the computing task, where k is the index of the computing unit in the resource pool. is the execution time, is the cost of computing power unit k unit time; The cost function of the strategy with the shortest execution time is Cost function of the strategy with the shortest execution time Indicates the execution time of the nth stage of the computing task in a cycle, wt i Indicates the task waiting time, Represents the computational task t i The task execution time; The cost function of the optimal energy consumption strategy is The cost function of the optimal energy consumption strategy is It represents the energy consumption cost of the nth stage of the computing task in a cycle. represents the computational energy consumption of computing hardware resources, It represents the energy consumption per unit time. Indicates non-computing energy consumption.

6. The intelligent scheduling method for multi-center heterogeneous computing power according to claim 4 or 5, characterized in that: The step of evaluating the candidate computing resource allocation schemes in the candidate computing resource allocation scheme list based on the task execution strategy requirements to determine the final computing resource allocation scheme includes: Use the Bayesian optimization method to evaluate the candidate computing resource allocation schemes in the candidate computing resource allocation scheme list, find the optimal solution under the task execution strategy selected by the user, and obtain the final computing resource allocation scheme; Among them, the objective function and proxy model of Bayesian optimization are determined according to the model training task; Determine the sampling function of Bayesian optimization according to the task execution strategy selected by the user; The expression of the sampling function is A(X)=A0(X)-ωp(x), A0(X) is the original acquisition function, ω is the weight parameter, and p(X) represents the penalty function; A0(X)=E[max{0,f(X + )-f(X)}] Among them, f(X) represents the cost function, X + To fit the current optimal solution in the solution process, E[.] represents the expectation.

7. The intelligent scheduling method for multi-center heterogeneous computing power according to any one of claims 1 to 6, characterized in that: The method further comprises: Monitor computing resource usage in real time and update computing resource reserve information based on computing resource usage; Obtain the computing task execution status information of the model training task. The computing execution status information is used to characterize the computing task execution process of the computing power unit of each computing node; The computing resource allocation plan is dynamically adjusted based on the updated computing resource reserve information and computing task execution status information with a period of time T.

8. A multi-center heterogeneous computing power intelligent scheduling platform, characterized in that: include: A computing power center, which includes a number of computing power resources of different types from different regions; A task decomposition module, which is used to parse the training task requirement information to obtain computing power requirement information; A resource pooling module, which is used to pool and integrate multi-center heterogeneous computing resources; A scheduling module, the scheduling module is used to execute the intelligent scheduling method of multi-center heterogeneous computing power as described in one of claims 1 to 7, and control the computing power resources of the computing power center to execute the computing power resource allocation plan.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor (10); and, a memory (11) communicatively connected to the at least one processor (10); The memory (11) stores a computer program that can be executed by the at least one processor (10), and the computer program is executed by the at least one processor (10) so that the at least one processor (10) can execute the intelligent scheduling method of multi-center heterogeneous computing power as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer storage medium stores a computer program; when the computer program is executed by a processor, the intelligent scheduling method for multi-center heterogeneous computing power as described in any one of claims 1 to 7 is implemented.

Images