Calculation power scheduling method and system under provincial regional calculation power network
Patent Information
- Application Number
- CN202411142489.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-11-12
Smart Images

Figure CN118939402A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of computing power network scheduling, and more specifically, relates to a distributed computing power scheduling method and system under a provincial regional computing power network. Background Art
[0002] The demand for computing power continues to expand in all walks of life, and the demand for inclusive, flexible and efficient computing power services is bursting out. The increasing number of intelligent products and applications are also constantly putting forward higher requirements for computing power, making computing power networks a focal area of global technological innovation and industrial service competition. Various provinces in my country have also begun to build computing power network-related infrastructure, such as supercomputing centers and data centers, hoping to rely on these facilities to establish provincial regional computing power networks to improve the computing power within the province.
[0003] At present, the research on provincial regional computing power scheduling is still in its infancy. The first method is to gradually build a scheduling strategy by selecting the current optimal solution through a greedy algorithm, such as Earliest Deadline First (EDF), and select the earliest completion time in each step; the second method is to search for the optimal strategy through a genetic algorithm to simulate natural selection and genetic mechanisms. It mainly searches for the optimal scheduling strategy by encoding, evaluating, selecting, mutating, and repeating the strategy; the third method is to predict task execution time and resource requirements by training neural networks (such as recurrent neural networks RNN), thereby guiding scheduling strategies.
[0004] However, the above existing scheduling methods all have some non-negligible defects:
[0005] First, the scheduling strategy using the greedy algorithm is likely to cause the final scheduling strategy to fall into the local optimal situation. This is mainly because it selects the current optimal solution at each step and lacks a global perspective.
[0006] Second, the scheduling strategy using genetic algorithms may result in a long search time due to the large scale of the problem or the general effect of each mutation, which leads to a long wait and inability to schedule;
[0007] Third, genetic algorithms cannot handle problems with constraints. If the tasks in the computing network have deadline constraints, then using genetic algorithms will not be able to obtain the optimal solution that satisfies the constraints.
[0008] Fourth, since the computing power network environment may change, the pre-trained neural network may find it difficult to adapt to the scheduling strategy based on the neural network, which may cause the scheduling strategy it predicts to no longer be appropriate. Summary of the invention
[0009] In response to the above defects or improvement needs of the prior art, the present invention provides a distributed computing power scheduling method and system under a provincial regional computing power network, which aims to solve the technical problem that the existing scheduling strategy using a greedy algorithm easily causes the final scheduling strategy to fall into a local optimal situation, and the technical problem that the existing scheduling strategy based on a genetic algorithm may cause the search time to be too long due to the large scale of the problem or the general effect of each mutation, resulting in a long waiting time and inability to schedule, and the technical problem that the existing scheduling strategy based on a genetic algorithm cannot obtain the optimal solution that satisfies the constraints when the tasks in the computing power network have deadline constraints, and the technical problem that when the computing power network environment changes, the existing scheduling strategy based on a neural network will have difficulty in adapting to such changes with a pre-trained neural network, resulting in its predicted scheduling strategy being no longer suitable.
[0010] To achieve the above object, according to one aspect of the present invention, a computing power scheduling method under a provincial regional computing power network is provided, comprising the following steps:
[0011] (1) Obtain heterogeneous computing resources within the province and model them using a two-level linkage architecture based on "K8S+OpenStack" to obtain multiple unified computing resources and unified operation tasks;
[0012] (2) All unified job tasks obtained in step (1) are matched with unified computing resources to obtain the optimal scheduling strategy in the global case.
[0013] (3) Obtain a service information flow table according to the optimal scheduling strategy in the global situation obtained in step (2), and schedule all unified job tasks obtained in step (1) according to the generated service information flow table to obtain a scheduling result.
[0014] Preferably, step (1) comprises the following steps:
[0015] (1-1) Obtaining heterogeneous computing resources in the province, and using OpenStack to virtualize the basic physical resources in the computing resources, where the basic physical resources include computing resources and storage resources, to obtain multiple virtualized virtual machines;
[0016] (1-2) Using a K8S cluster to further virtualize the multiple virtual machines obtained in step (1-1) to obtain multiple containers, each container including multiple computing resources and multiple job tasks;
[0017] (1-3) Obtain the unified computing resources and unified job tasks corresponding to each container according to the computing resources and job tasks in each container obtained in step (1-2);
[0018] Preferably, the unified computing resources of the container are specifically:
[0019]
[0020] Among them C br represents unified computing power resources, n represents the total number of logic operation chips in the container, m represents the total number of parallel computing chips in the container, p represents the total number of neural network acceleration chips in the container, f() is the computing power mapping function, a i represents the i-th logic operation chip in the container, α i represents the mapping ratio coefficient of the i-th logic operation chip in the container, β j represents the mapping ratio coefficient of the jth parallel computing chip in the container, b j represents the jth parallel computing chip in the container, γ k represents the mapping ratio coefficient of the kth neural network acceleration chip in the container, c k It represents the kth neural network acceleration chip in the container. q1(), q2() and q3() respectively represent the redundant computing power of logical operations, redundant computing power, and redundant computing power of neural network acceleration of the container. TOPS represents the number of logical operations that the processor in the container can perform per second. FLOPS represents the floating-point computing capability of the container.
[0021] Preferably, for the jth job task in the cth container, its corresponding unified job task W c,j It can be represented as the following triple:
[0022] W c,j =(α c,j ,d c,j ,G c,j )
[0023] where c∈[1, the total number of containers obtained in step (1-2)], α c,j represents the arrival time of the jth unified job task in the cth container, j∈[1, the total number of job tasks in the container], d c,j represents the deadline of the jth unified job task of the cth container, G c,j Represents the data dependency structure of the jth unified job task of the cth container, and the data dependency structure can be represented as a directed acyclic graph:
[0024] G c,j =(V c,j ,E c,j )
[0025] Where V c,j = {v j,1 ,v j,2 ,...,vj,m} represents the jth unified job task W of the cth container c,j The unit task set consisting of all m unit tasks in , Represents the set of directed arcs between the sets of unit tasks in the jth unified job task of the cth container.
[0026] Preferably, step (2) comprises the following steps:
[0027] (2-1) Set the iteration counter cnt = 1, and use the particle swarm optimization algorithm to match all the unified computing resources obtained in step (1) with the unified job tasks to obtain multiple feasible scheduling solutions;
[0028] (2-2) Take all feasible scheduling schemes obtained in step (2-1) as the population and obtain the personal best particle of each particle in the population at the cnt iteration And the global best particle gB of the population cnt , and judge whether cnt is greater than the preset maximum number of iterations i max If yes, go to step (2-5), otherwise go to step (2-3);
[0029] (2-3) The personal best particle of the kth particle in the population at the cntth iteration obtained using step (2-2) And the global best particle gB of the population cnt Get the kth particle at the cnt+1th iteration, and all the particles at the cnt+1th iteration form a new population;
[0030] (2-4) Set counter cnt=cnt+1 and return to step (2-2);
[0031] (2-5) The global best particle gB of the population cnt It is output as the optimal scheduling strategy in the global case.
[0032] Preferably, step (2-1) comprises the following sub-steps:
[0033] (2-1-1) For the cth container D among all the containers obtained in step (1-2) c For the jth unified task W corresponding to the cth container obtained in step (1-3), c,j Perform allocation processing to obtain the jth unified operation task W corresponding to the cth container c,j The corresponding waiting queue Q c,j ;
[0034] (2-1-2) For each queue Q to be processed obtained in step (2-1-1) c,jFor example, the waiting queue Q c,j Process the highest priority entry unit task v j,1 Dispatched to the pending queue Q c,j Corresponding container D c to process;
[0035] (2-1-3) In the pending queue Q c,j Corresponding container D c After processing the entry unit task obtained in step (2-1-2), obtain the entry unit task v j,1 The task calculation time t tc (v j,1 ,D c );
[0036] (2-1-4) Traverse the unified job task W c,j All remaining successor unit tasks v in j,m , to obtain the data transmission time and task calculation time of each subsequent unit task, and for the unified job task W c,j The data transmission time and task calculation time of all unit tasks in the task are summed up to obtain the unified task W. c,j The total data transmission cost t dt (c,j) and total task computation time t tc (c, j), where m∈[2, the total number of unit tasks of the jth unified job task];
[0037] (2-1-5) Each unified task W obtained according to step (2-1-4) c,j The total data transmission cost t dt (c,j) and total task computation time t tc (c,j) Get the unified job task W c,j The computational cost t c,j , and obtain multiple feasible scheduling solutions based on the computational costs generated by all unified job tasks;
[0038] Preferably, step (2-1-3) is to obtain the entry unit task v according to the following formula: j,1 The task calculation time t tc (v j,1 ,D c ):
[0039]
[0040] where τ j,1 Indicates the entry unit task v j,1 The total amount of calculation, C br Represents container D cThe unified computing resources, that is, the computing power of the container;
[0041] Step (2-1-4) is to calculate v according to the following formula j,m Data transfer time
[0042]
[0043] in represents the mth successor unit task v of the jth unified task j,m Need to receive its pth predecessor unit task v j,p (refers to the need to v j,m The data transmitted by the unit task that was previously executed first is e, is the mth successor unit task v of the jth unified task j,m Need to receive its pth predecessor unit task v j,p The total amount of data transmitted, b p,m is the task v from the pth predecessor unit j,p To the mth successor unit task v j,m The transmission bandwidth is p∈[1,m].
[0044] Step (2-1-4) is to calculate the container D according to the following formula c The subsequent unit task v j,m The task calculation time t tc (v j,m ,D c ):
[0045]
[0046] where τ j,m Indicates that the successor unit task v j,m The total amount of calculation, C br Represents container D c The unified computing resources of the container, that is, the computing power of the container.
[0047] In step (2-1-5), each unified task W c,j The computational cost of c,j Including its total task calculation time t tc (c,j) and the total data transmission cost t dt (c,j), if its corresponding scheduling scheme does not meet the deadline constraint, that is, any unified job task W c,j The arrival time α c,j Plus the computational cost t c,j Exceed its deadline c,j, then the scheduling scheme is not feasible and is abandoned; otherwise, the scheduling scheme for the unified task is feasible. Finally, the feasible scheduling schemes corresponding to all unified tasks are obtained.
[0048] Preferably, step (2-2) is specifically,
[0049] First, the following formula is used to calculate the kth particle in the population at the cntth iteration: The fitness of QPF based on quadratic penalty function is:
[0050]
[0051] where k∈[1, total number of particles in the population], represents the kth particle in the population at the cntth iteration; σ cnt represents the penalty factor σ of the cnt-th iteration; represents the penalty function, which is used to force the current particle to move towards the feasible region. Specifically, the particle's time constraint is used as its penalty term. c,j represents the arrival time of the jth unified job task of the cth container, t c,j is the computational cost of the jth unified job task of the cth container, d c,j represents the deadline of the jth unified job task of the cth container; represents the kth particle in the population at the cntth iteration The computational cost t of the jth unified job task of the cth container c,j .
[0052] Then, the particle corresponding to the maximum value is selected from the multiple QPF fitness of the kth particle in the first iteration, the second iteration, ... the cntth iteration as the personal best particle of the kth particle at the cntth iteration.
[0053] Finally, the particle with the maximum QPF fitness is selected from the personal best particles of all particles in the population as the global best particle gB of the population at the cntth iteration. cnt ;
[0054] Step (2-3) is to obtain the kth particle at the cnt+1th iteration according to the following formula:
[0055]
[0056] Among them, c1 and c2 are acceleration coefficients, and ⊙ represent the crossover operator and mutation operator of the genetic algorithm respectively, w is the inertia factor, and:
[0057]
[0058] where w max and w min represent the maximum and minimum values of w respectively, It indicates whether the kth particle is updated at the cntth iteration, NP represents the size of the population, and F() represents the QPF fitness of the particle.
[0059] Preferably, step (3) comprises the following sub-steps:
[0060] (3-1) Obtain the unified computing power resources for each container according to the optimal scheduling strategy in the global situation obtained in step (2);
[0061] (3-2) Loading the unified computing resources obtained in step (3-1) into the IP protocol or routing protocol through a router;
[0062] (3-3) performing network topology calculation according to the loaded IP protocol or routing protocol obtained in step (3-2) to obtain a service information flow table, and sending the service information flow table to the router;
[0063] Specifically, this step first constructs a topology map containing all containers and their connection relationships based on the information of the loaded IP protocol or routing protocol. Then, according to the optimal scheduling strategy and the topology map obtained in step (2), a corresponding flow table entry is generated for each unified job task. The flow table entry specifies the matching rules and corresponding processing actions of each unified job task and the path of its flow in the topology map. All flow table entries constitute a service information flow table.
[0064] (3-4) All unified job tasks obtained in step (1) are scheduled through the router according to the generated service information flow table to obtain a scheduling result.
[0065] According to another aspect of the present invention, a computing power scheduling system under a provincial regional computing power network is provided, comprising:
[0066] The first module is used to obtain heterogeneous computing resources within the province, and model the computing resources using a two-level linkage architecture based on "K8S+OpenStack" to obtain multiple unified computing resources and unified operation tasks;
[0067] The second module is used to match all unified job tasks obtained by the first module with unified computing resources to obtain the optimal scheduling strategy under global conditions.
[0068] The third module is used to obtain a service information flow table according to the optimal scheduling strategy in the global situation obtained by the second module, and schedule all unified job tasks obtained by the first module according to the generated service information flow table to obtain a scheduling result.
[0069] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:
[0070] (1) The present invention adopts step (2-2) and step (2-3), which updates the personal best particle and the global best particle in each iteration, and uses the inertia factor to adjust the search ability of the algorithm, thereby solving the technical problem that the existing scheduling method based on the greedy algorithm is easy to fall into the local optimum;
[0071] (2) The present invention adopts step (2-2), which uses QPF fitness to evaluate particles and uses a penalty factor to speed up the acquisition of feasible solutions, thereby solving the technical problem that the existing scheduling method based on genetic algorithm has a long search time due to the increase in problem scale;
[0072] (3) Since the present invention adopts step (2-1-5), it abandons the corresponding scheduling plan when the corresponding scheduling plan does not meet the deadline constraint, thereby solving the technical problem that the existing scheduling method based on genetic algorithm cannot handle constraint conditions;
[0073] (4) Since the present invention adopts step (2), it sets iterations and calculates the task execution time and data transmission time in each iteration, and continuously updates the optimal strategy through iteration. Therefore, it can solve the technical problem that the existing scheduling method based on neural network cannot adapt to changes in the network environment. BRIEF DESCRIPTION OF THE DRAWINGS
[0074] Figure 1 It is a flow chart of the computing power scheduling method under the provincial regional computing power network of the present invention;
[0075] Figure 2 is an example diagram of the unified operation task obtained in steps (1-3) of the method of the present invention;
[0076] Figure 3 is a detailed flow chart of step (2) in the method of the present invention;
[0077] Figure 4 It is an application environment diagram of the method of the present invention. DETAILED DESCRIPTION
[0078] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0079] The basic idea of the present invention is to firstly unify the modeling of heterogeneous computing resources through the existing virtualization technology; then adopt an improved particle swarm optimization algorithm as the scheduling strategy, and adopt a quadratic penalty function suitable for multi-constraint combinatorial optimization problems to speed up the acquisition of feasible solutions to obtain the optimal scheduling strategy; finally, build a service information flow table based on the computing priority network protocol to guide the scheduling of the final unified job tasks in the network.
[0080] The present invention designs a computing power scheduling method under a provincial regional computing power network, and the process is as follows: Figure 1 As shown in the figure, this strategy is based on the load modeling technology of cross-domain scheduling, which can perceive and quantify resources and distribute them with the goal of optimization. The implementation process involves Remote Procedure Call Protocol (RPCP), Real-time Communication Docker (RTCD), Load Balancing, Computing First Network (CFN), etc.
[0081] like Figure 1 As shown, the present invention provides a computing power scheduling method under a provincial regional computing power network, comprising the following steps:
[0082] (1) Obtain heterogeneous computing resources within the province and model them using a two-level linkage architecture based on "K8S+OpenStack" to obtain multiple unified computing resources and unified operation tasks;
[0083] Specifically, this step includes the following steps:
[0084] (1-1) Obtain heterogeneous computing resources in the province, and use OpenStack to virtualize the basic physical resources in the computing resources, including computing resources (such as CPU) and storage resources (such as SSD), to obtain multiple virtualized virtual machines;
[0085] (1-2) Using a K8S cluster to further virtualize the multiple virtual machines obtained in step (1-1) to obtain multiple containers (Docker), each container including multiple computing resources and multiple job tasks;
[0086] (1-3) Obtain the unified computing resources and unified job tasks corresponding to each container according to the computing resources and job tasks in each container obtained in step (1-2);
[0087] Specifically, the unified computing resources of a container are expressed as follows:
[0088]
[0089] Among them C br represents unified computing power resources, n represents the total number of logic operation chips in the container, m represents the total number of parallel computing chips in the container, p represents the total number of neural network acceleration chips in the container, f() is the computing power mapping function, a i represents the i-th logic operation chip in the container, α i represents the mapping ratio coefficient of the i-th logic operation chip in the container, β j represents the mapping ratio coefficient of the jth parallel computing chip in the container, b j represents the jth parallel computing chip in the container, γ k represents the mapping ratio coefficient of the kth neural network acceleration chip in the container, c k It represents the kth neural network acceleration chip in the container. q1(), q2() and q3() respectively represent the redundant computing power of logical operations, redundant computing power, and redundant computing power of neural network acceleration of the container. TOPS represents the number of logical operations that the processor in the container can perform per second. FLOPS represents the floating-point computing capability of the container.
[0090] For the jth job task in the cth container (where c∈[1, the total number of containers obtained in step (1-2)]), its corresponding unified job task W c,j It can be represented as the following triple, where j∈[1, the total number of job tasks in the container]:
[0091] W c,j =(α c,j ,d c,j ,G c,j )
[0092] where α c,j represents the arrival time of the jth unified job task of the cth container, d c,j represents the deadline of the jth unified job task of the cth container, G c,jrepresents the data dependency structure of the jth unified job task of the cth container, and the data dependency structure can be represented as a directed acyclic graph (such as Figure 2 shown):
[0093] G c,j =(V c,j ,E c,j )
[0094] Where V c,j = {v j,1 ,v j,2 ,...,v j,m} represents the jth unified job task W of the cth container c,j The unit task set consisting of all m unit tasks in , Represents the set of directed arcs between the sets of unit tasks in the jth unified job task of the cth container.
[0095] The advantage of this step is that it can unify the heterogeneous computing resources within the province and manage them with the same indicators, which facilitates the monitoring, scheduling and optimization of the computing network, effectively improves resource utilization and management efficiency, and realizes centralized management and efficient analysis of task information.
[0096] (2) All unified job tasks obtained in step (1) are matched with unified computing resources to obtain the optimal scheduling strategy in the global case.
[0097] Specifically, this step adopts an improved particle swarm optimization algorithm as a scheduling strategy and uses a quadratic penalty function suitable for multi-constrained combinatorial optimization problems.
[0098] like Figure 3 As shown, this step includes the following steps:
[0099] (2-1) Set the iteration counter cnt = 1, and use the particle swarm optimization algorithm to match all the unified computing resources obtained in step (1) with the unified job tasks to obtain multiple feasible scheduling solutions;
[0100] Specifically, the algorithm sets an iteration counter cnt to record the number of iterations; then evaluates and calculates all unified job tasks, such as the required computing time, data transmission time, etc., and then calculates the cost of all scheduling schemes in the population, and excludes those that do not meet the time constraint (i.e., unified job task W c,j The arrival time α c,j Plus the computational cost t c,j Exceed its deadline c,j ) and finally obtain multiple feasible scheduling solutions (each scheduling solution is called a particle).
[0101] This step includes the following sub-steps:
[0102] (2-1-1) For the cth container D among all the containers obtained in step (1-2) c For the jth unified task W corresponding to the cth container obtained in step (1-3), c,j Perform allocation processing to obtain the jth unified operation task W corresponding to the cth container c,j The corresponding waiting queue Q c,j ;
[0103] Specifically, this step first obtains each container D c The corresponding j-th unified task W c,j , then, traverse the unified job task W c,j All unit tasks in , put all entry unit tasks (specifically, unit tasks that can be executed immediately without waiting for data transmission) into the pending queue Q corresponding to the container c,j In the above example, the unified task W c,j The remaining unit tasks are placed in the task waiting pool corresponding to the container;
[0104] (2-1-2) For each queue Q to be processed obtained in step (2-1-1) c,j For example, the waiting queue Q c,j Process the highest priority entry unit task v j,1 Dispatched to the pending queue Q c,j Corresponding container D c to process;
[0105] Specifically, if container D in the computing resource c The pending queue Q c,j If it is not empty, the entry unit task v with the highest priority in the queue will be executed. j,1 Take it from the queue and dispatch it to container D c On the entry unit task v j,1 Perform execution processing;
[0106] (2-1-3) In the pending queue Q c,j Corresponding container D c After processing the entry unit task obtained in step (2-1-2), obtain the entry unit task v j,1 The task calculation time t tc (v j,1 ,D c );
[0107] Specifically, after the entry unit task is processed, the entry unit task v is obtained according to the following formula:j,1 The task calculation time t tc (v j,1 ,D c ):
[0108]
[0109] where τ j,1 Indicates the entry unit task v j,1 The total amount of calculation, C br Represents container D c The unified computing resources, that is, the computing power of the container;
[0110] (2-1-4) Traverse the unified job task W c,j All remaining successor unit tasks v in j,m (refers to the unit tasks remaining in the unified operation task except the entry unit task, which all need to be executed after data transmission), obtain the data transmission time and task calculation time of each unit task, and calculate the unified operation task W c,j The data transmission time and task calculation time of all unit tasks in the task are summed up to obtain the unified task W. c,j The total data transmission cost t dt (c,j) and total task computation time t tc (c, j), where m∈[2, the total number of unit tasks of the jth unified job task];
[0111] Specifically, the unified task W c,j The total data transmission cost t dt (c,j) includes the data transmission cost of all subsequent unit tasks, and the total task computing time t tc (c,j) includes the task calculation time t of its entry unit task tc (v j,1 ,D c ), and the task calculation time t of all its successor unit tasks tc (v j,m ,D c ).
[0112] The algorithm traverses the entry unit task v j,1 All the successor unit tasks, and then the successor unit task v j,m The required data is transferred to the container that executes the task, and v is calculated according to the following formula j,m Data transfer time
[0113]
[0114] in represents the mth successor unit task v of the jth unified task j,m Need to receive its pth predecessor unit task v j,p (refers to the need to v j,m The data transmitted by the unit task that was previously executed first is e, is the mth successor unit task v of the jth unified task j,m Need to receive its pth predecessor unit task v j,p The total amount of data transmitted, b p,m is the task v from the pth predecessor unit j,p To the mth successor unit task v j,m The transmission bandwidth is p∈[1,m].
[0115] Whenever a successor unit task v j,m , m∈[2, the total number of unit tasks of the jth job task], has received all the data from its predecessor unit task, and the algorithm will wait for the task v in the pool for this container j,m Put it in the pending queue and wait for processing. The pending queue will continuously take out the tasks with the highest execution priority and schedule them to the container for processing, and calculate the container D according to the following formula c The subsequent unit task v j,m The task calculation time t tc (v j,m ,D c ) until the queue is empty:
[0116]
[0117] where τ j,m Indicates that the successor unit task v j,m The total amount of calculation, C br Represents container D c The unified computing resources, that is, the computing power of the container;
[0118] After all subsequent unit tasks have been executed, the task calculation time t of the entry unit task obtained in step (2-1-3) is calculated. tc (v j,1 ,D c ) and the task calculation time t of all subsequent unit tasks obtained in this step tc (v j,m ,D c ) are added to obtain the unified task W c,j The total task computation time t tc (c,j); the data transmission time of all subsequent unit tasks obtained in this step Add together to get the total data transmission cost t of the unified task dt (c,j);
[0119] (2-1-5) Each unified task W obtained according to step (2-1-4) c,j The total data transmission cost t dt (c,j) and total task computation time t tc (c,j) Get the unified job task W c,j The computational cost t c,j , and obtain multiple feasible scheduling solutions based on the computational costs generated by all unified job tasks;
[0120] Specifically, each unified job task W c,j The computational cost of c,j Including its total task calculation time t tc (c,j) and the total data transmission cost t dt (c,j), if its corresponding scheduling scheme does not meet the deadline constraint (i.e. any unified job task W c,j The arrival time α c,j Plus the computational cost t c,j Exceed its deadline c,j ), then the scheduling scheme is not feasible and is abandoned; otherwise, the scheduling scheme for the unified task is feasible. Finally, the feasible scheduling schemes corresponding to all unified tasks are obtained.
[0121] The advantage of this step is that it can handle the deadline constraints of unified job tasks well, ensure that the scheduling plans are feasible, and allow users to specify the deadlines of the unified job tasks they submit, thereby improving the task execution efficiency of the computing network.
[0122] (2-2) Take all feasible scheduling schemes obtained in step (2-1) as the population and obtain the personal best particle of each particle in the population at the cnt iteration And the global best particle gB of the population cnt , and judge whether cnt is greater than the preset maximum number of iterations i max (its value range is 50 to 1000, preferably 300), if yes, proceed to step (2-5), otherwise proceed to step (2-3);
[0123] Specifically, this step is as follows: First, the following formula is used to calculate the kth (k∈[1, total number of particles in the population]) particle in the population at the cntth iteration The fitness based on the quadratic penalty function (QPF) is equal to:
[0124]
[0125] in represents the kth particle in the population at the cntth iteration; σ cnt represents the penalty factor σ of the cnt-th iteration (its value range is 0.05×2 G to 0.15×2 G , preferably 0.1×2 G ), which is used to force the current particle to fully and quickly approach the feasible domain, thereby speeding up the acquisition of feasible solutions); represents the penalty function, which is used to force the current particle to move towards the feasible region. Specifically, the particle's time constraint is used as its penalty term. c,j represents the arrival time of the jth unified job task of the cth container, t c,j is the computational cost of the jth unified job task of the cth container, d c,j represents the deadline of the jth unified job task of the cth container; represents the kth particle in the population at the cntth iteration The computational cost t of the jth unified job task of the cth container c,j .
[0126] Then, the particle corresponding to the maximum value is selected from the multiple QPF fitness of the kth particle in the first iteration, the second iteration, ... the cntth iteration as the personal best particle of the kth particle at the cntth iteration.
[0127] Finally, the particle with the maximum QPF fitness is selected from the personal best particles of all particles in the population as the global best particle gB of the population at the cntth iteration. cnt .
[0128] The advantage of this step is that the use of penalty functions and penalty factors speeds up the acquisition of feasible solutions and avoids the situation where the optimal scheduling strategy cannot be obtained after the iteration when the problem scale is too large.
[0129] (2-3) The personal best particle of the kth particle in the population at the cntth iteration obtained using step (2-2) And the global best particle gB of the population cnt Get the kth particle at the cnt+1th iteration, and all the particles at the cnt+1th iteration form a new population;
[0130] Specifically, the kth particle at the cnt+1th iteration is obtained according to the following formula:
[0131]
[0132] Wherein, c1 and c2 are acceleration coefficients (the value range of c1 is 0.2 to 0.9, preferably 0.5, and the value range of c2 is 0.4 to 0.9, preferably 0.7), and ⊙ represent the crossover operator and mutation operator of the genetic algorithm respectively, w is the inertia factor, and:
[0133]
[0134]
[0135] where w max and w min Respectively represent the maximum and minimum values of w (w max The value range is 0.8 to 1.5, preferably 0.9, w min The value range of is 0.1 to 0.5, preferably 0.4), It indicates whether the kth particle is updated at the cntth iteration, NP represents the size of the population, and F() represents the QPF fitness of the particle.
[0136] This mechanism can adaptively adjust the algorithm's search capability based on the difference between the current particle and the previous generation of particles. At the beginning of the iteration, the population is updated frequently, so the value of w should be large to enhance the global search capability of the particles. In the later stages of the iteration, the local search capability of the particles should be increased relatively small to perform a refined search to improve the search convergence of the scheduling strategy.
[0137] The advantage of this step is that the adaptive adjustment of the inertia factor w improves the algorithm's adaptability to the computing power network environment, and it can adapt itself when the environment changes without manual adjustment.
[0138] (2-4) Set counter cnt=cnt+1 and return to step (2-2);
[0139] (2-5) The global best particle gB of the population cnt As the optimal scheduling strategy output in the global case;
[0140] (3) Obtain a service information flow table according to the optimal scheduling strategy in the global situation obtained in step (2), and schedule all unified job tasks obtained in step (1) according to the generated service information flow table to obtain a scheduling result.
[0141] Specifically, the original intention of the design of the computing priority network is to solve the problems of complex deployment, low efficiency, and low resource reuse rate of mobile edge computing. It enables the network to have the ability to dynamically route built-in computing services, by publishing computing resource status and network status as routing information to the network, and routing computing task messages to the most suitable computing nodes based on virtual service IDs. The computing priority network protocol inherits the design ideas of the traditional label forwarding protocol, carries on the IP network, establishes sessions between adjacent routers that support the protocol, and publishes the acquired computing resource information to adjacent computing priority network routers with the help of routing protocols, realizing the full network diffusion of computing resource information. At the same time, the computing priority network router builds a service routing information table according to different services, guides business messages to be forwarded with the service ID as the destination address, and thus realizes the utilization of decentralized computing resources in a service manner.
[0142] The protocol adopted by the present invention is different from the traditional computing-first network protocol. Considering that the number of containers is huge compared to routers, if each container needs to communicate with the provincial regional computing power network controller, then the pressure on the controller will be too great. Therefore, the present invention ultimately adopts the router to continue to bear the responsibility of collecting computing resource information. Figure 4 The network architecture of the present invention is presented.
[0143] This step includes the following sub-steps:
[0144] (3-1) Obtain the unified computing power resources for each container according to the optimal scheduling strategy in the global situation obtained in step (2);
[0145] Specifically, for unified computing resources, the container is used to register its unified computing resources with the router;
[0146] (3-2) Loading the unified computing resources obtained in step (3-1) into the IP protocol or routing protocol through a router;
[0147] (3-3) performing network topology calculation according to the loaded IP protocol or routing protocol obtained in step (3-2) to obtain a service information flow table, and sending the service information flow table to the router;
[0148] Specifically, first, a topology map containing all containers and their connection relationships is constructed based on the information of the loaded IP protocol or routing protocol. Then, a corresponding flow table entry is generated for each unified job task based on the optimal scheduling strategy and the topology map obtained in step (2). The flow table entry specifies the matching rules and corresponding processing actions (such as forwarding, modification, discarding, etc.) of each unified job task and the path of its flow in the topology map. All flow table entries constitute a service information flow table.
[0149] (3-4) All unified job tasks obtained in step (1) are scheduled through the router according to the generated service information flow table to obtain a scheduling result.
[0150] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A computing power scheduling method under a provincial regional computing power network, characterized in that: The following steps are involved: (1) Obtain heterogeneous computing resources within the province and model them using a two-level linkage architecture based on "K8S+OpenStack" to obtain multiple unified computing resources and unified operation tasks; (2) All unified job tasks obtained in step (1) are matched with unified computing resources to obtain the optimal scheduling strategy in the global case. (3) Obtain a service information flow table according to the optimal scheduling strategy in the global situation obtained in step (2), and schedule all unified job tasks obtained in step (1) according to the generated service information flow table to obtain a scheduling result.
2. The computing power scheduling method under the provincial regional computing power network according to claim 1 is characterized in that: Step (1) comprises the following steps: (1-1) Obtaining heterogeneous computing resources in the province, and using OpenStack to virtualize the basic physical resources in the computing resources, where the basic physical resources include computing resources and storage resources, to obtain multiple virtualized virtual machines; (1-2) Using a K8S cluster to further virtualize the multiple virtual machines obtained in step (1-1) to obtain multiple containers, each container including multiple computing resources and multiple job tasks; (1-3) Obtain the unified computing resources and unified job tasks corresponding to each container based on the computing resources and job tasks in the container obtained in step (1-2).
3. The computing power scheduling method under the provincial regional computing power network according to claim 1 or 2 is characterized in that: The unified computing resources of the container are as follows: Among them C br represents unified computing power resources, n represents the total number of logic operation chips in the container, m represents the total number of parallel computing chips in the container, p represents the total number of neural network acceleration chips in the container, f() is the computing power mapping function, a i represents the i-th logic operation chip in the container, α i represents the mapping ratio coefficient of the i-th logic operation chip in the container, β j represents the mapping ratio coefficient of the jth parallel computing chip in the container, b j represents the jth parallel computing chip in the container, γ k represents the mapping ratio coefficient of the kth neural network acceleration chip in the container, c k It represents the kth neural network acceleration chip in the container. q1(), q2() and q3() respectively represent the redundant computing power of logical operations, redundant computing power, and redundant computing power of neural network acceleration of the container. TOPS represents the number of logical operations that the processor in the container can perform per second. FLOPS represents the floating-point computing capability of the container.
4. The computing power scheduling method under the provincial regional computing power network according to any one of claims 1 to 3, characterized in that: For the jth job task in the cth container, its corresponding unified job task W c,j It can be represented as the following triple: W c,j =(a c,j ,d c,j ,G c,j ) where c∈[1, the total number of containers obtained in step (1-2)], α c,j represents the arrival time of the jth unified job task in the cth container, j∈[1, the total number of job tasks in the container], d c,j represents the deadline of the jth unified job task of the cth container, G c,j Represents the data dependency structure of the jth unified job task of the cth container, and the data dependency structure can be represented as a directed acyclic graph: G c,j =(V c,j ,E c,j ) Where V c,j = {v j,1 ,v j,2 ,...,v j,m } represents the jth unified job task W of the cth container c,j The unit task set consisting of all m unit tasks in , Represents the set of directed arcs between the sets of unit tasks in the jth unified job task of the cth container.
5. The computing power scheduling method under the provincial regional computing power network according to claim 4 is characterized in that: Step (2) comprises the following steps: (2-1) Set the iteration counter cnt = 1, and use the particle swarm optimization algorithm to match all the unified computing resources obtained in step (1) with the unified job tasks to obtain multiple feasible scheduling solutions; (2-2) Take all feasible scheduling schemes obtained in step (2-1) as the population and obtain the personal best particle of each particle in the population at the cnt iteration And the global best particle gB of the population cnt , and judge whether cnt is greater than the preset maximum number of iterations i max If yes, go to step (2-5), otherwise go to step (2-3); (2-3) The personal best particle of the kth particle in the population at the cntth iteration obtained using step (2-2) And the global best particle gB of the population cnt Get the kth particle at the cnt+1th iteration, and all the particles at the cnt+1th iteration form a new population; (2-4) Set counter cnt=cnt+1 and return to step (2-2); (2-5) The global best particle gB of the population cnt It is output as the optimal scheduling strategy in the global case.
6. The computing power scheduling method under the provincial regional computing power network according to claim 5 is characterized in that: Step (2-1) includes the following sub-steps: (2-1-1) For the cth container D among all the containers obtained in step (1-2) c For the jth unified task W corresponding to the cth container obtained in step (1-3), c,j Perform allocation processing to obtain the jth unified operation task W corresponding to the cth container c,j The corresponding waiting queue Q c,j ; (2-1-2) For each queue Q to be processed obtained in step (2-1-1) c,j For example, the waiting queue Q c,j Process the highest priority entry unit task v j,1 Dispatched to the pending queue Q c,j Corresponding container D c to process; (2-1-3) In the pending queue Q c,j Corresponding container D c After processing the entry unit task obtained in step (2-1-2), obtain the entry unit task v j,1 The task calculation time t tc (v j,1 ,D c ); (2-1-4) Traverse the unified job task W c,j All remaining successor unit tasks v in j,m , to obtain the data transmission time and task calculation time of each subsequent unit task, and for the unified job task W c,j The data transmission time and task calculation time of all unit tasks in the task are summed up to obtain the unified task W. c,j The total data transmission cost t dt (c,j) and total task computation time t tc (c, j), where m∈[2, the total number of unit tasks of the jth unified job task]; (2-1-5) Each unified task W obtained according to step (2-1-4) c,j The total data transmission cost t dt (c,j) and total task computation time t tc (c,j) Get the unified job task W c,j The computational cost t c,j , and obtain multiple feasible scheduling solutions based on the computational costs incurred by all unified job tasks.
7. The computing power scheduling method under the provincial regional computing power network according to claim 6 is characterized in that: Step (2-1-3) is to obtain the entry unit task v according to the following formula: j,1 The task calculation time t tc (v j,1 ,D c ): where τ j,1 Indicates the entry unit task v j,1 The total amount of calculation, C br Represents container D c The unified computing resources, that is, the computing power of the container; Step (2-1-4) is to calculate v according to the following formula j,m Data transfer time in represents the mth successor unit task v of the jth unified task j,m Need to receive its pth predecessor unit task v j,p (refers to the need to v j,m The data transmitted by the unit task that was previously executed first is e, is the mth successor unit task v of the jth unified task j,m Need to receive its pth predecessor unit task v j,p The total amount of data transmitted, b p,m is the task v from the pth predecessor unit j,p To the mth successor unit task v j,m The transmission bandwidth is p∈[1,m]. Step (2-1-4) is to calculate the container D according to the following formula c The subsequent unit task v j,m The task calculation time t tc (v j,m ,D c ): where τ j,m Indicates that the successor unit task v j,m The total amount of calculation, C br Represents container D c The unified computing resources of the container, that is, the computing power of the container. In step (2-1-5), each unified task W c,j The computational cost of c,j Including its total task calculation time t tc (c,j) and the total data transmission cost t dt (c,j), if its corresponding scheduling scheme does not meet the deadline constraint, that is, any unified job task W c,j The arrival time α c,j Plus the computational cost t c,j Exceed its deadline c,j , then the scheduling scheme is not feasible and is abandoned; otherwise, the scheduling scheme for the unified task is feasible. Finally, the feasible scheduling schemes corresponding to all unified tasks are obtained.
8. The computing power scheduling method under the provincial regional computing power network according to claim 7 is characterized in that: Step (2-2) is specifically: First, the following formula is used to calculate the kth particle in the population at the cntth iteration: The fitness of QPF based on quadratic penalty function is: where k∈[1, total number of particles in the population], represents the kth particle in the population at the cntth iteration; σ cnt represents the penalty factor σ of the cnt-th iteration; represents the penalty function, which is used to force the current particle to move towards the feasible region. Specifically, the particle's time constraint is used as its penalty term. c,j represents the arrival time of the jth unified job task of the cth container, t c,j is the computational cost of the jth unified job task of the cth container, d c,j represents the deadline of the jth unified job task of the cth container; represents the kth particle in the population at the cntth iteration The computational cost t of the jth unified job task of the cth container c,j . Then, the particle corresponding to the maximum value is selected from the multiple QPF fitness of the kth particle in the first iteration, the second iteration, ... the cntth iteration as the personal best particle of the kth particle at the cntth iteration. Finally, the particle with the maximum QPF fitness is selected from the personal best particles of all particles in the population as the global best particle gB of the population at the cntth iteration. cnt ; Step (2-3) is to obtain the kth particle at the cnt+1th iteration according to the following formula: Among them, c1 and c2 are acceleration coefficients, and ⊙ represent the crossover operator and mutation operator of the genetic algorithm respectively, w is the inertia factor, and: where w max and w min represent the maximum and minimum values of w respectively, It indicates whether the kth particle is updated at the cntth iteration, NP represents the size of the population, and F() represents the QPF fitness of the particle.
9. The computing power scheduling method under the provincial regional computing power network according to claim 8 is characterized in that: Step (3) includes the following sub-steps: (3-1) Obtain the unified computing power resources for each container according to the optimal scheduling strategy in the global situation obtained in step (2); (3-2) Loading the unified computing resources obtained in step (3-1) into the IP protocol or routing protocol through a router; (3-3) performing network topology calculation according to the loaded IP protocol or routing protocol obtained in step (3-2) to obtain a service information flow table, and sending the service information flow table to the router; Specifically, this step first constructs a topology map containing all containers and their connection relationships based on the information of the loaded IP protocol or routing protocol. Then, according to the optimal scheduling strategy and the topology map obtained in step (2), a corresponding flow table entry is generated for each unified job task. The flow table entry specifies the matching rules and corresponding processing actions of each unified job task and the path of its flow in the topology map. All flow table entries constitute a service information flow table. (3-4) All unified job tasks obtained in step (1) are scheduled through the router according to the generated service information flow table to obtain a scheduling result.
10. A computing power dispatching system under a provincial regional computing power network, characterized in that: include: The first module is used to obtain heterogeneous computing resources within the province, and model the computing resources using a two-level linkage architecture based on "K8S+OpenStack" to obtain multiple unified computing resources and unified operation tasks; The second module is used to match all unified job tasks obtained in the first module with unified computing resources to obtain the optimal scheduling strategy under global conditions. The third module is used to obtain a service information flow table according to the optimal scheduling strategy in the global situation obtained by the second module, and schedule all unified job tasks obtained by the first module according to the generated service information flow table to obtain a scheduling result.
Citation Information
Cited By
Gateway-based computing power allocation method, device, gateway device and storage medium
CN119759581A
