A multi-stage dynamic orchestration method for modelica task parallel compilation load

Abstract

The application discloses a multi-stage dynamic arrangement method for Modelica task parallel compilation load, the method designs an arrangement optimization strategy according to the granularity of each stage, the front compilation stage improves the translation speed by setting the number of parallel CPUs for single-machine compilation of the task, but if multiple tasks are compiled concurrently on the single machine, the speed will decrease exponentially, so when the execution nodes are distributed, all the tasks currently arrived are arranged globally; the number of parallel compilation sub-threads of the total splitting and the distributed compilation optimization Modelica task parallel compilation load are set in the post-compilation stage, aiming at solving the problem that the Modelica model cannot be distributed compiled, improving the overall compilation efficiency of the system, and reducing the slow simulation speed problem caused by the single-machine computing power problem.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Modelica cloud simulation, and particularly relates to a multi-stage dynamic scheduling method for Modelica task parallel compilation load. BACKGROUND

[0002] The life cycle of a Modelica cloud simulation task is divided into four stages: pre-processing, compilation, solving and pooling. The compilation is a kind of work load that produces a large overhead on CPU, memory, IO and other computing resources of a computing node. The current computing service platform for Modelica modeling simulation focuses on the compilation of simulation tasks on each node, which mostly adopts a traditional scheduling scheme, i.e. a centralized allocation of a single arrived compilation task through a polling-based load balancing mechanism. Different single Modelica model compilation brings different work load to the computing node. The static equal scheduling mode has a certain lag in the adaptive allocation of computing resources required by each task, which may cause some machines to be overloaded due to the high complexity of the tasks. The present application divides the compilation process into pre-compilation and post-compilation based on the characteristics of Modelica task compilation, introduces a distcc distributed compiler, and designs a multi-stage Modelica model compilation load scheduling. Different load scheduling strategies are designed based on the two-stage compilation characteristics of Modelica tasks, which improves the overall potential parallel efficiency of the computing cluster, accelerates the compilation speed of different Modelica tasks, and solves the problem of long time consumption in the Modelica compilation stage due to insufficient single machine computing power. SUMMARY

[0003] The pre-compilation of Modelica compilation is to translate the standard model into C description code and parameter files through the invocation of an OMCompiler single machine compiler. The post-compilation is to generate a solving program corresponding to the model by making a C code. Both stages are completed in a compilation service cluster, each node can complete the pre-compilation and post-compilation, and a distcc distributed compilation environment is deployed. The present application designs a scheduling optimization strategy according to the granularity of each stage based on the several stages of Modelica simulation. The pre-compilation stage improves the translation speed by setting the number of parallel CPUs of the node for single machine compilation of the task. However, if multiple tasks are compiled concurrently on a single machine, the speed will decrease exponentially, so the current arrived tasks need to be scheduled globally when assigning the execution nodes. The post-compilation stage sets the number of parallel compilation sub-threads and optimizes the parallel compilation load of Modelica tasks through distributed compilation, aiming to solve the problem of distributed compilation of Modelica models, improve the overall compilation efficiency of the system, and reduce the problem of slow simulation speed caused by single machine computing power.

[0004] The application provides a multi-stage dynamic arrangement method for Modelica task parallel compilation load, and specifically comprises the following steps.

[0005] Step S1): obtain CPU usage RAM memory utilization Disk and network I / O occupancy The number of tasks n being compiled cpl According to formula (1) and formula (2), the calculation load L of each node i in the current compilation cluster is calculated i And the calculation performance R i .

[0006] Calculation load:

[0007] Wherein, The weight parameters of CPU usage, RAM memory utilization, disk and network I / O occupancy are 0.5, 0.3 and 0.2 respectively, because single machine compilation is a computationally intensive task.

[0008] Calculation performance:

[0009] Wherein The disk remaining rate, I / O remaining rate, CPU remaining rate and RAM memory remaining rate of node i are respectively, The CPU core number of node i.

[0010] Step S2): the pre-compilation load of any Modelica simulation task j is related to the component number m j And the connection number cnt j According to formula (3), the expected pre-compilation workload l j Of all the simulation tasks to be compiled in the current random arrival system is evaluated, wherein 1≤j≤m and j is an integer, and m is the total number of Modelica simulation tasks currently arriving in the system.

[0011]

[0012] Step S3): according to the results of formula (1) and (2), the work pool facing the simulation task j in the cluster is obtained through the work pool election strategy, one work pool is divided into one Client and several Servers, the total number of Servers as execution Modelica post-compilation calculation nodes is X, and the Server is set in the environment parameter DistccHosts of the Client.

[0013] ​S4) The load orchestrator assigns the simulation task j to the corresponding Client, and calculates the single-machine pre-compilation parallelism of task j according to formula (4) j As a system commend parameter, the OMCompiler is called through the system commend parameter to complete single-machine parallel pre-compilation, and the Modelica simulation task j standard model is translated into C code.

[0014]

[0015] Here, is the CPU core number of the Client itself.

[0016] S5) The optimal parallel compilation job number O of the simulation task j in the compilation field is calculated according to formula (5) j The compilation job number θ that can be distributed by each Server is calculated according to formula (6) x ,

[0017]

[0018]

[0019] wherein χ = 1, 2,..., X, χ represents the Server computing node number, X represents the total number of Server computing nodes, is the CPU core number of the Server computing node numbered χ, is the CPU remaining rate of the Server computing node numbered χ, represents rounding down.

[0020] The Distcc job online that can be received by each Server is set according to the calculation results of formula (5) and formula (6).

[0021] S6) The Client splits the simulation task j into a plurality of compilation jobs and distributes them to the Servers according to the optimal parallel compilation job number calculated in step S5), receives the compilation results executed by the Servers and integrates and outputs the Modelica solving program through the command line invocation of the distcc distributed compilation tool.

[0022] The application discloses a multi-stage dynamic orchestration method for Modelica task parallel compilation load. The method converts Modelica single-machine compilation into multi-machine parallel compilation mode, and designs a multi-stage dynamic orchestration method for parallel compilation load according to the characteristics of Modelica simulation task compilation stage and the computing power characteristics of each computing node itself, so as to improve the utilization rate of computing cluster overall computing power resources, dynamically balance node computing load, and reduce the situation that the overall compilation time is relatively long due to insufficient single-machine computing power. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is the overall architecture diagram of the multi-stage dynamic orchestration method for parallel compilation of Modelica tasks in the present invention. DETAILED DESCRIPTION

[0024] The specific implementation manner and working principle of the present invention will be further described in detail below with reference to the accompanying drawings.

[0025] This embodiment provides a multi-stage dynamic orchestration method for parallel compilation of Modelica tasks. The workflow is as follows: Figure 1 As shown, the specific steps include:

[0026] Step S1): Get the CPU usage of node i RAM memory utilization Disk and network I / O usage The number of tasks being compiled, n cpl The computational load L of each node i in the current compilation cluster is calculated according to formula (1) and formula (2). i and computing performance R i .

[0027] Calculate the load:

[0028] in, They are the weight parameters of the CPU usage, RAM memory utilization, disk and network I / O occupancy of node i. Since stand-alone compilation is a computationally intensive task, Set to 0.5, 0.3, and 0.2 respectively.

[0029] Computing performance:

[0030] in are the disk surplus rate, I / O surplus rate, CPU surplus rate, and RAM memory surplus rate of node i, respectively. is the number of CPU cores of node i.

[0031] like Figure 1 ,As shown in the architecture diagram, there are 7 computing nodes in the computing cluster.

[0032] Step S2): The pre-compilation load of any Modelica simulation task j and the number of components m constituting the standard model of the simulation task j 、Number of connections cnt j According to formula (3), the estimated pre-compilation workload l of all the simulation tasks to be compiled in the current random arrival system is evaluated.j wherein, 1≤j≤m and j is an integer, m is the total number of Modelica simulation tasks currently arriving at the system.

[0033]

[0034] Step S3): According to the results of formula (1) and (2), the work pool facing the simulation task j in the cluster is obtained through the work pool election strategy. One work pool is divided into one Client and several Servers. The total number of Servers as the execution nodes of Modelica post-compilation is X, which is set in the environment parameter DistccHosts of the Client.

[0035] Further, the work pool election strategy is specifically:

[0036] S31) First, select the computing nodes in the cluster whose computing load is lower than l j ;

[0037] S32) Among the computing nodes selected in step S31), the one with the largest computing performance / load ratio and is the Client, and the others are the Servers, to ensure the efficiency of the pre-compilation stage;

[0038] S33) If the Server selected in step S32) has , then replace it with the computing node with the lowest computing load among the unselected nodes until the disk and network I / O occupancy rates of the Servers in the work pool are all lower than 0.5.

[0039] In Figure 1 , according to the load and performance of each node obtained in steps S1) and S2), four computing nodes with a current work load lower than l1 are elected as the Worker pool, and the Client and the Servers are selected according to the work pool election strategy. The selected Client is called Clientl.

[0040] Step S4): The load orchestrator assigns the simulation task j to the corresponding Client, completes the single-machine parallel pre-compilation by calling the OMCompiler, and translates the Modelica simulation task j standard model into C code. The single-machine pre-compilation parallel degree a j of the simulation task j is related to the effective CPU core number of the Client itself, as shown in formula (4).

[0041]

[0042] The single-machine pre-compilation parallel degree a jAs a system commend parameter, through the system commend

[0043] Parameter calls OMCompiler to complete single-machine parallel pre-compilation, and translates Modelica standard model into C code.

[0044] Up to now, the pre-compilation phase of Modelica ends.

[0045] Step S5): Distributed parallel post-compilation for Modelica simulation tasks. Here, the size of the task itself needs to be considered. If the size is not large, setting the parallel degree of distributed compilation too high will reduce the compilation efficiency. Through formula (5), the complexity and size of the Modelica model are quantitatively calculated to obtain the optimal number of parallel compilation jobs Oi of the simulation task in the compilation field, and formula (6) calculates the number of distributable compilation jobs θ of each Server x (x = 1, 2,..., X), χ represents the number of Server computing nodes, and X represents the total number of Server computing nodes; according to the calculation results of formulas (5) and (6), the online jobs that can be received by each Server are set to prevent the distributed compilation from failing due to the overload of some nodes.

[0046]

[0047]

[0048] wherein, is the number of CPU cores of the Server computing node numbered χ, is the CPU remaining rate of the Server computing node numbered χ, represents rounding down.

[0049] Step S6): The Client calls the distcc distributed compilation tool through the make-jo i command according to the optimal number of parallel compilation jobs obtained in step S5) to split the simulation task into a plurality of compilation jobs and distribute them to the Servers, receives the compilation results executed by the Servers, and integrates and outputs the Modelica solver program.

[0050] In Figure 1 , the Client1 calls the distcc distributed compilation tool through the make-jo i command according to the optimal number of parallel compilation jobs obtained in step S5) to split the simulation task into a plurality of compilation jobs and distribute them to the Servers, receives the compilation results executed by the Servers, and integrates and outputs the Modelica solver program.

[0051] The above merely describes a specific implementation of the present application, any feature disclosed in the specification can be replaced by other equivalent or similar purpose alternative features unless specifically described, all features disclosed or steps in all methods or processes can be combined in any manner except mutually exclusive features and / or steps, any non-essential addition or replacement made by a person skilled in the art according to technical features of the technical solution of the present application is within the protection scope of the present application.

Claims

1. A multi-stage dynamic orchestration method for parallel compilation of Modelica tasks, characterized by: The method comprises the following steps: S1) Quantify the computational load L of each node i in the current compilation cluster i and computing performance R i ; S2) quantifying the estimated pre-compilation workload of all simulation tasks to be compiled in the current random arrival system; the quantification of the estimated pre-compilation workload of all simulation tasks to be compiled in the current random arrival system is specifically: Get the number of components m of any standard model of Modelica simulation task j j and the number of connections cnt j , calculate the estimated pre-compilation workload l of all simulation tasks to be compiled in the current random arrival system according to formula (3) j , where 1≤j≤m and j is an integer, m is the total number of Modelica simulation tasks currently arriving at the system, S3) Based on the results of S1) and S2), a work pool for any simulation task j in the cluster is obtained through a work pool election strategy. A work pool is divided into a client and several servers. The work pool election strategy specifically includes the following steps: S31) First select the cluster whose computing load is lower than l j Compute nodes; S32) Among the computing nodes selected in step S31), the one with the highest single-machine computing performance / load ratio and disk and network I / O occupancy rate less than 0.3 is the client, and the others are servers; S33) If the server selected in step S32) Then, the node with the lowest computing load is selected from the unselected nodes to replace them until the disk and network I / O usage of the servers in the work pool are all lower than 0.5; S4) The load scheduler assigns simulation task j to the corresponding Client and calculates the pre-compilation parallelism α of task j. j As the system command parameter, OMCompiler is called through the system command parameter to complete the single-machine parallel pre-compilation and translate the Modelica simulation task j standard model into C code; S5) Calculate the optimal number of parallel compilation jobs for simulation task j in the compilation farm and the number of compilation jobs that can be allocated to each server, and set the upper limit of the Distcc jobs that can be received by each server based on the above two calculation results; S6) The client uses the optimal number of parallel compilation jobs calculated in step S5) to call the distcc distributed compilation tool through the command line to split the simulation task j into several compilation jobs and distribute them to each server. The client receives the compilation results executed by each server and integrates them to output the Modelica solver.

2. The multi-stage dynamic orchestration method for parallel compilation of Modelica tasks according to claim 1, characterized in that: The computational load L of each node i in the current compilation cluster is quantified i and computing performance R i Specifically: Get the CPU usage of any node i RAM memory utilization Disk and network I / O usage The number of tasks being compiled, n cpl Indicator, calculate the computing load L of node i in the current compilation cluster according to formula (1) and formula (2) i and computing performance R i , Calculate the load: in, are the weight parameters of CPU usage, RAM memory utilization, disk and network I / O occupancy of node i; Computing performance: in are the disk surplus rate, I / O surplus rate, CPU surplus rate, and RAM memory surplus rate of node i, respectively. is the number of CPU cores of node i.

3. The multi-stage dynamic orchestration method for parallel compilation of Modelica tasks according to claim 1, characterized in that: The parallelism α of the single-machine pre-compilation of task j is obtained j Specifically: According to formula (4), the parallelism α of task j before single-machine compilation is calculated j , Here The number of CPU cores of the client.

4. The multi-stage dynamic orchestration method for parallel compilation of Modelica tasks according to claim 1, characterized in that: The optimal number of parallel compilation jobs for the computation simulation task j in the compilation farm and the number of compilation jobs that can be allocated to each server are specifically: According to formula (5), the optimal number of parallel compilation jobs O for simulation task j in the compilation farm is calculated. j , calculate the number of compilation jobs that can be assigned to each server according to formula (6) x , Where x = 1, 2, ..., X, x represents the server computing node number, X represents the total number of server computing nodes, Calculate the number of CPU cores of the Server node numbered x. Calculate the CPU remaining rate of the server node numbered x. Indicates rounding down.

5. The multi-stage dynamic orchestration method for parallel compilation of Modelica tasks according to claim 2, characterized in that: described The values ​​are 0.5, 0.3, and 0.2 respectively.

Images