Multi-stage dynamic arrangement method for parallel compiling load of Modelica task

Abstract

The invention discloses a multi-stage dynamic arrangement method for parallel compiling load of a Modelica task, which comprises the following steps of: designing an arrangement optimization strategy according to the granularity of each stage, and increasing the translation speed by setting the number of parallel CPUs (Central Processing Unit) for single-machine compiling of the task by a node in a front compiling stage; however, if a single machine concurrently compiles a plurality of tasks to cause exponential decline of the speed, all the currently arrived tasks are globally arranged when the execution nodes are allocated; in the post-compiling stage, by setting the number of parallel compiling sub-threads of overall splitting and the parallel compiling load of distributed compiling optimization Modelica tasks, the problem that a Modelica model cannot be compiled in a distributed mode is solved, the overall compiling efficiency of a system is improved, and the problem that the simulation speed is low due to the single-machine computing power problem is solved.

Description

Technical Field

[0001] The present invention relates to the field of Modelica cloud simulation technology, and in particular to a multi-stage dynamic orchestration method for parallel compilation loads of Modelica tasks. Background Art

[0002] The life cycle of Modelica cloud simulation tasks is divided into four stages: pre-processing, compilation, solving, and pooling. Compilation is a kind of workload that generates large overhead for computing resources such as CPU, memory, and IO in computing nodes. The current computing service platform for Modelica modeling and simulation mostly adopts traditional scheduling schemes for the compilation of simulation tasks at each node, that is, a single arriving compilation task is centrally allocated through a load balancing mechanism based on polling. The workloads brought to computing nodes by different single Modelica model compilations are not the same. The static equal scheduling mode has a certain lag in the adaptive allocation of computing resources required for each task, which may cause some machines that are repeatedly arranged to have tasks with higher compilation complexity to be overloaded. The present invention starts from the compilation characteristics of Modelica tasks, divides the compilation process into pre-compilation and post-compilation, introduces the distcc distributed compiler, designs a multi-stage Modelica model compilation load scheduling, designs different load scheduling strategies based on the two-stage compilation characteristics of Modelica tasks, improves the overall potential parallel efficiency of the computing cluster, accelerates the compilation speed of different Modelica tasks, and solves the problem that the Modelica compilation stage is time-consuming due to insufficient computing power of a single machine. Summary of the invention

[0003] The pre-compilation of Modelica compilation is to translate the standard model into C-described code and parameter files by calling the OMCompiler stand-alone compiler, and the post-compilation is to compile the C code to generate the solution program corresponding to the model by make. Both stages are completed in the compilation service cluster, and each node can complete the pre-compilation and post-compilation, and deploy the distcc distributed compilation environment. The present invention starts from several stages of Modelica simulation and designs the arrangement optimization strategy according to the granularity of each stage. In the pre-compilation stage, the translation speed is improved by setting the number of parallel CPUs of the node for the single-machine compilation of the task, but if the single-machine concurrent compilation of multiple tasks will cause an exponential decline in speed, it is necessary to globally arrange all currently arrived tasks when allocating execution nodes; in the post-compilation stage, the number of parallel compilation sub-threads of the overall split and the distributed compilation optimize the parallel compilation load of the Modelica task, aiming to solve the problem that the Modelica model cannot be distributed compiled, improve the overall compilation efficiency of the system, and reduce the problem of slow simulation speed caused by the computing power problem of the single machine.

[0004] The present invention proposes a multi-stage dynamic scheduling method for parallel compilation load of Modelica tasks, which specifically includes the following steps:

[0005] Step S1): Get CPU usage RAM memory utilization Disk and network I / O usage The number of tasks being compiled n cpl According to the indicators such as (1) and (2), the computing load L of each node i in the current compilation cluster is calculated: i and computing performance R i .

[0006] Calculate the load:

[0007] in, They are the weight parameters of CPU usage, RAM memory utilization, disk and network I / O occupancy. Since stand-alone compilation is a computationally intensive task, Set to 0.5, 0.3, and 0.2 respectively.

[0008] Computational performance:

[0009] in are the disk surplus rate, I / O surplus rate, CPU surplus rate, and RAM memory surplus rate of node i respectively. is the number of CPU cores of node i.

[0010] Step S2): The pre-compilation load of any Modelica simulation task j and the number of components m constituting the standard model of the simulation task j 、Number of connections cnt j According to formula (3), the estimated pre-compilation workload l of all simulation tasks to be compiled in the current random arrival system is evaluated. j , where 1≤j≤m and j is an integer, and m is the total number of Modelica simulation tasks currently arriving at the system.

[0011]

[0012] Step S3): According to the results of formulas (1) and (2), the work pool in the cluster for the simulation task j is obtained through the work pool election strategy. A work pool is divided into a client and several servers. The total number of servers as computing nodes for executing Modelica post-compilation is X, which is set in the client's environment parameter DistccHosts.

[0013] S4) The load scheduler assigns simulation task j to the corresponding Client and calculates the single-machine pre-compilation parallelism α of task j according to formula (4): j As a system commend parameter, OMCompiler is called through the system commend parameter to complete the single-machine parallel pre-compilation and translate the Modelica simulation task j standard model into C code;

[0014]

[0015] Here The number of CPU cores of the Client.

[0016] S5) Calculate the optimal number of parallel compilation jobs O for simulation task j in the compilation farm according to formula (5) j , calculate the number of compilation jobs that can be allocated to each server according to formula (6) x ,

[0017]

[0018]

[0019] Where χ=1,2,...,X, χ represents the server computing node number, X represents the total number of server computing nodes, The number of CPU cores of the server node numbered x is calculated. Calculate the CPU surplus rate of the server node numbered x. Indicates rounding down.

[0020] According to the calculation results of formula (5) and formula (6), the Distcc of each server is set to accept the job online.

[0021] S6) The Client uses the best number of parallel compilation jobs calculated in step S5) to call the distcc distributed compilation tool through the command line to split the simulation task j into several compilation jobs and distribute them to each Server, receive the compilation results executed by each Server and integrate and output the Modelica solver.

[0022] The present invention discloses a multi-stage dynamic orchestration method for parallel compilation load of Modelica tasks. The method converts Modelica single-machine compilation into multi-machine parallel compilation mode, and designs a multi-stage dynamic orchestration method for parallel compilation load according to the compilation stage characteristics of Modelica simulation tasks and the computing power characteristics of each computing node itself, aiming to improve the overall computing power resource utilization of the computing cluster, dynamically balance the node computing load, and reduce the situation where the overall compilation time is long due to insufficient computing power of a single machine. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is an overall architecture diagram of the multi-stage dynamic orchestration method for parallel compilation load of Modelica tasks according to the present invention. DETAILED DESCRIPTION

[0024] The specific implementation manner and working principle of the present invention are further described in detail below with reference to the accompanying drawings.

[0025] This embodiment provides a multi-stage dynamic orchestration method for parallel compilation load of Modelica tasks. The workflow is as follows: Figure 1 As shown, the specific steps include:

[0026] Step S1): Get the CPU usage of node i RAM memory utilization Disk and network I / O usage The number of tasks being compiled n cpl According to the indicators such as (1) and (2), the computing load L of each node i in the current compilation cluster is calculated: i and computing performance R i .

[0027] Calculate the load:

[0028] in, are the weight parameters of the CPU usage, RAM memory utilization, disk and network I / O occupancy of node i. Since stand-alone compilation is a computationally intensive task, Set to 0.5, 0.3, and 0.2 respectively.

[0029] Computational performance:

[0030] in are the disk surplus rate, I / O surplus rate, CPU surplus rate, and RAM memory surplus rate of node i respectively. is the number of CPU cores of node i.

[0031] like Figure 1 ,As shown in the architecture diagram, there are 7 computing nodes in the computing cluster.

[0032] Step S2): The pre-compilation load of any Modelica simulation task j and the number of components m constituting the standard model of the simulation task j 、Number of connections cnt j According to formula (3), the estimated pre-compilation workload l of all simulation tasks to be compiled in the current random arrival system is evaluated.j , where 1≤j≤m and j is an integer, and m is the total number of Modelica simulation tasks currently arriving at the system.

[0033]

[0034] Step S3): According to the results of formulas (1) and (2), the work pool in the cluster for the simulation task j is obtained through the work pool election strategy. A work pool is divided into a client and several servers. The total number of servers as computing nodes for executing Modelica post-compilation is X, which is set in the client's environment parameter DistccHosts.

[0035] Furthermore, the work pool election strategy is as follows:

[0036] S31) First select the cluster whose computing load is lower than l j Compute nodes;

[0037] S32) Among the computing nodes selected in step S31), the single-machine computing performance / load ratio is the largest and The first one is the Client, and the others are the Server, which ensures the efficiency of the pre-compilation stage;

[0038] S33) If the Server selected in step S32) Then, the node with the lowest computing load is selected from the unselected nodes for replacement until the disk and network I / O occupancy rates of the servers in the working pool are all lower than 0.5.

[0039] exist Figure 1 In the example, according to the load and performance of each node obtained in steps S1) and S2), the four computing nodes whose current workload is lower than l1 are elected as Worker pool 1, and the Client and Server are selected according to the worker pool election strategy. The selected Client is called Client 1.

[0040] Step S4): The load scheduler assigns simulation task j to the corresponding Client, completes the single-machine parallel pre-compilation by calling OMCompiler, and translates the Modelica simulation task j standard model into C code. The single-machine pre-compilation parallelism α set for simulation task j j It is related to the number of effective CPU cores of the client itself, as shown in formula (4).

[0041]

[0042] The calculated pre-compilation parallelism of the simulation task j is α jAs a system commend parameter, through the system commend

[0043] The parameters call OMCompiler to complete the single-machine parallel pre-compilation and translate the Modelica standard model into C code.

[0044] At this point, the pre-compilation stage of Mode1ica is over.

[0045] Step S5): Distributed parallel post-compilation of Modelica simulation tasks. Here, the scale of the task itself needs to be considered. If the scale is not large, setting the parallelism of distributed compilation too high will reduce the compilation efficiency. The complexity and scale of the Modelica model are quantified by formula (5), and the optimal number of parallel compilation jobs Oi for the simulation task in the compilation field is obtained. Formula (6) calculates the number of compilation jobs θ that can be allocated to each server. x (χ=1,2,...,X), χ represents the server computing node number, and X represents the total number of server computing nodes; according to the calculation results of formulas (5) and (6), the Distcc of each server is set to accept jobs online to avoid overloading some nodes and causing distributed compilation failure.

[0046]

[0047]

[0048] in, The number of CPU cores of the server node numbered x is calculated. Calculate the CPU surplus rate of the server node numbered x. Indicates rounding down.

[0049] Step S6): The Client calls the distcc distributed compilation tool through the command line according to the optimal number of parallel compilation jobs obtained in step S5) to split the simulation task into several compilation jobs and distribute them to each Server, receive the compilation results executed by each Server and integrate and output the Modelica solver.

[0050] exist Figure 1 In the process, Client 1 obtains the number of parallel compilation jobs according to step S5, and the post-compilation load scheduler outputs the make-j0 i command to call the distcc distributed compilation tool to split the simulation task into multiple compilation jobs and distribute them to Server 1, Server 2, and Server 3, receives the compilation results executed by the Server, and integrates and outputs the Modelica solver.

[0051] The above description is only a specific implementation mode of the present invention. Any feature disclosed in this specification, unless otherwise stated, can be replaced by other alternative features that are equivalent or have similar purposes; all disclosed features, or all steps in the methods or processes, except for mutually exclusive features and / or steps, can be combined in any way; any non-essential additions and substitutions made by technicians in this field based on the technical features of the technical solution of the present invention belong to the protection scope of the present invention.

Claims

1. A multi-stage dynamic orchestration method for parallel compilation of Modelica tasks, characterized in that: The method comprises the following steps: S1) Quantify the computational load L of each node i in the current compilation cluster i and computing performance R i ; S2) quantifying the estimated pre-compilation workload among all simulation tasks to be compiled that currently arrive randomly in the system; S3) According to the results of S1) and S2), the work pool for any simulation task j in the cluster is obtained through the work pool election strategy. A work pool is divided into a client and several servers; S4) The load scheduler assigns simulation task j to the corresponding Client and calculates the pre-compilation parallelism α of task j. j As a system commend parameter, OMCompiler is called through the system commend parameter to complete the single-machine parallel pre-compilation and translate the Modelica simulation task j standard model into C code; S5) Calculate the optimal number of parallel compilation jobs of simulation task j in the compilation farm and the number of compilation jobs that can be allocated to each server, and set the Distcc receivable job upper limit of each server according to the above two calculation results; S6) The Client uses the best number of parallel compilation jobs calculated in step S5) to call the distcc distributed compilation tool through the command line to split the simulation task j into several compilation jobs and distribute them to each Server, receive the compilation results executed by each Server and integrate and output the Modelica solver.

2. The multi-stage dynamic orchestration method for parallel compilation load of Modelica tasks according to claim 1, characterized in that: The quantification of the computational load L of each node i in the current compilation cluster i and computing performance R i Specifically: Get the CPU usage of any node i RAM memory utilization Disk and network I / O usage The number of tasks being compiled n cpl Indicator, calculate the computing load L of node i in the current compilation cluster according to formula (1) and formula (2) i and computing performance R i , Calculate the load: in, are the weight parameters of CPU usage, RAM memory utilization, disk and network I / O occupancy of node i; Computational performance: in are the disk surplus rate, I / O surplus rate, CPU surplus rate, and RAM memory surplus rate of node i respectively. is the number of CPU cores of node i.

3. The multi-stage dynamic orchestration method for parallel compilation load of Modelica tasks according to claim 2, characterized in that: The estimated pre-compilation workload of all simulation tasks to be compiled in the quantified current random arrival system is specifically: Get the number of components m of any standard model of Modelica simulation task j j and the number of connections cnt j , calculate the estimated pre-compilation workload l of all simulation tasks to be compiled that are currently arriving randomly in the system according to formula (3) j , where 1≤j≤m and j is an integer, and m is the total number of Modelica simulation tasks currently arriving at the system.

4. The multi-stage dynamic orchestration method for parallel compilation load of Modelica tasks according to claim 3 is characterized in that: The work pool election strategy specifically includes the following steps: S31) first select a computing node in the cluster whose computing load is lower than lj; S32) Among the computing nodes selected in step S31), the single-machine computing performance / load ratio is the largest and The ones are Clients, and the others are Servers; S33) If the Server selected in step S32) Then, the node with the lowest computing load is selected from the unselected nodes for replacement until the disk and network I / O occupancy rates of the servers in the working pool are all lower than 0.

5.

5. The multi-stage dynamic orchestration method for parallel compilation load of Modelica tasks according to claim 4, characterized in that: The single-machine pre-compilation parallelism α of task j is obtained j Specifically: According to formula (4), the pre-compilation parallelism α of task j is calculated as j , Here The number of CPU cores of the Client.

6. The multi-stage dynamic orchestration method for parallel compilation load of Modelica tasks according to claim 5, characterized in that: The optimal number of parallel compilation jobs for the computation simulation task j in the compilation farm and the number of compilation jobs that can be allocated to each server are specifically: According to formula (5), the optimal number of parallel compilation jobs O for simulation task j in the compilation farm is calculated: j , calculate the number of compilation jobs that can be allocated to each server according to formula (6) x , Where x = 1, 2, ..., X, x represents the server computing node number, and X represents the total number of server computing nodes. The number of CPU cores of the server node numbered x is calculated. Calculate the CPU surplus rate of the server node numbered x. Indicates rounding down.

7. The multi-stage dynamic orchestration method for parallel compilation load of Modelica tasks according to claim 6, characterized in that: Said The values ​​are 0.5, 0.3, and 0.2 respectively.

Images