Single computational analysis program multi-node operation system and method

Abstract

The invention belongs to the technical field of software development, and discloses a single calculation and analysis program multi-node operation system, which comprises a calculation task composer, N calculation program coordinators and N single calculation and analysis programs, the calculation task composer decomposes the received calculation analysis task into a plurality of sub-tasks and groups the plurality of sub-tasks into N groups; each computing program coordinator receives a group of subtasks and establishes a network flow matched with the group of subtasks; distributing the established network flow to a corresponding monomer calculation analysis program; the single calculation analysis program obtains data from the calculation task composer through the network flow, and the data is fed back to the calculation task composer through the network flow after calculation.

Description

Technical Field

[0001] The present invention belongs to the technical field of software development, and in particular relates to a multi-node operation system and method for a single computing analysis program. Background Art

[0002] In the field of engineering computing, there are a large number of existing single-body computing and analysis programs. These programs were written a long time ago, using older technologies and without considering the need for parallel computing. However, they have been written and maintained for a long time, have accumulated a long period of experience, have stable functions, and are highly reliable, making them difficult to transform through reengineering. In scenarios such as multi-objective optimization and batch computing, this type of single-node serial computing program cannot meet the needs.

[0003] The existing method of converting a single program to run on multiple nodes generally decomposes the computing and analysis tasks into multiple subtasks. For complex engineering calculations, the coupling relationship between the subtasks is complex and cannot be split independently. They are usually decomposed into multiple subtasks manually and then run separately by multiple computers. In addition, most single-body computing and analysis programs use standard input streams as input. When executing in batches, files and input redirection commands are usually used for execution. At this time, computing traces will be left on the computing nodes, which has poor confidentiality and is not conducive to the distribution of computing tasks in specific scenarios. Summary of the invention

[0004] Purpose of the invention: In view of the low serial running efficiency of existing single-body computing and analysis programs and the difficulty in handling task association relationships during parallel processing, the present invention proposes a multi-node running system and method for single-body computing and analysis programs, a computing program coordinator, and realizes multi-node running of single-body computing and analysis programs in cooperation with the single-body computing and analysis programs to be called.

[0005] Technical Solution

[0006] The technical solution of the present invention provides a single computing analysis program multi-node operation system, including: a computing task scheduler, N computing program coordinators, and N single computing analysis programs, wherein the N computing program coordinators and the N single computing analysis programs are deployed on N computer nodes to provide computing power.

[0007] The computing task scheduler decomposes the received computing and analysis task into multiple subtasks, and groups the multiple subtasks into N groups;

[0008] Each computing program coordinator receives a group of subtasks, establishes a network flow matching the group of subtasks, and distributes the established network flow to the corresponding single computing analysis program;

[0009] The single computing analysis program obtains data from the computing task scheduler via the network flow, and then feeds back to the computing task scheduler via the network flow after calculation. It directly uses the network flow as input data instead of pre-downloading it to the local as a computing parameter file, which can achieve calculation without leaving any trace and has a certain information confidentiality function.

[0010] When grouping, the relationship between each subtask needs to be considered to support the improvement of computability and computing efficiency. The relationship between multiple subtasks is divided into three types: independent operation, dependent operation, and inherited operation;

[0011] Independently running subtasks do not require any prerequisites for running;

[0012] The dependent subtask needs to complete the execution of the dependent subtask, and then continue to run based on the calculation result of the dependent subtask.

[0013] The inherited subtask can run independently or inherit the calculation results of the specified subtask to speed up its own operation.

[0014] Furthermore, multiple subtasks are grouped according to the number of computing nodes and the dependency relationship of the subtasks, so that subtasks with dependency relationships are grouped together, subtasks with inheritance relationships are grouped together as much as possible, and the number of subtasks in each group is as even as possible. The grouping process of multiple subtasks is as follows:

[0015] Step A: Group the subtasks with dependencies into M trees, with the root node of the tree being the independently running subtask;

[0016] Step B: When M≤N, go to step D, otherwise, merge the trees. The process is as follows:

[0017] S1: Sort the tree from high to low according to the number of nodes;

[0018] S2: K = N + 1, merge the Kth tree with the tree with the least number of nodes;

[0019] S3: Update the number of nodes in each tree, K = K + 1, return to S1, until K = M + 1, merge M trees into N trees;

[0020] Step C: Sort the N trees in depth-first order to form N subtask sequences that can be executed sequentially, and then proceed to step E;

[0021] Step D: Sort the M trees in depth-first order to form M subtask sequences that can be executed sequentially, and then fill NM empty sequences to form N subtask sequences, and then proceed to step E;

[0022] Step E: Fill the subtasks with inheritance relationship into the subtask sequence with the least number of nodes; each time a subtask with inheritance relationship is filled, the subtask sequence with the least number of nodes is updated;

[0023] Step F: The remaining independently running subtasks are filled into the N subtask sequences according to the method of step E.

[0024] Furthermore, in step E, if there are multiple subtask sequences with the least number of current nodes, the subtasks to be filled will be filled into the subtask sequences with which they have an inheritance relationship first. If there is no inheritance relationship, they will be filled randomly. Under normal circumstances, even if the subtasks with an inheritance relationship inherit the calculation results of the designated subtasks and speed up their own operation, the improvement in their overall efficiency is not as good as parallel operation. Therefore, this step takes the approach of giving priority to parallel operation and then considering the inheritance relationship.

[0025] Furthermore, when assigning subtasks, the computing task scheduler assigns a different identifier to each subtask, so that a downlink interface for obtaining subtask information and an uplink interface for uploading subtask output can be provided for each subtask according to the identifier.

[0026] Furthermore, each computing program coordinator obtains the network response stream from the downstream interface of the computing task scheduler according to the identifier of the subtask currently being executed; and simultaneously starts the network request stream;

[0027] Start the single computing analysis program as a child process, redirect the network response stream as its input, and redirect the network request stream as its output. The original input method of the single computing analysis program is the standard input and output stream. By replacing the input and output streams, the single computing analysis program can also be driven to run and the calculation can be realized without leaving any trace.

[0028] Based on the aforementioned system, the technical solution of the present invention provides a method for running a single computing analysis program on multiple nodes, comprising the following steps:

[0029] 1) Inputting the computing and analysis tasks to be executed into the computing task scheduler;

[0030] 2) The computing task scheduler decomposes the computing analysis task into multiple subtasks, and assigns the subtasks to N task groups according to the aforementioned subtask grouping process, and assigns each task group to a corresponding computing program coordinator node;

[0031] 3) Each computing program coordinator node calls the single computing analysis program as its child process, and transmits the input information of each subtask in each task group stored in the computing task scheduler to the single computing analysis program through input redirection from the downlink interface via the network flow, and transmits the output result of the execution of the single computing analysis program to the computing task scheduler by uploading the subtask output information to the network uplink interface according to the subtask identifier through output redirection via the network flow;

[0032] 4) The computing task scheduler collects the results of subtask execution by each computing program coordinator node and integrates them into the final computing analysis results.

[0033] In summary, the beneficial effects of the present invention are as follows:

[0034] The core of the present invention is to solve the problem of multi-task distribution with correlation in engineering calculations, and to shield the calculation traces through the input and output redirection technology of network flow, so that the single calculation analysis program originally designed to be executed on a single machine can be completed in parallel on multiple computers, thereby improving the calculation efficiency and performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a schematic diagram of a multi-node running system for a single computing analysis program. DETAILED DESCRIPTION

[0036] According to the technical solution of the present invention, the specific implementation method may be as follows.

[0037] Example 1: Use nodejs to build a computing task orchestration server, and use Python scripts as computing program coordinators to implement multi-node operation of F programs.

[0038] This embodiment provides a system for realizing multi-node operation of a single computing analysis program F, including: a computing task orchestration server written in nodejs, N computing program coordinator scripts, and deployment of N single computing analysis programs F, which are deployed on N computer nodes to provide computing power.

[0039] The single computational analysis program F is a computational analysis program written in C++. To ensure its portability and versatility, F relies on the underlying interface of the operating system as little as possible during its writing process, and both input and output use standard input and output streams. Its input file has the following form (Job / Task / Parameter), which is divided into three levels: job level, computational task level, and parameter level. Among them: the job level is the root node, and there is only one definition in the input file; each computational task is divided into three categories: balancing calculation, transient calculation, and stability calculation, among which transient calculation and stability calculation depend on the previous balancing calculation results, and in the continuous balancing calculation task, the balancing results of the previous example can be reused between multiple balancing states of a model state, thereby improving the computational efficiency.

[0040] The computing task scheduling service provides the following interfaces:

[0041] a) The computing job submission interface runJob(jobInfo) parses the received computing analysis job information jobInfo, generates a job description object job, and stores it in the service memory. It is decomposed into multiple L subtasks Task[L]. Each Task[i] contains the decomposed task information Task[i].info and is assigned a Task[i].id. The subtask arrangement arrangement is called to group the L subtasks Task[L] into N groups TaskGroup[N]. Each TaskGroup[i] contains a group Task[]. After the above operations are completed, a job.id is assigned to the job and returned to the user.

[0042] b) The subtask information is obtained through the downstream interface getTaskInfo(task_id), which finds task[i] according to the task_id identifier passed in, and returns task[i].info as the response stream;

[0043] c) Subtask calculation result output upload interface uploadTaskResult(task_id), which receives POST type requests, finds task[i] according to the incoming task identifier (task_id), creates result object attributes in task[i], and uses task[i].result as the data node to store the requestBody information in the POST request; each time the interface is called, a completed flag is added to task[i]. Finally, when all tasks under a job are marked as completed, they are integrated into the final calculation and analysis results and recorded in the job.summary of the job object residing in the memory described in interface a).

[0044] d) The task allocation interface allocTasks(group) is used to push the task information of the corresponding group to the computing program coordinator. When the interface is called, the computing task scheduling service finds the TaskGroup[i] information described in the above interface a) according to the passed-in group identifier, outputs all the Task[] information identifiers in TaskGroup[i], and adds a new identifier allocated in TaskGroup[i].

[0045] e) The computing job viewing interface getJobResult(jobid) obtains the job.summary attribute of the job object residing in the service based on the passed-in jobid. If yes, it returns; if not, it returns a prompt "the job is not completed".

[0046] The subtask arrangement described in the above interface a) is as follows:

[0047] 1) First, according to the input rules of the F software and the identification of all task information in the job, the dependency relationship is established. Multiple balancing calculations under the same model state are identified as successive successions, and stability and transient calculations under the same balancing state are identified as dependent balancing calculations.

[0048] 2) Group multiple subtasks according to the number of computing nodes and the dependencies of the subtasks, so that subtasks with dependencies are grouped together, subtasks with inheritance relationships are grouped together as much as possible, and the number of subtasks in each group is as even as possible. The grouping process of multiple subtasks is as follows:

[0049] 2-A) Group the dependent subtasks into M trees, with the root nodes of the trees being the independently running subtasks;

[0050] 2-B) When M≤N, go to step 2-D), otherwise, merge the trees. The process is as follows:

[0051] 2-B-S1) Sort the tree according to the number of nodes from high to low;

[0052] 2-B-S2) K = N + 1, merge the Kth tree with the tree with the least number of nodes;

[0053] 2-B-S3) Update the number of nodes in each tree, K = K + 1, return to 2-B-S1), until K = M + 1, merge the M trees into N trees;

[0054] 2-C) Sort the N trees in depth-first order to form N subtask sequences that can be executed sequentially, and proceed to step 2-E);

[0055] 2-D) Sort the M trees in depth-first order to form M subtask sequences that can be executed sequentially, and then fill NM empty sequences to form N subtask sequences, and proceed to step 2-E);

[0056] 2-E) Fill the subtasks with inheritance relationship to the subtask sequence with the least number of nodes; each time a subtask with inheritance relationship is filled, the subtask sequence with the least number of nodes is updated;

[0057] 2-F) The remaining independently running subtasks are filled into the N subtask sequences according to the method of step E.

[0058] In step 2-E), if there are multiple subtask sequences with the least number of current nodes, the subtasks to be filled are first filled into the subtask sequences with which there is an inheritance relationship; if there is no inheritance relationship, they are filled randomly.

[0059] In the computing node where the computing program coordinator Python script resides, when it is idle, the task allocation interface allocTasks(group) of the computing task scheduling service is called every 1 minute to obtain the task information list taskid[]. Then, the calculation is performed in sequence according to each taskid. The specific process is as follows:

[0060] 1) Call the subtask information of the computing task orchestration service to obtain the downstream interface getTaskInfo, pass in the task_id, and obtain the response stream ResponseStream;

[0061] 2) Create a POST type network request stream RequestStream, pointing to the subtask computing result output upload interface uploadTaskResult of the computing task orchestration service, with task_id given in the parameter;

[0062] 3) Use subprocess.run in the subprocess module in Python to start the single-body computing analysis program F, in which the redirection of input and output streams is specified in the run method, the standard input stream is replaced by ResponseStream, and the standard output stream is replaced by uploadTaskResult.

[0063] 4) When the single computing analysis program F is running, the generated output is sent to the uploadTaskResult interface. After the execution is completed, the writing of RequestStream is terminated. During the execution of the program, the computing node computer does not generate any file reading and writing traces from beginning to end.

[0064] The specific use process of this embodiment includes the following steps:

[0065] 1) The user passes the input file of the single computing analysis program F to the computing task submission interface runJob(job) of the computing task orchestration service and obtains a jobid identifier;

[0066] 2) During the execution of runJob, the computing task orchestration service decomposes the computing and analysis task into multiple subtasks;

[0067] 3) The computational program coordinator Python script obtains the task list assigned to its own node according to its group;

[0068] 4) The computing program coordinator Python script of each node executes each item in the assigned task list in turn. According to the taskid of each item, it first obtains the downstream network response stream and the upstream network request stream by calling the subtask information of the computing task orchestration service through the downstream interface getTaskInfo and the subtask computing result output upload interface uploadTaskResult, and calls the monomer computing analysis program F as a subroutine, using the downstream network response stream and the upstream network request stream as redirections to replace the original standard input and output streams of the F program. During the operation of F, the output content is continuously written to the upstream network request stream, and these output contents are passed to the uploadTaskResult interface of the computing task orchestration service;

[0069] 5) Each time the uploadTaskResult interface call of the computing task orchestration service is completed, the completion status of the corresponding Task is updated. Finally, when all Tasks under a Job are completed, uploadTaskResult is integrated into the final computing analysis result.

[0070] 6) The user can call getJobResult at any time with the jobid obtained in step 1) to check the calculation result status. When it no longer returns the prompt "The job is not completed", it indicates that the calculation is completed and the result is successfully obtained.

[0071] Through the technical solution described in this embodiment:

[0072] The parallel operation of the original single calculation and analysis program F is realized, and the dependency between its continuous balancing, transient, stability and balancing is well resolved. Some key parameter settings required for the execution of F do not leave traces on the computer providing the computing power, eliminating the user's concerns about the diffusion of parameter settings during parallel execution and improving the calculation efficiency and performance of the F program.

Claims

1. A single computing and analysis program multi-node operation system, characterized by: The system includes: Computing task scheduler, N computing program coordinators, N single computing analysis programs; The computing task scheduler decomposes the received computing and analysis task into multiple subtasks, and groups the multiple subtasks into N groups; Each computing program coordinator receives a set of subtasks and establishes a network flow matching the set of subtasks; Assign the established network flows to the corresponding monomer computational analysis programs; The single computing analysis program obtains input data from the computing task scheduler via the network flow, and after calculation, it feeds back output data to the computing task scheduler via the network flow.

2. The system according to claim 1, characterized in that: There are three types of associations between multiple subtasks: independent operation, dependent operation, and inherited operation; Independently running subtasks do not require any prerequisites for running; The dependent subtask needs to complete the execution of the dependent subtask, and then continue to run based on the calculation result of the dependent subtask. The inherited subtask can run independently or inherit the calculation results of the specified subtask to speed up its own operation.

3. The system according to claim 2, characterized in that: The grouping process of multiple subtasks is as follows: Step A: Group the subtasks with dependencies into M trees, with the root node of the tree being the independently running subtask; Step B: When M≤N, go to step D, otherwise, merge the trees. The process is as follows: S1: Sort the tree from high to low according to the number of nodes; S2: K = N + 1, merge the Kth tree with the tree with the least number of nodes; S3: Update the number of nodes in each tree, K = K + 1, return to S1, until K = M + 1, merge M trees into N trees; Step C: Sort the N trees in depth-first order to form N subtask sequences that can be executed sequentially, and then proceed to step E; Step D: Sort the M trees in depth-first order to form M subtask sequences that can be executed sequentially, and then fill NM empty sequences to form N subtask sequences, and then proceed to step E; Step E: Fill the subtasks with inheritance relationship into the subtask sequence with the least number of nodes; each time a subtask with inheritance relationship is filled, the subtask sequence with the least number of nodes is updated; Step F: The remaining independently running subtasks are filled into the N subtask sequences according to the method of step E.

4. The system according to claim 3, characterized in that: In step E, if there are multiple subtask sequences with the least number of current nodes, if the subtask inherited by the subtask to be filled exists, the subtask to be filled will be filled first into the subtask sequence where the subtask inherited by it is located; if the subtask sequence with the least number of current nodes does not have any subtask inherited by the subtask to be filled, it will be filled randomly.

5. The system according to claim 4, characterized in that: The computing task scheduler assigns a different identifier to each subtask, and respectively assigns an upstream interface and a downstream interface according to the identifier.

6. The system according to claim 5, characterized in that: Each computing program coordinator obtains the network response stream from the downstream interface of the computing task scheduler according to the identifier of the subtask it is currently executing; at the same time, it opens the network request stream pointing to the upstream interface of the computing task scheduler; The computing program coordinator starts the single computing analysis program as a child process, redirects the network response stream as its input, and redirects the network request stream as its output.

7. The system according to claim 6, characterized in that: When the computing program coordinator starts the single computing analysis program as a child process, it uses the input and output redirection technology to replace the original standard input and output streams of the single computing analysis program.

8. A method for running a single computing analysis program on multiple nodes, based on the system operation described in any one of the above, characterized in that: The following steps are involved: 1) Inputting the computing and analysis tasks to be executed into the computing task scheduler; 2) The computing task scheduler decomposes the computing analysis task into multiple subtasks, assigns the subtasks to N task groups, and assigns each task group to a corresponding computing program coordinator node; 3) Each computing program coordinator node calls the single computing analysis program as its child process, and transmits the input information of each subtask in each task group stored in the computing task scheduler to the single computing analysis program through input redirection from the downlink interface via the network flow, and transmits the output result of the execution of the single computing analysis program to the uplink interface of the network by uploading the subtask output information according to the subtask mark via the network flow through output redirection, and returns it to the computing task scheduler; 4) The computing task scheduler collects the results of subtask execution by each computing program coordinator node and integrates them into the final computing analysis results.

Images