Parallelization / distribution processing system of application code

Abstract

To reduce a processing time by appropriately distributing the processing time according to contents of an execution of an application program and an environment.SOLUTION: A parallelization / distribution processing system of an application code, comprises: a parallelization part that extracts a portion having parallelism to generate a parallel code from codes of an application program; a cluster selection processing calling part that calls selection processing of a cluster optimal for each program parallelized in the parallel code generated by the parallelization part; a parallel code block performance requirement defining part that measures a performance requirement of the parallel code generated by the parallelization part; a cluster performance measurement part that measures a dynamic performance in the case where the parallel code is executed in the cluster as an execution target of the parallel code; and a cluster selection part that selects a cluster satisfying the performance requirement measured by the performance requirement definition part based on a measurement result in the cluster performance measurement part.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a parallel and distributed processing system for application code. [Background technology]

[0002] It has been a common practice to partially rewrite existing application code and process programs in a distributed system in a distributed and parallel manner. For example, a distributed framework called Ray is known to provide APIs and libraries that instruct the execution of programs in a distributed and parallel manner from the code, as well as functions for building Ray clusters that serve as the platform for program execution. Ray clusters can be composed of a single node or multiple nodes, and distributing programs on a Ray cluster and executing them in parallel can speed up processing execution.

[0003] Patent Document 1 describes a technology based on OpenACC and CUDA that automatically offloads application processing to a GPU, parallelizes parts of a program that can be parallelized, and calculates and arranges parallel code blocks (blocks of code that have been parallelized to make parts of a program that can be parallelized) that minimize an objective function related to user satisfaction evaluation. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2023 / 228369 Summary of the Invention [Problem to be solved by the invention]

[0005] When using Ray to distribute and parallelize an application program, the entire program is not executed in parallel at once; rather, the application code is written using Ray's API and libraries and executed in parallel. Therefore, it is necessary to examine the application code to determine which parts of the application code can be executed in parallel. Furthermore, if you configure one Ray cluster on the node running the application program and one or more other Ray clusters on nodes different from the node running the application program, you must verify whether distributing the program to Ray clusters consisting of nodes different from the node running the application program reduces execution time by applying it to an actual program and environment. For example, the technology described in Patent Document 1 is a technology that determines only the offloading placement based on an objective function, but does not determine the need for distribution based on dynamic observation of the program. In other words, the technology described in Patent Document 1 has the problem that it does not determine the need for distribution based on the content and environment of program execution.

[0006] In view of these points, the present invention aims to provide a parallelized and distributed processing system for application code that can appropriately distribute application programs according to their execution content and environment, thereby achieving higher speeds. [Means for solving the problem]

[0007] In order to solve the above problems, for example, the configurations described in the claims are adopted. The present application includes a number of means for solving the above problems, examples of which are as follows. As a parallelization and distributed processing system for application code, it comprises a parallelization unit that rewrites parts of the application program code that have program parallelism into code that can be processed in parallel; a cluster selection process call unit that calls the process to select the optimal cluster for the program code that will be processed in parallel within the application code generated by the parallelization unit (hereinafter referred to as the parallel code block); a parallel code block performance requirements definition unit that defines the performance requirements of the parallel code block; a cluster performance measurement unit that measures the dynamic performance when the parallel code block is executed on a cluster, which is a computing resource; and a cluster selection unit that selects a cluster that satisfies the performance requirements defined by the performance requirements definition unit based on the measurement results of the cluster performance measurement unit. [Effects of the Invention]

[0008] This invention automatically generates code to parallelize part of a program by analyzing the application code, and dynamically determines whether to distribute the program based on the content of the program to be parallelized and the performance of clusters prepared in advance. This allows the application program to be executed on an appropriate cluster, thereby reducing the execution time of the application program. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a configuration diagram illustrating an example of a parallelized and distributed processing system for application code according to a first embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a remote cluster according to a first embodiment of the present invention. [Figure 3] FIG. 1 is a diagram illustrating an example of the hardware configuration of a computer according to a first embodiment of the present invention. [Figure 4] 1 is a diagram showing an example of the configuration of a parallelized and distributed processing system for application code according to a first embodiment of the present invention, and an overview of the processing; [Figure 5] FIG. 2 is a diagram showing an example of application code of a program having parallelism according to the first embodiment of the present invention. [Figure 6] FIG. 3 is a diagram showing an example of parallelized application code according to the first embodiment of the present invention. [Figure 7] FIG. 2 is a diagram showing an example of analysis data according to the first embodiment of the present invention. [Figure 8] 4 is a flowchart showing an example of parallelization processing of application code according to the first embodiment of the present invention. [Figure 9] 10 is a flowchart showing an example of a cluster selection process according to the first embodiment of the present invention. [Figure 10] 10 is a flowchart illustrating an example of a process for calculating a predicted value of a processing time in a remote cluster according to the first embodiment of the present invention. [Figure 11] FIG. 3 is a diagram showing an example of a parallel code block database according to the first embodiment of the present invention. [Figure 12] FIG. 10 is a configuration diagram illustrating an example of a parallelized and distributed processing system for application code according to a second embodiment of the present invention. [Figure 13] FIG. 10 is a diagram illustrating an example of a cluster selection process according to the second embodiment of the present invention. [Figure 14] 10 is a flowchart showing an example of a cluster selection process according to a second embodiment of the present invention. [Figure 15] FIG. 10 is a diagram illustrating an example of a performance database of a remote cluster according to the second embodiment of the present invention. [Figure 16] FIG. 10 is a diagram showing an example of a performance requirement database for parallel code blocks according to the second embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0010] <First embodiment> A parallelized and distributed processing system for application code according to a first embodiment of the present invention will be described below with reference to FIGS.

[0011] [Overall configuration of parallelized and distributed processing system for application code] 1 shows the overall configuration of an application code parallelization and distribution processing system 100 according to this embodiment. The configuration shown in FIG. 1 is seen from the perspective of the function of processing an application program. The application code parallelization and distribution processing system 100 parallelizes an application program prepared in a computer system 4 when parallelization is possible.

[0012] The application program 42 prepared in the computer system 4 is executed by the computer system 4 unless parallel processing, which will be described later, is performed. Here, in this embodiment, if the application program 42 can be parallelized, at least a portion of the application program is parallelized and the application code 41 is rewritten in processing in the application code parallelization and distributed processing system 100. In addition, at least a portion of the parallelized program is distributed. Furthermore, a program that has been parallelized but not distributed in processing in the application code parallelization and distributed processing system 100 is executed in a local cluster 43 prepared in the computer system 4.

[0013] The distributed program of application program 42 is then executed in a first remote cluster 5, a second remote cluster 6, or the like, which are prepared separately from computer system 4, thereby shortening the execution time of the application program. Analysis data 7 is data processed by the execution of the application program. In this embodiment, analysis data 7 is sample data for parallelization and distribution.

[0014] First, the configurations of the computer system 4, the first remote cluster 5, and the second remote cluster 6 will be described. The computer system 4 obtains the application program 42 from the application code 41. The application code 41 is a description of the instructions of the application program 42. The application program 42 includes a cluster selection process calling unit 421. Of the application programs 42 , those that have been parallelized but not distributed are executed on a local cluster 43 provided in the computer system 4 .

[0015] The local cluster 43 has worker nodes 432 that schedule program execution and perform processing operations, and a head node 431 that manages the local cluster 43 and the worker nodes 432 and sends program execution commands to the worker nodes 432. However, among the application programs 42, distributed programs are not executed in the local cluster 43.

[0016] The first remote cluster 5 and the second remote cluster 6 are each a collection of computer systems connected via a predetermined network 310 so as to be able to transfer data with the computer system 4. In the first remote cluster 5 and the second remote cluster 6, each computer system is assigned the role of a head node 51, 61 and a worker node 52, 62. For example, when a command to execute a program in the first remote cluster 5 is transmitted from the computer system 4, the first remote cluster 5 executes the corresponding program in the worker node 52 according to the command from the head node 51. The execution result in the worker node 52 is transmitted to the computer system 4.

[0017] Similarly, when a program execution command for the second remote cluster 6 is transmitted from the computer system 4, the second remote cluster 6 executes the corresponding application code in the worker node 62 according to the execution command from the head node 61. The execution result in the worker node 62 is similarly transmitted to the computer system 4. It should be noted that a plurality of worker nodes 52 and 62 may be provided as will be described with reference to FIG. The local cluster 43, the first remote cluster 5, and the second remote cluster 6 are collectively referred to as a cluster group 101. The cluster group 101 functions as a computational resource.

[0018] The decision as to which cluster in the cluster group 101 each parallel code block should be executed in is initiated by a cluster selection process calling unit 421 provided in the application program 42 . Note that the provision of two remote clusters 5 and 6 as shown in FIG. 1 is merely an example, and the number of remote clusters may be any number greater than or equal to one.

[0019] Next, the configuration of the application code parallelization and distribution processing system 100 that parallelizes the application code 41 acquired by the computer system 4 will be described. The parallelization and distributed processing system 100 for application code comprises a parallelization unit 1, a distribution unit 2, and a storage device 3. The parallelization and distributed processing system 100 for application code, which is made up of the parallelization unit 1, the distribution unit 2, and the storage device 3, is connected to a computer system 4 and remote clusters 5 and 6 via a network, and data can be transferred between them. 1, the application code parallelization and distributed processing system 100 may be configured as a computer separate from the computer system 4. For example, the application code parallelization and distributed processing system 100 may be configured to be included within the computer system 4.

[0020] The parallelization unit 1 includes a parallelization pattern detection unit 11 and a parallel code generation and rewriting unit 12 . The distribution unit 2 includes a parallel code block performance requirement definition unit 21, a remote cluster performance measurement unit 22, and a cluster selection unit . The storage device 3 includes a parallel code block database 31 and a database 32 for input data to the parallel code blocks. The functions executed by each processing unit in the application code parallelization and distribution processing system 100 will be described in detail later. The parallelization unit 1, distribution unit 2, storage device 3, computer system 4, and respective remote clusters 5 and 6 can communicate with each other via a network 8. The network 5 may be the Internet, a local area network (LAN), a dedicated communication network, or a virtual network such as a VLAN.

[0021] [Example of remote cluster configuration] FIG. 2 shows an example of the hardware configuration of each of the remote clusters 5 and 6. As described in FIG. 1, the remote clusters 5 and 6 are composed of head nodes 51 and 61 and worker nodes 52 and 62. The head nodes 51 and 61 and the worker nodes 52 and 62 are configured as a computer system as shown in FIG.

[0022] That is, the head node 51 of the first remote cluster 5 is configured as a computer system, and the worker nodes 52 that perform calculations under the control of the head node 51 are also configured as computer systems. In the example of FIG. 2, three worker nodes 52 are provided, each configured as an individual computer system. Similarly, the head node 61 of the second remote cluster 6 is configured as a computer system, and the three worker nodes 62 that perform calculations under the control of the head node 61 are also configured as individual computer systems.

[0023] The application code that each worker node 52 , 62 processes is supplied from the storage device 3 , and the results of the processing in each worker node 52 , 62 are transferred to the storage device 3 . Each head node 51, 61 also operates by exchanging data with the computer system 4. In the configuration shown in Figure 2, the parallelized and distributed processing system 100 for application code shown in Figure 1 is built into the computer system 4, and the application code 41 is stored in a storage device within the computer system 4.

[0024] [Example of computer system hardware configuration] FIG. 3 shows an example of the hardware configuration of a computer system 300 serving as the computer system 4 shown in FIG. 1, and the head nodes 51 and 61 and worker nodes 52 and 62 shown in FIG. 3, the computer system 300 includes a processor 301, a memory 302, a storage device 303, an input device 304, an output device 305, and a communication interface 306. The processor 301, the memory 302, the storage device 303, the input device 304, the output device 305, and the communication interface 306 are connected to each other via an internal bus 307.

[0025] The processor 301 is an arithmetic processing unit that reads out a program stored in the memory 302 or the storage device 303 and executes arithmetic processing in the memory 302. The processor 301 is also called a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). The memory 302 is a work memory that executes arithmetic processing under the control of the processor 301, but the memory 302 may also store programs. The storage device 303 stores programs such as the application program 42 and the operating system, etc. The storage device 303 also stores the results of calculations performed in the memory 302 and data required for the calculations.

[0026] The input device 304 receives input data from the outside. The input data received by the input device 304 is stored in the storage device 303 or the like. The output device 305 outputs the results of calculations executed under the control of the processor 301 to the outside. The communication interface 306 communicates with other computer systems via the network 8 and transfers data. For example, communication between the computer system 4 functioning as the local cluster 43 and the computer systems functioning as the first remote cluster 5 and the second remote cluster 6 is performed via the communication interface 306 provided in each computer system.

[0027] [Example of an overview of a parallelized and distributed processing system for application code] FIG. 4 shows an example of an outline of a parallelized and distributed processing system 100 for application code according to this embodiment. First, the application code parallelization / distributed processing system 100 acquires the application code 41 and causes the parallelization unit 1 to read it.

[0028] The parallelization unit 1 is composed of a parallelization pattern detection unit 11 and a parallel code generation and rewriting unit 12. The parallelization pattern detection unit 11 detects parallelization patterns in the application code 41. The parallel code generation and rewriting unit 12 generates parallel code from the detected parallelization patterns. The parallel code generation and rewriting unit 12 then executes a write process to rewrite the parallelization patterns in the application code 41 into the generated parallel code. The application program 42 applies the rewritten application code 41 as an application for execution.

[0029] The application program 42 issues an execution command to one of the clusters 43, 5, and 6 in the cluster group 101, which is a computational resource, to execute parallel processing on a parallel code block, which is a portion of the application code 41 where parallel code has been generated. The application program 42 then obtains the results based on this execution command. Immediately before this execution command, the cluster selection process calling unit 421 prepares processing, which starts the cluster selection process in the distribution unit 2. The cluster selection process will be explained later. When the distribution unit 2 sends the result of the cluster selection process to the application program 42, it connects to one of the clusters in the cluster group 101 based on the result, and sends an execution command for the parallel processing of the parallel code block to the connected cluster.

[0030] The distribution unit 2 is composed of a parallel code block performance requirement definition unit 21, a remote cluster performance measurement unit 22, and a cluster selection unit 23. The parallel code block performance requirement definition unit 21 defines the performance requirements for the parallel code block generated by the parallelization unit 1. A parallel code block is a block of code in which parallelizable portions of a program are parallelized, and is supplied to a cluster for parallel execution. The local cluster 43 processes sample analysis data 7 using the parallelized application code, measures the processing time of the parallelized code during execution, and collects data to define the performance requirements for the parallel code block based on that data. The data input to the parallel code block is linked to the corresponding parallel code and stored in the input database 32 for the parallel code block.

[0031] Furthermore, the remote cluster performance measurement unit 22 measures the performance requirements of the remote clusters 5 and 6 of the cluster group 101. At this time, the remote clusters 5 and 6 use the parallel code to obtain data associated with the corresponding parallel code block from the database 32 for input data to parallel code blocks, and then process the obtained data. Note that this process is limited to the execution of one parallel code block. From the processing results, the processing time, CPU capacity, and memory capacity of one parallel code block in the remote cluster are measured. However, the CPU capacity, memory capacity, etc. of the remote cluster may be obtained in advance, as described below.

[0032] Then, the cluster selection unit 23 selects an appropriate cluster based on the performance requirements of the parallel code block defined by the parallel code block performance requirement definition unit 21 and the performance of the remote clusters 5 and 6 defined by the remote cluster performance measurement unit 22. The selection result of the cluster selection unit 23 is communicated to the application program 42, and the parallel processing is executed in the selected cluster. The measurement of the performance requirements of the parallel code blocks in the distribution unit 2 and the measurement of the performance requirements of each of the clusters 43, 5, and 6 are performed by reading from and writing to the storage device 3, as described above.

[0033] [Examples of parallelizable and parallelized code] Next, examples of application code detected by the parallelization pattern detection unit 11 of the parallelization unit 1 in the application code parallelization and distributed processing system 100 of this embodiment will be described with reference to Figures 5 and 6. The application code shown in Figures 5 and 6 is application code that can be parallelized.

[0034] 5, in the second line "for i in range(1000)", the third line "num=i*i" and the fourth line "list1.append(num)" are executed multiple times (1000 times). The parallelization pattern detection unit 11 detects such a location where a program is processed multiple times using the same method, with only different data given.

[0035] Based on the detection by the parallelization pattern detection unit 11, the parallel code generation and rewriting unit 12 generates and rewrites parallel code at the corresponding location. Figure 6 shows an example of application code in which the parallelization pattern shown in Figure 5 has been rewritten into parallel code. The example in Figure 6 is written in a notation for executing processing on a Ray cluster (local cluster or remote cluster).

[0036] In the example in Figure 6, "input_data1=pd.read_pickle("forest.pkl")" on line 9 and "raycluster_for_f=ray.select_cluster(f,1000,input_data1)" on line 10 contain ray.select_cluster (function name, loop count, input data for one run of a program that is processed multiple times), which indicates the execution of the cluster selection process. The data read in line 9 here is generated during code analysis during parallel processing, written to a database for input data to the parallel code block, and read during the cluster selection process. The 11th line of the application code shown in Figure 6, "ray.init(address=raycluster_for_f)", indicates the Ray settings, specifies the cluster to perform processing, and initiates a connection to that cluster. Also, "num = f.remote(i)" on line 15 of the application code shown in Figure 6 indicates the execution call of function f on the cluster. Also, "list1 = ray.get(ray_obj)" on line 18 of the application code shown in Figure 6 indicates obtaining the execution results on the cluster.

[0037] [Example of analysis data] 7 shows an example of a list of analysis data 7 supplied to the parallelized and distributed application code processing system 100 of this embodiment. The analysis data 7 is sample data, and is data in a format that can be processed by the application program 42. In the example of FIG. 7, a sequence number (seq), a status (status), and an action (action#1, action#2, . . . ) are shown. The analysis data 7 shown in FIG. 7 is data that is processed by the execution of the application code 41.

[0038] [Application code parallelization process] FIG. 8 is a flowchart showing the procedure of the parallelization process of application code performed by the parallelization unit 1. First, the parallelization pattern detection unit 11 of the parallelization unit 1 detects and extracts parallelizable code from the prepared application code 41 by pattern recognition or the like (step S801). An example of this detected parallelizable code is the code described in FIG.

[0039] Next, the parallel code generator / rewriter 12 of the parallelizer 1 generates parallelized code from the parallelizable code detected by the parallelization pattern detector 11 (step S802). At this time, the parallel code generator / rewriter 12 generates parallelized code using, for example, a large-scale language model (LLM). An example of this parallelized code is the code described in FIG. 6.

[0040] The code generated by the parallel code generating and rewriting unit 12 in step S802 is stored in the parallel code block database 31 of the storage device 3 (step S803). Then, the parallelized code stored in the parallel code block database 31 is reflected in the application code (step S804). The application code parallelized in step S804 is output from the parallel code generating and rewriting unit 12 to the computer system 4 (step S805), and is reflected in the application code 41 held by the computer system 4 (step S806).

[0041] [Cluster selection process procedure] FIG. 9 is a flowchart showing the procedure of the process performed by the distribution unit 2 to select a cluster of parallelized code. First, the distribution unit 2 acquires a parallel code block, which is the code parallelized in the process of FIG. 8, input data (for one iteration) which is data to be processed by the parallel code, and the loop count (step S901). Then, the parallel code block performance requirement definition unit 21 of the distribution unit 2 executes one processing of the parallel code block in the local cluster 43 and measures the performance at that time (step S902). The performance measured here includes, for example, execution time, CPU or GPU performance, memory usage, etc.

[0042] Next, the remote cluster performance measurement unit 22 of the distribution unit 2 acquires the performance requirements of the remote clusters 5 and 6 from the computer systems configured as each of the remote clusters 5 and 6. Then, the remote cluster performance measurement unit 22 executes one processing of the parallel code block in each of the remote clusters 5 and 6, and measures the performance at that time (step S903).

[0043] Here, the case where two remote clusters 5 and 6 are provided is described, but if the number of remote clusters is three or more, the remote cluster performance measurement unit 22 measures the performance of each cluster. The performance measured here includes, for example, execution time, CPU or GPU performance, memory usage, etc.

[0044] The remote cluster performance measurement unit 22 may acquire the performance requirements of the remote clusters 5 and 6 in step S903 according to the flow shown in the flowchart of Fig. 9, or may acquire them in advance. However, it is preferable to acquire them sequentially according to the flow shown in the flowchart of Fig. 9, because this allows dynamic acquisition of the performance requirements of the remote clusters 5 and 6.

[0045] Next, the cluster selection unit 23 of the distribution unit 2 acquires the measurement results from steps S902 and S903, and executes a process of calculating a predicted value of the processing time in the remote cluster (step S904). Details of this processing time calculation process will be described with reference to FIG. Then, based on the processing time obtained in step S904, the cluster selection unit 23 determines the cluster that will process the parallel code block with the shortest total time (step S905), and outputs the determined cluster as the appropriate cluster (step S906). If the shortest total time is achieved by using only the local cluster 43 without using the remote clusters 5 and 6, the cluster selection unit 23 outputs a selection result that does not perform distribution.

[0046] When an appropriate cluster for processing the parallel code block is obtained in this way, computer system 4 initiates connection to the appropriate cluster based on the results of the cluster selection process, and the corresponding parallel code is processed in parallel on that cluster. However, if the cluster with the shortest total time obtained by cluster selection unit 23 selects local cluster 43, it is determined that distribution of processing is unnecessary, and the corresponding parallel code block is processed only on local cluster 43 without being distributed to other computer systems.

[0047] [Total time calculation process] FIG. 10 is a flowchart showing the details of the process of calculating the predicted value of the processing time in the remote cluster in step S904. First, the cluster selection unit 23 acquires performance information of the remote clusters 5 and 6 (step S1001). Next, the cluster selection unit 23 calculates latency from the data transfer speed and the size of the input data to the parallel code block (step S1002). Here, latency is the time from when a data transfer instruction is issued to when data transmission is completed, and is the time (latency) between the computer system 4 and each remote cluster 5, 6.

[0048] Furthermore, the cluster selection unit 23 calculates the processing time of the entire process from the processing times in the worker nodes 52 and 62 of the remote clusters 5 and 6, the number of loops, and the number of worker nodes 52 and 62 (step S1003). Thereafter, the cluster selection unit 23 adds the latency obtained in step S1002 to the processing time of the entire process in step S1003 to calculate the total time (step S1004). The total time calculated in step S1004 is the processing time obtained in step S904 of the flowchart in FIG.

[0049] [Parallel Code Block Database Example] FIG. 11 shows an example of data stored in the parallel code block database 31 that constitutes the storage device 3. 11, the parallel code block database 31 has code block IDs 311 and codes 312. Codes 312 are stored as parallel code blocks for each code block ID 311, which is an identifier, in the parallel code block database 31. A parallel code block is a group of codes obtained by parallelizing parts of a program that can be run in parallel, and is supplied to a cluster for parallel execution.

[0050] The parallel code blocks stored in the parallel code block database 31 are supplied to the clusters 43, 5, and 6 selected from the cluster group 101. The cluster selection process calling unit 421 determines which cluster the code blocks are to be supplied to.

[0051] [Effects of the first embodiment] As described above, according to this embodiment, it is possible to automatically create code that appropriately parallelizes application code by analyzing the application code, and it is also possible to measure the performance of the prepared clusters and, based on the results, execute a portion of the parallelized code program on one of the prepared clusters that has the shortest processing time. In particular, by sequentially measuring the performance of remote clusters and local clusters for a program to be processed in parallel, dynamic performance can be determined, making it possible to select an appropriate cluster according to the content of the program to be processed in parallel.

[0052] <Second embodiment> Next, a parallelized and distributed processing system for application code according to a second embodiment of the present invention will be described with reference to Figures 12 to 16. In Figures 12 to 16, parts corresponding to those in Figures 1 to 11 described in the first embodiment are given the same reference numerals, and duplicated explanations will be omitted.

[0053] [Overall configuration of parallelized and distributed processing system for application code] Fig. 12 shows the overall configuration of an application code parallelization and distributed processing system 100 according to this embodiment. The difference between the application code parallelization and distributed processing system 100 shown in Fig. 1 and the system shown in Fig. 1 is that the distribution unit 2 includes a parallel code block performance requirement database 24 and a remote cluster performance database 25. Other parts of the parallelized and distributed application code processing system 100 shown in FIG. 12 have the same configuration as the parallelized and distributed application code processing system 100 shown in FIG.

[0054] The parallel code block performance requirement database 24 stores the performance requirements of the parallel code blocks measured by the parallel code block performance requirement definition unit 21 . The remote cluster performance database 25 stores the performance and processing time prediction values ​​of the remote clusters 5 and 6 measured and acquired by the remote cluster performance measurement unit 22 .

[0055] Here, the reason why the application code parallelization and distribution processing system 100 of this embodiment is provided with the parallel code block performance requirement database 24 and the remote cluster performance database 25 will be explained. The application program 42 and the parallel processing cluster run on a Kubernetes cluster, which is a collection of nodes that executes, for example, any application program. The application program 42 receives the analysis data 7 as input data and performs large-scale data processing, and the application code 42 that describes the application program 42 is written in, for example, Python code.

[0056] The clusters that perform parallel processing are Ray clusters, which are computational resources consisting of multiple instances. However, not all clusters have the same performance, and the performance of the constituent instances differs for each cluster.

[0057] A Ray cluster consists of at least one head node and multiple worker nodes. For example, as explained in Figure 2, each remote cluster 5, 6 has worker nodes 52, 62 located under head nodes 51, 61, and the head nodes 51, 61 distribute and process tasks received from outside to the worker nodes 52, 62.

[0058] The local cluster 43 is a Ray cluster that is located in the same computer system 4 as the device (server) that executes the application. The local cluster 43 processes the received task using the processor and memory resources on that computer. On the other hand, the remote clusters 5 and 6 are computational resources independent of the computer system 4 on which the application server runs. For this reason, the parallel code block performance requirement database 34 and the remote cluster performance database 25 monitor the execution of tasks in the Ray cluster and accumulate logs and the like.

[0059] [Example of an overview of a parallelized and distributed processing system for application code] FIG. 13 shows an example of the outline of a parallelized and distributed processing system 100 for application code according to this embodiment. The processing example in Fig. 13 differs from the processing example in the first embodiment shown in Fig. 4 in that the cluster selector 23 selects an appropriate cluster based on the storage process in the parallel code block performance requirement database 24 and the storage process in the remote cluster performance database 25, and on the data stored in these databases 24 and 25. The rest of the processing is the same as the cluster selection processing shown in Fig. 4.

[0060] The parallel code block performance requirement database 24 stores the performance requirements of the parallel code blocks measured by the parallel code block performance requirement definition unit 21. Examples of the performance requirements stored in the parallel code block performance requirement database 24 will be described later with reference to FIG. The remote cluster performance database 25 stores the performance of the remote clusters 5 and 6 measured by the remote cluster performance measurement unit 22. Examples of performance requirements stored in the remote cluster performance database 25 will be described later with reference to FIG.

[0061] The distribution unit 2 acquires workflow logs and metrics during execution of parallel code blocks executed in the cluster, and when the size of input data to the parallel code block is changed or the cluster environment is changed, the cluster selection unit 23 performs the cluster selection process again to reselect an appropriate cluster. At this time, the cluster selection unit 23 uses the data stored in the parallel code block performance requirement database 24 and the data stored in the remote cluster performance database 25 to reselect an appropriate cluster, depending on the conditions, without re-executing measurements of performance requirements, etc.

[0062] [Cluster selection process procedure] FIG. 14 is a flowchart showing the procedure of the process performed by the distribution unit 2 to select a cluster of parallelized code. In the flowchart shown in FIG. 14, the processing shown in steps S901 to S906 indicates the same processing content as steps S901 to S906 shown in FIG. First, the distribution unit 2 acquires a parallel code block, input data (for one iteration) that is data to be processed by the parallel code, and the number of loops (step S901). Then, the distribution unit 2 determines whether the contents of the parallel code block have changed since the previous cluster selection (step S1401).

[0063] In step S1401, if the contents of the parallel code block have not changed since the last time a cluster was selected (Yes in step S1401), the distribution unit 2 determines whether the size of the input data (one batch) is the same as when it was previously processed (selected in the past) (step S1402). In step S1402, if the size of the input data (for one run) is the same as when it was previously processed (Yes in step S1402), the distribution unit 2 determines whether the number of loops of the parallel code block is the same as when it was previously processed (selected in the past) (step S1403).

[0064] In step S1403, if the number of loops of the parallel code block is the same as when it was previously processed (Yes in step S1403), the cluster selection unit 23 obtains an appropriate cluster from the stored data in the performance requirement database 24 for the parallel code block, and obtains the cluster selection result (step S1404). After obtaining the cluster selection results from the data stored in the performance requirement database 24 in step S1404, the cluster selector 23 proceeds to the process of step S906 and outputs the cluster selection results.

[0065] Furthermore, in step S1401, if the contents of the parallel code block have changed since the previous cluster selection (No in step S1401), or in step S1402, if the size of the input data (for one run) is different from that in the previous processing (No in step S1402), the distribution unit 2 proceeds to the processing of step S902. Note that the processing also proceeds to this step S902 when a cluster is selected for the first time. In step S902, the parallel code block performance requirement definition unit 21 of the distribution unit 2 executes one processing of the parallel code block in the local cluster 43 and measures the performance at that time.

[0066] After measuring the performance in step S902, the distribution unit 2 stores the measurement value obtained in step S902 in the performance requirement database 24 of the parallel code block (step S1405), and proceeds to the processing of step S903. In step S903, the remote cluster performance measurement unit 22 executes one processing of the parallel code block in each of the remote clusters 5 and 6, and measures the performance at that time. After measuring the performance in step S903, the distribution unit 2 stores the measurement value obtained in step S902 in the remote cluster performance database 25 (step S1406), and proceeds to the processing of step S904.

[0067] In step S904, the cluster selection unit 23 of the distribution unit 2 acquires the measurement results from steps S902 and S903, and calculates the processing time, which is the total time obtained by adding the latency to the processing time obtained from the measurement results, as a predicted value of the processing time in the remote cluster. Thereafter, the process proceeds to step S905, where the cluster selection unit 23 determines the cluster (appropriate cluster) that processes the parallel code block based on the processing time obtained in step S904, with the shortest total time. After determining the cluster in step S905, the distribution unit 2 stores the resulting cluster in the appropriate cluster section of the parallel code block performance requirement database (step S1407), and then proceeds to processing in step S906, where the cluster selection unit 23 outputs the cluster selection result.

[0068] [Example of a remote cluster performance database] FIG. 15 shows an example of the configuration of the remote cluster performance database 25. As shown in FIG. As shown in FIG. 15, the remote cluster performance database 25 includes a cluster ID 251, a cluster address 252, a maximum number of nodes in the cluster 253, memory information 254 of the cluster, and processor information 255 of the cluster.

[0069] The cluster ID 251 is an ID that identifies each remote cluster. Address 252 is an address on the network of remote clusters 5, 6, etc. The maximum number of nodes, 253, is the maximum number of worker nodes, 52, 62, etc., in remote clusters 5 and 6. The memory information 254 is performance information such as the transfer speed, clock frequency, and storage capacity of the memory of each of the remote clusters 5, 6, etc. The processor information 255 is performance information such as the clock frequency, the number of cores, and the number of threads.

[0070] [Example of a performance requirements database for parallel code blocks] FIG. 16 shows an example of the configuration of the performance requirement database 34 for parallel code blocks. As shown in FIG. 16, the performance requirement database 34 for the parallel code block has data on a data size 341, a loop count 342, a performance requirement 343, a processing time 344, and an appropriate cluster 345. These data 341 to 345 are configured as tables 24-1, 24-i, and 24-n (i and n are each an arbitrary integer) for each code block (function name).

[0071] [Effects of the second embodiment] As described above, according to this embodiment, similar to the first embodiment, it is possible to automatically create appropriately parallelized code by analyzing application code, and it is also possible to measure the performance of the prepared clusters and, based on the results, execute a part of the parallelized code in a cluster among the prepared clusters that has the shortest processing time. In particular, by sequentially measuring the performance of remote clusters and local clusters for a program to be processed in parallel, dynamic performance can be determined, making it possible to select an appropriate cluster according to the content of the program to be processed in parallel. In this embodiment, workflow logs and metrics are acquired during execution of parallel code blocks executed in the cluster. When the input data size is changed or the cluster environment is changed, the cluster selection unit 23 performs the cluster selection process again to reselect an appropriate cluster. This allows for dynamic cluster reselection, ensuring that parallel processing is always performed on an appropriate cluster for processing the program.

[0072] Furthermore, when the cluster selection unit 23 reselects an appropriate cluster, if there are no changes to the contents of the parallel code block, the size of the input data is the same as when the cluster was previously selected, and the number of loops is the same as when the cluster was previously selected, the cluster selection unit 23 can reselect from the performance requirements stored in the performance requirements database 24 of the parallel code block, thereby making it possible to omit the remeasurement process. However, reselection from the performance requirements stored in the parallel code block performance requirement database 24 is just one example, and performance may also be measured during reselection. By measuring the performance requirements again without using the performance requirements stored in the parallel code block performance requirements database 24, the parallel code block performance requirements database 24 becomes unnecessary, and the system configuration can be simplified accordingly.

[0073] <Modification> The embodiment described above has been described in detail to clearly explain the present invention, and is not necessarily limited to an embodiment having all of the configurations described. For example, in the configuration diagrams shown in Figures 1 and 12, only control lines and information lines that are considered necessary for explanation are shown, and not all control lines and information lines in the product are necessarily shown. In reality, it can be assumed that almost all components are interconnected. Furthermore, the processing flows shown in the flowcharts of Figures 8, 9, 10, and 14 are also examples, and as long as the processing results are the same, the order of some of the processing may be changed or multiple processes may be executed simultaneously.

[0074] In addition, in each of the above-described embodiments, the computer system is provided with the processing configuration shown in Fig. 1 or 12, but an existing computer system may be equipped with a program for realizing the processing configuration shown in Fig. 1 or 12 to perform similar processing. In this case, the program may be prepared in a storage device within the computer system, or may be stored and transferred on a recording medium such as an external memory, an IC card, an SD card, or an optical disk. [Explanation of symbols]

[0075] 1...Parallelization section 2...Dispersion section 3...Storage device 4. Computer Systems 5...First remote cluster 6...Second remote cluster 7. Analysis data 8. Network 11...Parallelization pattern detection unit 12...Parallel code generation and rewriting part 21...Parallel code block performance requirement definition section 22...Remote cluster performance measurement unit 23...Cluster selection section 24...Performance requirements database for parallel code blocks 25...Remote cluster performance database 31...Parallel code block database 32...Database for input data 41...Application Code 42...Application Program 43...Local cluster 51...Head node 52...Worker node 61...head node 62...Worker node 100...Parallelization and distributed processing system for application code 101...Cluster group 300...Computer Systems 301...Processor 302...Memory 303...Storage device 304...Input device 305...Output device 306...Communication interface 307...Internal bus 421...Cluster selection process call unit 431...head node 432...Worker node

Claims

1. a parallelization unit that extracts parallel parts from the code of an application program and generates parallel code; a cluster selection process calling unit that calls a process for selecting an optimal cluster for a parallel code block, which is a program code to be processed in parallel within the application code generated by the parallelization unit; a parallel code block performance requirement definition unit that measures performance requirements of the parallel code block; a cluster performance measurement unit that measures dynamic performance when the parallel code is executed in a cluster that is a target for executing the parallel code; a cluster selection unit that selects a cluster that satisfies the performance requirements measured by the performance requirement definition unit based on the measurement result by the cluster performance measurement unit. A parallel and distributed processing system for application code.

2. As the clusters, a local cluster configured on a computer that executes the application program and one or more remote clusters connected to the computer so as to be able to transfer data are provided, The cluster performance measurement unit measures the performance of the remote cluster.

2. The parallel and distributed processing system for application code according to claim 1.

3. The cluster selection unit includes, as an option when selecting a cluster, a situation in which the remote cluster is not selected.

3. The parallel and distributed processing system for application code according to claim 2.

4. When measuring the dynamic performance of the local cluster and the remote cluster, the cluster performance measurement unit measures the processing time when executing one loop of the parallel code, and obtains the total processing time of the process of the parallel code from the measured processing time for one loop and the number of loops.

4. The parallel and distributed processing system for application code according to claim 3.

5. Furthermore, the cluster performance measurement unit calculates latency from a data transfer rate between the cluster and the remote cluster and an amount of data to be transferred, The cluster selection unit selects a cluster that has the shortest total of the calculated latency and the total processing time of the process.

5. The parallel and distributed processing system for application code according to claim 4.

6. The data to be executed by the parallel code on the local cluster and the remote cluster is analysis data as sample data prepared in advance.

6. The parallel and distributed processing system for application code according to claim 5.

7. The cluster selection unit acquires workflow logs and metrics during execution of parallel code blocks executed in a cluster, and reselects an appropriate cluster when the input data size or cluster environment is changed.

3. The parallel and distributed processing system for application code according to claim 2.

8. a parallel code block performance requirement database that stores the performance requirements of the parallel code measured by the parallel code block performance requirement definition unit; a remote cluster performance database that stores the performance of the remote cluster measured by the cluster performance measurement unit; When the cluster selection unit reselects an appropriate cluster, if there is no change in the content of the parallel code, the size of the input data is the same as when the cluster was previously selected, and the number of loops is the same as when the cluster was previously selected, the cluster selection unit reselects an appropriate cluster from the performance requirements stored in the performance requirement database of the parallel code block.

8. The parallel and distributed processing system for application code according to claim 7.

Images