Management computer, management computer system, management computer program, and management computer method

Abstract

To perform appropriate processing parallelization in performing unsteady processing in a management calculator by using an on-demand cloud resource that is billed based on quantity.SOLUTION: A management calculator performs table parallelization processing of assigning a plurality of tables to a core of a processing execution calculator to parallelize data processing table by table, and performs record parallelization processing of, when the table has a size larger than a predetermined data size, dividing the table with a large data size into a plurality of records, and assigning the plurality of records to the core of the processing execution calculator to parallelize the data processing record by record.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a management computer, a management computing system, a management computing program, and a management computing method.

Background Art

[0002] Conventionally, dedicated servers, storage, networks, and other devices called on-premises were purchased to operate an application system. However, in recent years, cloud-based systems that enable on-demand use of IT resources such as servers, storage, and networks have emerged, enabling rapid and flexible procurement of IT resources. As a result, for example, non-steady processing such as batch processing, data backup, and analysis processing has increasingly been processed using cloud resources that are charged on a per-use basis on a demand basis.

[0003] In information processing on on-demand cloud resources, there are cases where a job that handles a plurality of tasks having no dependency on the execution order is executed. In many cases, each of these tasks is processed in parallel in any order for each virtual core of a virtual server that is a cloud resource.

[0004] As a technique for scheduling a plurality of types of tasks having different processing times, for example, there is Patent Document 1. Patent Document 1 describes a technique for determining devices or cloud services to be used, such as servers and storage, and making estimates of performance, availability, etc. according to business requirements.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] By using Patent Document 1, processing units obtained by dividing tasks can be processed efficiently. However, the technology in Patent Document 1 does not allow for a reduction in processing time through parallelization while considering the overhead of increased processing time caused by task division and the scope of the load on the system, and parallelization decisions must be made by an expert.

[0007] The objective of this invention is to enable proper parallel processing when non-routine processing is performed on an on-demand, pay-per-use cloud resource in a management computer. [Means for solving the problem]

[0008] A management computer in one aspect of the present invention is a management computer that performs data processing on data to be processed having multiple tables using a processing execution computer on a cloud service, and comprises a processor and an input / output device, wherein the processor generates a processing plan generation processing unit that uses data information input via the input / output device to generate a processing plan to be executed using on-demand, pay-per-use cloud resources and displays the generated processing plan on the input / output device, and the processor manages the execution of the processing execution computer so that the data processing is executed according to the processing plan selected via the input / output device, The processing plan generation processing unit is characterized in that it assigns a plurality of tables to the cores of the processing execution computer, parallelizes the data processing on a table-by-table basis to process tasks on a table-by-table basis, and performs table parallelization processing in which the cores of the processing execution computer each execute the tasks, and if the table is larger than a predetermined data size, it divides the table with a large data size into a plurality of records, assigns the plurality of records to the cores of the processing execution computer, parallelizes the data processing on a record-by-record basis to process tasks on a record-by-record basis, and performs record parallelization processing in which the cores of the processing execution computer each execute the tasks. [Effects of the Invention]

[0009] According to one aspect of the present invention, a management computer can perform appropriate parallel processing when processing non-routine processes using on-demand, pay-per-use cloud resources. [Brief explanation of the drawing]

[0010] [Figure 1] This is a block diagram showing an example configuration of the computer system according to Embodiment 1 of the present invention. [Figure 2] This figure shows an example of an input screen. [Figure 3] This figure shows an example of a processing plan display screen. [Figure 4] This diagram shows a flowchart illustrating an example of the procedure for generating a processing plan. [Figure 5] This figure shows an example of evaluation value table information. [Figure 6] This diagram shows a flowchart illustrating an example of the procedure for determining whether to apply partitioning and parallelization. [Figure 7] This diagram shows a flowchart illustrating an example of the procedure for assigning a processing server. [Figure 8] This is a conceptual diagram illustrating an example of the execution of the processing server allocation process. [Figure 9] This diagram shows a flowchart illustrating an example of the procedure for determining the integration of processing servers. [Figure 10] This is a conceptual diagram illustrating the first application example of the processing plan generation process. [Figure 11] This is a conceptual diagram illustrating a second application example of the processing plan generation process. [Figure 12] This is a conceptual diagram illustrating an example of bottleneck detection processing. [Figure 13] This figure shows an example of intermediate data in the processing server integration decision processing flow. [Figure 14] This diagram shows the relationship between table parallelism and record parallelism. [Figure 15] This is a block diagram showing an example configuration of a computer system according to Embodiment 2 of the present invention.

Best Mode for Carrying Out the Invention

[0011] In the following description, a "memory" is one or more memory devices, and typically may be a main memory device. At least one of the memory devices in the memory may be a volatile memory device or a non-volatile memory device.

[0012] Also, in the following description, a "persistent storage device" is one or more persistent storage devices. A persistent storage device is typically a non-volatile storage device (e.g., an auxiliary storage device), and specifically, for example, an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

[0013] Also, in the following description, a "storage device" may be either the above "memory" or the above "persistent storage device".

[0014] Also, in the following description, a "processor" is one or more processor devices. At least one of the processor devices is typically a microprocessor device such as a CPU (Central Processing Unit), but may also be another type of processor device such as a GPU (Graphics Processing Unit). Also, at least one of the processor devices may be single-core or multi-core. Also, at least one of the processor devices may be a processor core. Also, at least one of the processor devices may be a hardware circuit (e.g., an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit)) that performs part or all of the processing, which is a processor device in a broad sense.

[0015] In the following description, the information from which an output is obtained for an input may be described by an expression such as "xxx table", but the information may be data of any structure or a learning model such as a neural network that generates an output for an input. Therefore, "xxx table" can be rephrased as "xxx information". Also, in the following description, the configuration of each table is an example, and one table may be divided into two or more tables, or all or part of two or more tables may be one table, or it may include some data fields not shown in the figures.

[0016] In the following description, the processing may be described with "program" as the subject. However, since the program is executed by a processor to perform the defined processing while appropriately using a storage device and / or an interface device, etc., the subject of the processing may be the processor (or a device such as a controller having that processor). The program may be installed from a program source into a device such as a computer. The program source may be, for example, a program distribution server or a computer-readable (e.g., non-transitory) recording medium. Also, in the following description, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0017] Furthermore, in the following explanation, functions may be described using expressions such as "xxx section," but such functions may be implemented by the execution of one or more computer programs by a processor, or by one or more hardware circuits (e.g., FPGA or ASIC). When a function is implemented by the execution of a program by a processor, the defined processing is carried out using memory and / or interface devices as appropriate, so such functions may be at least a part of the processor. Also, processing described with a function as the subject may be processing performed by the processor or a device having that processor. In addition, programs may be installed from program source. Program source may be, for example, a program distribution computer or a computer-readable recording medium (e.g., a non-temporary recording medium). The description of each function is an example, and multiple functions may be combined into one function, or one function may be divided into multiple functions. Furthermore, in the following explanation, a "computer system" is a system that includes one or more physical computers. These physical computers may be general-purpose computers or dedicated computers.

[0018] Furthermore, the control lines and information lines shown are those deemed necessary for explanation purposes and do not necessarily represent all control lines and information lines required for implementation. In reality, it can be assumed that almost all components are interconnected.

[0019] Hereafter, a set of one or more computers that manages the information processing system and displays the display information in this embodiment may be referred to as a management system. When a management computer (hereinafter referred to as the management computer) displays the display information, the management computer is the management system, and a combination of a management computer and a display computer is also a management system. Furthermore, to speed up and improve the reliability of management processing, multiple computers may perform the same processing as the management computer, in which case the multiple computers (including the display computer if the display computer performs the display) constitute the management system.

[0020] It should be noted that the present invention is not limited to the embodiments described above, but includes various modifications and equivalent configurations within the spirit of the attached claims. For example, the embodiments described above are described in detail for the purpose of clearly illustrating the present invention, and the present invention is not necessarily limited to having all the described configurations. Furthermore, some of the configurations of one embodiment may be replaced with those of another embodiment. Furthermore, configurations of other embodiments may be added to the configuration of one embodiment. Furthermore, some of the configurations of each embodiment may be added, deleted, or replaced with those of other embodiments.

[0021] Hereinafter, an embodiment 1 of the present invention will be described with reference to the drawings. [Examples]

[0022] Figure 1 is a block diagram showing an example configuration of a computer system according to Embodiment 1 of the present invention.

[0023] The computer system of Example 1 illustrates a case in which the processor 111 of the management computer 101 performs task scheduling for a migration process execution computer 135, which is an on-demand, pay-per-use cloud resource on the cloud service 103, to migrate the data to be migrated 122 stored in the storage device 121 of the source computer 102 to the storage device 132 of the destination computer 131.

[0024] The computer system of Example 1 is a system in which a management computer 101, a source computer 102 for data transfer processing, a destination computer 131 on the cloud service 103, and a migration processing execution computer 135 on the cloud service 103 can be interconnected via a network 104.

[0025] The management computer 101 includes a processor 111, a storage device 112, and an input / output device 113. The input / output device 113 does not necessarily have to be provided by the management computer 101, and may be connected via the network 104. Here, the input / output device 113 refers to, for example, a touch panel, a tablet terminal, a display, a keyboard, a mouse, etc.

[0026] The processor 111 performs a processing plan generation process 114 and a processing plan execution management process 115 by loading the program into the storage device 112 and executing it. Here, the processing plan generation process 114 is called the "processing plan generation processing unit," and the processing plan execution management process 115 is called the "processing plan execution management processing unit."

[0027] Furthermore, the storage device 112 stores data corresponding to the processing plan generation process 114 and the processing plan execution management process 115, as well as an evaluation value table 116. The processes and information stored in the storage device 112 may be stored in different storage devices, or they may be stored in storage devices (not shown) connected via the network 103.

[0028] The processing plan generation process 114 calculates a processing plan for data transfer processing to be executed using on-demand, pay-per-use cloud resources, based on the source data connection information and destination data connection information input via the input / output device 113, and displays the calculation result via the input / output device 113.

[0029] The processing plan execution management process 115 is a process that performs execution operations on the migration processing execution computer 135 via the network 103 so that data transfer processing is executed according to the processing plan selected via the input / output device 141.

[0030] Network 104 is a communication path connected by wire or wireless means. For example, it could be a wired LAN cable or a wireless Wi-Fi (registered trademark), but is not limited to these.

[0031] Cloud service 103 includes network devices, server devices, storage devices, etc., and provides a wide variety of IT services that meet the specified requirements, such as virtual servers and database services. For example, it may use services provided by a company that provides cloud services as a business, or it may be a computing system owned independently by an individual or company.

[0032] Figure 2 shows an example of a condition input screen for setting the source data connection information and destination data connection information to be input to the processing plan generation process 114 in Embodiment 1 of the present invention.

[0033] The condition input screen 200 includes a source data connection information field 211 for receiving connection information for source data, a destination data connection information field 212 for receiving connection information for destination data, an advanced settings button 221 for calling up an advanced settings screen (not shown), and a plan display button 222 for instructing the start of the processing plan generation process.

[0034] The detailed settings screen (not shown) here refers to a screen for entering other detailed information that cannot be entered in the source data connection information 211 and destination data connection information 212, such as the username and password for logging in, as well as constraints for generating the processing plan, such as the target processing time to complete the process and the upper limit of available cloud resources. If the detailed settings screen (not shown) is already displayed in the condition input screen 200, the condition input screen 200 does not need to include the detailed settings button 221.

[0035] Figure 3 shows an example of a processing plan display screen output by the processing plan generation process 114 in Embodiment 1 of the present invention.

[0036] The processing plan display screen 300 includes a condition setting display area 301 for entering the conditions of the processing plan, a processing plan display area 302 for displaying information about the processing plan, a recalculation button 303 for executing a process to generate a processing plan again based on the conditions entered in the condition setting display area 301, and a data migration start button 304 for instructing the start of the data migration process based on the calculated processing plan. The condition setting display area 301 includes a target time field 311 for entering the target time for completing the process, and a bandwidth limit field 312 for entering the limit of cloud resources available for the process.

[0037] The condition setting display area 301 may include fields not shown for inputting processing plan generation conditions, in addition to the target time field 311 and the bandwidth limit field 312. If the process of regenerating the processing plan by inputting processing conditions from the user is not executed, the processing plan display screen 300 does not need to include the condition setting display area 301 and the recalculation button 303.

[0038] The processing plan display area 302 includes a graph area 321 that displays the processing schedule graphically on the axes of processing time and processing parallelism, a processing time field 322 that displays the estimated processing time, a processing fee field 323 that displays the on-demand additional resource charges required for the data migration process, and an estimated load field 324 that displays the estimated load that the data migration process places on the system.

[0039] The processing plan display screen 300 does not necessarily have to include the graph area 321, and may include information fields not shown that display other processing plan information. The processing plan display screen 300 does not have to include the data migration start button 304, for example, if the user has been given prior permission to execute the data migration process and does not need to input instructions to execute the data migration.

[0040] Figure 4 is a flowchart showing an example of the procedure for the processing plan generation process 114 of the management computer 101.

[0041] The processing plan generation process 114 illustrated in this flowchart is executed by an instruction from the plan display button 222 displayed on the condition input screen 222, or by an instruction from the data migration start button 304 displayed on the processing plan display screen 300. Alternatively, it may be executed by an instruction from some program.

[0042] In Figure 4, the management computer 101 executes the following processes: data acquisition process (S401), processing time calculation process for each task (S402), resource limit acquisition process (S403), theoretical shortest processing time calculation process (S404), partition parallel application determination process (S405), processing server integration determination process (S406), and data processing job creation process (S407). The processing plan generation process flow 400 may include other processing steps not shown. Also, the execution order of some processes may be changed or executed in parallel as long as no discrepancies occur in input / output. If the processing plan generation process flow 400 is executed multiple times, some processing steps may have their values ​​from previous executions stored in advance and output the same values ​​without being re-executed.

[0043] In the processing plan generation flow 400, during the data acquisition process (S401) for data to be processed, the management computer 101 acquires information about the data to be migrated 122 stored in the storage assumption 121 on the source computer 102, based on the information entered on the condition input screen 200. This information includes, for example, the amount of data in each table stored in the database.

[0044] In the processing time calculation process (S402) for each task in the processing plan generation process flow 400, the management computer 101 uses the information of the data to be migrated 122 obtained in the data acquisition process (S401) and the information of the evaluation value table 116 to calculate the processing time for each data to be migrated.

[0045] For example, if there are 10 tables to be migrated, the process time for each table will be calculated, such as 20 hours for one table and 1 hour for another.

[0046] In the resource limit acquisition process (S403) for each task in the processing plan generation process flow 400, the specification information of the source computer 102, the destination computer 131, and the network 104 is acquired as information about the processing execution environment. The bottleneck of the processing is identified, and the degree of parallelism of the available processing is calculated using the specification value of that location and the information in the evaluation value table 500.

[0047] In the resource limit acquisition process (S403), the system compares the upper limit value specified in the detailed settings screen (not shown) on the condition input screen 200, or in the bandwidth limit field 312 on the processing plan display screen 300, with the location of the previous bottleneck, and prioritizes outputting the upper limit of the processing parallelism calculated from the smaller value. For example, if the value specified in the bandwidth limit field 312 is lower than the data transfer bandwidth available on the network 104, which had the lowest specifications among the values ​​acquired as information on the processing execution environment, the system outputs the processing parallelism calculated from the value specified in the bandwidth limit field 312.

[0048] The parallelism of the processing output by the resource limit acquisition process (S403) is a value that takes into account the number of cores of the instance available as the processing execution computer 135, as shown in the support size field 505 of the evaluation value table 500.

[0049] For example, if the minimum value of the support size field 505 is 2, and the degree of parallelism of the processing calculated by the resource limit acquisition process (S403) is 11, then it is not possible to achieve 11 regardless of the combination of instances. In this case, the resource limit acquisition process (S403) outputs 10, which is the maximum number of cores that can be achieved by combining instances and is less than or equal to the upper limit of the degree of parallelism of processing, which is 11.

[0050] In the processing plan generation flow 400, the theoretical shortest processing time calculation process (S404) for each task calculates the processing time required when the data to be processed is parallelized to the maximum extent possible, assuming there is no overhead when processing tasks in parallel, and within the upper limit of the degree of parallelism of processing determined in the resource limit acquisition process (S403).

[0051] For example, if the total migration time of the tables to be migrated, calculated by the processing time calculation process for each task (S402), is 100 hours, and the degree of parallelism of the processes, calculated by the resource limit acquisition process (S403), is 10, then the theoretical minimum processing time can be calculated as 100 ÷ 10 = 10, which is 10 hours.

[0052] However, as illustrated in Figure 11, 1101, there may be some tables that are too large to be processed using record parallelization. If there are tables whose processing time is longer than the theoretical shortest processing time calculated by the shortest processing time calculation process (S404), the processing time of the largest table to which record parallelization cannot be applied is defined as the theoretically fastest transition time.

[0053] Record parallelization will be explained later. In the theoretical shortest processing time calculation process (S404), for example, if the target time for data processing time is entered from the target time field 311 of the plan display screen 300 displayed on the input / output device 113, and the aforementioned target time for data processing time is greater than the theoretical shortest processing time calculated by the process described above, the aforementioned target time for data processing time may be used instead of the value of the theoretical shortest processing time.

[0054] In the processing plan generation flow 400, the task division parallel application determination process (S405) calculates a task schedule that combines two parallelization methods to shorten processing time. The details of the task division parallel application determination process (S405) are explained using Figure 6.

[0055] In the processing server integration determination process (S406) for each task in the processing plan generation process flow 400, a task schedule is calculated that shortens the total operating time of the migration processing execution computer 135 used in the data transfer process. Details of the processing server integration determination process (S406) are explained using Figure 9.

[0056] The data processing job creation process (S407) in the processing plan generation process flow 400 calculates the operation settings necessary to operate the processing execution computer 135 according to the task schedule calculated in the processing server integration determination process (S406). The operation settings can be in a configuration file or in the form of commands to operate the API.

[0057] Figure 5 is an explanatory diagram illustrating the information held by the evaluation value table 116.

[0058] The evaluation value table 500 includes a processing classification field 501, a tool name field 502, a time coefficient field 503, a bandwidth coefficient field 504, a support size field 505, and an overhead calculation formula field 506. Some of the data fields in the evaluation value table 500 may be absent, and it may also include several data fields that are not shown.

[0059] The processing classification field 501 and the tool name field 502 are identifier information used to determine which column of data should be retrieved when acquiring information from the evaluation value table 500 in the processing plan generation processing flow 400. In this embodiment, these fields may be specified, for example, by a configuration file (not shown) or by a configuration screen (not shown).

[0060] The time coefficient field 503 is a coefficient used in the processing plan generation processing flow 400 to calculate the processing time for each task. For example, in the processing time calculation process (S402), the management server 101 calculates the processing time for the data table to be migrated by multiplying the amount of data in the data table to be migrated, obtained in the data acquisition process (S401), by the value of the time coefficient field 503 in the evaluation value table 500.

[0061] The bandwidth coefficient field 504 is a coefficient used in the processing plan generation processing flow 400 to calculate the degree of parallelism of the processing of the migration processing execution computer 135. For example, in the resource limit acquisition process (S403), the management server 101 calculates the degree of parallelism of processing by dividing the bandwidth performance information of the storage device 132 acquired in the resource limit acquisition process (S403) by the value of the bandwidth coefficient field 504 in the evaluation value table 500 and truncating the decimal part of the result.

[0062] The support size field 505 is a numerical value that represents a candidate for the degree of parallelism of partitioned parallelism to be applied to large tasks that become bottlenecks in reducing migration time in the processing plan generation processing flow 400. These values ​​are derived from the number of instance cores supported by the cloud resources available as the migration processing execution computer 135, and are used as constraints for applying partitioned parallelism without incurring core latency on the migration processing computer 135 that executes partitioned parallelism.

[0063] The overhead calculation formula field 506 is a formula for calculating the overhead of increased processing time that occurs when parallelization of processing is applied in the processing plan generation processing flow 400. In this embodiment, if table parallelization is used, where the migration processing of the data table to be migrated is allocated to each core of the processor 136 of the migration processing execution computer 135 and parallelized, no overhead of increased processing time due to parallelization occurs. However, if record parallelization is used, where the data table to be migrated is divided into multiple records and processed in parallel, overhead of increased processing time due to parallelization occurs.

[0064] For example, if a table takes 10 hours to migrate before record parallelization, and the table is split into four parts and record parallelization is applied, without overhead, 10 ÷ 4 = 2.5, meaning each of the four cores would process for 2.5 hours to complete the migration. However, in reality, extra processing occurs to split and transfer the table, so it might end up taking 3.5 hours.

[0065] In this example, the overhead is 3.5 hours - 2.5 hours = 1.0 hours. The overhead calculation formula field 506 stores the formula used to calculate this overhead time. For example, if the calculation formula field 506 records the transition time before parallelization × overhead coefficient α (0.1), then when applying record parallelization to a table where the transition time before applying record parallelization was 10 hours, an overhead of 1.0 hours will occur, regardless of the degree of parallelism of the record parallelization, calculated as 10 × 0.1 = 1.0.

[0066] The processing time for record parallelization can be calculated from the transition time before parallelization ÷ degree of parallelism + overhead. Therefore, if this table is parallelized by 2 using record parallelization, the transition time will be 10 ÷ 2 + 1.0 = 6.0 hours, and if this table is parallelized by 10 using record parallelization, the transition time will be 10 ÷ 10 + 1.0 = 2.0 hours.

[0067] Figure 6 is a flowchart showing a detailed example of the procedure for the division parallel application determination process (S405) for each task in the processing plan generation process flow 400.

[0068] In Figure 6, the management computer 101 executes the following processes: data selection process for applying partitioned parallelization (S601), parallelism determination process (S602), task parallelization process (S603), bottleneck determination process (S604), recalculation process (S605), and processing server allocation process (S6060). The partitioned parallelization application determination process flow 600 may include other processing steps not shown, and some of these steps may not be executed.

[0069] The purpose of the partitioned parallelism application determination processing flow 600 is to calculate a task schedule that appropriately combines partitioned parallelism and task parallelism, taking into account the constraints of the migration processing execution computer 135, in order to shorten the processing time of the data transfer process.

[0070] In database data transfer, data is typically transferred from multiple tables, but the data transfer process is parallelized on a table-by-table basis. In other words, each table is processed as a task, and the cores of the processing server execute each task. This is called task parallelism, but here we will refer to it as table parallelism.

[0071] When dealing with extremely large data tables, even with table parallelization, the transfer time of these large data tables becomes a bottleneck, preventing a reduction in the overall data transfer time. In such cases, the records that make up the large data table can be divided and transferred in parallel. This is called partitioning parallelization, and here we will refer to it as record parallelization.

[0072] Now, referring to Figure 14, we will explain the relationship between table parallelism and record parallelism.

[0073] (a) shows the case where the table size is smaller than a predetermined value. In this case, cores are allocated on a per-table basis to perform table parallel processing.

[0074] (b) shows the case where the table size is larger than a predetermined value. In this case, the table is divided into record units, and each record is allocated to a core for record parallel processing.

[0075] As explained in the overhead calculation formula for field 506 in Figure 5, record parallelization involves a time overhead. This overhead increases processing time, which in turn increases the operating time of the migration processing execution computer, leading to increased costs. Therefore, we want to avoid applying record parallelization unnecessarily.

[0076] Note that while table parallelism can be executed across multiple migration execution computers 135, record parallelism must be executed on a single migration execution computer 135. To prevent surplus core waiting time when the migration execution computer 135 executes the record parallel transfer process, the number of cores available to the instance of the migration execution computer 135 is set as the degree of parallelism for the record parallel transfer process.

[0077] In the partitioned parallelization application determination processing flow 600, in the partitioned parallelization application data selection processing (S601), the management computer 101 selects the tables to which record parallelization will be applied from the group of tables to be migrated.

[0078] Among the tables to be migrated, those whose transfer processing time is greater than the theoretical shortest processing time calculated in the processing plan generation flow 400's theoretical shortest processing time calculation process (S404) are selected as targets for record parallelization.

[0079] For example, Figure 10 illustrates the case of parallelizing 13 tables within a range of 12 parallelism. The vertical axis represents processing time, and the horizontal axis represents the degree of parallelism. Each box has a height corresponding to the processing time per task. Since the horizontal axis represents the degree of parallelism, it can be seen as the processing assigned to each core. For example, the leftmost core processes 10 tables, the second core from the left processes 9 tables, and the third core from the right processes 0.5 tables first, then another 0.5 tables, and so on.

[0080] Figure 1001 illustrates how the tables to be migrated are arranged in 12 parallel processes based on a greedy algorithm, sorted by processing time from longest to shortest. Because the arrangement is based on a greedy algorithm, any tables that exceed the parallel processing limit will be placed in the slot with the shortest processing time.

[0081] The total processing time for the table in Figure 1001 is 10 + 9 + 8 + 2 + 1.5 + 1 + 1 + 1 + 1 + 0.5 + 0.5 + 0.5 + 0.5 = 36.5, and the theoretically fastest transition time when 12 parallel processes are used is 36.5 ÷ 12 ≈ 3.

[0082] In Figure 10, at point 1001, three tables with processing times of 10, 9, and 8 are selected to be subjected to record parallelization. In the case illustrated in Figure 11, at point 1101, the processing time of the table with a processing time of 8, to which record parallelization cannot be applied, theoretically results in the fastest transition time. Therefore, the tables with processing times of 10 and 9 are selected to be subjected to record parallelization.

[0083] The parallelism determination process (S602) in the divided parallelism application determination process flow 600 determines the degree of parallelism when applying record parallelism. The upper limit of the degree of parallelism determined in the parallelism determination process (S602) is the value of the processing parallelism calculated in the resource limit acquisition process (S403), and the values ​​listed in the support size field 505 of the evaluation value table 500 are used as candidates.

[0084] For example, if the upper limit of parallelism is 12 and the values ​​of the support size field 505 are 2, 4, 8, 16, 32, 64, and 128, then the parallelism will be determined using 2, 4, and 8 as candidates. The parallelism determined by the parallelism determination process (S602) is the smallest parallelism that results in a processing time when record parallelism is applied that is lower than the theoretical shortest processing time calculated in the theoretical shortest processing time calculation process (S404) for each task.

[0085] If the maximum possible degree of parallelism does not result in a processing time lower than the theoretically shortest processing time, then that value should be treated as the value that determined the maximum degree of parallelism. Note that the processing time when record parallelism is applied here takes into account the overhead of applying record parallelism.

[0086] In the case illustrated in Figure 10, 1001, the processing parallelism for record parallelization is determined as 8 parallelisms for the table with a processing time of 10, and 4 parallelisms for the tables with processing times of 9 and 8. In the case illustrated in Figure 11, 1101, the processing parallelism for record parallelization is determined as 2 parallelisms for both the table with a processing time of 10 and the table with a processing time of 9.

[0087] The task parallelization process (S603) in the partitioned parallelization application determination process flow 600 calculates processing schedules for tables to which record parallelization is applied, tables to which record parallelization is not applied, and tables to which both are processed using table parallelization. Within the upper limit of processing parallelism, all tables should be sorted in order of priority, where the degree of parallelism of record parallelization > processing time, and then sorted using a greedy algorithm.

[0088] The arrangement exemplified in Figure 10, 1002, is a visualization of the task schedule after the partition parallelism determination process (S405), and is used as a reference diagram for the arrangement by the task parallelization process (S603). Figure 10, 1002, exemplifies parallel processing using two migration processing execution computers 135 with 8 cores and 4 cores. It can be seen that the 8-core migration processing execution computer 135 operates for 1.9 + 2.5 = 4.4 hours, and the 4-core migration processing execution computer 135 operates for 2.8 + 1 = 2.8 hours, so it takes 4.4 hours to complete the migration processing for all 13 tables. Comparing Figures 10, 1001 and 1002, it can also be seen that the processing time has been reduced from 10 hours to 4.4 hours.

[0089] Figure 11, item 1102, shows that processing is performed using six 2-core processing servers, and the processing time is 8, which is the processing time for an indivisible table.

[0090] In the bottleneck determination process (S604) of the partitioned parallel application determination process flow 600, the task that is the bottleneck in the task schedule calculated by the task parallelization process (S603) is selected. It can be determined that the table with the longest processing time among the tables that the core with the longest processing time is scheduled to process is the bottleneck.

[0091] For example, in the case illustrated in Figure 10, 1002, the four leftmost cores have the longest processing time, and the bottleneck table is one that originally had a processing time of 8, but after record parallelization, it is processed in 4 parallel lines, resulting in a processing time of 2.5 hours.

[0092] In the recalculation process (S605) of the partitioned parallel application determination process flow 600, the task schedule is calculated for when one-stage record parallelization is applied to the table obtained in the bottleneck determination process (S604). It is determined whether the migration processing time can be shortened and whether the cost reduction effect from shortening the migration time outweighs the total amount of overhead due to record parallelization. If both the migration processing time and cost can be reduced, it is decided to apply record parallelization to the table that was the bottleneck. If it is determined that there is no effect, the process determines a schedule without applying record parallelization.

[0093] Figure 12 illustrates the result of calculating a schedule where the parallelism is increased by one level in the recalculation process (S605) when the bottleneck determination process (S604) determines that the table with an original processing time of 8 is the bottleneck in the 1201 schedule (same as 1002). In this case, the processing time has not been reduced by increasing the parallelism of record parallelization, and no cost reduction can be expected. Therefore, it is determined that the process of increasing the parallelism of record parallelization should not be performed, and the 1201 schedule is output as the optimal schedule.

[0094] The processing server allocation process (S606) in the divided parallel application determination processing flow 600 determines the allocation of the processing server that minimizes processing cost in executing the task schedule calculated up to the recalculation process (S605). Details and examples of the processing server allocation process (S606) are explained using Figures 7 and 8.

[0095] For example, if a process scheduled in 6 parallel processes is assigned to an 8-core processing server (corresponding to the migration process execution computer 135), the two unused cores result in surplus costs. Additionally, because there is a difference in the processing time of tasks assigned to each core, cores that have finished their assigned processes must wait until the core with the longest processing time has finished, resulting in surplus costs.

[0096] Furthermore, while starting up a processing server takes time, charges are incurred during that time. Therefore, frequently starting up processing servers of different sizes depending on the time of day will result in excessive charges. For this reason, it is necessary to allocate processing servers in a way that avoids unnecessary core waiting time while minimizing startup overhead.

[0097] Figure 7 is a flowchart showing a detailed example of the processing server allocation process (S606) of the divided parallel application determination processing flow 600.

[0098] In Figure 7, the management computer executes the initial allocation process (S701), the same-size processing server integration process (S702), the different-size processing server integration process (S703), and the fee calculation process (S704). The processing server allocation process flow 700 may also include other processing steps not shown. Figure 8 is an example diagram illustrating how the processing server allocation process flow 700 is applied to a task schedule that processes seven tables A to G in six parallel processes to determine the allocation of processing servers. Figure 8, 801, shows the state before the server allocation process.

[0099] In the initial allocation process (S701) of the processing server allocation process flow 700, the finest-grained processing servers are allocated. Processing servers are allocated one-to-one to tables to which record parallelization is applied, and 2-core processing servers are allocated sequentially to the other groups of tables. Figure 8, 802, shows an example of processing server allocation after the execution of the initial allocation process (S701).

[0100] In the same-size processing server integration process (S702) of the processing server allocation process flow 700, if processing servers of the same size are allocated consecutively, the processing allocations are integrated into a single processing server. Figure 8, 803 shows an example of server allocation after the same-size processing server integration process (S702) has been executed. In Figure 8, 802, the allocation of (3) 2 cores and the allocation of (5) 2 cores were consecutive, but in Figure 8, 803, it can be seen that they have been integrated into one (3) 2 core.

[0101] The S703 processing server integration process (different size processing server integration process) in the 700 processing server allocation process flow determines the integration of consecutive processing servers of different sizes and assigns them accordingly. While integrating into a larger processing server increases the core waiting time of the processing server, it reduces the startup overhead of the later processing server.

[0102] If the startup overhead of a later processing server is greater than the increase in core latency, the servers will be merged. Figure 804 shows the case where the smaller, later processing server is merged as a result of this determination. If the increase in core latency is greater than the reduction in startup overhead of the smaller, later processing server, the servers will not be merged. Figure 805 shows the case where the smaller, later processing server is not merged as a result of this determination.

[0103] In the fee calculation process (S704) of the processing server allocation process flow 700, the additional charges incurred during the data migration process are calculated using the allocation of processing servers finally determined in the separate-size processing server integration process (S703) and instance fee information (not shown).

[0104] Figure 9 is a flowchart showing a detailed example of the processing server integration determination process (S406) in the processing plan generation process flow 400. In Figure 9, the management computer 101 executes the core standby time utilization determination process (S901), the integration plan generation process (S902), the disposable time utilization determination process (S903), and the integration plan generation process (S904). The processing server integration determination process flow 900 may include other processing steps not shown, and some steps may not be executed.

[0105] The processing server integration decision processing flow 900 aims to reduce the costs charged to processing servers by shortening the operating time of the processing servers, as illustrated in Figure 10, 1003 and Figure 11, 1103. Costs are reduced by shortening processing time by having the processing server with the longest core standby time process tasks assigned to other processing servers, and by integrating processes that span multiple processing servers into a single processing server, to the extent that the overall processing completion time does not worsen, thereby reducing the startup overhead of the processing servers. Reducing the cost of processing servers also leads to a reduction in the power consumption required to operate the processing servers, thus contributing to environmental protection.

[0106] The core standby time utilization determination process (S901) of the processing server integration determination process flow 900 calculates the core standby time of each processing server defined by the processing schedule generated in the division parallel application determination process (S405) of the processing plan generation process flow 400, selects the processing server with the largest core standby time, determines whether the tasks scheduled to be processed on the other cores fit within the core standby time, and whether the operating time of the processing server originally scheduled to process them can be reduced, and if the operating time of the processing server originally scheduled to process them can be reduced, reassigns the processing of those tasks to the processing server with the largest core standby time.

[0107] For example, in Figure 10, at point 1002, the core wait time of the 8-core processing server on the left is the maximum, and it can be determined that among the tasks scheduled to be processed by the 4-core processing server, the tables with processing times of 1, 1, 0.5, and 0.5 will fit within the core wait time of the 8-core processing server. In Figure 11, at point 1102, the core wait time of the 2-core processing server processing the table with an indivisible processing time of 8 is the maximum, and it can be determined that among the tasks scheduled to be processed by the other processing servers, the processing of the tables with processing times of 1.5, 1, 1, 1, 1, and 0.5 will fit within the core wait time.

[0108] The integration plan generation process (S902) of the processing server integration determination process flow 900 generates a plan in which the tables determined to be integrable in the core standby time utilization determination process (S901) are assigned to the target processing server for integration. Figure 1003 shows the process generated by the integration plan generation process (S902). The state illustrated in Figure 1301 is an intermediate state between 1102 and 1103 in Figure 11, and is a processing schedule generated when the core standby time utilization determination process (S901) and the integration plan generation process (S902) have been executed only once.

[0109] The processing server integration determination process flow 900 recursively calls the core standby time utilization determination process (S901) and the integration plan generation process (S902) of the processing server integration determination process flow 900. If it is no longer possible to reduce the core standby time any further, the disposable time determination process (S903) is executed.

[0110] In the disposable time utilization determination process (S903) of the processing server integration determination process flow 900, it is determined whether the uptime of a processing server that was originally scheduled to process a task can be shortened by using the difference in processing time between a certain processing server and the processing server with the longest processing time to process the task scheduled to be processed on another processing server. If there are multiple processing servers, the determination should be made in order of the smallest disposable time. In 1302 of Figure 13, the leftmost 2-core processing server has the smallest disposable time, and it can be determined that the uptime of the rightmost and second-to-right processing servers of 1302 can be reduced to zero by integrating the tasks scheduled to be processed on the rightmost and second-to-right processing servers of 1302 during that disposable time.

[0111] The integration plan generation process (S904) of the processing server integration determination process flow 900 is a process that generates a plan in which the tables that were determined to be integrable in the disposable time utilization determination process (S903) are assigned processing to the target processing server. The state illustrated in Figure 11, 1103 (Figure 13, 1103 is the same figure) is the processing schedule generated by the integration plan generation process (S904).

[0112] As described above, according to Example 1, the management computer 101 calculates the processing time of the tables containing the data to be migrated and the degree of parallelism of the migration process from the input connection information of the source database and the connection information of the destination database, shortens the processing time by combining record parallelism and table parallelism, determines the allocation of processing servers to perform the migration process while considering the cost of the migration processing servers, and executes the data migration process.

[0113] As described above, the computer system of Embodiment 1 comprises a processor 111, a storage device 112, and an input / output device 113. The storage device 112 stores at least an evaluation value table 116 that shows evaluation values ​​for the migration processing computer that executes the migration process. The input / output device 113 receives data connection information between the source and destination. The processor 111 uses the data connection information between the source and destination and the evaluation value table 116 to calculate the data processing time and the amount of resources to be used. Considering the constraints of the migration processing execution computer 135 stored in the evaluation value table 116, it calculates a processing schedule that shortens the processing time by combining record parallelization and table parallelization. It then calculates the allocation of the migration processing execution computer 135 to execute the processing schedule that can reduce surplus costs and determines the processing plan. The input / output device 113 outputs the determined processing plan.

[0114] Therefore, the computer system of Example 1 can search for a task schedule that appropriately combines record parallelism and table parallelism to reduce processing time under the constraints of the processing server, and can also determine the assignment of tasks to the processing server in a cost-reducing manner, thereby supporting the formulation of an appropriate processing plan for data migration.

[0115] Furthermore, the input / output device 113 receives the execution instruction for the processing plan, and the processor 111 performs the execution instruction processing for the migration processing execution computer 135. Therefore, data migration jobs can be easily executed.

[0116] As described above, according to Embodiment 1, the management computer 101 calculates the processing time of the tables containing the data to be migrated and the degree of parallelism of the migration process from the input connection information of the source database and the connection information of the destination database, shortens the processing time by combining record parallelism and table parallelism, determines the allocation of processing servers to perform the migration process while considering the cost of the migration processing servers, and executes the data migration process.

[0117] Thus, according to Example 1, it is possible to shorten processing time by parallelizing processing while taking into account the overhead of increased processing time caused by task division and the scope of the load on the system, and to support scheduling that reduces additional charges incurred by on-demand cloud resources used to execute tasks. [Examples]

[0118] Figure 15 is a block diagram showing an example configuration of the computer system according to Embodiment 2 of the present invention.

[0119] The computer system of Embodiment 1 shown in Figure 1 illustrates a case where, as an on-demand, pay-as-you-go cloud resource, the processor 111 of the management computer 101 performs task scheduling for the migration of data to be migrated 122 stored in the storage device 121 of the source computer 102 to the storage device 132 of the destination computer 131, via a migration processing execution computer 135, which is an on-demand, pay-as-you-go cloud resource on the cloud service 103.

[0120] In contrast, the computer system of Embodiment 2 shown in Figure 14 illustrates a case where, as an on-demand, pay-as-you-go cloud resource, the batch processing execution computer 1235, which is an on-demand, pay-as-you-go cloud resource on the cloud service 1203, schedules batch jobs for processing target data 1233 in the storage device 1232 of the computer 1231, with the processor 1211 of the management computer 1201 performing the scheduling.

[0121] The computer system in Example 2 is a system in which the management computer 1201, the batch processing execution computer 1235 on the cloud service 1203, and the computer 1231 on the cloud service 103 are interconnected via the network 1204. The other configurations are the same as those of the computer system in Example 1 shown in Figure 1, so their explanation is omitted.

[0122] In the above embodiment, considering the overhead of increased processing time due to parallelization by dividing tasks, task-level parallelization and task-dividing parallelization are combined to perform task scheduling that reduces processing time. Furthermore, tasks are assigned to processing servers in a way that minimizes the waiting time for the processing server cores.

[0123] Furthermore, task reallocation is performed by utilizing the waiting time of the processing server core to shorten the processing server's uptime and startup overhead.

[0124] Furthermore, tasks are reallocated to utilize the processing server's available time, thereby reducing the processing server's uptime and startup overhead. The execution management of the processing server is also performed to ensure it operates according to the calculated task schedule.

[0125] According to the above embodiment, it is possible to shorten processing time by parallelizing processing while taking into account the overhead of increased processing time caused by task division and the scope of the load on the system, and to support scheduling that reduces additional charges incurred by on-demand cloud resources used to execute tasks.

[0126] It should be noted that the present invention is not limited to the embodiments described above, and various modifications are included. For example, the embodiments described above are explained in detail to make the present invention easier to understand, and are not necessarily limited to those having all the configurations described. Furthermore, it is possible to replace or add configurations, not just delete them. For example, task scheduling may be performed in data backup. [Explanation of Symbols]

[0127] 101 Management computer 102 Source computer 103 Cloud Services 111 processors 112 Storage device 113 Input / Output Devices 114 Processing plan generation process 115 Processing plan execution management process 116 Evaluation Value Table 121 Storage device 122 Data to be processed 135. Migration Processing Execution Computer 136 processors 137 Storage device 131 Destination computer 132 Storage device

Claims

1. A management computer that uses a processing execution computer on a cloud service to perform data processing on data to be processed, which has multiple tables, It has a processor and input / output devices, The processor generates a processing plan generation processing unit that uses data information input via the input / output device to generate a processing plan to be executed using on-demand, pay-per-use cloud resources, and displays the generated processing plan on the input / output device. The processor includes a processing plan execution management processing unit that manages the execution of the processing execution computer so that the data processing is executed according to the processing plan selected via the input / output device, The processing plan generation processing unit, A table parallelization process is performed in which multiple tables are assigned to the cores of the processing execution computer, the data processing is parallelized on a table-by-table basis, tasks are processed on a table-by-table basis, and the cores of the processing execution computer each execute the tasks. If the table is larger than a predetermined data size, the table with the larger data size is divided into multiple records, the multiple records are assigned to the cores of the processing execution computer, and the data processing is parallelized on a record-by-record basis, processing the task on a record-by-record basis, and the cores of the processing execution computer each execute the task in a record-parallel processing manner. The processing plan generation processing unit is controlled by the processor, A management computer characterized by generating a processing plan that shortens the processing time of the data processing by combining the table parallelization processing and the record parallelization processing.

2. The aforementioned processing execution computer is: The aforementioned cloud service is used to execute migration processing, The aforementioned management computer is, As part of the data processing, the migration processing execution computer is used to perform data transfer processing of the data to be processed from the source computer to the destination computer on the cloud service. The processing plan generation processing unit, The processor generates a processing plan for the data transfer process to be executed using the cloud resources, based on the source data connection information and destination data connection information input via the input / output device. The aforementioned processing plan execution management processing unit: The management computer according to claim 1, characterized in that the processor performs the execution management of the migration processing execution computer so that the data transfer processing is executed according to the processing plan selected via the input / output device.

3. The aforementioned processing execution computer is: The aforementioned cloud service is a batch processing execution computer, The aforementioned management computer is, As part of the data processing, batch processing of the data to be processed is performed on a batch computer. The processing plan generation processing unit, The processor generates a processing plan for the batch process to be executed using the cloud resources. The aforementioned processing plan execution management processing unit: The management computer according to claim 1, characterized in that the processor performs the execution management of the batch processing execution computer so that the batch processing is executed according to the processing plan selected via the input / output device.

4. The processing plan generation processing unit is controlled by the processor, The management computer according to claim 1, characterized in that the tasks are assigned to the processing execution computer in such a way that the waiting time of the core of the processing execution computer is reduced.

5. The processing plan generation processing unit is controlled by the processor, The management computer according to claim 4, characterized in that it reallocates tasks by utilizing the waiting time of the cores of the processing execution computer in order to shorten the operating time and startup overhead of the processing execution computer.

6. The processing plan generation processing unit is controlled by the processor, The management computer according to claim 1, characterized in that it reallocates tasks by utilizing the disposable time of the processing execution computer in order to shorten the operating time of the processing execution computer and the startup overhead of the processing execution computer.

7. The processing plan generation processing unit is controlled by the processor, The management computer according to claim 1, characterized in that, among a plurality of tables, the table for which the time required for data processing is greater than a predetermined minimum processing time is determined to be the target of the record parallelization.

8. The processing plan generation processing unit is controlled by the processor, Select the task that is the bottleneck in the processing plan, The processing plan is regenerated when the record parallelization with increased parallelism is applied to the bottleneck task, The management computer according to claim 1, characterized in that when the processing time and cost of the aforementioned data processing can be reduced, the record parallelization process with an increased degree of parallelism is applied to the table which is the bottleneck.

9. The processing plan generation processing unit is controlled by the processor, The management computer according to claim 1, characterized in that the allocation of the cores of the processing execution computer is determined so as to minimize the processing cost of the data processing in the execution of the processing plan.

10. The aforementioned input / output device is It has a processing plan display screen, The management computer according to claim 1, characterized in that the processing plan display screen displays a graph showing the relationship between the processing time of the data processing and the degree of parallelism of the processing.

11. The management computer according to claim 1, The processing execution computer on the aforementioned cloud service, A management computing system connected via a network.

12. Using a processing execution computer on a cloud service, a management computer performs data processing on the target data which has multiple tables. A processing plan generation function that generates a processing plan to be executed using on-demand, pay-per-use cloud resources using data information input via an input / output device, and displays the generated processing plan on the input / output device, A management calculation program that causes a processor to execute a processing plan execution management processing function, which manages the execution of the processing execution computer so that the data processing is performed by the processor according to the processing plan selected via the input / output device, The aforementioned processing plan generation function is: Multiple tables are assigned to the cores of the processing execution computer, the data processing is parallelized on a table-by-table basis to process tasks on a table-by-table basis, and the cores of the processing execution computer perform table parallelization processing in which each of the tasks is executed. If the table is larger than a predetermined data size, the table with the larger data size is divided into multiple records, the multiple records are assigned to the cores of the processing execution computer, and the data processing is parallelized on a record-by-record basis, processing the task on a record-by-record basis, and the cores of the processing execution computer each execute the task in a record parallelization process. The aforementioned processing plan generation function is: A management calculation program characterized in that the processor generates a processing plan that shortens the processing time of the data processing by combining the table parallelization processing and the record parallelization processing.

13. A management calculation method that performs data processing on data to be processed having multiple tables using a processing execution computer on a cloud service, A processing plan generation step involves a processor generating a processing plan using data information input via an input / output device, executing the plan using on-demand, pay-as-you-go cloud resources, and displaying the generated processing plan on the input / output device. The processor has a processing plan execution management processing step that manages the execution of the processing execution computer so that the data processing is executed according to the processing plan selected via the input / output device, The processing plan generation step described above is: A table parallelization process is performed in which multiple tables are assigned to the cores of the processing execution computer, the data processing is parallelized on a table-by-table basis, tasks are processed on a table-by-table basis, and the cores of the processing execution computer each execute the tasks. If the table is larger than a predetermined data size, the table with the larger data size is divided into multiple records, the multiple records are assigned to the cores of the processing execution computer, and the data processing is parallelized on a record-by-record basis, processing the task on a record-by-record basis, and the cores of the processing execution computer each execute the task in a record parallelization process. The processing plan generation step described above is: A management calculation method characterized in that the processor generates a processing plan that shortens the processing time of the data processing by combining the table parallelization processing and the record parallelization processing.

Images