Batch system analysis device and batch system analysis method

Abstract

To provide information for specifying a part for dividing a batch system.SOLUTION: A batch system analysis device includes: a time-series analysis part for analyzing an execution order of jobs constituting a batch system based on schedule information for instructing execution of the jobs, and determining execution order information on the jobs; an I / O information analysis part for determining job I / O information indicating input / output of data used in the jobs from a source code of the jobs; a dependence analysis part for determining dependence information between the jobs from the job I / O information; a separation information analysis part for determining separation information which is a part capable of separating a batch system from the execution order information and the dependence information; and an output part for outputting the separation information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a batch system analysis device and a batch system analysis method. [Background technology]

[0002] Batch systems operated using computers such as mainframes have become complex due to years of modifications, and in many cases there is no documentation describing the entire batch system, or the documentation has not been updated.

[0003] In addition, the number of engineers using the COBOL language, which has been commonly used to develop batch systems, is decreasing, and even if there are source programs that correspond to current batch systems, it is becoming difficult for engineers to analyze the operation of the batch systems.

[0004] In this situation, new improvements are needed to address issues that arise in the current system, such as migrating from mainframe computers to cloud systems, creating online systems, and optimizing computer resources.

[0005] To address such issues, analysis of the batch system is necessary, and in Patent Document 1, multiple jobs contained in log data are classified according to which of multiple time intervals the job end time belongs to.

[0006] Then, for multiple jobs included in the first time interval, first data indicating the execution order relationship of the jobs is generated based on the end times of the jobs, and for multiple jobs included in a second time interval subsequent to the first time interval, second data indicating the execution order relationship of the jobs is generated based on the end times of the jobs.

[0007] An analysis device is disclosed that analyzes the execution order relationship between jobs included in the first time interval and jobs included in the second time interval based on the end times of jobs included in the first time interval and jobs included in the second time interval, and generates data indicating the execution order relationship of jobs from the first time interval to the second time interval. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-128828 Summary of the Invention [Problem to be solved by the invention]

[0009] The method of Patent Document 1 is an analysis device that generates data indicating the relationship between job execution orders from job end times using log data that is a record of job execution.

[0010] Therefore, if an error occurs during execution of a job or if the content or amount of input data for the job changes, the end time of the job may change.

[0011] Furthermore, different jobs may be executed depending on the input data, and it is conceivable that the behavior of the actual batch system may not match the data indicating the relationship between the job execution order, which is the output of the analytical device.

[0012] Furthermore, the method of Patent Document 1 does not analyze the inter-job dependencies regarding files input and output by the jobs, and therefore has the problem that job analysis results regarding inter-job dependencies cannot be obtained.

[0013] Generally, when modifying a large-scale batch system, it is technically and financially difficult to modify the entire batch system at once, so the batch system is divided into several parts and modifications are made to each of the divided parts.

[0014] Therefore, without proper division, it is not possible to run the modified part together with the unmodified part.

[0015] To solve this problem, it is necessary to analyze not only the job execution order but also the job dependencies. [Means for solving the problem]

[0016] The above-mentioned object is achieved by a batch system analysis device that includes a time series analysis unit that analyzes the execution order of jobs that make up a batch system based on schedule information that instructs job execution and obtains job execution order information, an I / O information analysis unit that obtains job I / O information that indicates the input and output of data used in the job from the job source code, a dependency analysis unit that obtains dependency information between jobs from the job I / O information, a separation information analysis unit that obtains separation information that is the point at which the batch system can be separated from the execution order information and dependency information, and an output unit that outputs the separation information. [Effects of the Invention]

[0017] It can provide information to identify where to split the batch system. [Brief explanation of the drawings]

[0018] [Figure 1] FIG. 1 is a block diagram illustrating an example of the configuration of a batch system analyzer according to an embodiment of the present invention. [Figure 2] A diagram explaining the job nets that make up a batch system [Figure 3] Schematic diagram of a job net [Figure 4] Diagram explaining schedule information [Figure 5] Diagram explaining CRUD information [Figure 6] Example of batch configuration information showing the calling relationship of jobs in the embodiment of this invention [Figure 7]Examples of job data dependency relationship information in an embodiment of this invention [Figure 8] Example of batch I / O information in an embodiment of the present invention [Figure 9] An example of a diagram showing job dependencies when time series is not taken into consideration in an embodiment of this invention. [Figure 10] Example of batch schedule information showing a job execution schedule in an embodiment of the present invention [Figure 11] An example of a diagram showing job dependencies based on time series in an embodiment of the present invention [Figure 12] FIG. 10 is a diagram illustrating the relationship between jobs in chronological order in an embodiment of the present invention. [Figure 13] Examples of separation information in an embodiment of the present invention [Figure 14] Example of analysis screen in an embodiment of the present invention [Figure 15] 1 is a flowchart illustrating the processing of a batch system analyzer according to an embodiment of the present invention. [Figure 16] 10 is a flowchart illustrating a process for creating batch I / O information from source file information in an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0019] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In each drawing for explaining the embodiments, the same components are given the same names and reference numerals as much as possible, and repeated explanations thereof will be omitted.

[0020] The present invention is not limited to the following examples, and includes various modifications and equivalent configurations within the spirit and scope of the appended claims. For example, the above examples have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations.

[0021] Furthermore, the processing units and processing modules described in the embodiments may be realized in part or in whole in hardware, for example by designing them as integrated circuits, or may be realized in software by having a processor interpret and execute a program that realizes each function.

[0022] The information explained in the embodiment may be a table, a database (DB), or data stored in the main memory. [Example]

[0023] FIG. 1 is a block diagram showing an example of the configuration of a batch system analyzer according to an embodiment of the present invention.

[0024] In this embodiment, the batch system analysis device 1 is implemented as a standalone computer having a processing device configured with a CPU (Central Processing Unit) 2 and the like, a main memory device 3 configured with a RAM (Random Access Memory), a ROM (Read Only Memory), and the like, an external memory device 4 configured with an HDD (Hard Disk Drive), an SSD (Solid State Drive), and the like, and an input / output unit 5 configured with a NIC (Network Interface Card), and the like, but it may also be implemented using a cloud system that provides computer resources.

[0025] The main memory device 3 stores a time series analysis unit that analyzes the execution order of jobs and jobnets in chronological order, an I / O information extraction unit 11 that analyzes the input and output of jobs, a dependency extraction unit 12 that analyzes the dependency of jobs based on the execution order of jobs and the dependency of the data used, and a separation information extraction unit 13 that obtains information for separating batch systems.

[0026] These processing units are realized as software modules, and are executed by the CPU 2 with reference to information stored in the external storage device 4 .

[0027] The external storage device 4 stores batch configuration information 20, batch schedule information, source files 22, and batch I / O information 23, and based on this information, a processing unit stored in the main storage device 3 performs analysis and creates time series information 24, dependency information 25, and separation information 26.

[0028] This information may be in the form of an independent file or table, or may be written in JASON format with multiple pieces of information stored in one file.

[0029] The input / output unit 5 receives information such as batch configuration information 20 of the batch system from the user, and outputs the results of processing by each processing unit stored in the main memory unit 3 based on the received information, and provides them to the user.

[0030] FIG. 2 is a diagram illustrating a job network that constitutes a batch system.

[0031] This example explains an example of a batch system described in JSON (JavaScript Object Notation). A batch system consists of an array of multiple root jobnets and is expressed in the format Batch process = [Root jobnet 1, Root jobnet 2, Root jobnet 3...].

[0032] The configuration information for one job net has a tree structure. The root of the tree is the root job net, and the identifier childNodes stores the jobs or job nets included in the job net. The identifier links stores pointers to the job nets or jobs.

[0033] In the case of a jobnet, it holds the value defined by the following identifier. name: Name of the jobnet childNodes: Name of the job net or job contained in the job net links: Pointer to the job net or job to be executed type: set to "Jobnet" In case of a job, it holds the values ​​defined by the following identifiers. name : The name of the job childNodes: empty value links : empty value script: A pointer to the script the job will execute. type: set to "Job" In this example, a configuration example is defined in which a job net named Root calls jobs A, B and a lower-level job net C, and the lower-level job net C calls jobs D and E.

[0034] Figure 3 shows a schematic representation of the batch configuration information 20 written in JSON format source code shown in Figure 2. The circles represent jobs, and the arrows represent jobnets.

[0035] FIG. 4 is a diagram illustrating schedule information.

[0036] Schedule information consists of the name of the root jobnet and multiple schedule details, and the root jobnet is executed according to one or more schedules.

[0037] The schedule information has the following data structure: name: Name of the root jobnet scheduleInfos: All schedule details for the root jobnet startYear: Year in which the root jobnet starts executing startMonth: Root jobnet execution start month startDay: Execution start date of the root jobnet startHour: 0-23 is specified for the "time" when the root jobnet starts executing. startMinute: The minute the root jobnet starts executing (0-59) cycle: Execution cycle of the root jobnet If you do not specify an item, the value should be set to -1. For daily, cycle="PER_DAY", for monthly, cycle="PER_MONTH", specify the number of days in startDay, and to specify "end of the month", specify "32". For weekly, cycle="PER_WEEK", specify the day of the week in startDay, and specify the month to day from 1 to 7.

[0038] To execute every X days, specify the first execution date with Cycle="X", startYear, startMonth, and startDay. If the first execution date is 7 / 7 / 2006, set startYear=2006, startMonth=7, startDay=7.

[0039] In this example, a job named Root will be started every day at 7:00 AM.

[0040] FIG. 5 is a diagram illustrating CRUD information.

[0041] The CRUD information consists of the script name and the script's CRUD, and the data structure is as follows: Script: The name of the script cruds : All CRUD information for the script DataName: Name of the access data crud: Operation details for the data The operation contents are expressed by C, R, U, and D, which mean C:Create, R:Read, U:Update, and D:Delete.

[0042] In this example, the script A.sh is defined to create TABLE1 and read and update TABLE2.

[0043] FIG. 6 shows an example of batch configuration information 20 that indicates the calling relationship of jobs in the embodiment of the present invention.

[0044] This table shows the call relationships based on the job schedule information written in JSON format. The calling job 30, the called job, and the job net 32 ​​that contains the job are stored in association with each other.

[0045] FIG. 7 shows an example of data dependency information for jobs in an embodiment of the present invention.

[0046] The dependency information 25 stores, in association with each other, the dependency source 50, which is the name of the job that is the dependency source; the dependency destination 51, which is the name of the job that is the dependency destination; the dependency type 52, which indicates whether the dependency is RAW (Read After Write), WAR (Write After Read), or WAW (Write After Write); and the dependency data 53, which is the name of the dependent data.

[0047] The types of dependency are explained here. For example, if there is a RAW dependency in which job J reads the content output by job K, if the job is split ignoring this dependency, the reading of job J will be executed first, resulting in data inconsistency.

[0048] If there is a WAR dependency where job J writes out data written by job K again, and the data is split while ignoring this dependency, the write order of jobs J and K will change, and a data inconsistency will occur when the subsequent job Z reads the data.

[0049] If there is a WAR in which job J writes out data read by job K again, if the division is performed ignoring this dependency, the execution order of job J and job K will change and the execution result of job K will change, resulting in data inconsistency.

[0050] FIG. 8 is an example of batch I / O information in an embodiment of the present invention.

[0051] The batch I / O information 23 stores Job# 60 indicating the job number and data access 61 indicating the name of the data accessed by the job and the access type READ / WRITE in pair, in association with each other.

[0052] FIG. 9 is an example of a diagram showing job dependencies when time series is not taken into consideration in the embodiment of the present invention.

[0053] The job call relationships shown in Figure 6 and the dependencies based on the presence or absence of data dependency information shown in Figure 7 are shown in an effective diagram. The solid arrows indicate the execution order based on the call relationships, and the thin dotted lines indicate data dependencies.

[0054] In this example, the only data dependency from Jobnet3 to Jobnet2 is the dependency of job 14 on job 11. Also, the only data dependency from Jobnet5 to Jobnet3 is the dependency of job 18 on job 15. Therefore, the two locations indicated by the thick dotted lines, division points s1 and s2, are candidates for division.

[0055] Here, the execution schedule of the job will be considered.

[0056] FIG. 10 shows an example of batch schedule information representing a job execution schedule in an embodiment of the present invention.

[0057] Figure 10 shows schedule information written in JSON format rewritten in table format. The job name (Job#40), execution year (StartYear41), execution month (StartMonth42), execution date (StartDay43), execution time (StartHour), execution minute (StartMinute45), and execution cycle (Cycle46) are stored in association with each other.

[0058] FIG. 11 is an example of a diagram showing job dependencies based on time series in an embodiment of the present invention.

[0059] Figure 11 shows the rearrangement of the effective diagram in Figure 6 based on the schedule in Figure 9. When arranged in chronological order, it can be seen that the division point s1 (t=t1) between Jobnet3 and Jobnet2, which was a candidate division point in the dependency relationships in Figure 9, only has data dependency from job 14 to job 11, but the division point s2 (t=t2) between Jobnet5 and Jobnet3 also has data dependency from job 18 to job 13, from job 17 to job 13, and from job 19 to job 13.

[0060] Furthermore, as a candidate for a split point, there is no execution order dependency between Jobnet1 and Jobnet2 at t=t0, only data dependency, and considering that the number of data dependencies at t=t2 is 4, the number of data dependencies at t=0 is 3, so it can be seen that there is a candidate split point.

[0061] FIG. 12 is an example of a diagram showing job dependencies based on time series in an embodiment of the present invention.

[0062] When simulating (extracting) the execution order within a job net, a set of jobs that start to be executed in parallel is treated as a time series unit by a simulation algorithm.

[0063] Furthermore, if there are job nets that start execution at the same time, there may be jobs that belong to multiple job nets in one time series unit.

[0064] If we divide the job net in Figure 11 into chronological units, it will look like Figure 12. If we illustrate the execution order of job net 1, jobs 1 and 3 will start executing at the same time. Jobs 5 and 6 will also start executing at the same time.

[0065] Therefore, jobs 1 and 3 belong to the same time series unit, and jobs 5 and 6 belong to the same time series unit.

[0066] FIG. 13 shows an example of separation information in an embodiment of the present invention.

[0067] The separation information 26 stores, in association, the separation time 79, the processed job network 80, the unprocessed job network 81, the score 82 representing the separation possibility index (difficulty level) obtained at the separation candidate point, the source 50 which is the name of the job that is the source of the data dependency, the destination 51 which is the name of the job that is the destination of the data dependency, the type 52 of the data dependency, and the dependent data 53 which is the name of the data on which the dependency is made.

[0068] In this example, it is stored after sorting in ascending order of the score. The score is calculated based on the weights according to the number of dependencies and the type of dependencies.

[0069] In this example, the weight of RAW is 1, the weight of WAW is 0.5, and the weight of WAR is 0.2. At t = t0, since there are 2 RAW dependencies and 1 WAR dependency, the score is 1 * 2 + 0.2 = 2.2.

[0070] Similarly, at t = t1 it is 0.5, and at t = t2 it is 4, and the magnitude of the dependencies is in the order of t1 < t0 < t2.

[0071] FIG. 14 is an example of an analysis screen in an embodiment of the present invention. It receives input of information indicating a batch system to be analyzed, such as the batch configuration 71, the schedule 72, and the source file 73. In this embodiment, examples of screens for inputting each piece of information are described, but when the information indicating the batch system is included in one file, it is only necessary to specify one file name.

[0072] If it is configured to receive specification of the maximum score 74 which is the upper limit of the difficulty level of the division when dividing and the maximum number 75 of division candidates to be displayed, it is possible to display only the division candidate locations with high consideration value.

[0073] By pressing the analysis button 76, the analysis is started, and an analysis screen 77 that displays the dependencies using a directed diagram and division candidate locations 78 which are the results of calculating the scores of each division point candidate are output.

[0074] A division candidate location 78 displays a division candidate indicated by a time 79 in association with a processed job net 80, an unprocessed job net 81, and a score 82 at that time.

[0075] In addition, by specifying the division point candidate for which you want to calculate the score using the arrow indicating the time axis displayed on the analysis screen 77 and pressing the analysis button 76, you can obtain the score of any division point candidate on the displayed time axis.

[0076] FIG. 15 is an example of a flowchart showing the processing of a batch system analyzer in an embodiment of the present invention.

[0077] First, batch configuration information 20, schedule information, and source file designation are accepted from the user (S1).

[0078] Next, the time series analysis unit 10 creates the batch configuration information 20 based on the received batch configuration information 20 and schedule information, arranges the jobs in chronological order, and creates time series information (S2).

[0079] The I / O information extraction unit 11 creates batch I / O information 23 from the source file information (S3), and the dependency extraction unit 12 creates dependency information 25 from the time-series information and batch I / O information (S4).

[0080] The separation information extraction unit 13 creates separation information 26 from the dependency relationship information 25 (S5), and the input / output unit 5 outputs an analysis screen 70 including an overall view of the batch (S6).

[0081] The input / output unit 5 receives the specification of a divisible point on the time axis of the overall batch diagram on the analysis screen 70 (S7). The separation information extraction unit 13 calculates the dependency relationship and divisibility index (score) at the specified divisible point, and the output unit outputs them to the candidate divisible point 78 (S8).

[0082] FIG. 16 is an example of a flowchart of a process for creating batch I / O information from source file information in an embodiment of the present invention.

[0083] The dependency extraction unit 12 sets the time series unit to be executed first to A, sets the time series unit to be executed after A to B (S20), sets the first job of A to a1, sets the first job of B to b1, and determines whether jobs a1 and b1 read and write the same data (S21).

[0084] If there is a read or write, and job a1 writes and job b1 reads (S23), job b1 is determined to be WAR dependent on job a1, and this is entered in the dependency type 52 of the dependency information 25 (S24).

[0085] If job a1 is a write job and job b1 is a write job (S25), job b1 is determined to be WAW dependent on job a1, and this is entered in the dependency type 52 of the dependency information 25 (S26).

[0086] If job a1 is a read job and job b1 is a write job (S27), job b1 is determined to be RAW dependent on job a1, and this is entered in the dependency type 52 of the dependency information 25 (S28).

[0087] If the answer is No in (S27), the dependency type is confirmed and stored in the dependency type 52 of the dependency information 25, and then it is determined whether there are other jobs in the time series unit B (S29), and if there are, the next job is set in b1 and the process returns to S22 (S30).

[0088] If there is no other job in B, it is determined whether there is another job in time series unit A (S31), and if there is, the next job is set in a1, the first job is set in b1, and the process returns to S22 (S32).

[0089] If there is no other job in A, it is determined whether there is another time series unit to be executed after job net B (S33), and if there is, the next time series unit is set in B and the process returns to S22 (S34).

[0090] If there are no other jobs in B, it is determined whether A is the last job (S35), and if it is not the last job, the next time series unit is set in A, the time series unit to be executed next after the set job net is set in B, and the process returns to S22 (S36). If it is the last job, the process ends and returns.

[0091] This process makes it possible to check the data dependencies between jobs included in all time series units within the batch system. [Explanation of symbols]

[0092] 1: Taxi ticket management system, 2: CPU, 3: Main memory, 4: External storage device, 5: Input / output section, 10: Time series analysis section, 11: I / O information extraction unit, 12: Dependency extraction part, 13: Separation information extraction section, 20: Batch configuration information, 21: Batch schedule information, 22: Source files, 23: Batch I / O information, 24: Time series information, 25: Dependency information, 26: Separation information

Claims

1. In batch system analyzers, a time series analysis unit that analyzes the execution order of jobs constituting a batch system based on schedule information that instructs job execution, and obtains job execution order information; an I / O information analysis unit that obtains job I / O information indicating input and output of data used in a job from a source code of the job; a dependency analysis unit that obtains dependency information between jobs from job I / O information; a separation information analysis unit that obtains separation information indicating where the batch system can be separated from the execution order information and dependency information; A batch system analyzer having an output unit that outputs separation information.

2. 2. The batch system analyzer of claim 1, The batch system analysis device in accordance with the present invention is characterized in that the schedule information analyzed by the time series analysis unit is written in JSON format.

3. 2. The batch system analyzer of claim 1, The batch system analyzer includes a batch system analyzer, and the I / O information obtained by the I / O information analysis unit is information relating to reading from and writing to a file.

4. 4. The batch system analyzer of claim 3, The dependency information obtained by the dependency analysis unit is a WAR in which the first job writes data and the second job reads data, RAW, where the first job reads the data and then the second job writes the data, or A batch system analyzer includes a dependency type indicating whether a second job writes data after a first job writes data.

5. 5. The batch system analyzer of claim 4, The separation information analysis unit is a batch system analysis device that obtains separation information of the batch system at the candidate division point based on the job dependencies that occur at the candidate division point corresponding to the time included in the execution sequence information and the dependency type of the dependencies.

6. 6. The batch system analyzer of claim 5, The separation information analysis unit multiplies the number of dependencies for each dependency type occurring at the split candidate location by a weight determined according to the dependency type, and calculates a total separability index for all dependency types; The output unit is a batch system analysis device that outputs processed jobs, unprocessed jobs, and separability indicators at the candidate division locations.

7. 7. The batch system analyzer of claim 6, The output unit is a batch system analysis device that outputs a predetermined number of candidate division points when the separability indexes are arranged in ascending order.

8. In the batch system analysis method, The time series analysis unit analyzes the execution order of jobs that make up the batch system based on the schedule information that instructs the execution of the jobs, and obtains job execution order information. The I / O information analysis unit obtains job I / O information indicating input and output of data used in the job from the source code of the job, The dependency analysis unit obtains dependency information between jobs from the job I / O information, The separation information analysis unit obtains separation information, which is a point where the batch system can be separated, from the execution order information and dependency information. A batch system analysis method in which an output unit outputs separation information.

Images