History management system, history management method, program, and storage apparatus

Abstract

To support the reproduction of research data in research using an HPC system.SOLUTION: A history management system includes: a first collection unit for collecting job management number information representing a management number of a job and job script identification information identifying a job script from a job queue which sequentially assigns an input job to one of a plurality of server computers, and generates first job information; a second collection unit for collecting job management number information and program identification information identifying a program corresponding to a process from a server computer which executes the process based on the job, and generates second job information; and a storage unit for generating and storing first history information by using the first job information acquired from the first collection unit and the second job information acquired from the second collection unit.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to a history management system that manages history information, a history management method, a program, and a storage device. [Background technology]

[0002] In research using HPC (High-Performance Computing) systems, research data is managed using provenance information.

[0003] As a related technique, Non-Patent Document 1 discloses introducing a lightweight provenance service (LPS) into an HPC system to capture history information. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Dong Dai, Yong Chen, Philip Carns, John Jenkins, Robert Ross, “Lightweight Provenance Service for High-Performance Computing”, [online], Date Added to IEEE Xplore: 02 November 2017, 2017 26th International Conference on Parallel Architectures and Compilation Techniques (PACT), Retrieved 15 September 2023, Internet<URL:https: / / doi.org / 10.1109 / PACT.2017.14> Summary of the Invention [Problem to be solved by the invention]

[0005] However, when reproducing research data, the history information in Non-Patent Document 1 and the like does not include information necessary for reproduction, so the research data cannot be easily reproduced. In other words, in the past, the history information had to be manually analyzed to reproduce the research data.

[0006] One example of the purpose of this disclosure is to assist in reproducing research data in research using HPC systems. [Means for solving the problem]

[0007] In order to achieve the above object, a history management system according to one aspect of the present disclosure includes: a first collection unit that collects job control number information representing a control number of a job and job script identification information identifying a job script from a job queue that sequentially assigns an input job to one of a plurality of server computers, and generates first job information; a second collection unit that collects the job control number information and program identification information that identifies a program corresponding to the process from the server computer that executes a process based on the job, and generates second job information; a storage unit that generates and stores first history information using the first job information acquired from the first collection unit and the second job information acquired from the second collection unit; The present invention is characterized by having the following.

[0008] In order to achieve the above object, a history management method according to one aspect of the present disclosure includes: obtaining job control number information representing a control number of the job and job script identification information identifying a job script from a job queue that sequentially assigns an input job to one of a plurality of server computers, and generating first job information; acquiring the job control number information and process identification information for identifying the process from a server computer that executes a process based on the job, and generating second job information; generating first history information using the first job information and the second job information and storing the first history information in a storage device; It is characterized by:

[0009] In order to achieve the above object, a program according to one aspect of the present disclosure includes: A computer having a job queue that sequentially assigns submitted jobs to one of a plurality of server computers, acquiring job control number information indicating a control number of the job and job script identification information identifying a job script from the job queue, generating first job information, and transmitting the first job information to a storage device; It is characterized by:

[0010] In order to achieve the above object, a program according to one aspect of the present disclosure includes: A process start command is acquired from a job queue that sequentially assigns submitted jobs to one of a plurality of server computers, and the server computer executes a process based on the job. acquiring job control number information indicating a control number of the job and process identification information for identifying the process, generating second job information, and transmitting the second job information to a storage device; It is characterized by:

[0011] Furthermore, in order to achieve the above object, a program according to one aspect of the present disclosure includes: A computer having a job queue which sequentially assigns submitted jobs to one of a plurality of server computers, the server computer which acquires a process start command from the job queue and executes a process based on the job, and a storage device electrically connected thereto receiving, from the computer, first job information generated using job control number information representing a control number of the job and job script identification information identifying a job script, both of which are collected from the job queue; receiving second job information from the server computer, the second job information being generated using job control number information representing a control number of the job and process identification information identifying the process, the second job information being collected from the server computer; generating first history information using the received first job information and the received second job information and storing the first history information; It is characterized by: Effect of the Invention

[0012] As described above, the present disclosure can assist in reproducing research data in research using HPC systems. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram for explaining an example of a job script for generating research data. [Diagram 2] FIG. 2 is a diagram for explaining an example of the operation of generating research data. [Diagram 3] FIG. 3 is a diagram illustrating an example of a history management system. [Figure 4] FIG. 4 is a diagram for explaining an example of generation of process file history information. [Diagram 5] FIG. 5 is a diagram for explaining an example of the operation of the history management system. [Figure 6] FIG. 6 is a diagram illustrating an example of a computer, a server computer, and a storage device that realizes the history management system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] First, an overview will be given to facilitate understanding of the embodiments described below. FIG. 1 is a diagram for explaining an example of a job script for generating research data. The job script file "Job1.sh" shown in FIG. 1 is a file for generating research data "result.csv" in an HPC system. Note that the job script file is not limited to the example shown in FIG. 1.

[0015] Specifically, when the job script file "Job1.sh" is entered into the job queue, the job is assigned to a server computer, and then the assigned server computer generates the research data "result.csv." In other words, the research data "result.csv" is generated according to the contents of the job script file "Job1.sh."

[0016] The generation of research data will be explained with reference to FIG. 2. FIG. 2 is a diagram for explaining an example of the operation of generating research data. First, when input data "indata1" is input based on the description "prog1 indata1 > outdata1" in the job script file "Job1.sh" shown in FIG. 1, the program "prog1" generates output data "outdata1" and stores it in a storage device such as a storage or memory. Next, when input data "indata2" is input based on the description "prog2 indata2 > outdata2" in the job script file "Job1.sh", the program "prog2" generates output data "outdata2" and stores it in a storage device such as a storage or memory.

[0017] Next, based on the statement "prog3 outdata1 outdata2 > result.csv" in the job script file "Job1.sh," the program "prog3" reads "outdata1" and "outdata2" as input data from the storage device mentioned above, and using the read "outdata1" and "outdata2," the program "prog3" generates a file named "result.csv" containing research data, and stores (writes) it in a storage device such as a storage or memory.

[0018] In addition, conventional history information (process file history information) is stored, for example, in the format shown below, that is, information representing the relationship between input data, programs, and output data (information corresponding to "->" shown below) in, for example, storage, memory, storage device, etc.

[0019] [Process file history information] indata1 -> prog1 -> outdata1 indata2 -> prog2 -> outdata2 outdata1, outdata2 -> prog3 -> result.csv

[0020] As explained above, in the example of Figure 2, it is known in advance that the contents of the process file history information match the contents of the job script file "Job1.sh", so it is clear that the research data can be reproduced by using the contents of the process file history information described above.

[0021] However, in reality, after creating research data, in order for a user to reproduce the research data he or she desires, the user must manually analyze the process file history information and create a job script file that can reproduce the research data he or she desires.

[0022] Specifically, to reproduce the research data "result.csv" shown in Figure 2, a job script file equivalent to the job script file "Job1.sh" shown in Figure 1 must be created by analyzing multiple process file history information stored in a storage device.

[0023] Therefore, there is a need for the development of a mechanism that can assist users in easily creating job script files that can reproduce the research data they desire, without the need to analyze multiple process file history information.

[0024] Through this process, the inventors discovered that the conventional methods described above had the problem that research data could not be easily reproduced, and they also came up with a means for solving this problem.

[0025] Hereinafter, an embodiment will be described with reference to the drawings. In the drawings described below, elements having the same or corresponding functions are denoted by the same reference numerals, and repeated description thereof may be omitted.

[0026] (Embodiment) The configuration of the history management system 100 in the embodiment will be described with reference to Fig. 3. Fig. 3 is a diagram for explaining an example of the history management system.

[0027] [System Configuration] The history management system 100 shown in Fig. 3 is, for example, an HPC system. The history management system 100 includes a computer 10, a plurality of server computers 20 (20a, 20b, 20c, etc.), and a storage device 30. Note that, although the number of server computers 20 is three in Fig. 3, it is not limited to three and may be more than three.

[0028] Furthermore, the computer 10, the multiple server computers 20 (20a, 20b, 20c, . . . ), and the storage device 30 are electrically connected via a communication network.

[0029] The communication network is a general network constructed using communication lines such as the Internet, a LAN (Local Area Network), a dedicated line, a telephone line, an in-house network, a mobile communication network, Bluetooth (registered trademark), and Wi-Fi (Wireless Fidelity) (registered trademark).

[0030] The computer 10 is an information processing device having, for example, a CPU (Central Processing Unit), a programmable device such as an FPGA (Field-Programmable Gate Array), a GPU (Graphics Processing Unit), or a circuit equipped with any one or more of these.

[0031] Each of the server computers 20 (20a, 20b, 20c, . . . ) is an information processing device having, for example, a CPU, or a programmable device such as an FPGA, or a GPU, or a circuit equipped with any one or more of them.

[0032] The storage device (storage unit) 30 is a database, a server computer, a circuit having a memory, etc. The storage device 30 stores, for example, at least information such as history information. The storage device 30 may be provided inside the HPC system 100 or may be provided outside the HPC system 100.

[0033] [Device configuration] The computer 10 includes a job queue 11 and a first collection unit 12. Each of the server computers 20 includes a second collection unit 21 (21a, 21b, 21c). The storage device 30 stores first history information 31.

[0034] The first collection unit 12 is provided in a computer that executes the job queue 11. For example, it is software, but it may be configured as hardware.

[0035] Specifically, the first collection unit 12 first collects job control number information indicating the control number of the job and job script identification information identifying the job script from the job queue 11, which sequentially assigns submitted jobs to one of the multiple server computers 20, to generate first job information. Next, the first collection unit 12 transmits the generated first job information to the storage device 30.

[0036] The job script identification information may be, for example, a file path in a storage device where the job script is stored, or may be a file path and timestamp information of the file, or may be a file path and version information of the file.

[0037] The second collection unit 21 is, for example, software included (provided) in the operating system 23 that runs on the server computer 20. However, it may also be configured as hardware.

[0038] Specifically, the second collection unit 21 first collects job control number information and program identification information for identifying a program corresponding to the process from the server computer 20 that executes a process based on a job, and generates second job information. Next, the second collection unit 21 transmits the generated second job information to the storage device 30.

[0039] The storage device (storage unit) 30 generates and stores first history information (job process history information) 31 using the first job information acquired (received) from the first collection unit 12 and the second job information acquired (received) from the second collection unit 21.

[0040] In this manner, in the embodiment, since it is possible to generate first history information 31 having information necessary for reproducing research data, it is possible to reproduce research data in research using an HPC system.

[0041] (Example) Specifically, when the job script file shown in FIG. 1 is used, information in the following format, for example, that represents the relationship between the job script file and the program, is stored in the storage device 30 as job process history information.

[0042] [Process file history information] Job1.sh - prog1 Job1.sh - prog2 Job1.sh - prog3

[0043] This will be described in more detail with reference to Fig. 4. Fig. 4 is a diagram for explaining an example of generation of process file history information.

[0044] 4, in order to easily explain the functions of the server computer 20, a process 22 and an OS 23 are shown in addition to the second collection unit 21. Also, it is shown that each of the server computers 20 has a third collection unit 24.

[0045] The process 22 is a program executed by a process start (command) from the job queue 11. The OS 23 is an operating system (software) installed in the computer server.

[0046] The third collection unit 24 is, for example, software included (provided) in the operating system 23 that runs on the server computer 20. However, it may also be configured as hardware.

[0047] Specifically, the third collection unit 24 first collects process information and file access information from the server computer 20 in order to generate the above-mentioned process / file history information.

[0048] The process information includes, for example, process management number information indicating the management number of a process, program identification information identifying a program corresponding to a process, etc. The file access information includes, for example, information regarding reading a file, information regarding writing a file, input data identification information identifying input data input to a program, output data identification information identifying output data output from a program, and information indicating the input / output relationship between input data and output data for a program, etc.

[0049] The input data identification information may be, for example, a file path in a storage device where the input data is stored as a file. Alternatively, the input data identification information may be composed of a file path and timestamp information of the file. Alternatively, the input data identification information may be a file path and version information of the file.

[0050] The output data identification information may be, for example, a file path in a storage device where the output data is stored as a file. Alternatively, the output data identification information may be composed of a file path and timestamp information of the file. Alternatively, the output data identification information may be a file path and version information of the file.

[0051] Next, the third collector 24 transmits the process information and the file access information to the storage device 30. Thereafter, the storage device 30 generates and stores process and file history information 32 using the process information and the file access information acquired (received) from the third collector 24.

[0052] In the example of FIG. 4, when the job management number is "1003", the first collection unit 12 collects the job management number information "1003" and the job script identification information "Job1.sh" to generate the first job information. The second collection unit 21 collects the job management number information "1003" and the program identification information "prog1", "prog2", and "prog3" to generate the second job information. After that, the storage device 30 generates and stores the above-mentioned job process history information (first history information) 31 based on the job management number information "1003" that is common to the acquired first job information and second job information.

[0053] Therefore, the job script file ("Job1.sh") used when generating the research data is associated with the executed processes ("prog1", "prog2", "prog3"), so that the user can easily create a job script file equivalent to the job script file "Job1.sh" by using the process file history information 32.

[0054] [Device operation] Next, the operation of the history management system in the embodiment will be described with reference to FIG. 5. FIG. 5 is a diagram for explaining an example of the operation of the history management system. In the following description, the diagram will be referred to as appropriate. Also, in the embodiment, the history management method is implemented by operating the history management system. Therefore, the description of the history management method in the embodiment will be replaced with the following description of the operation of the history management system.

[0055] 5, the first collection unit 12 first collects job control number information representing the control number of the job and job script identification information identifying the job script from the job queue 11, which sequentially assigns submitted jobs to one of multiple server computers 20, and generates first job information (step A1). Next, the first collection unit 12 transmits the generated first job information to the storage device 30 (step A2).

[0056] Next, the second collection unit 21 collects job control number information and program identification information for identifying a program corresponding to the process from the server computer 20 that executes the process based on the job, and generates second job information (step A3). Next, the second collection unit 21 transmits the generated second job information to the storage device 30 (step A4).

[0057] Next, the storage device (storage unit) 30 acquires (receives) the first job information and the second job information (step A5). Next, the storage device 30 generates and stores the first history information (job / process history information) 31 using the first job information and the second job information (step A6).

[0058] [Effects of the embodiment] As described above, according to the embodiment, the relationship between the job script file and the process (program) of the process file history information is clarified, so that it is possible to support the reproduction of research data in research using an HPC system. In other words, a user can easily create a job script file that can reproduce the research data desired by the user without having to analyze multiple process file history information.

[0059] [program] The program in the embodiment may be a program that causes a computer 10 that executes a job queue that sequentially assigns submitted jobs to any of a plurality of server computers to execute steps A1 to A2 shown in Fig. 5. By installing and executing this program on the computer 10, the first collection unit 12 in the embodiment can be realized. Also, the program in the embodiment may be executed by a computer system constructed by a plurality of computers.

[0060] Furthermore, the program in the embodiment may be a program that acquires a process start command from a job queue that sequentially assigns an input job to one of a plurality of server computers, and causes the server computer 20 that executes a process based on the job to execute steps A3 to A4 shown in Fig. 5. By installing and executing this program in the server computer 20, the second collection unit 21 in the embodiment can be realized.

[0061] Furthermore, the program in the embodiment may be a program that causes the storage device (storage unit) 30 to execute steps A5 to A6 shown in Fig. 5. By installing this program in the computer of the storage device 30 and executing it, the storage device 30 in the embodiment can be realized.

[0062] [Physical configuration] Here, the computer 10, server computer 20, and storage device 30 that realize the history management system by executing the program in the embodiment will be described with reference to Fig. 6. Fig. 6 is a diagram for explaining an example of the computer, server computer, and storage device that realize the history management system in the embodiment. The computer 110 shown in Fig. 6 corresponds to the computer 10, server computer 20, and storage device 30, respectively.

[0063] 6, the computer 110 includes a CPU (Central Processing Unit) 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader / writer 116, and a communication interface 117. These components are connected to each other via a bus 121 so as to be able to communicate data with each other. Note that the computer 110 may include a GPU or an FPGA in addition to the CPU 111 or instead of the CPU 111.

[0064] The CPU 111 loads a program in the embodiment, which is composed of a group of codes and is stored in the storage device 113, into the main memory 112, and executes each code in a predetermined order to perform various calculations. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory).

[0065] Moreover, the program in the embodiment is provided in a state stored in a computer-readable recording medium 120. Note that the program in the embodiment may be distributed on the Internet connected via the communication interface 117.

[0066] Specific examples of the storage device 113 include a hard disk drive and a semiconductor storage device such as a flash memory. The input interface 114 mediates data transmission between the CPU 111 and input devices 118 such as a keyboard and a mouse. The display controller 115 is connected to a display device 119 and controls the display on the display device 119.

[0067] Data reader / writer 116 mediates data transmission between CPU 111 and recording medium 120, reads programs from recording medium 120, and writes processing results in computer 110 to recording medium 120. Communication interface 117 mediates data transmission between CPU 111 and other computers.

[0068] Specific examples of recording medium 120 include general-purpose semiconductor storage devices such as CF (Compact Flash (registered trademark)) and SD (Secure Digital), magnetic recording media such as a flexible disk, or optical recording media such as a CD-ROM (Compact Disk Read Only Memory).

[0069] In addition, the computer 10, the server computer 20, and the storage device 30 in the embodiment can be realized by using hardware corresponding to each part, for example, electronic circuits, instead of a computer with a program installed. Furthermore, the computer 10, the server computer 20, and the storage device 30 may be realized in part by a program and the remaining part by hardware. In the embodiment, the computer is not limited to the computer shown in FIG. 6.

[0070] Although the invention has been described above with reference to the embodiments, the invention is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the invention within the scope of the invention. [Industrial Applicability]

[0071] According to the above description, it is possible to support the reproduction of research data in research using an HPC system, and is useful in fields where research is conducted using an HPC system. [Explanation of symbols]

[0072] 10. Computers 11 Job Queue 12 First collection section 20, 20a, 20b, 20c Server computers 21, 21a, 21b, 21c Second collection section 22 Process 23 OS (Operating System) 24 Third Collection Section 30 Storage device (storage unit) 100 Systems 110 Computer 111 CPU 112 Main memory 113 Storage device 114 Input Interface 115 Display Controller 116 Data Reader / Writer 117 Communication Interface 118 Input Devices 119 Display device 120 Recording media 121 Bus

Claims

1. a first acquisition means for acquiring job control number information representing a control number of a job and job script identification information identifying a job script from a job queue which sequentially assigns an input job to one of a plurality of server computers, and generating first job information; a second collection means for collecting the job control number information and program identification information for identifying a program corresponding to the process from the server computer that executes a process based on the job, and generating second job information; a storage unit that generates and stores first history information using the first job information acquired from the first collection unit and the second job information acquired from the second collection unit; A provenance management system having:

2. the storage means stores second history information having the program identification information, input data identification information for identifying input data input to the program, output data identification information for identifying output data output from the program, and information representing an input / output relationship between the input data and the output data for the program; The system of claim 1 .

3. the first collection means is provided in a computer that executes the job queue, The second collection means is provided in an operating system running on the server computer. The system according to claim 1 or 2 .

4. obtaining job control number information representing a control number of the job and job script identification information identifying a job script from a job queue that sequentially assigns an input job to one of a plurality of server computers, and generating first job information; acquiring the job control number information and process identification information for identifying the process from a server computer that executes a process based on the job, and generating second job information; generating first history information using the first job information and the second job information and storing the first history information in a storage device; Provenance management methods.

5. A computer having a job queue that sequentially assigns submitted jobs to one of a plurality of server computers, acquiring job control number information indicating a control number of the job and job script identification information identifying a job script from the job queue, generating first job information, and transmitting the first job information to a storage device; program.

6. A process start command is acquired from a job queue that sequentially assigns submitted jobs to one of a plurality of server computers, and the server computer executes a process based on the job. acquiring job control number information indicating a control number of the job and process identification information for identifying the process, generating second job information, and transmitting the second job information to a storage device; program.

7. A computer having a job queue which sequentially assigns submitted jobs to one of a plurality of server computers, the server computer which acquires a process start command from the job queue and executes a process based on the job, and a storage device electrically connected thereto receiving, from the computer, first job information generated using job control number information representing a control number of the job and job script identification information identifying a job script, both of which are collected from the job queue; receiving second job information from the server computer, the second job information being generated using job control number information representing a control number of the job and process identification information identifying the process, the second job information being collected from the server computer; generating first history information using the received first job information and the received second job information and storing the first history information; storage device.

Images