Information processing device and information processing method, computer program, and information processing system

Abstract

Provided is an information processing device that manages files related to data analysis.　The information processing device comprises: a reception unit that receives designation of a file from a user; and a control unit that causes a display unit to display a data lineage related to a plurality of files used in the process of generating the designated file. The reception unit displays, in a second pane of a screen, a file recorded under a directory designated by the user in a first pane of the screen, and receives designation of a file from the user. The control unit causes the display unit to display, in a third pane, the data lineage related to the plurality of files used in the process of generating the designated file.

Description

Information processing device, information processing method, computer program, and information processing system

[0001] The technology disclosed in this specification (hereinafter referred to as "the present disclosure") relates to an information processing device, an information processing method, a computer program, and an information processing system that perform information processing related to data analysis work.

[0002] Data analysis work involves exploratory testing, often involving multiple analytical methods and numerous parameter changes within each method. Reproducing past data analyses is a common need. To obtain the same analytical results, it is necessary to accurately reproduce all of the data sets, analytical code, commands, and parameters involved in the previous analysis steps.

[0003] Although version control technologies for programs and data already exist, these technologies were not designed for data analysis, which involves trial and error, and there are many challenges in applying them to data analysis work. Existing version control tools such as Git, Subversion, and DVC (Data Version Control) are used to reproduce code and data from past trials, and manage versions at the function level of addition, modification, and deletion, but do not address the need to reproduce data analysis results.

[0004] JP 2017-525039 A JP 2020-46993 A

[0005] An object of the present disclosure is to provide an information processing device, an information processing method, a computer program, and an information processing system that manage files related to data analysis.

[0006] The present disclosure has been made in consideration of the above-mentioned problems, and a first aspect thereof is an information processing device including: a reception unit that receives a file specification from a user; and a control unit that displays data lineage related to multiple files used in the process of generating the specified file on a display unit.

[0007] The receiving unit displays, in a second pane of the screen of the display unit, files recorded under a directory specified by a user in a first pane of the screen, and receives a file specification from the user in the second pane.The control unit then causes the display unit to display, in a third pane of the screen, data lineage related to multiple files used in the process of generating the specified file.

[0008] The process of generating the specified file includes a plurality of sections, and the data lineage includes a plurality of section lineages defined for each section.

[0009] In each interval, an output file is generated that stores the resulting dataset obtained by executing a command and processing the dataset stored in the input file. The interval lineage includes a list of file IDs for the input files in the corresponding interval, a list of parameters for the execution command, and a list of file IDs for the output files. The parameters for the execution command include the file IDs for the code file, input file, and output file. The file ID is a hash value of the file or a value that can identify the version of the file.

[0010] A second aspect of the present disclosure is an information processing method including: a receiving step of receiving a file specification from a user; and a control step of displaying data lineage related to multiple files used in the process of generating the specified file on a display unit.

[0011] Furthermore, a third aspect of the present disclosure is a computer program written in a computer-readable format to cause a computer to function as: a reception unit that receives a file specification from a user; and a control unit that displays, on a display unit, data lineage related to multiple files used in the process of generating the specified file.

[0012] A computer program according to a third aspect of the present disclosure defines a computer program written in a computer-readable format to perform predetermined processing on a computer. The computer program can be provided to a computer capable of executing various program codes in a computer-readable format via a storage medium or communication medium, such as an optical disk, a magnetic disk, or a semiconductor memory, or a communication medium such as a network. By installing the computer program according to the third aspect of the present disclosure on a computer via any of these media, a cooperative effect is exerted on the computer, and the same effects as those of the information processing device according to the first aspect of the present disclosure can be obtained.

[0013] A fourth aspect of the present disclosure is an information processing device including: a section lineage creation unit that creates section lineage for each section that executes a command to process a dataset stored in an input file and generate an output file that stores the dataset resulting from the processing.

[0014] The section lineage creation unit creates a section lineage including a list of file IDs of input files in the corresponding section, a list of parameters of the execution command, and a list of file IDs of output files. The file IDs consist of a hash value of the file or a value that can identify the version of the file. The parameters of the execution command include the file IDs of the code file, input file, and output file.

[0015] The section lineage creation unit associates a section lineage file describing the section lineage with the output file. Specifically, the section lineage creation unit names the section lineage file using a naming rule that includes the file ID of the output file.

[0016] A fifth aspect of the present disclosure is an information processing method, comprising: a section lineage creation step of creating section lineage for each section in which a command is executed to process a dataset stored in an input file and generate an output file storing the dataset resulting from the processing.

[0017] A sixth aspect of the present disclosure is a computer program written in a computer-readable format to cause a computer to function as: a section lineage creation unit that creates section lineage for each section that executes a command to process a dataset stored in an input file and generate an output file storing the dataset resulting from the processing.

[0018] A seventh aspect of the present disclosure is an information processing system including: a recording device including a section lineage creation unit that creates section lineage for each section that executes a command to process a dataset stored in an input file and generate an output file storing the dataset resulting from the processing; and a reproduction device including: a reception unit that receives file designation from a user; and a control unit that displays data lineage related to multiple files used in the process of generating the designated file on a display unit.

[0019] However, the term "system" used here refers to a logical collection of multiple devices (or functional modules that realize specific functions), regardless of whether each device or functional module is contained within a single housing. In other words, both a single device consisting of multiple parts or functional modules and a collection of multiple devices are considered "systems."

[0020] According to the present disclosure, it is possible to provide an information processing device, an information processing method, a computer program, and an information processing system that manage files used in data analysis, reconstruct a data lineage showing multiple files used in the process of generating a file specified by the user, and present the data lineage to the user.

[0021] It should be noted that the effects described in this specification are merely examples, and the effects brought about by the present disclosure are not limited to these. Furthermore, the present disclosure may also bring about additional effects in addition to the effects described above.

[0022] Further objects, features, and advantages of the present disclosure will become apparent from the following detailed description based on the embodiments and accompanying drawings.

[0023] FIG. 1 is a diagram showing an example of data lineage in data analysis. FIG. 2 is a diagram showing an example of a simplified notation for the data lineage shown in FIG. 1. FIG. 3 is a diagram showing an example of a group of files related to data analysis. FIG. 4 is a diagram showing an example of a simplified notation for data lineage after file changes and file additions. FIG. 5 is a diagram showing an example of a group of files related to data analysis (after file additions and changes). FIG. 6 is a diagram showing a list of files stored in a storage directory before and after a version change. FIG. 7 is a diagram showing the definition of section lineage. FIG. 8 is a diagram showing a specific example of how sections are divided and section lineage definitions. FIG. 9 is a diagram showing how section lineage files, which describe section lineage defined in processing steps, are linked to output files. FIG. 10 is a diagram showing how the section lineage file for section i is traced to the section lineage file for section i-1. FIG. 11 is a diagram showing the procedure for reconstructing the data lineage for the entire data analysis from analysis result files. FIG. 12 is a diagram showing the procedure for reconstructing the data lineage for the entire data analysis from analysis result files. FIG. 13 is a diagram showing a procedure for reconstructing data lineage for the entire data analysis from an analysis result file. FIG. 14 is a diagram showing a procedure for reconstructing data lineage for the entire data analysis from an analysis result file. FIG. 15 is a diagram showing an example of performing preprocessing on a dataset and outputting an intermediate file. FIG. 16 is a diagram showing an example of managing files using a version control system. FIG. 17 is a diagram showing an example configuration of an experiment management system 1700 to which the present disclosure is applied. FIG. 18 is a diagram showing an example screen display of a recording device. FIG. 19 is a flowchart showing a processing procedure for recording input files, output files, commands, parameters, and section lineage files in the storage device 1730 when a command is executed. FIG. 20 is a diagram showing a specific example of a section lineage file. FIG. 21 is a diagram showing an example screen configuration of a GUI used to reproduce data analysis. FIG. 22 is a flowchart showing a processing procedure for reconstructing and visually displaying data lineage. FIG. 23 is a flowchart showing a processing procedure for restoring files.FIG. 24 is a diagram showing an example of the hardware configuration of the information processing device 2000.

[0024] Hereinafter, embodiments of the present disclosure will be described in the following order with reference to the drawings.

[0025] A. Trial and error in data analysis B. Presentation of data lineage C. Complete reproduction of data analysis C-1. Distinguishing between file versions C-2. Defining lineage for each section C-3. Method for constructing lineage for the entire data analysis C-4. Comparison with prior art D. Examples D-1. System configuration D-2. Recording device D-3. Operation of the recording device D-4. Reproduction device D-5. Operation of the reproduction device D-5-1. Reconstruction of data lineage D-5-2. Restoration of files E. Example configuration of information processing device

[0026] A. Trial and error in data analysis For example, consider the trial and error involved in data analysis to predict the sales of an ice cream shop.

[0027] In the first trial, sales were predicted using regression analysis with past temperature data and ice cream sales data, but the prediction accuracy was low. Therefore, in the second trial, precipitation data was added to predict sales. In this case, the precipitation data file was added to the input file, and the program was modified to read the precipitation data and add explanatory variables to the prediction model so that the precipitation information could be used in the prediction. In the third trial, sales forecasts were made by adding information about whether the day was a weekday or a holiday. In the fourth trial, sales forecasts were made by adding exchange rate information. In the fifth trial, sales forecasts were made by adding the Nikkei average stock price. In short, the data set used for analysis increased with each trial, and the program had to be modified each time the data set changed.

[0028] Here, the prediction accuracy improved in the second and third trials, but actually decreased in the fourth and fifth trials. Furthermore, in the fourth trial, it was discovered that there was an error in the sales data used in the first three trials, which meant that the prediction model in the first three trials was also incorrect. Therefore, in the sixth trial, the system returned to the third trial, which was expected to have high prediction accuracy, and made a sales forecast using the sales data corrected in the fourth trial. In the seventh trial, the analysis method was changed from multiple regression analysis to deep learning, and a sales forecast was made. In other words, as the number of trials increases, the analysis code may be overwritten or the dataset may be modified later.

[0029] As can be seen from the above example, analysis codes, parameters, and analysis methods are frequently added and updated during the trial-and-error process. Actual data analysis work involves many more processing steps, and the number of analysis code files and dataset files used is significantly greater. Naturally, the number of trial-and-error attempts is also significantly greater. This results in the accumulation of numerous dataset files, analysis code files, and analysis result files used during the trial-and-error process. In addition, many intermediate files are often generated by processing the input data in some way.

[0030] In short, in data analysis that involves trial and error, the numerous dataset files used in the trial and error process, the intermediate files generated, the analysis code that has been updated many times, and the analysis result files for each trial accumulate in a disorganized working directory. Many of the analysis result files can no longer be reproduced as is because the analysis code was later overwritten or the dataset was later modified. This makes it difficult to correctly perform actions such as "returning to the state of a past trial and then trying a different method" from a certain analysis result file.

[0031] To avoid disorganizing files used during the trial-and-error process, a separate directory can be created for each trial, storing all files involved in the trial, allowing the status of each trial to be managed. However, this management method is extremely time-consuming and places a heavy burden on analysts. Furthermore, this management method requires copying large dataset files for each trial, which can lead to storage capacity pressure. Of course, a management method could be considered in which dataset files are not copied each time, and only newly saved dataset files that have been changed are saved, but this would require separate management of which dataset file corresponds to which trial.

[0032] B. Presentation of data lineage Ideally, analysts should be able to easily and accurately reproduce and edit the analysis results obtained through several attempts, without having to organize each file involved in the data analysis.

[0033] For example, consider a data analysis that sequentially executes processing steps 1 to 4, as shown in Figure 1. However, in each processing step, a command specifying an analysis code and parameters is executed. The parameters here include the paths of input and output files for the dataset input to the processing and the dataset output by executing the command, parameters that determine the type of preprocessing, and numerical values ​​used in the preprocessing calculations.

[0034] Processing step 1 executes the analysis code "004.py" to process the dataset stored in the input file "aaa.csv," and outputs the intermediate file "aab.csv" into which the generated dataset is written. Processing step 1 is performed by an analyst inputting command 1 into the data analysis device, specifying the path to the analysis code file "004.py" and the paths of each parameter, including the input file "aaa.csv" and the intermediate file "aab.csv," and then executing command 1 on the data analysis device. Note that commands can be input to the data analysis device by keying in the paths of each file on a command prompt screen, or by specifying the pathnames of each file on a GUI (Graphical Use Interface) screen (the same applies below).

[0035] Furthermore, processing step 2 executes analysis code "007.py" to process the dataset stored in input file "ccc.csv" and outputs intermediate file "ccd.csv" into which the generated dataset is written. Processing step 2 is performed by executing command 2 on the data analysis device, which specifies the path of the analysis code file "007.py" and the paths of each parameter including the input file "ccc.csv" and the intermediate file "ccd.csv".

[0036] Furthermore, processing step 3 uses the intermediate file "aab.csv" output from processing step 1 and the intermediate file "ccd.csv" output from processing step 2 as input files, executes analysis code "010.py" to process the datasets stored in these input files, and outputs intermediate file "eee.csv" into which the generated dataset is written. Processing step 3 is performed by executing a command on the data analysis device that specifies the path to the analysis code file "010.py," the input files "aab.csv" and "ccd.csv," and the paths of each parameter including the intermediate file "eee.csv."

[0037] Processing step 4 then uses the intermediate file "eee.csv" output from processing step 3 as an input file, executes analysis code "011.py" to process the datasets stored in these input files, and outputs an analysis result file "AAA.xlsx" into which the generated dataset is written. Processing step 4 is performed by executing a command on the data analysis device that specifies the path to the analysis code file "011.py" and the paths of each parameter including the input file "eee.csv" and the analysis result file "AAA.xlsx."

[0038] In this specification, when a dataset output by a certain process is used as input for the next process, the dataset file is referred to as an "intermediate file," and the output file of the final process of data analysis is referred to as an "analysis result file."

[0039] When one trial is executed, the used input file "aaa.csv", input file "ccc.csv", generated intermediate file "aab.csv", intermediate file "ccd.csv", intermediate file "eee.csv", executed analysis code file "004.py", analysis code file "007.py", analysis code file "010.py", analysis code file "011.py", and the analysis result file "AAA.xlsx" output by executing the final process 4 are accumulated in the working directory without being organized.

[0040] Ideally, it would be desirable to be able to easily and accurately reproduce and edit the analysis result file "AAA.xlsx" without having to specifically organize each file involved in the data analysis shown in Figure 1. Specifically, when the user simply specifies the analysis result file "AAA.xlsx," files required in the data analysis process would be automatically copied to any location specified by the user, and data lineage would be presented that shows the processes performed up to the generation of this analysis result file "AAA.xlsx," the order in which each process was performed, and the input and output files for each process.

[0041] In the following description, for convenience, each process in the data analysis will be represented by a simplified notation of "input file name → analysis code file name → output file name (or intermediate file name)." For example, the data lineage shown in FIG. 1 is represented by a simplified notation as shown in FIG. 2. In this specification, commands are described only as commands that execute analysis code, so the meaning does not change due to such simplified notation of data lineage. Also, in this embodiment, for simplicity of description, the input file and intermediate file are fixed to CSV files, the analysis result file to XLSX files, and the analysis code executed in each process to py (Python) files. However, each file may be in any file format, and the present disclosure is not limited to any specific file format.

[0042] C. Complete Reproduction of Data Analysis This section explains how to automatically identify the files required in the data analysis process by simply specifying the analysis result file.

[0043] To reproduce a data analysis, all you need to do is uniquely identify all data sets, commands, parameters, and analysis codes related to the data analysis, and know the order in which each process is executed. Creating and managing the data lineage of the analysis result file makes it possible to extract the files necessary to derive the analysis result file from all the files used and generated during the trial and error process. Figure 3 shows the files necessary to derive the analysis result file "AAA.xlsx" from the files related to the data analysis (working directory) circled.

[0044] C-1. Distinguishing File Versions As attempts are made, it is common for the analysis code to be overwritten or the dataset to be modified later. For example, suppose that after obtaining the analysis result file "AAA.xlsx," the following changes (1) to (4) are made, and the analysis result file "BBB.xlsx" is obtained in the most recent data analysis.

[0045] (1) Correct and overwrite the errors in the input file "ccc.csv". (2) Add the input file "fff.csv" and the analysis code "009.py", run the analysis code "009.py" to process the dataset stored in the input file "fff.csv", and output the intermediate file "ffg.csv" into which the generated dataset is written. (3) Modify the analysis code "010.py" so that it further reads the intermediate file "ffg.csv" and executes the processing. (4) Modify the analysis code "011.py".

[0046] Figure 4 shows a simplified representation of the data lineage after the above changes (1) to (4) have been made. Figure 4 also lists the changes from the data lineage shown in Figure 2. Figure 5 also lists, circled, the files needed to derive the analysis result file "BBB.xlsx" from the most recent data analysis in the working directory after the above changes (1) to (4) have been made. Figure 5 also lists the differences from the files shown in Figure 3. Comparing Figures 4 and 5 with the data lineage shown in Figure 2 and the files shown in Figure 3, the input file "ccc.csv" is different, the input file "fff.csv," the analysis code file "009.py," and the intermediate file "ffg.csv" have been added, and the analysis code files "010.py" and "011.py" are different.

[0047] If you search for the file needed to derive the analysis result file "AAA.xlsx" from the working directory immediately after the analysis result file "BBB.xlsx" was output in the most recent data analysis using only the file name as a key, as can be seen by comparing Figures 3 and 5, a file with the same file name but different content from when the analysis result file "AAA.xlsx" was obtained is extracted.

[0048] In short, there is a problem in that the analysis result file cannot be faithfully reproduced simply by extracting the files included in the data lineage of the analysis result file based on the file name information.

[0049] Therefore, in this disclosure, a file ID is used to distinguish between different file versions. Specifically, a hash value of the file is created and used as the file ID. Even if the file name is the same, if the file content has changed, the hash value will change, allowing for clear distinction. Of course, information other than the file hash value may be used as the file ID as long as it can uniquely identify the file version.

[0050] Because files with changed content are required to reproduce the analysis result file, they are saved in a directory for storage at an appropriate time, such as when the analysis process is executed. Figure 6 shows an example of a list of files saved in a directory for storage before and after a version change. In the example shown in Figure 6, files originally with the same file name are differentiated into versions by file ID. Specifically, three data files with the same file name "ccc.csv" are saved, with versions differentiated by file ID, and two analysis code files with the same file name "011.py" are saved, with versions differentiated by file ID.

[0051] 6 shows an example in which the file name remains the same even when the version is changed, and two or more files with the same name exist in the same directory, for the sake of convenience, but in actual implementation, when saving a file after a version change, the file name is changed to a name different from the original file name to avoid file name collisions. Also, the correspondence between the file name in the saving directory and the original file name is managed in a settings file, and when a renamed file is called to reproduce data analysis, the settings file is referenced and the original file name is restored.

[0052] C-2. Definition of Lineage for Each Section Next, we will explain the lineage creation method according to the present disclosure. First, please fully understand that the lineage according to the present disclosure does not require the user to create a lineage configuration file or add lineage code to a program.

[0053] In this disclosure, the lineage is constructed by analyzing the execution of user commands. With this approach, it is difficult to automatically determine whether the output obtained as a result of the user's command execution is an analysis result file or an intermediate file without the user's assistance. In other words, it is difficult to determine the end of the analysis process. Therefore, in this disclosure, we have decided to represent the lineage in the form of a linked list that can be added sequentially.

[0054] Data analysis is realized by linking multiple processing steps, each of which is a unit of "data input (inputting data from an input file) → processing (executing analysis code to process the input data) → data output (outputting the processed data to an output file)" (see, for example, Figures 1 and 2). Each processing step is performed by executing a command that specifies parameters such as analysis code and input / output files. Therefore, in this disclosure, the data is divided into intervals, each of which is a unit of "data input → processing → data output," and a lineage for each interval (hereinafter referred to as "interval lineage") is created. Then, in this disclosure, the interval lineage is used to create a lineage for the entire data analysis. Note that in this specification, when the term "data lineage" or simply "lineage" is used, it refers to the lineage of the entire data analysis and is distinguished from "interval lineage."

[0055] A definition of the section lineage is shown in Figure 7. The section lineage is made up of three storage areas.

[0056] 7, the first recording area designated by the reference numeral 701 is an input file ID list recording area, which records a list of file IDs of all input files in the target section. The file ID is an ID that distinguishes between different file versions, and is, for example, a hash value of the file (as described above).

[0057] The second recording area, designated by the symbol 702, is an execution command and parameter list recording area. This area records all execution commands in the target section and the parameters passed to those commands. The parameters here include the file IDs of the preprocessing code file and analysis code file executed by the command, parameters that determine the type of processing, and numerical parameters used in processing calculations. The execution command is executed by receiving a parameter management file that lists parameters, including the file IDs of the code file, input file, and output file. The parameter management file is a file that manages parameters using methods other than command line arguments. For example, Json and Yaml are common formats for managing parameters other than command line arguments.

[0058] The third recording area, denoted by reference numeral 703, is an output file ID list recording area, in which the file IDs of all output files in the target section are recorded.

[0059] In principle, a section lineage can have file IDs of input and output files, execution commands, and parameters, a number of which is a positive integer including 0. Furthermore, for the sake of convenience in this specification, when showing specific examples of section lineage, only the "file ID of the execution code file executed by the command" is recorded in the execution command and parameter list recording area 702, except in special cases.

[0060] For ease of understanding, the method of dividing sections and the definition of section lineage will be specifically explained with reference to FIG. 8, using as an example the data lineage in the data analysis that leads to the analysis result "AAA.xlsx" shown in simplified form in FIG. 2.

[0061] If the data lineage shown in Figure 2 is divided into four sections, from processing step 1 to processing step 4, with "data input from input file → processing → data output to output file" as one unit, then section lineage is created for each section of each processing step according to the definition shown in Figure 7. At this time, the "section lineage file" that describes the section lineage is managed by linking it to the output file of that section.

[0062] Specifically, a section lineage file 1 describing the section lineage of processing step 1 is created and linked to the output file "aab.csv" of processing step 1. In FIG. 8 , the linking is indicated by a dotted line connecting the output file "aab.csv" and section lineage file 1. Similarly, a section lineage file 2 describing the section lineage of processing step 2 is created and linked to the output file "ccd.csv" of processing step 2. Furthermore, a section lineage file 3 describing the section lineage of processing step 3 is created and linked to the output file "eee.csv" of processing step 3. Furthermore, a section lineage file 4 describing the section lineage of processing step 4 is created and linked to the output file "AAA.xlsx" of processing step 4.

[0063] By linking the "section lineage file" that describes the section lineage with the output file for that section, when an output file is selected, the section lineage file for the section that output the output file is determined to be first.

[0064] The definition of section lineage for each section and the linking of section lineage files to output files are performed sequentially for each section when a processing step is executed. For example, in processing step 1, command 1 is executed, specifying the path of the analysis code file "004.py" and the paths of each parameter, including the input file "aaa.csv" and the intermediate file "aab.csv." When the data set resulting from the processing is written to the intermediate file "aab.csv" as the output file, the section lineage for processing step 1 is defined and the section lineage file 1 is linked to the intermediate file "aab.csv." Of course, the definition of section lineage for all processing steps and the linking of section lineage files to output files may be performed collectively when the entire data analysis is completed.

[0065] In this embodiment, the linking relationship between the section lineage file and the output file is clarified by using a naming convention in which the file name of the section lineage file is "output file ID of the relevant section"_lineage.json. "json (JavaScript Object Notation)" is a data format created with reference to how objects are written in the JavaScript programming language. In this embodiment, the section lineage file is in the json format, but any other file format may be used. However, the linking relationship between the section lineage file and the output file may be clarified using a different naming convention, or the linking relationship between the section lineage file and the output file may be clarified by means other than the file name.

[0066] FIG. 9 shows how the section lineage is defined and the section lineage file and output file are linked in processing step 4 according to the definition shown in FIG.

[0067] In processing step 4, command 4 is executed, specifying the path of the analysis code file "011.py" and the paths of each file including the input file "eee.csv" and the analysis result file "AAA.xlsx", and the processing result dataset is written to the analysis result file "AAA.xlsx".

[0068] When the hash value of a file is used as the file ID, the file ID "ACA3F1F...04" of the input file "eee.csv" and the file ID "CCD2A8B...D3" of the analysis code file "011.py" are already determined at the start of processing step 4. In addition, when command 4 is executed and the data set analysis result file "AAA.xlsx" of the processing results is stored, the file ID "FD2D5AB...19" of the analysis result file "AAA.xlsx" is determined at the time it is written out. Then, the file ID "ACA3F1F...04" of the input file "eee.csv" is recorded in the input file ID list recording area, the file ID "CCD2A8B...D3" of the analysis code file "011.py" is recorded in the execution command and parameter list recording area, and the file ID "FD2D5AB...19" of the analysis result file "AAA.xlsx" is stored in the output file ID list recording area, thereby creating the section lineage in processing step 4. In addition, the section lineage file "FD2D5AB...19_lineage.json" that describes this section lineage is linked to the analysis result file "AAA.xlsx."

[0069] The method of indicating the linking relationship between files is not particularly limited. For example, a lookup table or a setting file that describes the linked files may be prepared, or the linking relationship may be indicated by the file name.

[0070] C-3. Method for Reconstructing the Lineage of an Entire Data Analysis Section C-2 above explained how the lineage of an entire data analysis is defined by dividing it into intervals, each unit of which is "data input → processing → data output," and how interval lineage files containing the interval lineage are linked to the output files of those intervals. Next, Section C-3 explains how to reconstruct the lineage of an entire data analysis by linking the interval lineages based on the interval lineage files linked to the output files of each interval.

[0071] Since the interval lineage file that records the interval lineage and the output file for that interval are linked, when a certain file is specified, it is possible to identify the interval lineage file of the processing step that output that file. Furthermore, if it is possible to link the interval lineages created for each interval, it is possible to reconstruct the lineage of the entire data analysis.

[0072] Here, we focus on some sections of a series of data analysis processes, for example, the i-th section (hereinafter referred to as "section i") and the i-1-th section (hereinafter referred to as "section i-1"). Section i-1 is a section that was executed before section i, and the intermediate file output from section i-1 becomes the input file for section i. If the section lineage file for section i-1 can be traced from the section lineage file for section i, we can say that section i-1 and section i have been linked.

[0073] FIG. 10 shows the section lineage of section i-1 on the left side and the lineage of section i on the right side, illustrating how the section lineage file of section i is traced to the section lineage file of section i-1.

[0074] Suppose a section lineage file for section i is given. The input file ID "ACA3F1F...04" recorded in the input file ID list recording area of ​​the section lineage file for section i is the output file ID of the section executed before section i. Therefore, the output file ID list recording areas of all section lineage files can be checked to search for a section lineage file with an output file ID that matches "ACA3F1F...04". The one found is the section lineage file for section i-1.

[0075] To link the section lineages, opening all the section lineage files and searching their output file ID list recording areas would result in unnecessary processing. For example, in this embodiment, a file naming convention is used in which the file name of the section lineage file linked to the output file of each section includes the output file ID of each section. In the example shown in Figure 10, if the section lineage file name for section i-1 includes "ACA3F1F...04," which is the input file ID recorded in the input file ID list recording area for section i, the section lineage file for section i-1 can be easily found from the directory, eliminating the need for the unnecessary processing of opening the section lineage file.

[0076] Next, when the analysis result file "AAA.xlsx," which is the final output of the data analysis whose data lineage is shown in Figure 2, is selected, the procedure for reconstructing the data lineage of the entire data analysis up to the generation of this analysis result file "AAA.xlsx" will be described with reference to Figures 11 to 14. Reconstructing the data lineage uses section lineage files linked to the output files of each processing step. In this description, it is assumed that the file names of the section lineage files linked to the output files of each section are implemented to match the output file IDs of each section.

[0077] First, the section lineage of processing step 4, which generated the selected file "AAA.xlsx," is reconstructed. Specifically, the section lineage file 004 linked to the analysis result file "AAA.xlsx" is obtained, and the file IDs of the input file and the analysis code file are obtained from the respective recording areas of this section lineage file 004. Then, the input file "eee.csv" and the analysis code "011.py," which have matching file IDs, are obtained, and the section lineage of processing step 4 is reconstructed as shown in FIG. 11 . Note that in FIG. 11 , for simplicity's sake, the file IDs of the input file, analysis code file, and output file in the respective recording areas of the section lineage file are abbreviated as "input file ID004," "analysis code file ID004," and "output file ID004," respectively (the same applies to FIGS. 12 to 14 ). The file names of the input file and the analysis code file can be obtained by referencing the configuration file (described above) that records the correspondence between file IDs and original file names. In addition, the section lineage file 004 can be obtained by searching the directory for a file with the file name "output file ID 004_lineage.json."

[0078] Next, the section lineage of processing step 3, which was executed before processing step 4 and generated the input file "eee.csv" for processing step 4, is reconstructed. First, a file with the file name "output file ID 003_lineage.json" is searched for in the directory, and the section lineage file 003 linked to the file "eee.csv" is obtained. The input file "eee.csv" for processing step 4 is also the output file (intermediate file) output from processing step 3 and has the same file ID. The file IDs of the input file and the analysis code file are obtained from the respective recording areas of this section lineage file 003. Then, the file names of the two input files, "aab.csv" and "ccd.csv," and the analysis code file name "010.py" are obtained from the configuration file (described above) that records the correspondence between file IDs and original file names, and the section lineage of processing step 3 is reconstructed as shown in FIG. 12.

[0079] Next, the section lineage of processing step 1, which was executed before processing step 3 and generated one of the input files "aab.csv" for processing step 3, is reconstructed. First, a file with the file name "output file ID 001_lineage.json" is searched for in the directory, and the section lineage file 001 linked to the file "aab.csv" is obtained. One of the input files for processing step 3, "aab.csv," is also the output file (intermediate file) output from processing step 1 and has the same file ID. Each file ID of the input file and the analysis code file is obtained from each recording area of ​​this section lineage file 001. Then, the input file file name "aaa.csv" and the analysis code file name "004.py" are obtained from the setting file (described above) that records the correspondence between file IDs and original file names, and the section lineage of processing step 1 is reconstructed as shown in FIG. 13.

[0080] The section lineage of processing step 2, which was executed before processing step 3 and generated the other input file "ccd.csv" for processing step 3, is reconstructed. First, a file with the file name "output file ID002_lineage.json" is searched for in the directory, and the section lineage file 002 linked to the file "ccd.csv" is obtained. The other input file "ccd.csv" for processing step 3 is also the output file (intermediate file) output from processing step 2 and has the same file ID. The file IDs of the input file and the analysis code file are obtained from the respective recording areas of this section lineage file 002. Then, the input file file name "ccc.csv" and the analysis code file name "007.py" are obtained from the setting file (described above) that records the correspondence between file IDs and original file names, and the section lineage of processing step 2 is reconstructed as shown in FIG. 14.

[0081] In this way, by simply specifying the analysis result file "AAA.xlsx," which is the output of the entire data analysis consisting of a series of processing steps 1 to 4, the lineage of the entire data analysis can be reconstructed as shown in Figure 2 (or Figure 8) while sequentially obtaining the section lineage files of each processing step 1 to 4.

[0082] By visually displaying the data lineage reconstructed from the analysis result files in this way, users can easily understand the files used in the corresponding trial process of the data analysis, the parameters set, the versions of the files used, etc. Users can also export files linked to the visualized data lineage.

[0083] C-4. Comparison with Prior Art Data lineage is a technology for visualizing data history. For example, a method for managing lineage information that represents one or more lineage relationships between multiple data processing programs and multiple logical data sets (see Patent Document 1) and a device for creating data lineage across multiple systems (see Patent Document 2) have been proposed. However, while these technologies can visualize data lineage, they cannot reproduce the data lineage of the entire data analysis, from the data analysis result file to the generation of the data analysis result.

[0084] During the trial-and-error process of data analysis, analysis codes, parameters, and analysis methods are frequently added and updated. This poses a technical challenge: it is difficult to reproduce the data lineage of the entire data analysis up to the generation of a certain analysis result file. This disclosure aims to manage files used in data analysis so that the data lineage of the entire data analysis can be reproduced from the analysis result file.

[0085] Version control tools such as Git, Subversion, and DVC already exist in the industry. However, these version control tools are used to reproduce code and data from past experiments, and manage versions for each function that is added, modified, or deleted. However, these tools are only used for experiment management, and do not address the need to reproduce data analysis results.

[0086] As a simple example, consider the process of executing the analysis code "004.py" to preprocess a dataset stored in the input file "aaa.csv" and outputting the intermediate file "aab.csv" into which the generated dataset is written (see FIG. 15). This process corresponds to processing step 1, which is one unit included in the series of data analysis processes shown in FIGS. 1, 2, and 8. This process is executed, for example, by using the following command:

[0087] $ python. / 004py [parameters]

[0088] However, the [parameters] portion of the above command contains the paths of input and output files for the datasets input to the process and the datasets output by executing the command, parameters that determine the type of preprocessing, numerical values ​​used in the calculations for preprocessing, etc. For example, a user (such as an analyst) can execute a command by typing the above character string on the command prompt screen of the data analysis device.

[0089] Assume that the analysis code file "004.py," the input file "aaa.csv," and the intermediate file "aab.csv" are committed when the execution of the analysis code "004.py" is completed. In this case, a version control system such as Git manages the analysis code file "004.py," the input file "aaa.csv," and the intermediate file "aab.csv" as shown in FIG. 16. However, unlike the "interval lineage" disclosed herein, it is clear that information on commands, parameters, and processing flow is not saved. For this reason, such version control systems cannot completely reproduce the entire data analysis flow. In short, existing version control systems are fundamentally different from the file management method used in the data analysis disclosed herein.

[0090] D. Example In this section D, as an example of the present disclosure, an experiment management for managing files used in data analysis on a computer that performs the data analysis will be described.

[0091] D-1. System Configuration Figure 17 shows a schematic diagram of an example configuration of an experiment management system 1700 to which the present disclosure is applied. The illustrated experiment management system 1700 is composed of a recording device 1710, a reproduction device 1720, and a storage device 1730. The experiment management system 1700 may be, for example, experiment management software that runs on a computer that performs data analysis. Of course, each of the components of the recording device 1710, the reproduction device 1720, and the storage device 1730 may also be configured as dedicated hardware.

[0092] The recording device 1710 is software that runs on, for example, a terminal. The recording device 1710 has a role of recording input files, output files, commands, parameters, and section lineage files in the storage device 1730 when commands are executed for each processing step in data analysis. The recording device 1710 includes a user interface 1711, a section lineage creation unit 1712, and a control unit 1713. The user interface 1711 allows the user to input commands and displays a command prompt. The section lineage creation unit 1712 defines section lineage and creates a section lineage file each time a command is executed. The control unit 1713 provides overall control over the operations within the recording device 1710.

[0093] The storage device 1730 serves to store all files used in the data analysis, such as input files, output files, command files, parameters, and section lineage files, and also serves to transfer files between the recording device 1710 and the reproduction device 1720.

[0094] The reproduction device 1720 reconstructs and visualizes the data lineage of the entire data analysis from the analysis result file specified by the user until the analysis result is generated. Furthermore, when the user performs an export operation, the reproduction device 1720 copies all files linked to the data lineage from the storage device 1730 to a directory specified by the user. The reproduction device 1720 includes a user interface 1721, a lineage construction / file restoration unit 1722, and a control unit 1723. The user interface 1721 accepts a data lineage reconstruction request (or a selection of an analysis result file) from the user and presents the reconstructed data lineage. The lineage construction / file restoration unit 1722 reconstructs the data lineage of the entire data analysis from the analysis result file selected by the user until the analysis result file is generated, and restores the files used until the analysis result file is generated. The control unit 1723 provides overall control over the operation of the reproduction device 1720.

[0095] The following describes the operation of each of the recording device 1710 and the reproduction device 1720, assuming that the experiment management system 1700 is implemented as experiment management software that runs on a computer that performs data analysis.

[0096] D-2. Recording Device When starting trial-and-error data analysis, the user executes the following command from the terminal of the computer that will perform the data analysis: This causes the computer to start operating as a recording device 1710.

[0097] $ source ~ / liheage / activate

[0098] 18 shows an example of a display on the screen of the recording device 1710 (a terminal screen when a computer performing data analysis is performing a recording operation), assuming that the OS (Operating System) of the computer is Linux (registered trademark).

[0099] While the computer is operating as the recording device 1710, the character string "(lineage)" is added to the beginning of the command prompt to indicate that recording is in progress. To stop recording on the computer, execute the following command:

[0100] $ deactivate

[0101] While the recording device 1710 is operating (or while the computer performing the data analysis is performing a recording operation), each time a command to execute an analysis code file is executed, i.e., each time one unit of processing step, "data input → processing → data output," is performed, the recording device 1710 records a set of input files, output files, commands, parameters, and section lineage files in the storage device 1730.

[0102] D-3. Operation of Recording Device When executing a command for each processing step in data analysis, the recording device 1710 records the input file, output file, command, parameters, and section lineage file of the relevant section in the storage device 1730. Figure 19 shows, in the form of a flowchart, the processing procedure by which the recording device 1710 performs the recording operation in the storage device 1730.

[0103] When the user executes the analysis code file, the control unit 1713 issues a command to the user interface 1711, and switches the command prompt screen to a display indicating "Processing (or command not accepted)" and does not accept any command input from the user until the series of operations to record the entire set of files for the relevant section in the storage device 1730 is completed.

[0104] The control unit 1713 then issues a command to the section lineage creation unit 1712 to acquire the executed commands and parameters for the relevant section (step S1901). The acquired commands and parameters are temporarily stored in, for example, the computer's main storage device until they are written to the execution command and parameter list recording area of ​​the section lineage in subsequent processing.

[0105] Next, the control unit 1713 issues a command to the section lineage creating unit 1712 to obtain the path of the file read when the analysis code file is executed and the path of the file output (step S1902).

[0106] Next, the control unit 1713 passes the paths of each file acquired in step S1902 to the section lineage creation unit 1712 and issues a command to calculate an input file ID, an output file ID, and a code file ID that uniquely identify the version of each file from the input file, output file, and analyzed code file (step S1903). A hash function or the like is used to calculate the file IDs. Each calculated file ID is held, for example, in the computer's main storage device until it is written to each recording area of ​​the section lineage in subsequent processing.

[0107] Next, the control unit 1713 issues a command to the section lineage creation unit 1712 to create a section lineage file for the corresponding section. The section lineage creation unit 1712 creates a new section lineage file, records each file ID calculated in step S1903 in each section lineage recording area, and then saves the file in the storage device 1730 (step S1904). In this embodiment, the section lineage creation unit 1712 names the section lineage file in accordance with the naming convention "output file ID of each section"_lineage.json (described above), but the present disclosure is not limited to a specific filename naming convention.

[0108] 20 shows an example of a section lineage file. The section lineage includes an input file ID list recording area indicated by reference numeral 2010, an execution command and parameter list recording area indicated by reference numeral 2020, and an output file ID list recording area indicated by reference numeral 2030. The execution command and parameter list recording area 2020 includes an execution code file ID, a command ("python"), and an analysis code file specification (". / 004.py"), an input file specification (". / aaa.csv"), and an output file specification (". / aab.csv") as parameters, as indicated by reference numerals 2021 to 2025, respectively.

[0109] Next, the control unit 1713 copies the input file, output file, and analysis code used in the corresponding section to the storage device 1730 under different names (step S1905). At this time, the file name of each file is changed to the respective file ID. The extension is not changed.

[0110] If a file with the same name already exists in the storage device 1730 (Yes in step S1906), the control unit 1713 does not store the file (step S1909) and ends this process.

[0111] On the other hand, if a file with the same name does not exist in the storage device 1730 (No in step S1906), the control unit 1713 stores the file in the storage device 1730 (step S1908). The control unit 1713 also records the correspondence between the file ID and the file name in the setting file (described above) that manages the correspondence between the file ID in the storage device 1730 and the original file name (step S1908), and ends this processing.

[0112] When the control unit 1713 has completed the above processing for the relevant section, it issues a command to the user interface 1711 to return the command prompt screen to a display indicating "commands being accepted" and resume accepting commands from the user.

[0113] D-4. Reproduction Device The reproduction device 1720 is a client application that has a GUI that can display and select analysis result files on the computer that performed the data analysis. Furthermore, the reproduction device 1720 reconstructs the data lineage of the entire data analysis up to the generation of the analysis result file selected via the GUI, and the above GUI has an area that displays the reconstructed data lineage in the form of a directed graph.

[0114] Fig. 21 shows an example of the screen configuration of a GUI used to reproduce data analysis, which is included in the reproduction device 1720. The GUI screen shown in Fig. 21 includes a pane on the left side of the screen indicated by reference numeral 2101, a pane on the top right side of the screen indicated by reference numeral 2102, and a pane on the bottom right side of the screen indicated by reference numeral 2103. The control unit 1723 in the reproduction device 1720 issues a command to the user interface 1721 to display the GUI screen.

[0115] Pane 2101 displays the directory tree of the computer. The directory displayed in pane 2101 can be either a directory within the computer operating as the reproduction device 1720 or a directory of a computer on the network accessed via ssh (secure shell) or the like. The directory hierarchy is expressed using indentation. In pane 2101, the user selects a working directory that contains the analysis result file for which the user wants to visualize data lineage. In the example shown in FIG. 21 , an area 2111 containing the working directory selected by the user in pane 2101 is displayed in gray.

[0116] Pane 2102 displays a list of analysis result files recorded under the working directory 2111 selected in pane 2101. In the example shown in FIG. 21 , analysis result file "AAA.xlsx" and analysis result file "BBB.xlsx" are displayed. In pane 2102, the user selects an analysis result file for which the user wishes to reconstruct and visualize data lineage that shows the processes executed up to the time of generation, the order in which each process was executed, and the input / output files of each process. In the example shown in FIG. 21 , analysis result file "AAA.xlsx" is selected in pane 2102, and area 2121 including analysis result file "AAA.xlsx" is displayed in gray.

[0117] Pane 2103 visualizes and displays the data lineage of the analysis result file "AAA.xlsx" selected in pane 2102 in the form of a directed graph. When the analysis result file "AAA.xlsx" is selected in pane 2102, the reproduction device 1720 (or a computer operating as the reproduction device 1720) reconstructs, in the background, a data lineage that shows the processes executed until the analysis result file "AAA.xlsx" was generated, the order in which each process was executed, and the input and output files of each process. Then, as indicated by the reference numeral 2132, the obtained data lineage is displayed in pane 2103.

[0118] Because the number of processing steps executed before the analysis result file "AAA.xlsx" is generated is large and complex, if the size of the data lineage becomes large, it may be possible to provide zoom-in / zoom-out buttons such as those indicated by the symbol 2133 or vertical and horizontal scroll bars (not shown) for pane 2103, so that the entire data lineage can be displayed or some of the processing steps can be enlarged.

[0119] Here, for the sake of convenience, analysis result files are listed in pane 2102, and only the analysis result files are available as options for reconstructing and visualizing data lineage. However, intermediate files (i.e., output files of each processing step) may also be listed in pane 2102 and added to the user's options.

[0120] The restore button 2131 in the upper right corner of the pane 2103 is a button for instructing the file restore operation. The user uses this restore button 2131 when, for example, he wants to export and reuse a file used in a past data analysis. Details of the file restore operation will be described later.

[0121] D-5. Operation of the Reproduction Device D-5-1. Data Lineage Reconstruction Operation The reproduction device 1720 reconstructs and visualizes the data lineage in response to the selection of an analysis result file (or an intermediate file) on the GUI screen shown in FIG. 21, for example.

[0122] 22 is a flowchart showing the processing procedure for reconstructing and visualizing data lineage by the reproduction device 1720. Hereinafter, the data lineage reconstruction operation in the reproduction device 1720 will be described with reference to FIG.

[0123] When the user selects an analysis result file on the GUI screen shown in FIG. 21, the control unit 1723 issues a command to the lineage construction / files restoration unit 1722 to create a new lineage file (for the entire data analysis) (step S2201).

[0124] Next, the control unit 1723 creates a file ID from the analysis result file (step S2202). The lineage construction / file restoration unit 1722 must calculate the file ID using the same method as the section lineage creation unit 1712 in the recording device 1710.

[0125] Next, the control unit 1723 pushes the calculated file ID onto the stack (step S2203). From this point onwards, the graph search process for constructing data lineage is performed, and is repeated until the stack is empty. In this embodiment, a stack is used to construct data lineage using a depth-first search, but a queue can be used instead if a breadth-first search is to be performed. There is no essential difference between the two search methods, so either method can be implemented.

[0126] If there are still file IDs in the stack (Yes in step S2204), the control unit 1723 extracts one file ID from the stack (step S2205).

[0127] Next, the control unit 1723 generates a section lineage file name from the file ID of the analysis result file calculated in step S2202, in accordance with the naming rules for section lineage file names. Here, the control unit 1723 must generate the file name of the section lineage file using the same naming rules (described above) as those used by the section lineage creation unit 1712 in the recording device 1710. Then, the control unit 1723 reads out the section lineage file that matches the generated file name from the storage device 1730 (step S2206).

[0128] If a section lineage file with a matching file name is found in the storage device 1730 (Yes in step S2207), the control unit 1723 reads the input file ID, output file ID, command, and parameters from each recording area of ​​the section lineage file and adds them to the new lineage file created in step S2201 above (step S2208).

[0129] Furthermore, if an input file ID exists in the section lineage file, the control unit 1723 pushes the input file ID onto the stack (step S2209). After that, the process returns to step S2204, and the reproduction device 1720 repeats the above process. By pushing the input file ID onto the stack, the input file can be linked to the section lineage of the previously executed processing step that generated it (see, for example, FIG. 10).

[0130] On the other hand, if no section lineage file with a matching file name is found in the storage device 1730 (No in step S2207), the control unit 1723 outputs an error log indicating that the corresponding section lineage file does not exist (step S2209).The control unit 1723 then issues a command to the user interface 1721 to display an error on the GUI screen (step S2210), and then terminates this processing.

[0131] If the stack becomes empty (No in step S2204), the control unit 1723 saves the lineage file (of the entire data analysis) created as described above (step S2211).The control unit 1723 then issues a command to the user interface 1722 to visually display the reconstructed data lineage of the entire data analysis (i.e., based on the created lineage file) in the form of a directed graph in the pane 2103 of the GUI screen shown in FIG. 21 (step S2212), and ends this process.

[0132] D-5-2. File Restoration Operation There is a need to export and reuse files used in past data analyses. In this section D-5-2, we will explain the file restoration operation by the reproduction device 1720. The file restoration is started in response to a user instruction to restore files via the user interface 1721. Specifically, the user triggers the file restoration operation by clicking the restore button 2131 in the upper right corner of the pane 2103.

[0133] 23 is a flowchart showing the procedure for restoring files by the reproduction device 1720. The file restoration operation in the reproduction device 1720 will be described below with reference to FIG.

[0134] When the user clicks the restore button 2131 in the upper right corner of the pane 2103, the control unit 1723 in the reproduction device 1720 issues a command to the user interface 1721, pops up an input screen for the destination path for the restored files, and waits until the user enters the path (step S2301).

[0135] Then, when the user has finished inputting the destination path on the destination path input screen, the control unit 1723 saves the input destination path (step S2302).

[0136] Next, the control unit 1723 reads the lineage file of the data lineage visualized and displayed in the pane 2103 (step S2303).

[0137] Next, the control unit 1723 copies the files indicated by the file IDs for all file IDs written in the lineage file from the storage device 1730 to the destination path specified by the user (step S2304). At this time, the control unit 1723 performs processing to restore the original file names by referring to a setting file that manages the correspondence between file IDs and file names.

[0138] If a file with the same name already exists in the path specified by the user, the control unit 1723 issues a command to the user interface 1721 to display a message to the user saying, "The file cannot be restored because a file with the same name already exists in the specified path." If the path specified by the user does not exist, the control unit 1723 creates a new directory in the specified path and performs the restoration process.

[0139] Then, when the restoration of the files is complete, the control unit 1723 issues a command to the user interface 1721 to display a message indicating that the restoration of the files has been completed (step S2305), and ends this processing.

[0140] E. Configuration Example of Information Processing Device The experiment management system 1700 shown in Fig. 17 may be configured as a single information processing device such as a personal computer, or may be configured as the same information processing device as the data analysis device (for example, the experiment management system 1700 may be experiment management software that runs on the information processing device that performs the data analysis). Alternatively, the recording device 1710, the reproduction device 1720, and the storage device 1730 may each be configured as a physically independent information processing device.

[0141] 24 shows an example of the hardware configuration of an information processing device 2000. This information processing device 2000 includes a CPU (Central Processing Unit) 2001, a ROM (Read Only Memory) 2002, a RAM (Random Access Memory) 2003, a host bus 2004, a bridge 2005, an expansion bus 2006, an interface unit 2007, an input unit 2008, an output unit 2009, a storage unit 2010, a drive 2011, and a communication unit 2013. The information processing device 2000 is configured, for example, by an information terminal such as a personal computer, a tablet, or a smartphone.

[0142] The CPU 2001 controls the overall operation of the information processing device 2000 in accordance with various programs. When performing processing with a high computational load on the information processing device 2000, it is desirable that the CPU 2001 be a multi-core CPU (e.g., Apple M1 Max, etc.), or that the information processing device 2000 further be equipped with a multi-core processor (e.g., NVIDIA's "Quadro A6000") such as a GPU (Graphics Processing Unit) or GPGPU (General-purpose computing on graphics processing units) in addition to the CPU 2001. However, hereinafter, for convenience, these will be collectively referred to simply as the CPU 2001.

[0143] The ROM 2002 stores in a nonvolatile manner programs (such as a basic input / output system) and calculation parameters used by the CPU 2001. The RAM 2003 is used to load programs to be executed by the CPU 2001 and to temporarily store parameters such as working data that change as appropriate during program execution. Programs loaded into the RAM 2003 and executed by the CPU 2001 include, for example, various application programs and an operating system (OS).

[0144] The CPU 2001, ROM 2002, and RAM 2003 are interconnected by a host bus 2004, which includes a CPU bus and the like. The CPU 2001 executes various application programs in an execution environment provided by an OS through the cooperative operation of the ROM 2002 and RAM 2003, thereby enabling various functions and services. If the information processing device 2000 is a personal computer, the OS may be, for example, Microsoft Windows (registered trademark), Unix (registered trademark), or a successor OS. Examples of application programs executed on the information processing device 2000 include the following: (1) a data analysis program that executes a command specifying an analysis code and parameters; (2) a recording program that records input files, output files, commands, parameters, and section lineage files when a command is executed for each processing step in data analysis; and (3) a reproduction program that reconstructs and visualizes the data lineage of the entire data analysis, from the analysis result file specified by the user to the generation of the analysis result.

[0145] The host bus 2004 is connected to an expansion bus 2006 via a bridge 2005. The expansion bus 2006 is, for example, a PCI (Peripheral Component Interconnect) bus or PCI Express, and the bridge 2005 is based on the PCI standard. However, the information processing device 2000 does not need to be configured so that the circuit components are separated by the host bus 2004, bridge 2005, and expansion bus 2006, and may be implemented so that almost all circuit components are interconnected by a single bus (not shown).

[0146] The interface unit 2007 connects peripheral devices such as an input unit 2008, an output unit 2009, a storage unit 2010, a drive 2011, and a communication unit 2013 in accordance with the standards of the expansion bus 2006. However, not all of the peripheral devices shown in Fig. 24 are necessarily required, and the information processing device 2000 may further include peripheral devices not shown. Furthermore, the peripheral devices may be built into the main body of the information processing device 2000, or some of the peripheral devices may be externally connected to the main body of the information processing device 2000.

[0147] The input unit 2008 is composed of an input control circuit that generates an input signal based on user input and outputs it to the CPU 2001. If the information processing device 2000 is a personal computer, the input unit 2008 may include a keyboard, mouse, and touch panel, and may also include a camera and microphone used for remote conferences and face-to-face customer service. The output unit 2009 includes display devices such as a liquid crystal display (LCD) device, an organic electroluminescence (EL) display device, and an LED (light emitting diode), as well as an audio output device such as a speaker. User prompts are issued using the input unit 2008, and a GUI screen (see, for example, FIG. 21 ) is displayed using the output unit 2009.

[0148] The storage unit 2010 stores files such as programs (applications, OS, etc.) executed by the CPU 2001 and various data. The storage unit 2010 is configured, for example, with a large-capacity storage device such as an SSD (Solid State Drive) or an HDD (Hard Disk Drive), but may also include an external storage device. The storage unit 2010 functions as the storage device 1730 of the experiment management system 1700, and may be used to store files and lineage files used in data analysis, section lineage files created for each processing step, etc.

[0149] The removable storage medium 2012 is a storage medium configured as a cartridge, such as a microSD card. The drive 2011 performs read and write operations on the loaded removable storage medium 2012. The drive 2011 outputs data read from the removable storage medium 2012 to the RAM 2003 or the storage unit 2010, and writes data on the RAM 2003 or the storage unit 2010 to the removable storage medium 2012.

[0150] The communication unit 2013 is a device that performs wireless communication via Wi-Fi (registered trademark), Bluetooth (registered trademark), or cellular communication networks such as 4G and 5G. The communication unit 2013 may also include terminals such as a Universal Serial Bus (USB) or a High-Definition Multimedia Interface (HDMI) (registered trademark), and may further include a function for performing HDMI (registered trademark) communication with USB devices such as scanners and printers, displays, and the like. Programs executed on the information processing device 2000 are installed from an external device, for example, via the communication unit 2013. An acoustic signal that is the subject of the summary generation process according to the present disclosure is captured, for example, via the communication unit 2013.

[0151] The present disclosure has been described in detail above with reference to specific embodiments. However, the present disclosure should not be construed as being limited to the above-described embodiments, and it is obvious that those skilled in the art can modify or substitute the embodiments without departing from the spirit of the present disclosure. Furthermore, the effects described in this specification are merely examples, and the effects brought about by the present disclosure are not limited thereto, and additional effects not described in this specification may exist.

[0152] The present disclosure can be applied to the management of experiments conducted in various fields, and can suitably manage files used in the process of data analysis, and reconstruct and present to the user a data lineage showing multiple files used in the process of generating a file specified by the user.

[0153] In short, the present disclosure has been described in the form of examples, and the contents of the specification should not be interpreted as limiting. To determine the gist of the present disclosure, the claims should be taken into consideration.

[0154] The series of processes described in this specification can be executed by hardware, software, or a configuration that combines hardware and software. When executing processes by software, a program recording a processing sequence related to realizing the present disclosure is installed in memory in a computer incorporated in dedicated hardware and executed. It is also possible to install the program in a general-purpose computer capable of executing various processes and execute the processes related to realizing the present disclosure.

[0155] The program can be stored in advance on a recording medium installed in the computer, such as a HDD, SSD, or ROM. Alternatively, the program can be temporarily or permanently stored on a removable recording medium such as a flexible disk, CD-ROM (Compact Disc Read Only Memory), MO (Magneto Optical) disk, DVD (Digital Versatile Disc), BD (Blu-Ray Disc (registered trademark)), magnetic disk, or USB (Universal Serial Bus) memory. Using such a removable recording medium, a program related to the realization of the present disclosure can be provided as so-called package software.

[0156] The program may also be transferred wirelessly or via a wire from a download site to a computer via a network such as a wide area network (WAN) typified by cellular, a local area network (LAN), the Internet, etc. The computer can receive the program transferred in this manner and install it in a large-capacity storage device such as an HDD or SSD within the computer.

[0157] The present disclosure may also be configured as follows.

[0158] (1) An information processing device comprising: a reception unit that receives a file designation from a user; and a control unit that causes a display unit to display data lineage related to multiple files used in the process of generating the designated file.

[0159] (2) The information processing device described in (1) above, wherein the reception unit displays files recorded under a directory specified by the user in a first pane of the screen of the display unit in a second pane of the screen and receives a file specification from the user in the second pane, and the control unit causes the display unit to display data lineage for multiple files used in the process of generating the specified file in a third pane of the screen.

[0160] (3) The information processing device according to any one of (1) or (2), wherein the process of generating the specified file includes a plurality of sections, and the data lineage includes a plurality of section lineages defined for each section.

[0161] (4) In the information processing device described in (3) above, in each section, an output file is generated that stores the resulting dataset obtained by executing a command and processing the dataset stored in the input file, and the section lineage includes a list of file IDs of the input files in the corresponding section, a list of parameters of the execution command, and a list of file IDs of the output files.

[0162] (5) The information processing device according to (4), wherein the file ID is a hash value of the file or a value that can identify the version of the file.

[0163] (6) The information processing device according to any one of (4) or (5), wherein the parameters of the execution command include file IDs of the code file, the input file, and the output file.

[0164] (7) The information processing device according to any one of (4) or (5) above, wherein the execution command is executed by receiving a parameter management file in which parameters including file IDs of the code file, input file, and output file are written.

[0165] (8) The information processing device described in any one of (3) to (7) above, further comprising a lineage construction unit that reconstructs data lineage for the entire process of generating the specified file using a section lineage file that describes the section lineage of each section, and the control unit causes the display unit to display the data lineage reconstructed by the lineage construction unit.

[0166] (9) The information processing device described in (8) above, wherein a section lineage file describing the section lineage is linked to the output file of each section, and the lineage construction unit obtains the section lineage file linked to the specified file and constructs the lineage of the section.

[0167] (9-1) The information processing device described in (9) above, wherein the section lineage file is named using a naming rule that includes the file ID of the output file, and the lineage construction unit searches for the corresponding section lineage file from the file ID of the specified file based on the command rule.

[0168] (10) The information processing device described in any one of (8) or (9) above, wherein the lineage construction unit searches for a section lineage file of a first section executed before the second section, whose output file file ID is the same as the file ID of an input file held by the section lineage file of the second section, and concatenates the section lineage file of the first section before the section lineage file of the second section.

[0169] (11) An information processing method comprising: a receiving step of receiving a file designation from a user; and a control step of displaying data lineage relating to a plurality of files used in the process of generating the designated file on a display unit.

[0170] (12) A computer program written in a computer-readable format to cause a computer to function as: a reception unit that receives a file designation from a user; and a control unit that displays, on a display unit, data lineage related to multiple files used in the process of generating the designated file.

[0171] (13) An information processing device comprising: a section lineage creation unit that creates section lineage for each section that executes a command to process a data set stored in an input file and generate an output file storing the data set resulting from the processing.

[0172] (14) The information processing device according to (13), wherein the section lineage creation unit creates a section lineage including a list of file IDs of input files in the corresponding section, a list of parameters of execution commands, and a list of file IDs of output files.

[0173] (15) The information processing device according to (14), wherein the file ID is a hash value of the file or a value that can identify the version of the file.

[0174] (16) The information processing device according to any one of (14) or (15), wherein the parameters of the execution command include file IDs of the code file, the input file, and the output file.

[0175] (16-1) The information processing device according to any one of (14) or (15) above, wherein the execution command is executed by receiving a parameter management file in which parameters including file IDs of a code file, an input file, and an output file are described.

[0176] (17) The information processing device according to any one of (14) to (16), wherein the section lineage creation unit links a section lineage file describing the section lineage to the output file.

[0177] (17-1) The information processing device according to (17), wherein the section lineage creation unit names the section lineage file using a naming rule that names a file including the file ID of the output file.

[0178] (18) An information processing method, comprising: a section lineage creation step of creating section lineage for each section in which a command is executed to process a data set stored in an input file and generate an output file storing the data set resulting from the processing.

[0179] (19) A computer program written in a computer-readable format to cause a computer to function as a section lineage creation unit that creates section lineage for each section that executes a command to process a dataset stored in an input file and generate an output file storing the dataset resulting from the processing.

[0180] (20) An information processing system comprising: a recording device including a section lineage creation unit that creates section lineage for each section that executes a command to process a dataset stored in an input file and generate an output file that stores the dataset resulting from the processing; and a reproduction device including: a reception unit that receives file designations from a user; and a control unit that displays data lineage for multiple files used in the process of generating the designated file on a display unit.

[0181] 1700...experiment management system, 1710...recording device, 1711...user interface, 1712...section lineage creation unit, 1713...control unit, 1720...reproduction device, 1721...user interface, 1722...lineage construction / file restoration unit, 1723...control unit, 1730...storage device, 2000...information processing device, 2001...CPU, 2002...ROM, 2003...RAM, 2004...host bus, 2005...bridge, 2006...expansion bus, 2007...interface unit, 2008...input unit, 2009...output unit, 2010...storage unit, 2011...drive, 2012...removable recording medium, 2013...communication unit

Claims

1. An information processing device comprising: a reception unit that receives a file specification from a user; and a control unit that displays, on a display unit, data lineage relating to multiple files used in the process of generating the specified file.

2. The information processing device of claim 1, wherein the reception unit displays files recorded under a directory specified by the user in a first pane of the screen of the display unit in a second pane of the screen and receives a file specification from the user in the second pane, and the control unit causes the display unit to display data lineage related to multiple files used in the process of generating the specified file in a third pane of the screen.

3. The information processing device according to claim 1, wherein the process of generating the specified file includes a plurality of sections, and the data lineage includes a plurality of section lineages defined for each section.

4. The information processing device of claim 3, wherein in each section, an output file is generated that stores the resulting dataset obtained by executing a command and processing the dataset stored in the input file, and the section lineage includes a list of file IDs of the input files in the corresponding section, a list of parameters of the execution command, and a list of file IDs of the output files.

5. The information processing device according to claim 4, wherein the file ID is a hash value of the file or a value that can identify the version of the file.

6. The information processing device according to claim 4, wherein parameters of the execution command include file IDs of the code file, input file, and output file.

7. The information processing apparatus according to claim 4, wherein the execution command is executed by receiving a parameter management file in which parameters including file IDs of the code file, input file, and output file are written.

8. An information processing device as described in claim 3, further comprising a lineage construction unit that reconstructs the data lineage of the entire process of generating the specified file using a section lineage file that describes the section lineage of each section, and wherein the control unit causes the display unit to display the data lineage reconstructed by the lineage construction unit.

9. The information processing device according to claim 8, wherein a section lineage file describing the section lineage is linked to the output file of each section, and the lineage construction unit obtains the section lineage file linked to the specified file and constructs the lineage of that section.

10. The information processing device according to claim 8, wherein the lineage construction unit searches for a section lineage file of a first section executed before the second section, the section lineage file of which has the file ID of an input file held by the section lineage file of the second section as its output file file ID, and concatenates the section lineage file of the first section before the section lineage file of the second section.

11. An information processing method comprising: a receiving step of receiving a file specification from a user; and a control step of displaying on a display unit data lineage relating to a plurality of files used in the process of generating the specified file.

12. A computer program written in computer-readable format to cause a computer to function as: a reception unit that receives file designation from a user; and a control unit that displays on a display unit data lineage related to multiple files used in the process of generating the designated file.

13. An information processing device comprising: a section lineage creation unit that creates section lineage for each section that executes a command to process a dataset stored in an input file and generate an output file storing the dataset resulting from the processing.

14. The information processing device according to claim 13, wherein the section lineage creation unit creates a section lineage including a list of file IDs of input files in the corresponding section, a list of parameters of execution commands, and a list of file IDs of output files.

15. The information processing device according to claim 14, wherein the file ID is a hash value of the file or a value that can identify the version of the file.

16. The information processing device according to claim 14, wherein parameters of the execution command include file IDs of the code file, input file, and output file.

17. The information processing device according to claim 14, wherein the section lineage creation unit links a section lineage file describing the section lineage to the output file.

18. An information processing method, comprising: a section lineage creation step of creating section lineage for each section that executes a command to process a dataset stored in an input file and generate an output file storing the dataset resulting from the processing.

19. A computer program written in a computer-readable format to cause a computer to function as an interval lineage creator that creates interval lineage for each interval that executes commands to process a dataset stored in an input file and generate an output file storing the dataset resulting from the processing.

20. An information processing system comprising: a recording device including a section lineage creation unit that creates section lineage for each section that executes a command to process a dataset stored in an input file and generate an output file storing the dataset resulting from the processing; a reception unit that receives file designations from a user; and a reproduction device including a control unit that displays data lineage for multiple files used in the process of generating the designated file on a display unit.

Images