Data conversion device, data conversion method, and program

Abstract

This data conversion device improves operating efficiency for data conversion by comprising: an input unit configured to input tabular format data to be converted and conversion results for positive and negative examples related to a part of the tabular format data; a generation unit configured to generate one or more candidates for a program that outputs a conversion result containing the positive example but not containing the negative example when the tabular format data has been input; a searching unit configured to search for the program from the one or more candidates; and an output unit configured to output a conversion result obtained by the program with respect to the tabular format data.

Description

Technical Field

[0001] The present invention relates to a data conversion device, a data conversion method, and a program.

Background Art

[0002] With digital transformation, the utilization of digital technologies and data (information assets) is progressing. To enhance competitiveness, it is important not only for data scientists but also for general employees to incorporate data analysis into their daily work. Data preprocessing is required for data analysis, and it is said that approximately 80% of the operation time is spent on preprocessing. Since knowledge of conversion methods and programming is also required, it has been an obstacle to the spread of data analysis.

[0003] As a tool for supporting data analysis, there is an ETL tool. In the ETL tool, data conversion can be automatically performed by specifying a data conversion method. However, in addition to the need for knowledge of conversion methods, it takes time to specify the conversion method every time a new data conversion is performed.

[0004] AutoPandas (Non-Patent Document 1) is a technology for synthesizing a program that realizes the conversion from a small-scale example before and after data conversion that reflects the data conversion to be realized. By executing the synthesized program on the data to be converted, the desired converted data can be obtained.

[0005] Program synthesis technology is a technology for searching for a program that realizes a given data conversion example (input example and output example) from combinations of a previously prepared function set by search. The synthesized program candidates are given the input examples and executed, and it is confirmed whether the same output as the output example given by the user can be obtained. The program that has obtained the same output as the output example is output as a program that realizes the data conversion desired by the user.

Prior Art Documents

Non-Patent Documents

[0006] [Non-Patent Document 1] Rohan Bavishi, Caroline Lemieux, Roy Fox, Koushik Sen, and Ion Stoica. 2019. "AutoPandas: neural-backed generators for program synthesis." Proc. ACM Program. Lang. 3, OOPSLA, Article 168 (October 2019), 27 pages [Overview of the project] [Problems that the invention aims to solve]

[0007] AutoPandas has the advantage of being usable even without knowledge of transformation methods or programming, as it only requires providing concrete examples of data before and after transformation. However, users must prepare examples of data before and after transformation that reflect the desired specifications, and if the specifications are not sufficiently reflected in the examples before and after transformation and the desired program cannot be synthesized, the examples must be recreated, which is time-consuming.

[0008] The present invention has been made in view of the above points, and aims to improve the efficiency of data conversion work. [Means for solving the problem]

[0009] Therefore, in order to solve the above problem, the data conversion device comprises an input unit configured to input tabular data to be converted, and conversion results for positive and negative examples relating to a part of the tabular data, and a generation unit configured to generate one or more candidates for a program that outputs a conversion result that includes the positive examples but does not include the negative examples when the tabular data is input. Based on the similarity of the conversion results of the tabular data by each of the one or more candidates to the positive and negative examples,The system includes a search unit configured to search for the program from among the one or more candidates mentioned above, and an output unit configured to output the conversion result of the program to the tabular data. The generation unit is configured to increase the program depth and generate one or more candidates at each depth based on the candidates generated at depths shallower than the current depth, and the search unit is configured to delete some of the candidates generated at a certain depth based on the commonality of the output when the search unit is unable to find a program that outputs a conversion result that includes the positive example but does not include the negative example at a certain depth. . [Effects of the Invention]

[0010] This can improve the efficiency of data conversion work. [Brief explanation of the drawing]

[0011] [Figure 1] This figure shows an example of the hardware configuration of the data conversion device 10 in an embodiment of the present invention. [Figure 2] This figure shows an example of the functional configuration of the data conversion device 10 in an embodiment of the present invention. [Figure 3] This flowchart illustrates an example of the processing procedure performed by the data conversion device 10. [Figure 4] This figure shows an example of the pre-transformation dataset in this embodiment. [Figure 5] This figure shows an example of a DSL template in this embodiment. [Figure 6] This figure shows an example of a DSL for transforming a dataset before transformation. [Figure 7] This figure shows the user-requested join result for the pre-transformation dataset in this embodiment. [Figure 8] This figure shows an example of user-submitted data in this embodiment. [Figure 9] This figure shows an example of the program searched and the converted data in step S200. [Figure 10] This figure shows an example of user-submitted data with negative examples added. [Figure 11] This figure shows an example of the program searched and the converted data in step S400. [Figure 12]This is a flowchart for explaining an example of the processing procedure for generating converted data that reflects user-presented data. [Figure 13] This is a diagram for explaining the depth of a program. [Figure 14] This is a flowchart for explaining an example of the processing procedure for the initialization process of program bank PB. [Figure 15] This is a flowchart for explaining an example of the processing procedure for the generation process of all program sets PN with a depth of N. [Figure 16] This is a flowchart for explaining an example of the processing procedure for the calculation process of score S based on output O. [Figure 17] This is a flowchart for explaining an example of the processing procedure for the calculation process of the score for positive examples. [Figure 18] This is a diagram for explaining a specific example of the calculation process of the score for positive examples. [Figure 19] This is a flowchart for explaining an example of the processing procedure for the calculation process of the score for negative examples. [Figure 20] This is a diagram for explaining a specific example of the calculation process of the score for negative examples. [Figure 21] This is a flowchart for explaining an example of the processing procedure for the pruning process of program bank PB. [Figure 22] This is a diagram for explaining pruning based on the commonality and depth of the output. [Figure 23] This is a diagram for explaining pruning based on the score.

Embodiments for Carrying Out the Invention

[0012] In the technology disclosed in this embodiment, data conversion can be performed with minimal effort, even without knowledge of conversion methods or programming, by interactively performing data conversion between the user and a machine (data conversion device 10 described later). Specifically, the user provides the machine with a portion of the data to be converted, the machine synthesizes a conversion program that includes that data in the conversion result, and presents the converted data to the user. The user reviews the data and corrects any inappropriate parts of the converted data. The machine then synthesizes a conversion program that takes the corrections into account and presents the new converted data to the user again. By repeating this interactive process, the user can perform data conversion even without knowledge of conversion methods or programming. Furthermore, there is no need to create small-scale examples of the data before and after conversion that reflect the desired data conversion, as is the case with AutoPandas.

[0013] In existing program synthesis techniques, user intent is expressed through input / output examples. These examples can be seen as positive representations of a portion of the desired specifications. In contrast, in the interactive program synthesis described in this embodiment, user intent is added to the output results of the program synthesized by the machine. Since the machine's output results may contain inappropriate data, it is necessary to be able to express negative user intent as well. Furthermore, in program synthesis, it is necessary to be able to quantitatively evaluate the extent to which the synthesized program satisfies the given specifications. Evaluation methods for new ways of expressing intent also need to be considered.

[0014] Therefore, in this embodiment, the user's intent is expressed using positive and negative examples. The machine searches for a program that produces an output that includes a positive example but does not include a negative example (a program that satisfies both positive and negative examples). Furthermore, a new evaluation function is proposed that takes into account the degree to which the positive and negative examples are satisfied.

[0015] Embodiments of the present invention will be described below with reference to the drawings.

[0016] Figure 1 shows an example of the hardware configuration of a data conversion device 10 in an embodiment of the present invention. The data conversion device 10 in Figure 1 includes a drive device 100, an auxiliary storage device 102, a memory device 103, a CPU 104, an interface device 105, a display device 106, and an input device 107, etc., which are all interconnected by bus B.

[0017] The program that enables processing in the data conversion device 10 is provided on a recording medium 101 such as a CD-ROM. When the recording medium 101 containing the program is set in the drive device 100, the program is installed from the recording medium 101 to the auxiliary storage device 102 via the drive device 100. However, the program does not necessarily have to be installed from the recording medium 101; it may also be downloaded from another computer via a network. The auxiliary storage device 102 stores the installed program as well as necessary files and data.

[0018] The memory device 103 reads and stores a program from the auxiliary storage device 102 when a program startup command is received. The CPU 104 implements the functions related to the data conversion device 10 according to the program stored in the memory device 103. The interface device 105 is used as an interface for connecting to a network. The display device 106 displays a GUI (Graphical User Interface) etc., generated by a program. The input device 107 consists of a keyboard and mouse etc., and is used to input various operation commands.

[0019] Figure 2 shows an example of the functional configuration of a data conversion device 10 in an embodiment of the present invention. In Figure 2, the data conversion device 10 includes a program synthesis unit 11, a synthesized program evaluation unit 12, and a pruning unit 13. Each of these units is realized by a process in which one or more programs installed in the data conversion device 10 are executed by the CPU 104.

[0020] The following describes the processing procedure performed by the data conversion device 10. In this embodiment, we assume that the user wants to obtain the result of joining two tabular data (hereinafter referred to as "tables"). However, the data conversions possible in this embodiment are not limited to joining tables.

[0021] Figure 3 is a flowchart illustrating an example of the processing procedure performed by the data conversion device 10.

[0022] In step S101, the program synthesis unit 11 receives the pre-transformation dataset as input. In this embodiment, the pre-transformation dataset is a pair of two tables to be transformed (joined) (hereinafter, each table is referred to as "pre-transformation data"). The data structure of the pre-transformation dataset can be expressed as follows. <Pre-transformation dataset>::=Pre-transformation data+ Figure 4 shows an example of a pre-transformation dataset in this embodiment. Figure 4 shows two pre-transformation data sets that make up the pre-transformation dataset. The first pre-transformation data set is the employee table, and the second pre-transformation data set is the approval table.

[0023] In this embodiment, the pre-conversion dataset (Figure 4) is, for convenience, of a size that allows for easy identification of all data (all records). However, this embodiment can also be applied to pre-conversion datasets of enormous size, in which case even more significant effects can be obtained.

[0024] Next, the program synthesis unit 11 reads a template for the DSL (Domain-Specific Language) (S102).

[0025] In this embodiment, DSL refers to the definition of specifications related to data conversion rules. In this embodiment, the joining of two tables is required. Therefore, for example, the developer of the data conversion device 10 (the person who provides data conversion services to the user of the data conversion device 10) pre-defines a template DSL for joining two tables in response to the user's request (joining two tables) and saves it in the auxiliary storage device 102 or the like. The program synthesis unit 11 reads the template DSL from the auxiliary storage device 102 or the like.

[0026] As a prerequisite, the data conversion device 10 has various data conversion methods pre-implemented. For example, the interface specification for the method of joining two tables is as follows. merge(DF,DF,K0,K1,H) The first argument, DF, is the left-hand table to be joined. The second argument, DF, is the right-hand table to be joined. The third argument, K0, is the attribute (label name) that will serve as the join key for the left-hand table. The fourth argument, K1, is the attribute (label name) that will serve as the join key for the right-hand table. The fifth argument, H, is the type of join (left outer join, right outer join). Note that the terms "left-hand table" and "right-hand table" are related to the type of join.

[0027] The user's request, which developers are informed of in advance, is simply "to join two tables," without specifying the table structure. The developers determine that the merge method is the best solution for this request and create a DSL template based on the assumption that the merge method will be used.

[0028] Figure 5 shows an example of a template for the DSL in this embodiment. As shown in Figure 5, the DSL includes two parts, (1) and (2).

[0029] (1) is the part where candidate input values ​​for each argument of the method used (merge in this embodiment) are defined. For the first and second arguments, DF, the input values ​​can be the data before transformation, Table 0 and Table 1, and the combined table which is the output of the merge method. Therefore, these are considered candidate input values ​​for DF.

[0030] Since K0 and K1 depend on the data before conversion, developers who are unaware of the data before conversion cannot define them. Therefore, no candidate input values ​​are defined for K0 and K1.

[0031] For H, it is known to developers and others that left and right are candidate input values ​​according to the specification of the merge method. Therefore, left and right are defined as candidate input values ​​for H. Note that left means left outer join and right means right outer join.

[0032] (2) is the part that defines what the start symbol, non-terminal symbol, terminal symbol, and abstract symbol are for a program that performs the data conversion requested by the user.

[0033] A start symbol is a symbol that can become the root node when a program is represented as a tree structure. In this embodiment, it is conceivable to developers that Table 0, Table 1, and the merge method can be start symbols. Therefore, these are defined as start symbols. Note that a program in which Table 0 or Table 1 is the root node is a program that outputs Table 0 or Table 1 as is, and the output of such a program is unlikely to be the output desired by the user. However, assuming that developers do not know the specific output desired by the user, it is safer to include candidates in the definition whose possibility is not zero (i.e., it is possible that the user wants a transformation in which Table 0 is output as the result of combining Table 0 and Table 1). Therefore, in this embodiment, Table 0 and Table 1 are also considered start symbols. However, if it is clear that Table 0 or Table 1 is not an output, then Table 0 and Table 1 do not need to be included as start symbols.

[0034] A non-terminal symbol is a symbol that does not become a leaf node when a program is represented as a tree structure. In other words, a method corresponds to a non-terminal symbol. Therefore, in this embodiment, the merge method is defined as a non-terminal symbol.

[0035] Terminal symbols are symbols that can become leaf nodes when a program is represented as a tree structure. Since terminal symbols are constants given as arguments to the merge method, they depend on the table being merged. Therefore, terminal symbols are unknown to developers and others. For this reason, terminal symbols are not defined in Figure 5.

[0036] An abstract symbol is a symbol that represents a set of values. More specifically, an abstract symbol is a symbol defined in part (1). This symbol is known to developers, etc. That is, DF, K0, K1, and H are abstract symbols.

[0037] Next, the program synthesis unit 11 completes a DSL (specific to the pre-transformation data) related to the transformation of the pre-transformation dataset by filling in the missing parts in the template shown in Figure 5 based on the pre-transformation dataset (Figure 4) (S103).

[0038] Figure 6 shows an example of a DSL related to the transformation of the pre-transformation dataset. In the pre-transformation dataset (Figure 4), the employee table corresponds to Table 0, and the approval table corresponds to Table 1. The attributes (label names) of the employee table are employee ID and employee name. Therefore, the program synthesis unit 11 assigns employee ID and employee name as candidate input values ​​for K0. Similarly, the attributes (label names) of the approval table are approval ID, employee ID, and result. Therefore, the program synthesis unit 11 assigns approval ID, employee ID, and result as candidate input values ​​for K0. As a result, part (1) of the DSL is completed.

[0039] Furthermore, among the symbols defined in (1), the symbols other than non-terminal symbols are table0, table1, employeeID, employeeName, approvalID, employeeID, result, left, and right. Therefore, the program synthesis unit 11 assigns these symbols to terminal symbols. As a result, part (2) is also completed. In the following steps, a program that performs the conversion desired by the user is searched for using the completed DSL.

[0040] In the above, it is assumed that the structure of the data before conversion is unknown to the developers, etc. However, if the structure of the data before conversion is known to the developers, etc., the completed DSL may be defined by the developers, etc. In this case, the program synthesis unit 11 does not need to execute step S103.

[0041] Next, the program synthesis unit 11 receives user-provided data, which is created by the user based on the pre-conversion dataset (S104).

[0042] In this embodiment, the user desires a program that combines the employee table and the approval table to produce data as shown in Figure 7 (however, the user does not need to know all of the conversion results). In this case, the user inputs data such as the following as user-provided data.

[0043] Figure 8 shows an example of user-submitted data in this embodiment. User-submitted data is data that includes one or more positive examples and zero or more negative examples of the output of the program to be synthesized, as follows: <User-submitted data>::=Positive example Negative example As a positive example, a portion of the user's desired output is provided. Figure 8 shows an example where the first data in Figure 7 is given as a positive example. As a negative example, data that does not correspond to any of the user's desired outputs is provided. However, a negative example does not need to be provided initially. Figure 8 shows an example where no negative example is provided.

[0044] Next, the data conversion device 10 performs a process to generate converted data that reflects the user-provided data (S200). In this generation process, one or more program candidates are generated within the range that satisfies the DSL, and a program that satisfies both positive and negative examples is searched for from among the one or more candidates. If a suitable program is found, the converted data (data output by the program) is obtained by executing the program.

[0045] Figure 9 shows an example of a program searched for and converted data in step S200. In Figure 9, (1) is a program searched for that satisfies the user-provided data in Figure 8, and (2) is an example of the converted data by that program.

[0046] Next, the synthesis program evaluation unit 12 of the data conversion device 10 receives a response from the user regarding the suitability of the converted data (S250). If the user responds that the data is suitable (Yes in S250), the processing procedure shown in Figure 3 is completed. In this case, the converted data obtained in step S200 is used by the user.

[0047] On the other hand, if the input indicates that the data is inappropriate (No in S250), the program synthesis unit 11 accepts new user-submitted data from the user (S300).

[0048] For example, in the converted data shown in Figure 9, the second record (the record that does not include the payment ID) is a record of a person who has not made a payment, but the user wishes that this record not be included in the converted data. In this case, the user enters new user-submitted data in which this record is added as a negative example.

[0049] Figure 10 shows an example of user-submitted data with a negative example added. The user-submitted data shown in Figure 10 has the second record from Figure 9 added as a negative example to the user-submitted data shown in Figure 8.

[0050] In this embodiment, an example was shown in which negative examples are added to the new user-submitted data; however, positive examples may also be added to the new user-submitted data.

[0051] Next, the data conversion device 10 performs a process to generate converted data that reflects the new user-submitted data (S400). The algorithm in step S400 is the same as the algorithm in step S200. Therefore, the searched program and the converted data output by executing the program are obtained.

[0052] Figure 11 shows an example of a program searched for and converted data in step S400. In Figure 11, (1) is a program searched for that satisfies the user-provided data in Figure 10, and (2) is an example of the converted data by that program.

[0053] The program in Figure 11 changes from a left outer join to a right outer join compared to the program in Figure 9 by providing a negative example (the last argument changes from left to right). Furthermore, the transformed data in Figure 11 does not include negative examples.

[0054] Next, steps S250 onwards are repeated. Therefore, if the user is satisfied with the converted data obtained in step S400 (Yes in S250), the processing procedure in Figure 3 ends; otherwise (No in S250), steps S300 and S400 are executed again.

[0055] Next, we will explain the details of steps S200 and S400. Figure 12 is a flowchart illustrating an example of the processing procedure for generating converted data that reflects user-provided data.

[0056] In step S201, the program synthesis unit 11 assigns 1 to the variable N.

[0057] Next, the program synthesis unit 11 executes the initialization process of the program bank PB (S210). A program bank is data having the following structure. <Program Bank>::=[Program, Score, Output, Depth]+ In other words, a program bank is a collection of data where each set includes a synthesized program, a score for that program, the output of that program, and the depth of that program. Program banks function as a storage location for previously synthesized programs, eliminating the need to resynthesize the same program and thus streamlining (reducing the time) program synthesis. They are used in bottom-up program synthesis.

[0058] The score indicates the degree to which the program's output matches the user-provided data (similarity to the user-provided data).

[0059] Program depth refers to the maximum depth of a program when its structure is represented as a tree, using a combination of DSLs (Domain-Specific Languages).

[0060] Figure 13 is a diagram illustrating program depth. In Figure 13, program (1) has a depth of 0. Program (2) has a depth of 1.

[0061] Next, the program synthesis unit 11 performs the process of generating all program sets PN such that the depth is N (S220). A program set PN is a set of one or more programs (a set of candidate programs that satisfy both positive and negative examples).

[0062] Next, the synthesized program evaluation unit 12 executes a loop process for each program (candidate) included in the program set PN. The program being processed in this loop process is called "program P".

[0063] In step S231, the synthesis program evaluation unit 12 inputs the pre-conversion dataset (Figure 4) into program P and executes it to obtain the output O from program P. If program P does not take the pre-conversion dataset as input, the execution result of program P becomes the output O. If program P is a constant, that constant becomes the output O.

[0064] Next, the synthesized program evaluation unit 12 determines whether the beginning of program P is a start symbol (S232). The beginning of program P refers to the first element that appears in program P, which is created by combining DSLs. For example, the beginning of the program "merge(employee table, approval table, employee ID, employee ID, left)" is merge. In this embodiment, the start symbols are employee table, approval table, and merge(DF, DF, K0, K1, H). Therefore, in step S232, it is determined whether the beginning of program P is one of these.

[0065] If the beginning of program P is not a start symbol (No in S232), the synthesized program evaluation unit 12 sets the score S of program P to null (S233) and proceeds to step S235.

[0066] If the beginning of program P is a start symbol (Yes in S232), the synthesized program evaluation unit 12 calculates a score S based on the output O (S240).

[0067] Next, the synthesis program evaluation unit 12 determines whether the score S is 1.0 or not (S234). If the score S is 1.0 (Yes in S234), the synthesis program evaluation unit 12 outputs output O to the user and returns control of the process to the caller. In this case, program P becomes the synthesis result of a program that satisfies both the positive and negative examples, and output O becomes the converted data by program P. If the score S is not 1.0 (No in S233), the process proceeds to step S235.

[0068] In this way, based on the score S, it is determined whether the output O satisfies both positive and negative examples. In other words, calculating the score S corresponds to searching for programs that satisfy both positive and negative examples.

[0069] In step S235, the synthesis program evaluation unit 12 adds the set {program P, score S, output O, N} to the program bank PB.

[0070] Once the above steps are performed for all programs P included in the program set PN, the pruning unit 13 performs pruning on the program bank PB (S250). In the next step S220, a new program is synthesized based on the program bank PB. However, if there are many programs in the program bank PB, pruning is performed to prevent combinatorial explosion.

[0071] Next, the program synthesis unit 11 adds 1 to N (S261) and repeats steps S220 onward. That is, while increasing the program depth, candidate programs that satisfy positive and negative examples are generated for each depth based on the programs generated at depths shallower than the current depth, and a program that satisfies both positive and negative examples is searched for among these candidates.

[0072] Next, we will explain the details of step S210 in Figure 12. Figure 14 is a flowchart illustrating an example of the processing procedure for the initialization process of the program bank PB.

[0073] In step S211, the program synthesis unit 11 initializes the program bank PB to an empty state.

[0074] Next, the program synthesis unit 11 executes loop processing for each DSL termination symbol. Since the DSL termination symbols are "Employee Table | Approval Table | Employee ID | Employee Name | Approval ID | Employee ID | Result | left | right", loop processing is executed for each of these symbols. The termination symbol being processed in the loop processing is called the "termination symbol TS".

[0075] In step S212, the program synthesis unit 11 executes the terminal symbol TS to obtain output O. Basically, since the terminal symbol is a constant, its value becomes the output O. For example, the output O of the terminal symbol "Employee Table" is "Employee Table".

[0076] Next, the program synthesis unit 11 determines whether the termination symbol TS is a start symbol (S213). In this embodiment, the start symbols are the employee table, the approval table, and the merge method. Therefore, in step S213, it is determined whether the termination symbol TS is the employee table, the approval table, or the merge method.

[0077] If the terminal symbol TS is a start symbol (Yes in S213), the program synthesis evaluation unit 12 calculates a score S based on the output O (S240) and proceeds to step S215. The algorithm in step S240 here is the same as the algorithm in step S240 in Figure 12. If the terminal symbol TS is not a start symbol (No in S213), the program synthesis unit 11 sets the score S to null (S214) and proceeds to step S215.

[0078] In step S215, the program synthesis unit 11 adds {termination symbol TS, score S, output O, depth N} to the program bank PB. For example, if the termination symbol TS is an employee ID, then {employee ID, null, employee ID, 0} is added to the program bank PB.

[0079] Once the above has been executed for all terminal symbols of the DSL, the program bank PB is returned to the caller.

[0080] Next, we will explain the details of step S220 in Figure 12. Figure 15 is a flowchart illustrating an example of the processing procedure for generating all program sets PN such that the depth is N.

[0081] In step S221, the program synthesis unit 11 initializes the program set PN to an empty state.

[0082] Next, the program synthesis unit 11 executes loop processing for each non-terminal symbol of the DSL. The non-terminal symbol that is processed in the loop processing is called the "non-terminal symbol NS". In this embodiment, the only non-terminal symbol is "merge(DF,DF,K0,K1,H)".

[0083] In step S222, the program synthesis unit 11 generates a new program set by comprehensively applying programs from the program bank PB that can correspond to each non-terminal symbol NS. Here, a program from the program bank PB that can correspond to a certain abstract symbol is a program whose beginning belongs to that abstract symbol.

[0084] For example, suppose the non-terminal symbol NS is merge(DF,DF,K0,K1,H), and the program bank PB contains Table 0, Table 1, Employee ID, Employee Name, Approval ID, Result, left, and right. In this case, each DF can be either Table 0 or Table 1. K0 can be either Employee ID or Employee Name. K1 can be either Approval ID, Employee ID, or Result. H can be either left or right.

[0085] Therefore, in this case, a new program set PN is generated containing 2 × 2 × 2 × 3 × 2 = 48 types of programs (candidate programs that satisfy both positive and negative examples).

[0086] Next, the program synthesis unit 11 deletes all programs from the program set PN except those with a depth of N (S223).

[0087] Once steps S222 and S223 have been executed for all non-terminal symbols, the program set PN is returned to the caller.

[0088] Next, we will explain the details of step S240 in Figures 12 and 14. Figure 16 is a flowchart illustrating an example of the processing procedure for calculating the score S based on the output O.

[0089] In step S241, the synthesis program evaluation unit 12 calculates a score for output O relative to a positive example and assigns the calculation result to s_p.

[0090] Next, the synthesis program evaluation unit 12 calculates a score for the output O against negative examples and assigns the calculation result to s_n (S242).

[0091] The synthesis program evaluation unit 12 returns s_p - s_n as the score S.

[0092] Next, we will explain the details of step S241 in Figure 16. Figure 17 is a flowchart illustrating an example of the processing procedure for calculating the score for positive examples.

[0093] In step S2411, the synthesis program evaluation unit 12 initializes the similarity list S_L to an empty state.

[0094] Next, the synthesis program evaluation unit 12 calculates the similarity Sim_0 between the label name row C1 of output O and the label name row C2 of the positive example of user-submitted data, and adds the calculation result sim_0 to S_L (S2412). Here, the label name row of output O is, for example, the row "Settlement ID, Employee ID, Employee Name, Result" if output O is as shown in the converted data in Figure 9. The label name row of the positive example of user-submitted data is the row "Settlement ID, Employee ID, Employee Name, Result" in the positive examples in Figure 8 and Figure 10. The formula for calculating Sim_0 is as follows. Sim_0 ← Number of elements in (C1∩C2) / Number of elements in (C1∪C2) In other words, Sim_0 is the value obtained by dividing the number of elements contained in both C1 and C2 by the number of elements contained in at least one of C1 and C2.

[0095] Next, the synthesis program evaluation unit 12 executes a loop process (hereinafter referred to as "loop process A") including steps S2413 to S2416 for each row of the user-submitted data other than the label name row. The row of the positive example that is processed in loop process A is called "C1_N". In the positive examples in Figures 8 and 10, the rows other than the row containing "payment ID, employee ID, employee name, result" are candidates for C1_N.

[0096] In step S2413, the synthesis program evaluation unit 12 initializes the list-type variable S_L_temp to an empty state.

[0097] Next, the synthesis program evaluation unit 12 executes a loop process (hereinafter referred to as "loop process B") including steps S2414 and S2415 for each row of output O other than the label name row. The row of output O that is processed in loop process B is called "C2_N".

[0098] In step S2414, the synthesis program evaluation unit 12 calculates the similarity Sim_N_temp between C1_N and C2_N. The formula for calculating the similarity Sim_N_temp is as follows: Sim_N_temp ← Number of elements in (C1_N∩C2_N) / Number of elements in (C1∪C2) Next, the synthesis program evaluation unit 12 adds Sim_N_temp to S_L_temp (S2415).

[0099] When loop processing B has finished for all rows of output O except for the label name row, the synthesis program evaluation unit 12 adds the maximum value in SL_temp to S_L (S2416).

[0100] Once loop processing A has finished for all rows except the label name row of the positive example, the synthesis program evaluation unit 12 returns the average value of all elements in S_L as the score S for the positive example.

[0101] Figure 18 is a diagram illustrating a specific example of the score calculation process for positive examples. Figure 18 shows three specific examples, (1) to (3).

[0102] (1) is the case where output O contains all positive examples. In this case, the value of Sim_0 is 4 / 4 = 1.0. Also, the maximum value of Sim_N_temp is 4 / 4 = 1.0. Therefore, the score S, which is the average value within S_L, is (4 / 4 + 4 / 4) / 2 = 1.0. That is, the score S when output O contains all positive examples is 1.0.

[0103] (2) is the case where the label name row in output O matches the label name row in the positive example, but there are no matching rows between output O and the positive example for rows other than labels. In this case, the value of Sim_0 is 4 / 4 = 1.0. Also, the maximum value of Sim_N_temp is 3 / 4 = 0.75. Therefore, the score S, which is the average value within S_L, is (4 / 4 + 3 / 4) / 2 = 0.875.

[0104] (3) is the case where there are no matching rows between output O and the positive examples. In this case, the value of Sim_0 is 3 / 4 = 0.75. Also, the maximum value of Sim_N_temp is 2 / 4 = 0.5. Therefore, the score S, which is the average value within S_L, is (3 / 4 + 2 / 4) / 2 = 0.625.

[0105] Next, we will explain the details of step S242 in Figure 16. Figure 19 is a flowchart illustrating an example of the processing procedure for calculating scores for negative examples.

[0106] In step S2421, the synthesis program evaluation unit 12 initializes the similarity list S_L to an empty state and assigns an arbitrary (random) value (but a positive value greater than 0) to the variable weight.

[0107] Next, the synthesis program evaluation unit 12 executes a loop process (hereinafter referred to as "loop process C") including steps S2422 to S2425 for each row of the negative example of user-presented data other than the label name row. The row of the negative example that is processed in loop process C is called "C1_N". In the negative example in Figure 10, the rows other than the row containing "payment ID, employee ID, employee name, result" are candidates for C1_N.

[0108] In step S2422, the synthesis program evaluation unit 12 initializes the list-type variable S_L_temp to an empty state.

[0109] Next, the synthesis program evaluation unit 12 executes a loop process (hereinafter referred to as "loop process D") including steps S2423 and S2424 for each row of output O other than the label name row. The row of output O that is processed in loop process D is called "C2_N".

[0110] In step S2423, the synthesis program evaluation unit 12 calculates the similarity Sim_N_temp between C1_N and C2_N. The formula for calculating the similarity Sim_N_temp is as follows: Sim_N_temp ← Number of elements in (C1_N∩C2_N) / Number of elements in (C1∪C2) Next, the synthesis program evaluation unit 12 adds Sim_N_temp to S_L_temp (S2424).

[0111] When loop processing D has finished for all rows of output O except for the label name row, the synthesis program evaluation unit 12 adds the maximum value in SL_temp to S_L (S2425).

[0112] When loop processing C has finished for all rows except the label name row of the negative example, the synthesis program evaluation unit 12 returns the number of 1.0 values ​​in S_L multiplied by the weight as the score S for the negative example.

[0113] Note that if the user-submitted data does not contain any negative examples, loop C will not be executed. In this case, the number of 1.0 values ​​in S_L will be 0. Therefore, the score S will be 0.

[0114] Figure 20 illustrates a specific example of the score calculation process for negative examples. Figure 20 shows two specific examples, (1) and (2).

[0115] (1) is the case where all negative examples are included in output O. In this case, the number of 1.0 in S_L is 1. Therefore, if the weight value is 0.2, the score S will be 0.2.

[0116] (2) is the case where output O does not contain any negative examples. In this case, the number of 1.0 in S_L is 0. Therefore, the score S is 0.

[0117] Thus, the score S for a negative example increases as the output O becomes more similar to the negative example.

[0118] Next, we will explain the details of step S250 in Figure 12. Figure 21 is a flowchart illustrating an example of the pruning procedure for the program bank PB.

[0119] In step S251, the pruning unit 13 groups programs in program bank PB that have the same output O. In other words, programs in program bank PB are classified into groups based on the commonality of their output O.

[0120] Next, the pruning unit 13 removes all programs from the program bank PB except for the shallowest program within each group (S252). In other words, programs that return the same output are considered equivalent, and simpler (shallower) programs are retained. If there are multiple programs with the shallowest depth, the pruning unit 13 randomly selects one of the multiple shallowest programs and removes all other programs.

[0121] Figure 22 illustrates pruning based on the commonality and depth of outputs. Figure 22 shows an example where program bank PB contains programs 1 and 2, etc. Here, the outputs of programs 1 and 2 are common, as shown in the table below Figure 22. Therefore, programs 1 and 2 are classified into the same group. Within this group, program 1 has the shallowest depth. Therefore, program 2 is deleted, and program 1 remains.

[0122] Next, the pruning unit 13 keeps the program with the highest score among the programs whose scores are not null, and deletes the remaining programs whose scores are not null (S253). In other words, the programs with null scores and the program with the highest score among the programs whose scores are not null remain. The program with the highest score is kept because it is assumed to be the program that is most similar to the user-provided data. Programs with null scores (i.e., programs that do not return a table) are kept because they may become arguments for programs that return a table.

[0123] Figure 23 illustrates pruning based on scores. Figure 23 shows an example where programs 1-4 constitute part of program bank PB. Here, program 4 is retained because its score is null. Among programs 1-3, only program 2, which has the highest score, is retained (programs 1 and 3 are deleted).

[0124] As described above, this embodiment improves the efficiency of data conversion work. Specifically, users can perform data conversion with minimal effort, even without knowledge of conversion methods or programming. This makes it possible for non-data scientists to perform data analysis and to incorporate insights gained from data analysis into daily operations.

[0125] In this embodiment, for the sake of explanation, an example was described in which a program using only one method (the merge method) is synthesized and searched. However, this embodiment can also handle transformations using two or more methods. In this case, for example, the rows of the DF in the DSL may be defined as follows. DF::=EmployeeTable|ApprovalTable|merge(DF,DF,K0,K1,H)|Method2(...) Here, Method 2(DF,...) could be, for example, a method that outputs a table from which a specific column or row has been removed from DF. In this case, a program is synthesized that performs a transformation process that joins the two tables and then removes a specific row or column, or a transformation process that removes a column or row from one of the tables and then joins the two tables. Which program is ultimately selected depends on the user-provided data.

[0126] In this embodiment, the program synthesis unit 11 is an example of an input unit and a generation unit. The synthesized program evaluation unit 12 is an example of a search unit and an output unit.

[0127] Although embodiments of the present invention have been described in detail above, the present invention is not limited to these specific embodiments, and various modifications and changes are possible within the scope of the gist of the present invention as described in the claims. [Explanation of Symbols]

[0128] 10 Data conversion device 11 Program Synthesis Unit 12 Synthesis Program Evaluation Unit 13. Branch trimming section 100 drive unit 101 Recording media 102 Auxiliary storage 103 Memory device 104 CPU 105 Interface device 106 Display device 107 Input device B Bus

Claims

1. An input unit configured to input tabular data to be converted, and the conversion results of positive and negative examples relating to a portion of the tabular data, A generation unit is configured to generate one or more candidates for a program that outputs a conversion result that includes the positive example but does not include the negative example when the aforementioned tabular data is input. A search unit configured to search for the program from among the one or more candidates based on the similarity of the conversion results of the tabular data by each of the one or more candidates to the positive and negative examples, An output unit configured to output the conversion result of the program to the tabular data, It has, The generation unit is configured to increase the program depth and generate one or more candidates at each depth based on the candidates generated at depths shallower than the program depth. If the search unit fails to find a program that outputs a conversion result containing the positive example but not the negative example at a certain depth, the pruning unit is configured to delete a portion of the candidates generated at that depth based on the commonality of the output. A data conversion device characterized by having the following features.

2. An input unit configured to input tabular data to be converted, and the conversion results of positive and negative examples relating to a portion of the tabular data, A generation unit is configured to generate one or more candidates for a program that outputs a conversion result that includes the positive example but does not include the negative example when the aforementioned tabular data is input. A search unit configured to search for the program from among the one or more candidates based on the similarity of the conversion results of the tabular data by each of the one or more candidates to the positive and negative examples, An output unit configured to output the conversion result of the program to the tabular data, It has, The generation unit is configured to increase the program depth and generate one or more candidates at each depth based on the candidates generated at depths shallower than the program depth. If the search unit fails to find a program that outputs a conversion result that includes the positive example but does not include the negative example at a certain depth, the pruning unit is configured to delete a portion of the candidates generated at that depth based on the similarity. A data conversion device characterized by having the following features.

3. The search unit is configured to search for a program that also satisfies the newly input conversion result if a positive or negative conversion result is newly input after the output unit has output the conversion result. A data conversion device according to claim 1 or 2.

4. An input procedure for inputting tabular data to be converted, and the conversion results of positive and negative examples related to a portion of the tabular data, A generation procedure for generating one or more candidates for a program that outputs a conversion result that includes the positive example but does not include the negative example when the aforementioned tabular data is input, A search procedure for searching for the program from among the one or more candidates based on the similarity of the conversion results of the tabular data by each of the one or more candidates to the positive and negative examples, An output procedure for outputting the conversion result of the program to the aforementioned tabular data, The computer executes this, The generation procedure increases the program depth and generates one or more candidates at each depth based on the candidates generated at depths shallower than the said depth. If the search procedure fails to find a program that outputs a conversion result containing the positive example but not the negative example at a certain depth, a pruning procedure is configured to delete a portion of the candidates generated at that depth based on the commonality of the output. A data conversion method characterized by being performed by a computer.

5. An input procedure for inputting tabular data to be converted, and the conversion results of positive and negative examples related to a portion of the tabular data, A generation procedure for generating one or more candidates for a program that outputs a conversion result that includes the positive example but does not include the negative example when the aforementioned tabular data is input, A search procedure for searching for the program from among the one or more candidates based on the similarity of the conversion results of the tabular data by each of the one or more candidates to the positive and negative examples, An output procedure for outputting the conversion result of the program to the aforementioned tabular data, The computer executes this, The generation procedure increases the program depth and generates one or more candidates at each depth based on the candidates generated at depths shallower than the said depth. If the search procedure fails to find a program that outputs a conversion result containing the positive example but not the negative example at a certain depth, a pruning procedure is configured to delete a portion of the candidates generated at that depth based on the similarity. A data conversion method characterized by being performed by a computer.

6. An input procedure for inputting tabular data to be converted, and the conversion results of positive and negative examples related to a portion of the tabular data, A generation procedure for generating one or more candidates for a program that outputs a conversion result that includes the positive example but does not include the negative example when the aforementioned tabular data is input, A search procedure for searching for the program from among the one or more candidates based on the similarity of the conversion results of the tabular data by each of the one or more candidates to the positive and negative examples, An output procedure for outputting the conversion result of the program to the aforementioned tabular data, Have the computer run it, The generation procedure increases the program depth and generates one or more candidates at each depth based on the candidates generated at depths shallower than the said depth. If the search procedure fails to find a program that outputs a conversion result containing the positive example but not the negative example at a certain depth, a pruning procedure is configured to delete a portion of the candidates generated at that depth based on the commonality of the output. A program characterized by causing a computer to execute something.

7. An input procedure for inputting tabular data to be converted, and the conversion results of positive and negative examples related to a portion of the tabular data, A generation procedure for generating one or more candidates for a program that outputs a conversion result that includes the positive example but does not include the negative example when the aforementioned tabular data is input, A search procedure for searching for the program from among the one or more candidates based on the similarity of the conversion results of the tabular data by each of the one or more candidates to the positive and negative examples, An output procedure for outputting the conversion result of the program to the aforementioned tabular data, Have the computer run it, The generation procedure increases the program depth and generates one or more candidates at each depth based on the candidates generated at depths shallower than the said depth. If the search procedure fails to find a program that outputs a conversion result containing the positive example but not the negative example at a certain depth, a pruning procedure is configured to delete a portion of the candidates generated at that depth based on the similarity. A program characterized by causing a computer to execute something.

Images