Code conversion device, code conversion method, and program

Abstract

A code conversion device 10 comprises: a detection unit 11 that detects first codes, each including a first function code that combines key columns included in two-dimensional array data and executes a grouping operation for each combination of key columns; an extraction unit 12 that extracts, from the first codes, second codes for which the two-dimensional array data targeted by the first function code is the same and the aggregate operation code included in the first codes is the same; a selection unit 13 that, on the basis of the aggregate operation code included in each second code and the key columns of the target two-dimensional array data, selects key columns to be used in an intermediate table in which the key columns of the two-dimensional array data are reduced; a generation unit 14 that generates a third code using the first function code, the selected key columns, and the aggregate operation code, and adds the third code to the front of the second codes; and a conversion unit 15 that, on the basis of the third code, aligns the plurality of second codes with the third code and converts the second codes into a fourth code.

Description

[Technical Field]

[0001] The present disclosure relates to a code conversion device and a code conversion method for converting data into code, and also to a program for realizing these. Mu Regarding. [Background technology]

[0002] Preprocessing for generating training data for machine learning involves feature generation, which is known to be time-consuming.

[0003] Therefore, we want to shorten the time required for the feature generation process. The reason why the feature generation process takes time is because multiple columns contained in the two-dimensional array data are used as key columns, and a grouping operation is performed for each combination of key columns. In other words, if there are overlapping columns between key columns, duplicate processing is performed.

[0004] As a related technique, Patent Document 1 discloses a technique for reducing the number of combinations of aggregation results and creating aggregation results at high speed. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 11-003354 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the technology of Patent Document 1 does not convert the code of the grouping operation used in the feature generation process or the like into a code for speeding up (reducing the operation time).

[0007] An example of an objective of the present disclosure is to speed up (reduce calculation time) a grouping calculation using a plurality of key strings in a table (two-dimensional array data) included in an input code. [Means for solving the problem]

[0008] In order to achieve the above object, a code conversion device according to one aspect of the present disclosure includes: a detection unit that detects, from an input code that is input to be executed by a computer and that is stored in advance in a storage device, a first code including a first function code that combines a plurality of key strings included in the two-dimensional array data and executes a grouping operation for each combined key string; an extraction unit that extracts, from the detected plurality of first codes, a plurality of second codes whose first function codes target the same two-dimensional array data and whose first codes include the same aggregate operation code; a selection unit that selects a key string to be used in an intermediate table obtained by reducing the key string of the target two-dimensional array data, based on the aggregate operation code included in each of the second codes and the key string of the target two-dimensional array data; a generation unit that generates a third code using the first function code, a selected key string, and an aggregate operation code, and adds the third code to the front of the second code; a conversion unit that aligns the plurality of second codes with the third code based on the third code and converts them into a fourth code; The present invention is characterized by having the following.

[0009] In order to achieve the above object, a code conversion method according to one aspect of the present disclosure includes: The computer detecting a first code including a first function code for combining a plurality of key strings included in the two-dimensional array data and executing a grouping operation for each combined key string from input codes that are input to be executed by the computer and that are stored in advance in a storage device; extracting, from the detected plurality of first codes, a plurality of second codes whose first function codes target the same two-dimensional array data and whose first codes include the same aggregate operation code; selecting a key string to be used in an intermediate table obtained by reducing the key string of the target two-dimensional array data based on the aggregate operation code included in each of the second codes and the key string of the target two-dimensional array data; generating a third code using the first function code, the selected key string, and an aggregate operation code, and adding the third code to the front of the second code; aligning the plurality of second codes with the third code based on the third code and converting them into a fourth code; It is characterized by:

[0010] Furthermore, in order to achieve the above object, in one aspect of the present disclosure, program teeth, On the computer, detecting a first code including a first function code for combining a plurality of key strings included in the two-dimensional array data and performing a grouping operation for each combined key string from an input code that is input to be executed by the computer and that is stored in advance in a storage device; extracting, from the detected plurality of first codes, a plurality of second codes whose first function codes target the same two-dimensional array data and whose first codes include the same aggregate operation code; selecting a key string to be used in an intermediate table obtained by reducing the key string of the target two-dimensional array data based on the aggregate operation code included in each of the second codes and the key string of the target two-dimensional array data; generating a third code using the first function code, the selected key string, and an aggregate operation code, and adding the third code to the front of the second code; aligning the plurality of second codes with the third code based on the third code and converting them into a fourth code; It is characterized by: [Effects of the Invention]

[0011] As described above, according to the present disclosure, it is possible to speed up (reduce the calculation time) a grouping calculation using a plurality of key strings of a table (two-dimensional array data) included in an input code. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram for explaining Target Encoding. [Figure 2] FIG. 2 is a diagram for explaining Target Encoding when expanded to multiple categorical variables. [Figure 3] FIG. 3 is a diagram for explaining the Target Encoding code. [Figure 4] FIG. 4 is a diagram illustrating an example of a system including the code conversion device of the first embodiment. [Figure 5] FIG. 5 is a diagram for explaining the second code of the first embodiment. [Figure 6] FIG. 6 is a diagram for explaining the third code of the first embodiment. [Figure 7] FIG. 7 is a diagram for explaining code matching in the first embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of the operation of the code conversion device according to the first embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of a system including a code conversion device according to the second embodiment. [Figure 10] FIG. 10 is a diagram for explaining the second code of the second embodiment. [Figure 11] FIG. 11 is a diagram for explaining the third code of the second embodiment. [Figure 12] FIG. 12 is a diagram for explaining code matching in the second embodiment. [Figure 13]FIG. 13 is a diagram illustrating an example of the operation of the selection unit of the code conversion device according to the second embodiment. [Figure 14] FIG. 14 is a diagram illustrating an example of a computer that realizes the code conversion device according to the first and second embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0013] First, an overview will be given to facilitate understanding of the embodiments to be described below. Feature generation is a preprocessing step for generating learning data used in machine learning. Known examples of feature generation include Target Encoding (or Target Mean Encoding (Likelihood Encoding)), which quantifies (features) categorical variables. Target Encoding aggregates objective variables for each categorical variable and quantifies the aggregated values ​​(for example, maximum, minimum, sum, count, average, etc.).

[0014] Figure 1 is a diagram used to explain Target Encoding. When Table 1 shown in Figure 1 is used as input for machine learning, the data in the "Category" column of Table 1 is not numerical, so it cannot be used as is as input for machine learning.

[0015] Therefore, using Target Encoding, we convert the data in the "Category" column of Table 1, as shown in Figure 1, into a numerical value that aggregates the target variable, such as the data shown in the "Category Tgt-Mean" column of Table 3.

[0016] In this case, first, set the data in the "Category" column of Table 1 to information that does not have meaning in itself, such as integer values, for each of the categorical variables A, B, C, and D, as in the data shown in the "Category ID" column of Table 2. In the example in Figure 1, categorical variable A is set to 1, categorical variable B is set to 2, categorical variable C is set to 3, and categorical variable D is set to 4.

[0017] Next, using the data shown in the "Category ID" column of Table 2, calculate the mean value for each categorical variable, as shown in the "Category Tgt-Mean" column of Table 3. In the example in Figure 1, categorical variable A is quantified to 0.50 (= (1 + 0) / 2), categorical variable B is quantified to 0.33 (= (1 + 0 + 0) / 3), categorical variable C is quantified to 0.75 (= (1 + 0 + 1 + 1) / 4), and categorical variable D is quantified to 1.00 (= (1) / 1).

[0018] Next, an example of target encoding using not only one categorical variable but a combination of multiple categorical variables will be explained using Figure 2. Figure 2 is a diagram for explaining target encoding when expanded to multiple categorical variables.

[0019] In the example in Figure 2, Target Encoding is performed using four of the categorical variables "Category A," "Category B," "Category C," "Category D," and "Category E" shown in Table 4. Note that in the example in Figure 2, the data for each column has been omitted for convenience.

[0020] In the example in Figure 2, Target Encoding is performed using the categorical variables "CategoryA," "CategoryB," "CategoryC," and "CategoryD," as well as Target Encoding using the categorical variables "CategoryB," "CategoryC," "CategoryD," and "CategoryE."

[0021] As a result, the categorical variables "CategoryABCD Tgt-Mean" and "CategoryBCDE Tgt-Mean" in Table 5 shown in Figure 2 are generated.

[0022] This section explains Target Encoding using a table processing library. Figure 3 is a diagram for explaining the Target Encoding code. The code shown in Figure 3 is an example of code that uses "groupby" and "transform" from pandas, a Python table processing library.

[0023] Code 6 in Figure 3 is the Target Encoding code using one categorical variable as explained in Figure 1. Code 7 in Figure 3 is the Target Encoding code using multiple categorical variables as explained in Figure 2.

[0024] "groupby" used in Codes 6 and 7 is a function (or method) for grouping. "transform" is a function (or method) for rewriting data using acquired statistical information (e.g., maximum value, minimum value, sum, number, average value, etc.).

[0025] "Category," "CatA," "CatB," "CatC," "CatD," and "CatE" written in Codes 6 and 7 represent the columns "Category," "CategoryA," "CategoryB," "CategoryC," "CategoryD," and "CategoryE" shown in Figures 1 and 2. "Target" represents "Target" shown in Figures 1 and 2. "Category_TgtMean," "ABCD_TgtMean," and "BCDE_TgtMean" represent "Category Tgt-Mean," "CategoryABCD Tgt-Mean," and "CategoryBCDE Tgt-Mean" shown in Figures 1 and 2.

[0026] The processes executed by Codes 6 and 7 include a process of generating groups and a process of calculating aggregated values ​​for each group. In the case of Code 6, the group generation process generates groups GRP0, GRP1, GRP2, and GRP3 for each categorical variable as shown below.

[0027] The numerical values ​​representing the elements included in the groups GRP0 to GRP3 shown below are expressed using the row numbers shown in FIG.

[0028] GRP0:0,1 (Category A group) GRP1:2,3,4 (Category B group) GRP2:5,6,7,8 (Category C group) GRP3:9 (Category D group)

[0029] Furthermore, in the case of code 6, by calculating the aggregated values ​​for each group, the average value for each group is calculated as shown below.

[0030] GRP0: Average value of 0,1 (0.50) (Category Tgt-Mean A) GRP1: Average value of 2, 3, and 4 (0.33) (Category Tgt-Mean B) GRP2: Average value of 5, 6, 7, and 8 (0.75) (Category Tgt-Mean C) GRP3:9 average value (1.00) (Category Tgt-Mean D)

[0031] However, when "groupby" using multiple columns (key columns) is executed multiple times with different combinations of key columns, if there are overlapping columns among the key columns, duplicate processing (similar, useless processing) will be executed.

[0032] Specifically, as shown in Code 7, if you run "groupby" twice with two combinations of categorical variables, "CategoryA", "CategoryB", "CategoryC", and "CategoryD", and two combinations of categorical variables, "CategoryB", "CategoryC", "CategoryD", and "CategoryE", the categorical variables "CategoryB", "CategoryC", and "CategoryD" are duplicated, so duplicate processing (similar but useless processing) will be executed.

[0033] Therefore, the calculation speed of the feature generation process slows down (calculation time increases) by the time spent performing unnecessary processing. Furthermore, the amount of calculation increases as the number of key strings increases.

[0034] Through this process, the inventors have discovered the problem of how to increase the calculation speed (shorten the calculation time) of the feature generation process, and have also derived a means for solving this problem.

[0035] That is, the inventor has derived a means for converting a code used to execute a grouping operation using multiple key strings contained in two-dimensional array data (table) into a code that can speed up the operation (shorten the operation time), thereby making it possible to speed up the operation (shorten the operation time) of the feature generation process.

[0036] Hereinafter, embodiments will be described with reference to the drawings. In the drawings described below, elements having the same or corresponding functions are denoted by the same reference numerals, and repeated description thereof may be omitted.

[0037] (Embodiment 1) The configuration of the code conversion device 10 in the first embodiment will be described more specifically with reference to Fig. 4. Fig. 4 is a diagram showing an example of a system including a code conversion device.

[0038] [System Configuration] In the example of FIG. 4, the system 100 includes a transcoding device 10 and a storage device 20 .

[0039] The code conversion device 10 is, for example, an information processing device such as a CPU (Central Processing Unit), a programmable device such as an FPGA (Field-Programmable Gate Array), a GPU (Graphics Processing Unit), or a circuit equipped with one or more of these, a server computer, a personal computer, or a mobile terminal.

[0040] The code conversion device 10 is a device used to speed up (shorten the calculation time) a grouping calculation using multiple key strings included in a table (two-dimensional array data) of input codes. That is, the code conversion device 10 converts the code used in the grouping calculation included in the input code into code that reduces the number of rows in the original table used in the grouping calculation and performs an aggregation calculation on the reduced table (intermediate table), thereby reducing the number of calculations.

[0041] The storage device 20 stores computer-executable input code (pre-conversion code) used to generate training data. The storage device 20 also stores code (post-conversion code) that can increase the calculation speed (reduce the calculation time).

[0042] The code conversion device of the first embodiment will be specifically described. As shown in FIG. 4, the code conversion device 10 according to the first embodiment includes a detection unit 11, an extraction unit 12, a selection unit 13, a generation unit 14, and a conversion unit 15.

[0043] The specific code conversion process will be explained using code that uses "groupby" from pandas, a Python table processing library. However, the language used to write the code is not limited to Python.

[0044] The detection unit 11 detects a first code including a first function code that combines multiple key strings included in a table (two-dimensional array data) and executes a grouping operation for each combined key string from an input code that is input to be executed by a computer and that is stored in advance in the storage device 20.

[0045] The input code is, for example, code created by a user using Python etc. Specifically, the input code is code that includes a groupby method (a function belonging to an object) that executes an aggregate operation multiple times on the same table having multiple key columns by changing the combination of key columns.

[0046] The table (two-dimensional array data) is, for example, data in a Python two-dimensional data structure (DataFrame). The first function code is, for example, the groupby method of the Python table processing library pandas. The first code is, for example, code including the groupby method.

[0047] The extraction unit 12 extracts, from the detected plurality of first codes, a plurality of second codes whose target tables (two-dimensional array data) are the same as the first function codes and whose aggregate operation codes are included in the first codes.

[0048] The aggregate operation code is code used for aggregate operations such as the aggregate method and transform method of Python. The aggregate method and transform method are methods that execute multiple aggregate operations together.

[0049] Fig. 5 is a diagram for explaining the second codes of the first embodiment. The example of Fig. 5 shows four second codes 50 ((1)(2)(3)(4)) extracted by the extraction unit 12 from the plurality of first codes detected by the detection unit 11. Also, 51 in Fig. 5 shows a first function code (groupby()) and the same table targeted by the first function code.

[0050] 52 in Figure 5 contains the aggregate operation code (['val'].agg("sum")) contained in the second code. The "sum" in the aggregate operation code represents the sum function. The sum function is a function that calculates the total sum. In addition to the sum function, the max function (a function that calculates the maximum value), min function (a function that calculates the minimum value), count function (a function that calculates the number of items), mean function (a function that calculates the average value), etc. may also be used. ['val'] in the aggregate operation code is the aggregate operation column data that the aggregate operation code targets.

[0051] The selection unit 13 selects key strings to be used in an intermediate table obtained by reducing the key strings of the target two-dimensional array data, based on the aggregate operation code included in each second code and the key strings of the target two-dimensional array data.

[0052] Specifically, first, if the aggregate operation code included in the second code includes a function that calculates the maximum value, minimum value, sum, or number (sum function, max function, min function, count function), the selection unit 13 combines sets of key strings of the second codes and, for each combination, determines whether the sets of key strings of the target second codes included in the combination include sets of key strings of other second codes.

[0053] Next, if it is determined that the set of key strings of the target second code includes a combination that includes a set of key strings of another second code, the selection unit 13 selects the key strings of the second code included in that combination.

[0054] In the example of FIG. 5, the second codes include the aggregate operation code "['val'].agg("sum")".

[0055] In the example of Figure 5, the set of key sequences for the second chord shown in (1) is ['A', 'B', 'C', 'D']. The set of key sequences for the second chord shown in (2) is ['A', 'B', 'D', 'E']. The set of key sequences for the second chord shown in (3) is ['A', 'B', 'C', 'D', 'E']. The set of key sequences for the second chord shown in (4) is ['A', 'B', 'C', 'D', 'F'].

[0056] Next, in the combinations (1), (2), (3), and (4), there is no combination in which the set of key strings of the target second chord includes the set of key strings of another second chord.

[0057] Next, in the combinations (1)(2)(3), (1)(2)(4), (1)(3)(4), and (2)(3)(4), in the combination (1)(2)(3), the set of key strings in (3) includes the set of key strings in (1)(2), so in the example of Figure 5, the key strings in (1)(2)(3) are selected. In that case, the set of key strings in (4) is excluded.

[0058] In addition, the combinations (1)(2)(4), (1)(3)(4), and (2)(3)(4) are not selected because the set of key strings of the target second code does not include any other set of key strings of the second code.

[0059] The generation unit 14 generates a third code using the first function code, the key string used in the selected intermediate table, and the aggregate operation code, and adds it to the front of the second code.

[0060] Specifically, the generation unit 14 first generates a third code using the key string of the second code included in the selected combination, and then adds the generated third code to the front of the second code.

[0061] Fig. 6 is a diagram for explaining the third code of the first embodiment. In the example of Fig. 6, the combination of (1), (2), and (3) is selected, so the third code "tmp = table.groupby(['A', 'B', 'C', 'D', 'E'])['val'].agg("sum")" (underlined portion) is generated using the first function code "table.groupby," the set of key columns ['A', 'B', 'C', 'D', 'E'] (intermediate table) of the second code shown in (3), and the aggregate operation code "['val'].agg("sum")."

[0062] The conversion unit 15 converts the plurality of second codes into a fourth code by aligning them with the third code based on the third code. Specifically, the conversion unit 15 converts the table of the second codes included in the selected combination into a fourth code that uses an intermediate table of the third code based on the third code.

[0063] 7 is a diagram illustrating code matching in the first embodiment. In the example of FIG. 7, the combination of (1), (2), and (3) is selected, and therefore the second code of (1), (2), and (3) is converted based on the third code “tmp = table.groupby(['A', 'B', 'C', 'D', 'E'])['val'].agg("sum")” to convert it into the fourth code shown in FIG. 7 as (1) “tbl1 = tmp.groupby(['A', 'B', 'C', 'D'])['sum'].agg("sum")” (underlined), (2) “tbl2 = tmp.groupby(['A', 'B', 'D', 'E'])['sum'].agg("sum")” (underlined), and (3) “tbl3 = tmp” (underlined).

[0064] That is, the code is converted to execute the aggregation operation using an intermediate table tmp that is smaller in size than the original table table, rather than using the original large-sized table table.

[0065] In this way, the third code (tmp = table.groupby(['A', 'B', 'C', 'D', 'E'])['val'].agg("sum")) is generated, which performs a groupby sum operation once on table, and the second code is converted into a fourth code (tbl1 = tmp.groupby(['A', 'B', 'C', 'D'])['sum'].agg("sum"), tbl2 = tmp.groupby(['A', 'B', 'D', 'E'])['sum'].agg("sum"), tbl3 = tmp) which performs a groupby sum operation three times on the intermediate table tmp.

[0066] In embodiment 1, when it is determined that the set of key strings of the target second code includes a combination that includes a set of key strings of another second code, a third code is generated using the set of key strings of the target second code included in the selected combination, and based on the generated third code, the multiple second codes are transformed into a fourth code for aligning them with the third code.

[0067] Therefore, in the first embodiment, it is possible to speed up (shorten the calculation time) the grouping calculation using a plurality of key strings included in the table (two-dimensional array data). Also, in the first embodiment, it is possible to reduce the memory usage during the calculation.

[0068] [Device operation] Next, the operation of the code conversion device in embodiment 1 will be described with reference to Fig. 8. Fig. 8 is a diagram for explaining an example of the operation of the code conversion device in embodiment 1. In the following description, the diagram will be referenced as appropriate. Furthermore, in embodiment 1, the code conversion method is implemented by operating the code conversion device. Therefore, the description of the code conversion method in embodiment 1 will be replaced by the following description of the operation of the code conversion device.

[0069] As shown in FIG. 8, first, the detection unit 11 detects a first code including a first function code that combines multiple key strings included in a table (two-dimensional array data) and executes a grouping operation for each combined key string from an input code that is input to be executed by a computer and that is stored in advance in the storage device 20 (step A1).

[0070] Next, the extraction unit 12 extracts, from the detected plurality of first codes, a plurality of second codes whose target tables (two-dimensional array data) are the same as the first function codes and whose aggregate operation codes are included in the first codes (step A2).

[0071] Next, the selection unit 13 selects a key string to be used in the intermediate table to reduce the key string of the target two-dimensional array data, based on the aggregate operation code included in each second code and the key string of the target two-dimensional array data (step A3).

[0072] Next, the generation unit 14 generates a third code using the first function code, the key string used in the selected intermediate table, and the aggregate operation code (step A4), and adds it to the front of the second code (step A5).

[0073] Next, the conversion unit 15 converts the plurality of second codes into a fourth code by aligning them with the third code based on the third code (step A6).

[0074] Even if the input code contains a second code that uses multiple different tables, the input code can be converted into a code that allows high-speed grouping operations by repeating the processing of steps A1 to A6 described above.

[0075] [Effects of the First Embodiment] As described above, according to the first embodiment, when it is determined that the set of key strings of the target second code includes a combination that includes a set of key strings of another second code, a third code is generated using the set of key strings (intermediate table) of the target second code included in the selected combination, and based on the generated third code, multiple second codes are aligned with the third code and converted into a fourth code.

[0076] Therefore, in the first embodiment, it is possible to speed up (shorten the calculation time) the grouping calculation using a plurality of key strings included in the table (two-dimensional array data). Also, in the first embodiment, it is possible to reduce the memory usage during the calculation.

[0077] To give a more specific example, in the case of an input code that targets a table (1 million records) containing information on age ranges (6 levels), prefectures of residence (47 combinations), and blood types (4 combinations), and that calculates the maximum purchase amount for each combination of (age, prefecture of residence), (age, blood type), and (prefecture of residence, blood type), the code will perform aggregation three times using 1 million pieces of data, resulting in duplicate processing (similar but unnecessary processing).

[0078] However, according to the first embodiment, a table (1 million items) is used to first generate a third code (a single aggregation of 1 million items of data) that calculates the maximum purchase amount for the combination of (age, prefecture of residence, blood type). That is, an intermediate table (maximum 6 x 47 x 4 = 1128 items) is generated using the third code.

[0079] Next, using the data from the intermediate table (maximum 1128 items), a fourth code (three aggregations of 1128 data items) is generated that calculates the maximum value for each combination of (age, prefecture of residence), (age, blood type), and (prefecture of residence, blood type).

[0080] In this way, by converting the input code that performs three aggregations using 1 million pieces of data into code that performs one aggregation on 1 million pieces of data and three aggregations on 1,128 pieces of data, it is possible to speed up (shorten calculation time) grouping calculations using multiple key columns contained in the original table (two-dimensional array data).It also reduces memory usage during calculations.

[0081] [program] The program in the first embodiment may be any program that causes a computer to execute steps A1 to A6 shown in Fig. 8. By installing and executing this program in a computer, the code conversion device and code conversion method in the first embodiment can be realized. In this case, the processor of the computer functions as a detection unit 11, an extraction unit 12, a selection unit 13, a generation unit 14, and a conversion unit 15, and performs the processing.

[0082] The program in embodiment 1 may be executed by a computer system constructed by a plurality of computers, in which case, for example, each computer may function as one of the detection unit 11, extraction unit 12, selection unit 13, generation unit 14, and conversion unit 15.

[0083] (Embodiment 2) In the second embodiment, if the aggregate operation code includes the operation of the maximum value, minimum value, sum, count, or average value, it is determined whether it can be sped up, and if it is determined that it can be sped up, the input code is converted.

[0084] The configuration of the code conversion device in the second embodiment will be described with reference to Fig. 9. Fig. 9 is a diagram illustrating an example of the code conversion device in the second embodiment.

[0085] As shown in FIG. 9, a code conversion device 10a according to the second embodiment includes a detection unit 11, an extraction unit 12, a selection unit 13a, a generation unit 14, and a conversion unit 15.

[0086] Since the detection unit 11, extraction unit 12, generation unit 14, and conversion unit 15 have already been described, detailed description of the detection unit 11, extraction unit 12, generation unit 14, and conversion unit 15 will be omitted.

[0087] When the operation of the aggregate operation code included in the second code is a maximum value, a minimum value, a sum, a count, or an average value, the selection unit 13a determines whether the processing using the converted third code will be faster than the processing before conversion, based on the sum of the number of key strings included in each second code and the size of the set sum of the key strings in each second code.

[0088] Specifically, first, the selection unit 13a determines whether the aggregate calculation code included in the second code includes a function (sum function, max function, min function, count function, mean function) that calculates the maximum value, minimum value, sum, number, or average value.

[0089] Next, if the aggregation operation code included in the second code includes a function that calculates the maximum value, minimum value, sum, number, or average value, the selection unit 13a calculates the sum P of the number of columns of the key columns included in each second code and the size Q of the set sum of the key columns in each second code.

[0090] In the example of Figure 5, the number of columns in each key sequence of the second codes (1) to (4) containing the sum function is calculated. The number of columns in the key sequence ['A', 'B', 'C', 'D'] of (1) is 4, the number of columns in the key sequence ['A', 'B', 'D', 'E'] of (2) is 4, the number of columns in the key sequence ['A', 'B', 'C', 'D', 'E'] of (3) is 5, and the number of columns in the key sequence ['A', 'B', 'C', 'D', 'F'] of (4) is 5. Next, the sum P of the number of columns in (1) to (4) is calculated, and the sum P is 18 The result is (=4+4+5+5).

[0091] Also, in the example of Figure 5, the set sum of the key strings for each of the second codes (1) to (4) containing the sum function is ['A', 'B', 'C', 'D', 'E', 'F'], so the size Q is set to 6.

[0092] Next, the selection unit 13a calculates the cost X before the conversion of the process flow and the cost Y after the conversion of the process flow based on the sum P of the number of columns and the size Q of the set sum.

[0093] The cost X before the process flow conversion can be expressed, for example, using the sum P of the number of columns. Specifically, the cost X before the process flow conversion can be expressed using the area of ​​the table used in each groupby. Here, the area of ​​the table used in each groupby is expressed as the sum P of the number of key columns used in each groupby × the number L of rows in the original table.

[0094] In the example of Figure 5, if the number of rows in the original table table is L, the area of ​​(1) is 4L, the area of ​​(2) is 4L, the area of ​​(3) is 5L, and the area of ​​(4) is 5L. Therefore, in the example of Figure 5, the cost X before the process flow conversion is 4L + 4L + 5L + 5L = 18 It becomes L.

[0095] If the aggregation code contains a function that calculates the maximum, minimum, sum, or number of items, the cost Y after the processing flow conversion can be expressed as, for example, (size of the set sum Q × number of rows in the original table L) + ((coefficient α × number of rows in the original table L) × sum P of the number of key columns used in each groupby).

[0096] Specifically, the cost Y after converting the process flow can be expressed as the sum of the cost of generating the intermediate table and the cost of calculating groupby from the intermediate table. The cost of calculating groupby from the intermediate table can be expressed as P × (α × L), assuming that the number of rows in groupby, L, is reduced by a factor of α (0≦α<1). Therefore, the cost Y after converting the process flow can be expressed as (Q × L) + P × (α × L).

[0097] The coefficient α is set in advance to an arbitrary value within the range of 0≦α<1. The coefficient α is set on the assumption that the larger the coefficient α, the larger the intermediate table will remain and will not become smaller.

[0098] In the example of Figure 5, if the coefficient α is set to 0.2, the cost Y after the processing flow conversion in (1) to (4) is Y = 6L + 0.2L × 14 (= L × (6 + 0.2 × 14) = 8.8L), where Q is the size of the set sum, L is the number of rows in the original table, and P is the sum of the number of key columns used in each groupby.

[0099] Note that both the cost X before the process flow conversion and the cost Y after the process flow conversion include the number of rows L of the original table, but since they are the same number of rows, you can simply compare the cost X of 14 with the cost 8.8.

[0100] If the aggregation code contains a function that calculates the average value, the cost Y after converting the processing flow can be expressed as ((Q × L) + ((α × L) × P)) × 2. The reason for doubling is that in the case of the average value, the calculation is performed using the sum and the number of items.

[0101] Next, the selection unit 13a compares the cost X before the process flow conversion with the cost Y after the process flow conversion to determine whether or not an effect of speeding up can be obtained. That is, the selection unit 13a determines that an effect of speeding up can be obtained if the cost Y after the process flow conversion is smaller than the cost X before the process flow conversion (X>Y).

[0102] In the example of FIG. 5, the cost X before the process flow conversion is 14, and the cost Y after the process flow conversion is 8.8, so it can be determined that an effect of speeding up can be obtained.

[0103] Next, the selection unit 13a selects the key string of the second chord to be included in the combination determined to be effective in increasing the speed.

[0104] Next, the generation unit 14 generates a third code using the key string of the second code included in the combination determined to be effective in increasing the speed, and then adds the generated third code to the front of the second code.

[0105] Next, the conversion unit 15 converts the second code included in the selected combination into a fourth code based on the third code by aligning it with the third code.

[0106] Fig. 10 is a diagram for explaining the second code of the second embodiment. The example of Fig. 10 shows a plurality of second codes 50a ((1)(2)(3)(4)) extracted by the extraction unit 12 from a plurality of first codes detected by the detection unit 11. Also, 51a in Fig. 10 shows a first function code (groupby()) and the same table targeted by the first function code.

[0107] 10 includes an aggregate operation code (['val'].agg("mean")) included in the second code. "mean" in the aggregate operation code represents the mean function.

[0108] In the example of Figure 10, the set of key sequences for the second chord shown in (1) is ['A', 'B', 'C', 'D']. The set of key sequences for the second chord shown in (2) is ['A', 'B', 'D', 'E']. The set of key sequences for the second chord shown in (3) is ['A', 'B', 'C', 'D', 'E']. The set of key sequences for the second chord shown in (4) is ['A', 'B', 'C', 'D', 'F'].

[0109] 11 is a diagram illustrating the third code of the second embodiment. In the example of FIG. 11, in the combination of (1), (2), (3), and (4), the selection unit 13a determines that an effect of speeding up can be obtained if the cost Y after the processing flow conversion is smaller than the cost X before the processing flow conversion (X>Y). Therefore, the selection unit 13a generates the third code ("tmp = table.groupby(['A', 'B', 'C', 'D', 'E', 'F'])['val'].agg(['sum', 'count'])" (underlined portion)) using the first function code "table.groupby", the set of key strings ['A', 'B', 'C', 'D', 'E', 'F'] of the second code shown in (3), and the aggregate operation code "['val'].agg('sum')").

[0110] Fig. 12 is a diagram for explaining code matching in embodiment 2. Next, in the example of Fig. 12, the combination of (1), (2), (3), and (4) is selected, so the second code of (1), (2), (3), and (4) is converted into a fourth code based on the third code "tmp = table.groupby(['A', 'B', 'C', 'D', 'E', 'F'])['val'].agg(["sum", "count"])".

[0111] That is, the second code is "tmp1 = tmp.groupby(['A', 'B', 'C', 'D'])["sum","count"].agg("sum")" (underlined part), "tmp2 = tmp.groupby(['A', 'B', 'D', 'E'])["sum","count"].agg("sum")" (underlined part), "tmp3 = tmp.groupby(['A', 'B', 'C', 'D', 'E'])["sum","count"].agg("sum")" (underlined part), "tmp4 = tmp.groupby(['A', 'B', 'C', 'D', 'F'])["sum","count"].agg("sum")" (underlined part) in Figure 12. These are converted into the fourth code shown below: (1) pandas.DataFrame((tmp1["sum"] / tmp1["count"]).rename("mean")) (underlined part), (2) "tbl2 = pandas.DataFrame((tmp2["sum"] / tmp2["count"]).rename("mean"))" (underlined part), (3) "tbl3 = pandas.DataFrame((tmp3["sum"] / tmp3["count"]).rename("mean"))" (underlined part), (4) "tbl4 = pandas.DataFrame((tmp4["sum"] / tmp4["count"]).rename("mean"))" (underlined part).

[0112] In this way, the third code (tmp = table.groupby(['A','B','C','D','E','F']['val'].agg(["sum", "count"])) is generated, which performs a groupby sum calculation on table once, and the second code is converted into the fourth code, which calculates the groupby average value on the intermediate table tmp.

[0113] In the intermediate table tmp in Figure 12, the sum of the values ​​for each group obtained by groupby on A, B, C, D, E, and F is stored in the sum column, and the number of values ​​is stored in the count column. Furthermore, by performing groupby+agg(sum) on the intermediate table tmp, the sum and number of values ​​for each group can be calculated, and the average can be calculated by dividing the sum by the number.

[0114] In the second embodiment, if the aggregate operation code includes the operation of the maximum value, minimum value, sum, count, or average value, it is determined whether it can be sped up, and if it is determined that it can be sped up, the input code is converted.

[0115] Therefore, the embodiment 2 In the embodiment, it is possible to speed up (reduce the calculation time) the grouping calculation using a plurality of key strings included in the table (two-dimensional array data). 2 In this case, the amount of memory used during the calculation can be reduced.

[0116] [Device operation] Next, the operation of the code conversion device in the second embodiment will be described with reference to Fig. 13. Fig. 13 is a diagram for explaining an example of the operation of the selection unit of the code conversion device in the second embodiment. In the following description, the diagram will be referred to as appropriate. Furthermore, in the second embodiment, the code conversion method is implemented by operating the code conversion device. Therefore, the description of the code conversion method in the second embodiment will be replaced by the following description of the operation of the code conversion device.

[0117] In the second embodiment, the process of step A3 in the first embodiment described with reference to FIG. 8 is replaced with the processes of steps B1 to B5 shown below.

[0118] As shown in FIG. 13, first, the selection unit 13a determines whether the aggregate operation code included in the second code extracted in the processing of steps A1 to A2 in FIG. 8 includes a function (sum function, max function, min function, count function, mean function) that calculates the maximum value, minimum value, total sum, number, or average value (step B1).

[0119] Next, if the aggregate operation code included in the second code includes a function that calculates the maximum value, minimum value, sum, number, or average value, the selection unit 13a calculates the sum P of the number of key strings included in each second code and the size Q of the set sum of the key strings in each second code (step B2).

[0120] Next, the selection unit 13a calculates the cost X before the process flow conversion and the cost Y after the process flow conversion based on the sum P of the number of columns and the size Q of the set sum (step B3).

[0121] Next, the selection unit 13a compares the cost X before the process flow conversion with the cost Y after the process flow conversion to determine whether or not an effect of speeding up can be obtained (step B4). That is, in step B4, the selection unit 13a determines that an effect of speeding up can be obtained if the cost Y after the process flow conversion is smaller than the cost X before the process flow conversion (X>Y).

[0122] Next, the selection unit 13a selects the key string of the second chord included in the combination determined to be effective in increasing the speed (step B5), and then executes the processes of steps A4 to A6 in FIG.

[0123] [Effects of the second embodiment] As described above, according to the second embodiment, if the aggregate operation code includes the operation of the maximum value, minimum value, sum, count, or average value, it is determined whether it can be sped up, and if it is determined that it can be sped up, the input code is converted.

[0124] Therefore, in the second embodiment, it is possible to speed up (reduce the calculation time) the grouping calculation using a plurality of key strings included in the table (two-dimensional array data). 2 In this case, the amount of memory used during the calculation can be reduced.

[0125] [program] The program in the second embodiment may be a program that causes a computer to execute steps A1 to A2 and steps A4 to A6 shown in Fig. 8 and steps B1 to B5 shown in Fig. 13. By installing and executing this program in a computer, the code conversion device and code conversion method in the second embodiment can be realized. In this case, the processor of the computer functions as a detection unit 11, an extraction unit 12, a selection unit 13a, a generation unit 14, and a conversion unit 15, and performs the processing.

[0126] Also, embodiments 2 The program in the above may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may function as one of the detection unit 11, extraction unit 12, selection unit 13a, generation unit 14, and conversion unit 15.

[0127] [Physical configuration] A computer that realizes the code conversion device by executing the program according to the first and second embodiments will now be described with reference to Fig. 14. Fig. 14 is a diagram illustrating an example of a computer that realizes the code conversion device according to the first and second embodiments.

[0128] 14, the computer 110 includes a CPU (Central Processing Unit) 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader / writer 116, and a communication interface 117. These components are connected to each other via a bus 121 so as to be able to communicate data with each other. Note that the computer 110 may include a GPU or an FPGA in addition to or instead of the CPU 111.

[0129] The CPU 111 loads the programs (codes) in the embodiment stored in the storage device 113 into the main memory 112 and executes them in a predetermined order to perform various calculations. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory). The programs in the embodiment are provided in a state stored in a computer-readable recording medium 120. The programs in the embodiment may be distributed over the Internet connected via the communication interface 117. The recording medium 120 is a non-volatile recording medium.

[0130] Specific examples of the storage device 113 include a hard disk drive and a semiconductor storage device such as a flash memory. The input interface 114 mediates data transmission between the CPU 111 and input devices 118 such as a keyboard and a mouse. The display controller 115 is connected to a display device 119 and controls the display on the display device 119.

[0131] The data reader / writer 116 mediates data transmission between the CPU 111 and the recording medium 120, reads programs from the recording medium 120, and writes processing results from the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and other computers.

[0132] Specific examples of the recording medium 120 include general-purpose semiconductor storage devices such as CF (Compact Flash (registered trademark)) and SD (Secure Digital), magnetic recording media such as flexible disks, or optical recording media such as CD-ROMs (Compact Disk Read Only Memory).

[0133] The code conversion device in the first and second embodiments can be realized by using hardware corresponding to each unit, instead of a computer on which a program is installed. Furthermore, the code conversion device may be partially realized by a program and the remaining part by hardware.

[0134] Although the present invention has been described with reference to the embodiments, it is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the invention within the scope of the invention. [Industrial Applicability]

[0135] According to the above description, it is possible to speed up (reduce the calculation time) the grouping calculation using multiple key strings in a table (two-dimensional array data) included in an input code. Also, it is useful in fields where grouping calculation using multiple key strings included in two-dimensional array data (table) is required. [Explanation of symbols]

[0136] 10, 10a code conversion device 11 Detection unit 12 Extraction part 13, 13a Selection section 14 Generation part 15 Conversion unit 20 Storage device 100, 100a systems 110 Computer 111 CPU 112 main memory 113 Storage device 114 Input Interface 115 Display Controller 116 Data Reader / Writer 117 Communication Interface 118 Input Devices 119 Display Device 120 Recording Media 121 Bus

Claims

1. a detection means for detecting a first code from an input code that is input to be executed by a computer and that is stored in advance in a storage device, the first code including a first function code that combines a plurality of key strings included in the two-dimensional array data and executes a grouping operation for each combined key string; an extraction means for extracting, from the detected plurality of first codes, a plurality of second codes whose first function codes target the same two-dimensional array data and whose first codes include the same aggregate operation code; a selection means for selecting a key string to be used in an intermediate table obtained by reducing the key string of the target two-dimensional array data, based on the aggregate operation code included in each of the second codes and the key string of the target two-dimensional array data; a generating means for generating a third code using the first function code, a selected key string, and an aggregate operation code, and adding the third code to the front of the second code; a conversion means for converting the plurality of second codes into a fourth code by aligning the second codes with the third code based on the third code; A code conversion device having the following.

2. The selection means if the aggregate operation code included in the second code includes a function for calculating a maximum value, a minimum value, a sum, or a number, combine the sets of key strings of the second codes, and for each combination, determine whether the set of key strings of the target second code included in the combination includes a set of key strings of another second code; When it is determined that the set of key strings of the target second code includes a combination including the set of key strings of the other second code, selecting the set of key strings of the second code included in the combination. The code conversion device according to claim 1 .

3. The selection means if the operation of the aggregate operation code included in the second code is a maximum value, a minimum value, a sum, a count, or an average value, determine whether or not the processing using the converted third code can be performed faster than the processing before conversion based on the sum of the number of columns of the key columns included in each of the second codes and the size of the set sum of the key columns of each of the second codes; The code conversion device according to claim 1 .

4. The computer detecting a first code including a first function code for combining a plurality of key strings included in the two-dimensional array data and executing a grouping operation for each combined key string from input codes that are input to be executed by the computer and that are stored in advance in a storage device; extracting, from the detected plurality of first codes, a plurality of second codes whose first function codes target the same two-dimensional array data and whose first codes include the same aggregate operation code; selecting a key string to be used in an intermediate table obtained by reducing the key string of the target two-dimensional array data based on the aggregate operation code included in each of the second codes and the key string of the target two-dimensional array data; generating a third code using the first function code, the selected key string, and an aggregate operation code, and adding the third code to the front of the second code; aligning the plurality of second codes with the third code based on the third code and converting them into a fourth code; Code conversion method.

5. if the aggregate operation code included in the second code includes a function for calculating a maximum value, a minimum value, a sum, or a number, combine the sets of key strings of the second codes, and for each combination, determine whether the set of key strings of the target second code included in the combination includes a set of key strings of another second code; When it is determined that the set of key strings of the target second code includes a combination including the set of key strings of the other second code, selecting the set of key strings of the second code included in the combination. The code conversion method according to claim 4.

6. if the operation of the aggregate operation code included in the second code is a maximum value, a minimum value, a sum, a count, or an average value, determine whether or not the processing using the converted third code can be performed faster than the processing before conversion based on the sum of the number of columns of the key columns included in each of the second codes and the size of the set sum of the key columns of each of the second codes; The code conversion method according to claim 4.

7. On the computer, detecting a first code including a first function code for combining a plurality of key strings included in the two-dimensional array data and performing a grouping operation for each combined key string from an input code that is input to be executed by the computer and that is stored in advance in a storage device; extracting, from the detected plurality of first codes, a plurality of second codes whose first function codes target the same two-dimensional array data and whose first codes include the same aggregate operation code; selecting a key string to be used in an intermediate table obtained by reducing the key string of the target two-dimensional array data based on the aggregate operation code included in each of the second codes and the key string of the target two-dimensional array data; generating a third code using the first function code, the selected key string, and an aggregate operation code, and adding the third code to the front of the second code; aligning the plurality of second codes with the third code based on the third code and converting them into a fourth code; A program containing instructions.

8. If the aggregate operation code included in the second code includes a function for calculating a maximum value, a minimum value, a sum, or a number, the sets of key strings of the second codes are combined, and for each combination, it is determined whether the sets of key strings of the second codes included in the combination include sets of key strings of other second codes; when it is determined that the set of key strings of the target second code includes a combination including the set of key strings of the other second code, selecting the set of key strings of the second code included in the combination; The program according to claim 7.

9. if the operation of the aggregate operation code included in the second code is a maximum value, a minimum value, a sum, a count, or an average value, a determination is made as to whether or not the processing using the converted third code can be performed faster than the processing before conversion, based on the sum of the number of columns of the key columns included in each of the second codes and the size of the set sum of the key columns of each of the second codes; The program according to claim 7.

Images