Data processing method and data processing program

Abstract

To perform large-scale calculations on a GPU, the necessary data sets are processed and restructured into a format that allows for parallel, batch computation. [Solution] A data processing method for processing data in data frame format using a computer equipped with a GPU, comprising: a first step of reading all or part of the data to be processed by the computer and expanding it into a data frame in the GPU's memory; a second step of merging second data necessary for processing the first data expanded into the data frame into the data frame; a third step of determining whether each column of the data frame meets the conditions for executing a predetermined process; and a fourth step of extracting rows in the data frame in which the values ​​of each column meet the conditions, executing a predetermined process on each extracted row by parallel processing on the GPU, and recording the processing results in each row.

Description

Technical Field

[0001] The present invention relates to data processing technology in computers, and particularly relates to technology that is effectively applied to data processing methods and data processing programs that perform data processing by means of a GPU (Graphics Processing Unit).

Background Art

[0002] In recent years, GPGPU (General-purpose computing on GPU), which uses GPUs in fields requiring enormous arithmetic processing such as mining in blockchain technology, development of generative AI (Artificial Intelligence), machine learning, and simulation, has become widespread (see, for example, Non-Patent Document 1), and has achieved results in computationally intensive areas that are difficult for CPUs (Central Processing Units) to handle.

[0003] As a technology for using GPUs for general-purpose computing purposes other than their original image processing, for example, there is CUDA (Compute Unified Device Architecture) (Non-Patent Document 2) (registered trademark, the same applies hereinafter), a general-purpose parallel computing platform and programming model for GPUs developed and provided by NVIDIA (registered trademark).

Prior Art Literature

Non-Patent Literature

[0004]

Non-Patent Document 1

Non-Patent Document 2

[0005] In existing core systems built on mainframes and other infrastructure (so-called "legacy systems"), addressing issues such as the end of maintenance support for the running hardware and software is a challenge. Therefore, when upgrading systems, migrations such as changing the application language are sometimes performed. Legacy systems often involve massive data processing, including batch processing, making speeding up these processes crucial during migration.

[0006] Since mainframe batch processing involves data aggregation, it is expected that applying CUDA or similar technologies to enable GPU processing during migration will speed up the process.

[0007] As shown in Non-Patent Document 1, etc., GPU processing is suitable for processing large amounts of the same calculation in a single batch because it can perform parallel calculations using a large number of processing cores. However, in programs that include branching, where the content of the processing or calculations changes depending on the content or conditions of the data, the processing performance may drop drastically, and it may not be possible to achieve high speed. Programs such as batch processing in legacy systems may have branching that degrades processing performance because they were not designed with parallel processing by GPUs in mind, and there are challenges in achieving high speed.

[0008] Therefore, the object of the present invention is to provide a data processing method and a data processing program that process and reconstruct the necessary data sets into a form that can be computed in parallel in a single operation when performing large-scale calculations on a GPU. The aforementioned and other objects and novel features of the present invention will become clear from the description herein and the accompanying drawings. [Means for solving the problem]

[0009] A brief overview of some of the representative inventions disclosed in this application is as follows:

[0010] A representative embodiment of the present invention is a data processing method for processing data in data frame format using a computer equipped with a GPU, comprising: a first step of reading all or part of the data to be processed by the computer and expanding it into a data frame in the memory of the GPU; a second step of merging second data necessary for processing the first data expanded into the data frame into the data frame; a third step of determining whether the conditions for executing a predetermined process are met for each column of the data frame; and a fourth step of extracting rows in the data frame in which the values ​​of each column meet the conditions, executing the predetermined process on each extracted row by parallel processing of the GPU, and recording the processing results in each row.

[0011] Furthermore, a typical embodiment of the present invention can also be applied to a data processing program that causes a computer to execute the above-described data processing method. [Effects of the Invention]

[0012] The effects obtained by a representative embodiment of the invention disclosed in this application can be briefly explained as follows: In other words, according to a representative embodiment of the present invention, when performing a large amount of computation on a GPU, it becomes possible to process and reconstruct the necessary data sets into a form that can be computed in parallel in a single operation. [Brief explanation of the drawing]

[0013] [Figure 1] This figure outlines an example of a data processing method, which is one embodiment of the present invention. [Figure 2] This diagram provides an overview of a current example of data frame processing. [Figure 3] This figure outlines an example of data frame processing in one embodiment of the present invention. [Figure 4] This diagram provides an overview of an example of the current IF branching process. [Figure 5] This figure outlines an example of an IF branching process in one embodiment of the present invention. [Figure 6] This figure outlines an example of a specific calculation process in one embodiment of the present invention. [Figure 7] This figure outlines another example of data frame processing in one embodiment of the present invention. [Figure 8] This figure outlines another example of data frame processing in one embodiment of the present invention. [Figure 9] This figure outlines another example of data frame processing in one embodiment of the present invention. [Figure 10] This figure outlines another example of data frame processing in one embodiment of the present invention. [Modes for carrying out the invention]

[0014] Embodiments of the present invention will be described in detail below with reference to the drawings. In principle, the same parts will be denoted by the same reference numerals in all the drawings used to describe the embodiments, and repeated descriptions will be omitted. On the other hand, a part that is denoted by a reference numeral and described in one drawing may be referred to again in the description of another drawing, although it will not be shown again.

[0015] <Overview> In recent data processing, flexible and efficient processing is often implemented by using data frames (table-formatted data composed of rows and columns). In the aforementioned CUDA, cuDF (cuda based DataFrames, https: / / github.com / rapidsai / cudf), a library for performing data frame processing on GPUs, is also provided. The data processing method according to one embodiment of the present invention also achieves speedup by performing calculation processing in batch processing and the like through data frame processing on a GPU.

[0016] Figure 2 is a diagram outlining an example of conventional data frame processing. A simple example is shown here, in which a total amount is calculated by multiplying quantity data by a predetermined unit price. In conventional processing, the process of reading quantity records one by one from a file, acquiring unit price information corresponding to the record, multiplying the quantity by the unit price to calculate the total amount is repeated for the number of records in the file.

[0017] That is, as shown in the example of Figure 2, first, record data (quantity in the example of Figure 2) identified by a primary key (PK) is read one by one from a file and expanded into a data frame (data frame 2a). Then, corresponding unit price information is acquired from a database (DB) or the like and recorded in the data frame (data frame 2b), and the total amount obtained by multiplying the quantity by the unit price is recorded in the data frame (data frame 2c). The above processing is repeated for the number of records in the file, which is a loop processing.

[0018] Figure 3 is a diagram outlining an example of data frame processing according to one embodiment of the present invention. Here, an example of performing data frame processing on a GPU in the same example as the aforementioned Figure 2 is shown. After reading the entire file (or a chunk unit of a predetermined size) and merging the required unit price information, the GPU performs parallel calculation processing on a plurality of records.

[0019] That is, as shown in the example of FIG. 3, first, the entire file (PK and quantity) is read and expanded into a data frame (data frame 1a). Then, after acquiring necessary unit price information from a file or the like and merging it into the data frame (data frame 1b), the target total amount for a plurality of records is collectively calculated by parallel processing using a GPU (data frame 1d).

[0020] As described above, by making the calculation processing for a data frame column-oriented processing as shown in the example of FIG. 3 instead of the conventional row-oriented processing as shown in the example of FIG. 2, it is possible to achieve speedup by incorporating the processing into parallel computation using a GPU.

[0021] <IF branch processing> As a typical syntax of branch processing, there is an IF branch in which processing YY is performed when a condition XX is satisfied (or not satisfied).

[0022] FIG. 4 is a diagram outlining an example of conventional IF branch processing. An example of a COBOL program whose content is branched by an IF statement is shown in the upper part of the figure, and a data frame 2 similar to the example of FIG. 2 described above is shown in the lower part. In conventional processing, for each record (row) of data frame 2, it is sequentially checked whether the conditions of the IF statement ("CONDITION1", "CONDITION2") are satisfied, and only when both conditions are satisfied, the value of the record ("RECORDS(IDX)") is used to execute a predetermined processing ("PROC-A"), and this processing is repeated. In the example of data frame 2 in FIG. 4, it is shown that, assuming that the two records with PK "N01-102" and "N01-103" satisfy the condition, the total amount is calculated from the quantity and unit price as the predetermined processing.

[0023] Figure 5 is a diagram illustrating an example of IF branching processing in one embodiment of the present invention. The upper part of the figure shows an example of a Python (registered trademark, hereinafter the same) program using the cuDF library that performs processing when a condition is met, and the lower part shows a data frame 1 similar to the example in Figure 3 described above. In this embodiment, the condition is checked for each column of data frame 1 ("condition1_func", "condition2_func"), and only the matching rows ("cond1_series", "cond2_series" both True) are filtered, and a predetermined process ("procA") is performed all at once by parallel processing using the GPU. In the example of data frame 1 in Figure 5, it is shown that the condition is met for two records, PK "N01-102" and "N01-103", and the total amount is calculated from the quantity and unit price as a predetermined process for these records all at once.

[0024] Figure 1 is a diagram illustrating an example of a data processing method that is one embodiment of the present invention. Here, a more specific example of data frame processing related to the IF branching process shown in the example of Figure 5 above is shown. Similar to the example of Figure 3 above, first, the entire file (PK and quantity) is read and expanded into a data frame (data frame 1a). Then, the necessary unit price information is obtained from the file, etc., and merged into the data frame (data frame 1b). Then, it is determined whether each column meets the conditions, and the target value, the total amount, is calculated all at once using parallel processing by the GPU for only the rows that meet the conditions (data frame 1c).

[0025] Figure 1 shows an example where the rows that meet the quantity condition are 3 records with primary keys “N01-102”, “N01-103”, and “N01-104” (marked with a star in the quantity column in the figure), and the rows that meet the unit price condition are 3 records “N01-101”, “N01-102”, and “N01-103” (marked with a star in the unit price column in the figure). In this case, it is shown that only for the records “N01-102” and “N01-103”, which meet the conditions in both columns, the target value, the total amount, is calculated in a batch using parallel processing by the GPU, and the result is set in the total amount column.

[0026] Figure 6 is a diagram illustrating an example of a specific calculation process in one embodiment of the present invention. Here, an example is shown in which a target value is calculated based on the values ​​of predetermined columns, targeting only the rows in a data frame that meet certain conditions. The upper part of the figure shows an example of a Python program using cuDF, similar to the example in Figure 5 described above. The lower part shows an example of how the data frame is expanded on the memory of a computer's storage and GPU.

[0027] Figure 6 shows an example where the file to be processed, recorded on storage 10, is a fixed-length file, which is commonly used in batch processing of legacy systems. This fixed-length file contains, for example, n rows of data in four columns, “col0” to “col3”. The data from this fixed-length file is read and expanded as a data frame on the GPU memory, as shown on the GPU 20a, either through CPU processing (not shown) or, if possible, directly without CPU processing using DMA (Direct Memory Access) technology.

[0028] At this point, in order to effectively utilize parallel processing by the GPU, the data in each row is converted to an alignment where the rows and columns are swapped, as shown in the diagram, so that the data in each row is arranged in the direction of data reading by the GPU, and then loaded into GPU memory. This allows the processing of determining whether each column meets the conditions to be performed efficiently by parallel processing on the GPU.

[0029] Subsequently, conditional calculations are performed on the data frame according to the Python program. The program sets conditions for "col0" ("condition1_func") and "col1" ("condition2_func"), and as shown in GPU20b, the results of determining whether each row meets the respective conditions ("cond1_series", "cond2_series") are loaded into GPU memory.

[0030] Subsequently, the Python program calculates the target value for the data frame. The program instructs the system to execute a calculation ("result_func") based on the data in "col2" and "col3" for rows that meet the conditions (both "cond1_series" and "cond2_series" are True). As shown in GPU20c, the calculation result ("result") is displayed in the data frame only for rows that meet the conditions.

[0031] In this way, by performing preprocessing such as merging all the necessary data as a data frame on GPU memory, identifying rows that match the conditions for each column, filtering only the target rows, and performing calculations in a batch using parallel processing on the GPU, the processing can be significantly sped up. These processes can be implemented, for example, using a Python program with cuDF (the same applies to other processes shown in this embodiment).

[0032] <Data extraction processing using inequality signs, etc.> In addition to the IF branching mentioned above, other methods of processing data frames that are difficult to speed up through parallel processing include methods that extract and process data belonging to a predetermined range, such as the BETWEEN operator and conditional specification using inequality signs in COBOL. For example, this could occur when a financial institution calculates transaction fees for orders and the applicable consumption tax rate differs depending on the year the transaction takes place.

[0033] Figure 7 is a diagram illustrating another example of data frame processing in one embodiment of the present invention. The upper part of the figure shows an order data frame 3 and a consumption tax data frame 4. The order data frame 3 holds the order date and its contents for multiple orders, and the consumption tax data frame 4 holds the consumption tax rate for each year (application period) specified by the application start date.

[0034] In the example in Figure 7, when retrieving the consumption tax rate applicable to each order in the order dataframe 3, a conditional branching process is performed for each consumption tax rate record in the consumption tax dataframe 4 to determine whether the order date of each order in the order dataframe 3 falls within the applicable period for the target consumption tax rate, and the record contents of the orders that belong to that period are merged.

[0035] In the diagram, dataframe 5a shows that for records with a 10% consumption tax rate in the consumption tax dataframe 4, a record was created by extracting and merging order data from order dataframe 3 for dates after April 1, 2019, which was the effective date of the new rate. Furthermore, in dataframe 5b, a record was created for records with an 8% consumption tax rate by extracting and merging order data from order dataframe 3 for dates after April 1, 2014 (but before April 1, 2019), which was the effective date of the new rate.

[0036] By creating dataframes 5a and 5b using this method and then reconstructing the dataframes, it is possible to obtain the consumption tax rate applicable to the order date of each order. While this method is simple and easy to understand, it requires looping through the number of records in order dataframe 3 and consumption tax dataframe 4, which can be time-consuming.

[0037] Figure 8 is a diagram illustrating another example of data frame processing in one embodiment of the present invention. Here, similar to the example in Figure 7 above, with an order data frame 3 and a consumption tax data frame 4, the start dates for each consumption tax rate in the consumption tax data frame 4 are all expanded on a daily basis to create a consumption tax data frame 4a. Then, records where the dates (expanded start dates and order dates) match between the consumption tax data frame 4a and the order data frame 3 are joined to reconstruct the data frame.

[0038] In the example shown in Figure 8, the record for October 22, 2015, among the effective start dates expanded in the consumption tax dataframe 4a, matches the order date in the order dataframe 3. By joining these records, a consumption tax rate of 8% can be obtained as the applicable tax rate for that order. This method is suitable for GPU processing because it involves expanding and joining dataframes, and can be accelerated and made more efficient. However, since the expansion of the consumption tax dataframe 4a requires records for each date, the memory consumption on the GPU may increase significantly.

[0039] Figure 9 is a diagram illustrating another example of data frame processing in one embodiment of the present invention. Here, using the same order data frame 3 and consumption tax data frame 4 as in the example in Figure 7 above, these data frames are cross-combined and reconstructed, and only records that meet the conditions are extracted.

[0040] In the example in Figure 9, each record in the Order Dataframe 3 is merged with each record in the Consumption Tax Dataframe 4 to form Dataframe 6. Then, records that meet the condition that the order date belongs to the year specified by the effective date (in the example in the figure, records with an order date of October 22, 2015 and a consumption tax rate of 8%) are extracted. This method is suitable for GPU processing because it involves merging dataframes and filtering to extract only records that meet the conditions, allowing for faster and more efficient processing. However, since the total number of records in the resulting Dataframe 6 is the product of the number of records in the Order Dataframe 3 and the Consumption Tax Dataframe 4, the memory consumption on the GPU may increase significantly.

[0041] As shown in the examples in Figures 7 to 9, each data frame processing method has its advantages and disadvantages, so it is desirable to select a method appropriately according to the amount of data to be processed and other characteristics. While the examples in Figures 7 to 9 show the acquisition of the consumption tax rate applicable to an order, it goes without saying that the methods are not limited to such subjects or cases, but can be broadly and flexibly applied to other subjects and cases as well.

[0042] <When there are constraints such as dependencies or processing order between records> Another type of data frame processing that is difficult to speed up through parallel processing is when there are dependencies between records within or between data frames, or when there are constraints on the order of processing. For example, in securities trading, if multiple redemption orders are placed for a given stock in an account, the number of shares held may be insufficient to cover the total redemption quantity, requiring adjustment of the redemption quantity.

[0043] Figure 10 is a diagram illustrating another example of data frame processing in one embodiment of the present invention. Here, an example is shown where, when processing all cancellation orders related to the target account and securities in the order data frame 7 in the figure, the actual number of securities held in the holdings data frame 8 for those accounts and securities is insufficient (i.e., there are dependencies between records, and the holdings data frame 8 imposes constraints on the processing of the order data frame 7). For example, this can occur when cancellation orders are made by specifying an amount rather than a quantity, because the quantity to be canceled is not specified. In this case, for example, it is necessary to reduce the cancellation quantity related to the last / most recent order so that the total cancellation quantity is less than or equal to the number of securities held.

[0044] In the example in Figure 10, the order dataframe 7 already has the cancellation quantity calculated from the cancellation amount. The cancellation quantities for each record are aggregated and summed up by account and asset, and then compared by merging the records of the number of assets held for the target account and asset in the held assets dataframe 8 (dataframe 9). In the example in the figure, the total cancellation quantity is 500, while the actual number of assets held is only 400, showing a shortfall of 100.

[0045] Based on this comparison result, the difference is subtracted from the cancellation quantity of the last (most recent) record in order dataframe 7 to correct it (dataframe 7a), and the total cancellation quantity is recalculated and compared with the number of items held. In this way, when there are dependencies and constraints between records in a dataframe, preprocessing is performed to create an intermediate dataframe for determining whether the dependencies and constraints are satisfied, such as by aggregating, processing, and merging one or both dataframes, so that comparisons and processing can be performed. This allows the dataframe processing to be executed on the GPU.

[0046] As described above, according to one embodiment of the present invention, the data processing method involves performing preprocessing to expand all the necessary data as a data frame on the GPU memory, such as merging the data, and then processing and reconstructing the data frame so that it can be processed in a batch by parallel computing on the GPU, or is suitable for such processing, thereby speeding up the processing.

[0047] The present inventors have described the invention in detail based on embodiments above, but it goes without saying that the present invention is not limited to the above embodiments and can be modified in various ways without departing from its essence. Furthermore, the above embodiments are described in detail for the purpose of explaining the present invention in an easy-to-understand manner and are not necessarily limited to those having all the configurations described. In addition, it is possible to add, delete, or replace some of the configurations of the above embodiments with other configurations.

[0048] Furthermore, each of the above configurations, functions, processing units, and processing means may be implemented in hardware, in whole or in part, for example, by designing them as integrated circuits. Alternatively, each of the above configurations, functions, and means may be implemented in software by having the processor interpret and execute programs that implement each function. Information such as programs, tables, and files that implement each function can be stored in memory, hard disks, SSDs, or other recording devices, or in recording media such as IC cards, SD cards, or DVDs.

[0049] Furthermore, in the diagrams above, the control lines and information lines shown are those deemed necessary for explanation and do not necessarily represent all control lines and information lines that would be present in the actual implementation. In reality, it can be assumed that almost all components are interconnected. [Industrial applicability]

[0050] This invention can be used in data processing methods and data processing programs that perform data processing using a GPU. [Explanation of symbols]

[0051] 1, 1a~d...Data frame, 2, 2a~c...Data frame, 3...Order data frame, 4, 4a...Consumption tax data frame, 5a, b...Data frame, 6...Data frame, 7, 7a...Order data frame, 8...Deposit data frame, 9...Data frame, 10...Storage, 20a~c...GPU

Claims

1. A data processing method for processing data in data frame format using a computer equipped with a GPU (Graphics Processing Unit), The aforementioned computer, A first step involves reading all or part of the data to be processed and expanding it into a data frame in the memory of the GPU, A second step involves merging the second data necessary for processing the first data expanded into the aforementioned data frame into the aforementioned data frame, A third step involves determining whether the conditions for executing a predetermined process are met for each column of the data frame, A data processing method comprising: a fourth step of extracting rows in the data frame whose column values ​​match the above-mentioned conditions; executing the predetermined processing on each extracted row using parallel processing of the GPU; and recording the processing results in each row.

2. In the data processing method described in claim 1, In the second step, if each row of the second data is defined according to a range of values ​​in a predetermined column of the first data, the data processing method involves merging each row of the second data with the row of the first data to which the value of the predetermined column belongs to the range of that row, in order to form the data frame.

3. In the data processing method described in claim 1, In the second step, if each row of the second data is defined according to a range of values ​​in a predetermined column of the first data, the data processing method involves expanding all the values ​​in that range of the second data row by row, and combining each expanded row with the row in which the value in the predetermined column of the first data matches to form the data frame.

4. In the data processing method described in claim 1, The second step is a data processing method in which each row of the first data is multiplied by each row of the second data to form the data frame.

5. In the data processing method described in claim 1, In the second step, if there are constraints imposed by the second data when processing the first data, a data frame is created based on the first data and / or the second data for determining whether or not the constraints apply, and if it is determined based on the data frame that the constraints apply, the contents of the first data are modified based on the determination result to form the data frame.

6. In the data processing method described in claim 1, The first step is a data processing method in which, for each column of the data to be processed, the values ​​of each row related to that column are arranged in the GPU memory read direction and expanded into the data frame.

7. A data processing program that causes a computer equipped with a GPU (Graphics Processing Unit) to process data in data frame format, To the aforementioned computer, A first step involves reading all or part of the data to be processed and expanding it into a data frame in the memory of the GPU, A second step involves merging the second data necessary for processing the first data expanded into the aforementioned data frame into the aforementioned data frame, A third step involves determining whether the conditions for executing a predetermined process are met for each column of the data frame, A data processing program that performs a fourth step of extracting rows in the data frame whose column values ​​match the above conditions, executing the predetermined processing on each extracted row using parallel processing of the GPU, and recording the processing results in each row.

Images