Data processing device and data processing method

Abstract

To realize data area open / close suitable for processing speed, reliability and memory utilization.SOLUTION: A data processing device includes a memory and a first storage device, and reads a data area from a second storage device (a storage device having a lower speed than the first storage device and storing a plurality of data areas as components of data). The data processing device determines, for each processing block, whether an open / close area for processing a first processing block, which is the processing block, is all or a part of a data area based on whether or not the data area to be input / output in the processing of the first processing block can be accessed also in the processing of a second processing block, which is a processing block other than the first processing block, and whether or not the second processing block is being executed.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates generally to data processing, and more particularly to managing the opening and closing of data areas. [Background technology]

[0002] Examples of data areas, which are logical areas in which data is stored, include files in a file system and tables in a database. A data area may be called a data collection, and when it is open, the data in the data area can be written and / or read. In other words, while a data area is closed, the data in the data area cannot be written or read.

[0003] Regarding data area management, for example, there is a technology described in US5822780. According to this patent document, a hierarchical database is divided into several data areas, and the last access time is stored for each divided data area, and it is determined whether the data area is online or not. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] US5822780 Summary of the Invention [Problem to be solved by the invention]

[0005] In order to promote cloud services in the database field, there is a demand for the development of cache systems that use object storage. In particular, when multiple processes and multiple cache files exist, it is a challenge to ensure safety and reduce memory usage while maintaining processing speed.

[0006] There are two methods for opening and closing a cache file: opening the cache file each time it is accessed (e.g., each time it is accessed according to the query plan of a query), and leaving the cache file open throughout processing (e.g., processing according to the entire query plan of a query). The former method closes the cache file during time periods when it is not accessed. The latter method leaves the cache file open even if there are time periods when it is not accessed.

[0007] The former method slows down data processing speed because the cache file needs to be opened and closed frequently when different processes access the same cache file at different times.

[0008] The latter method avoids frequent opening and closing of cache files, but there is a possibility that the write status of many cache files may not be known correctly in the event of a failure, and it also consumes a large amount of main memory (e.g., work area).This is because the data in a cache file is written to main memory when the cache file is opened, and the data in the cache file remains in main memory while the cache file is open.

[0009] In this way, there is a trade-off between processing speed, reliability, and memory usage when opening and closing a cache file. In addition, such a problem may also occur in data areas other than the cache file. [Means for solving the problem]

[0010] The data processing device has a memory and a first storage device, and reads out a data area from a second storage device (a storage device that is slower than the first storage device and stores multiple data areas as components of data). The data processing device determines, for each processing block, whether the opening and closing area for the processing of the first processing block, which is the processing block, is the entire area of ​​the data area or a part of the area, based on whether the data area input / output in the processing of the first processing block can also be accessed in the processing of a second processing block, which is a processing block other than the first processing block, and whether the second processing block is being executed. Effect of the Invention

[0011] According to the present invention, it is possible to realize data area opening and closing that optimizes processing speed, reliability, and memory usage. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0012] [Figure 1] FIG. 2 is a configuration diagram of a database server according to the embodiment. [Figure 2A] FIG. 11 is a diagram showing a specific example of a cache file. [Figure 2B] FIG. 11 is a diagram showing a specific example of a cache file. [Diagram 3] FIG. 13 is a diagram showing a specific example of a table state management array. [Figure 4] 11 is a flowchart showing an example of an import process according to the embodiment. [Diagram 5] 5 is a flowchart of step a3 (mode determination) in FIG. 4. [Figure 6] 5 is a flowchart of the open / close region management in step a4 of FIG. 4; [Figure 7] 7 is a flowchart of step c3 in FIG. 6. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the embodiments described below do not limit the invention according to the claims, and all of the elements and combinations thereof described in the embodiments are not necessarily essential to the solution of the invention.

[0014] In the following description, an "interface unit" may be one or more interface devices. The one or more interface devices may be at least one of the following: An I / O interface apparatus which is one or more I / O (Input / Output) interface devices. The I / O (Input / Output) interface devices are interface devices to at least one of an I / O device and a remote display computer. The I / O interface device to the display computer may be a communications interface device. The at least one I / O device may be a user interface device, for example, either an input device such as a keyboard and a pointing device, or an output device such as a display device. A communication interface apparatus which is one or more communication interface devices. The one or more communication interface devices may be one or more homogeneous communication interface devices (e.g., one or more NICs (Network Interface Cards)) or two or more heterogeneous communication interface devices (e.g., a NIC and an HBA (Host Bus Adapter)).

[0015] In the following description, a "memory" refers to one or more memory devices, which are an example of one or more storage devices, and may typically be a primary storage device. At least one memory device in the memory may be a volatile memory device or a non-volatile memory device.

[0016] In the following description, a "persistent storage device" may be one or more persistent storage devices, which are an example of one or more persistent storage devices. A persistent storage device may typically be a non-volatile storage device (e.g., an auxiliary storage device), and more specifically, may be, for example, a hard disk drive (HDD), a solid state drive (SSD), a non-volatile memory express (NVME) drive, or a storage class memory (SCM).

[0017] Also, in the following description, "storage device" may be at least memory, including memory and persistent storage device.

[0018] Furthermore, in the following description, a "processor" may be one or more processor devices. The at least one processor device may typically be a microprocessor device such as a CPU (Central Processing Unit), but may also be other types of processor devices such as a GPU (Graphics Processing Unit). The at least one processor device may be a single-core or multi-core. The at least one processor device may be a processor core. The at least one processor device may also be a broader processor device such as a circuit that is a collection of gate arrays written in a hardware description language that performs part or all of the processing (e.g., an FPGA (Field-Programmable Gate Array), a CPLD (Complex Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit)).

[0019] In the following description, functions may be described using the expression "yyy unit", but the functions may be realized by one or more computer programs being executed by a processor, or by one or more hardware circuits (e.g., FPGA or ASIC), or by a combination thereof. When a function is realized by a program being executed by a processor, the function may be at least a part of the processor, since the specified processing is performed using a storage device and / or an interface device, etc., as appropriate. Processing described with a function as the subject may be processing performed by a processor or a device having the processor. A program may be installed from a program source. The program source may be, for example, a program distribution computer or a storage medium (e.g., a non-transitory storage medium) that can be read by a computer. The description of each function is an example, and multiple functions may be combined into one function, or one function may be divided into multiple functions.

[0020] In the following description, when elements of the same type are described without distinction, common reference symbols will be used, and when elements of the same type are described with distinction, reference symbols will be used.

[0021] FIG. 1 shows an example of the configuration of a database server according to an embodiment.

[0022] The database server 100 (an example of a data processing device) is a device that performs input and output to a database. The database server 100 is an on-premise server, but may alternatively be a server in a cloud such as a public cloud or a private cloud. The database server 100 has a memory 120, a high-speed storage 130, an input / output interface 141, a communication interface 142, and a CPU 110 connected thereto. The memory 120 and the high-speed storage 130 may be examples of storage devices. The input / output interface 141 and the communication interface 142 may be examples of interface devices. The CPU 110 may be an example of a processor.

[0023] The database server 100 is connected to a display device 101 and an input device 102 through an input / output interface 141. The display device 101 may be a liquid crystal panel, and the input device 102 may be a keyboard, a mouse, or the like.

[0024] The communication interface 142 communicates with the low-speed storage 150 via the network 11. The communication interface 142 may communicate with a client terminal (for example, a computer such as a personal computer or a smartphone) operated by an operator. That is, the operator may use the display device 101 and the input device 102, or may use a remote client terminal instead of or in addition to them. Also, at least a part of the database that stores the data to be processed may exist outside the database server 100. In this embodiment, the import of data and the use of the cache therefor will be mainly described, and a description of other processes will be omitted, but the configuration related to the use of the cache may be applicable to other processes as well.

[0025] The CPU 110 loads a data processing program into the memory 120 and executes a process included in the program, thereby realizing the functions of a query execution unit 121, a data input / output unit 122, and a communication unit 123.

[0026] The query execution unit 121 generates a query plan based on a query provided by a query source (e.g., an operator or an application executed inside or outside the database server 100). The query execution unit 121 determines a plan for opening and closing a cache file based on the query plan. In determining the plan, a "mode" is determined for each processing block. The "mode" is a mode for opening and closing a cache file, and is determined by an opening and closing unit (whether the entire cache file is opened or a part of it is opened) and an I / O type (the type of I / O for data in the cache file). The "processing block" corresponds to a unit of mode allocation, and may be a query, a partial query, an execution instance, or a query processing part. The "partial query" may be a query divided from a query. The "execution instance" may be an instance such as a table scan, a table creation, an index creation, a join, or a selection, and the execution instance may be called a database operator or a database operation. The "query processing part" may be a process corresponding to a part of the query plan and may be a process composed of one or more execution instances.

[0027] The data input / output unit 122 inputs and outputs data in executing a query based on a query plan. For example, the data input / output unit 122 reads a data aggregate (e.g., an object) from the low-speed storage 150 through the communication interface 142, and writes the data aggregate as a cache file 131 in the high-speed storage 130. Furthermore, during import processing, the data input / output unit 122 opens and closes the cache file 131 in the high-speed storage 130 using mode information provided by the query execution unit 121. Note that, in the description of this embodiment, as described above, import processing is mainly used as an example of processing, but the cache file 131 is also opened and closed as necessary in processing other than import processing.

[0028] The communication unit 123 communicates with the high-speed storage 130 and the input / output interface 141. For example, the communication unit 123 may cause information indicating a query plan and a mode for each query to be displayed on the display device 101 via the input / output interface 141. This allows the operator to know what kind of query plan has been created and what kind of mode has been assigned to it.

[0029] Management information such as the table state management array 24 is held in memory 120 .

[0030] The high-speed storage 130 is a storage having a faster data I / O speed for the CPU 110 (e.g., a storage having a shorter response time) than the low-speed storage 150, and is a non-volatile (or volatile) storage device such as an SSD (Solid State Drive). The high-speed storage 130 stores a cache file (CF) 131. The high-speed storage 130 may be a part of a memory. Typically, the storage capacity of the high-speed storage 130 is smaller than the storage capacity of the low-speed storage 150. Specifically, the storage capacity of the high-speed storage 130 is not large enough to store all data aggregates (e.g., objects) in the low-speed storage 150.

[0031] The low-speed storage 150 is a storage having a lower data I / O speed for the CPU 110 than the high-speed storage 130, and holds a plurality of data aggregates. For example, in this embodiment, the low-speed storage 150 holds Table A and Table B as import targets. In this embodiment, the low-speed storage 150 is an object storage in the cloud (public cloud). Therefore, the unit of input / output to the low-speed storage 150 may be an object unit. The low-speed storage 150 is not limited to an external storage of the database server 100 like a storage in the cloud, but may be an internal storage of the database server 100. For example, the high-speed storage 130 may be a part of a memory, and the low-speed storage 150 may be an auxiliary storage device such as an HDD (Hard Disk Drive).

[0032] In this embodiment, the term "cache file" refers to a file that has been read (cached) from the low-speed storage 150 to the high-speed storage 130. One cache file 131 may be the whole or a part of one or more objects.

[0033] The query execution unit 121 creates a query plan based on the query from the query source, and executes the query according to the query plan. In the query execution, for example, the data input / output unit 122 performs input / output to the low-speed storage 150 and input / output to the high-speed storage 130 as follows, for example. <I / O to slow storage 150>

[0034] If the target cache file 131 that is the data input / output destination cache file 131 is not present in the high-speed storage 130, the data input / output unit 122 reads an object including the data input / output destination column from the low-speed storage 150 to the high-speed storage 130 as the cache file 131.

[0035] When there is a dirty cache file 131 in the high-speed storage 130 (a cache file 131 that has been updated in the high-speed storage 130 but not reflected in the low-speed storage 150), the data input / output unit 122 performs so-called destaging (i.e., writes the dirty cache file 131 to the low-speed storage 150 as an object). When the dirty cache file 131 in the high-speed storage 130 is a cache file 131 to which data is input / output in the execution of a query waiting to be executed (for example, a query accumulated in a queue), the data input / output unit 122 may wait to write the cache file 131 to the low-speed storage 150. Furthermore, the cache file 131 to be destaged is a cache file 131 that has been updated in the execution of a query, and destaging may be performed in the execution of the query. Alternatively, destaging may be performed asynchronously with the execution of the query. <Input / output to high-speed storage 130>

[0036] As described above, objects are read from the slow storage 150 to the fast storage 130 as cache files 131 .

[0037] In the query execution, the data input / output unit 122 performs an open operation, which is an operation for opening the cache file 131 to which data is input / output (written and / or read) in the query execution. In the open operation for the cache file 131, an area to be opened in the cache file 131 (for example, the entire cache file 131 or a part of the cache file 131) and an I / O type of I / O for the cache file 131 may be specified. In response to the open operation, data in the cache file area (all or a part of the cache file 131) opened is read to the memory 120 (for example, a work area). Data is input / output to / from the data read from the cache file area to the memory 120, and the updated data in the memory 120 is stored in the cache file 131. In the query execution, the data input / output unit 122 performs a close operation, which is an operation for closing the open area of ​​the cache file 131. The cache file area is closed in response to the close operation. The closed and dirty cache file 131 is destaged to the low-speed storage 150.

[0038] 2A and 2B each show a specific example of a cache file.

[0039] Cache file 131A shown in FIG. 2A is a cache file of table A to be imported, and specifically, holds columns such as order date (A_orderdate) 201 and order number (A_orderkey) 202.

[0040] Cache file 131B shown in FIG. 2B is a cache file for table B to be imported, and specifically, holds columns such as item number (B_itemnumber) 211 and order number (B_orderkey) 212.

[0041] In this embodiment, the cache file includes the entire table in the database. Therefore, there is a one-to-one correspondence between the table and the cache file 131. However, the cache file 131 may include some columns (or rows) of one table, and therefore, multiple cache files 131 may exist for one table.

[0042] FIG. 3 is a concrete example of the table state management array 24. As shown in FIG.

[0043] The table status management array 24 has a row for each processing block, and the row holds information such as a query number 301 , a table identifier 302 , an open / close type 303 , a read / write 304 , an active flag 305 , a shared counter 306 , and a mode 307 .

[0044] The query number 301 indicates an identification number of a query, and the table identifier 302 indicates an identifier of a table.

[0045] The open / close type 303 indicates the open / close type of the open / close area of ​​the cache file 131. For each processing block, the "open / close area" is an area including an area in which all or part of the cache file 131 read into the high-speed storage 130 and input / output data is stored. The values ​​of the open / close type 303 include, for example, "partial open" in which only part of the cache file 131 is open / closed, and "full open" in which the entire cache file 131 is open / closed. Another value of the open / close type 303 is "full open during the entire process."

[0046] Read / write 304 indicates whether the input / output occurring in the execution of a query is read only ("read"), write only ("write"), or both read and write ("read & write").

[0047] The running flag 305 indicates whether or not a query is being executed ("True") ("False"). When the execution of a query has started, the running flag 305 becomes "True". When the execution of a query has ended, the running flag 305 becomes "False".

[0048] The shared counter 306 indicates the number of other queries using the table. A shared counter of "0" in the shared counter 306 means that the table is not being used by another query. The mode 307 indicates the mode number.

[0049] FIG. 4 is a flowchart showing an example of the import process.

[0050] (Step a1) Query execution unit 121 receives a query. The received query may be stored in memory 120 in association with information indicating a timestamp of the query (for example, the date and time of receipt of the query).

[0051] (Step a2) Query execution unit 121 creates a query plan based on the received query. The created query plan is stored in memory 120 in association with the query number of the query.

[0052] (Step a3) Query execution unit 121 determines the mode based on the created query plan.

[0053] (Step a4) The data input / output unit 122 executes an import process. In the import process, the data input / output unit 122 opens and closes all or a part of the cache file 131 based on the mode determined in step a3. The import process is an example of a process performed in the execution of a query.

[0054] FIG. 5 is a flow chart showing step a3 (mode determination) of FIG.

[0055] (Step b1) Query execution unit 121 refers to a new query plan. A "new query plan" is a query plan based on a query for which a mode has not yet been determined. If there are multiple new query plans, the query plan corresponding to the query with the oldest query reception date and time may be used. In the following description of FIG. 5, "new query plan" refers to the query plan referred to in step b1. Furthermore, hereinafter, a query corresponding to the "new query plan" may be referred to as a "new query."

[0056] (Step b2) Query execution unit 121 determines whether the query of the new query plan referenced in step b1 is an update. "Update" here may be, for example, any of import, delete, or update as a database process. If the determination result in step b2 is true (step b2: Yes), query execution unit 121 assigns mode 7 to the query corresponding to the new query plan referenced in step b1.

[0057] (Step b3) If the determination result in step b2 is false (step b2: No), based on the new query plan referenced in step b1 and the table to which data is input / output based on the new query plan, query execution unit 121 registers the query number corresponding to the new query plan and the table identifier of the table in table status management array 24. Here, one or more rows are added for the new query. For example, if data is input to or output from multiple tables in the execution of a new query, a row is added to table status management array 24 for each of the multiple tables. When a new query is executed, the in-operation flag 305 is "True" in the added row.

[0058] (Step b4) The data input / output unit 122 judges whether the table status management array 24 contains a row having the same table identifier as the table identifier registered in step b3, and whether a query corresponding to the row (a different query from the new query) is currently in operation (whether the in operation flag 305 is "True"). If the same table identifier does not exist, or if a query currently in operation for the same table identifier is not identified, the judgment result in step b4 is false (step b4: No), and the process proceeds to step b4-1. If the same table identifier exists, and a query currently in operation for the same table identifier is identified, the judgment result in step b4 is true (step b4: Yes), and the process proceeds to step b5.

[0059] (Step b4-1) For the step b4: Yes pair (pair of new query and table), the query execution unit 121 determines the mode based on the open / close type 303 of the table "partial open" and the I / O type of the new query. Specifically, since the open / close type 303 is "partial open", the mode is as follows according to the I / O type. If the I / O type is read & write (read / write 304 is “read & write”), mode 307 is “mode 2”. The I / O type is write only (if Read / Write 304 is "write"), Mode 307 is "Mode 4". The I / O type is read only (if Read / Write 304 is "Read"), Mode 307 is "Mode 6".

[0060] (Step b5) The query execution unit 121 judges whether the scale of the new query is large compared to all other queries corresponding to the same table identifier as the new query (the scale of all queries using the same table). If the query scale is not large (Step b5: No), the process proceeds to Step b6. If the query scale is large (Step b5: Yes), the process proceeds to Step b4-1. Note that "the query scale is large" may mean that the query scale exceeds a scale threshold. The scale threshold may be a threshold of a parameter value (e.g., the size of a query plan) for each of one or more parameters belonging to the query scale, and may be determined in advance based on the hardware specifications of the database server 100 (e.g., the hardware specifications of the CPU 110, etc.). For example, even if the query scale is the same, if the machine specifications are high, the query scale may be judged to be not large, and if the machine specifications are low, the query scale may be judged to be large.

[0061] (Step b6) Query execution unit 121 adds 1 to the value represented by shared counter 306 of each row having the same table identifier (rows corresponding to the new query and the different query).

[0062] (Step b7) If there is a row in which the shared counter 306 has been updated in step b6 and the value of the shared counter 306 is equal to or greater than "1", the query execution unit 121 changes the value of the open / close type 303 of that row from "partial open" to "full open". As a result, the open / close type 303 of the row corresponding to the pair of the new query and the relevant table (the table accessed in the execution of the new query) is "full open".

[0063] (Step b8) Query execution unit 121 determines the mode for the pair of the new query and the corresponding table based on open / close type 303 "all open" and the I / O type of the new query. Specifically, since open / close type 303 is "all open", the mode is as follows according to the I / O type. If the I / O type is read & write (read / write 304 is “read & write”), mode 307 is “mode 3”. The I / O type is write only (if Read / Write 304 is "write"), Mode 307 is "Mode 5". The I / O type is read only (if Read / Write 304 is "Read"), Mode 307 is "Mode 7".

[0064] In step b8, query execution unit 121 changes the value of mode 307 of a row having the same table identifier as that of the relevant table to the same mode 307 value as that of the relevant table.

[0065] FIG. 6 is a flow chart showing the open / close region management in step a4 of FIG.

[0066] (Step c1) Query execution unit 121 refers to the first row of table status management array 24.

[0067] (Step c2) The query execution unit 121 determines whether the running flag 305 of the referenced row is "True". If the running flag 305 is "True" (Step c2: Yes), the process proceeds to Step c3. If the running flag 305 is "False" (Step c2: No), the process proceeds to Step c2-1. In the following explanation of FIG. 6, the row referenced in Step c1 or c5 (the row most recently referenced) is referred to as the "referenced row".

[0068] (Step c2-1) Query execution unit 121 determines whether the relevant table (the table corresponding to table identifier 302 of the referenced row) is a table scheduled to be used (accessed) by a query waiting in the queue. Whether the relevant table is scheduled to be used can be determined by determining whether the row corresponding to the waiting query has the same table identifier as the relevant table. If the relevant table is scheduled to be used (step c2-1: Yes), processing proceeds to step c4. If the relevant table is not scheduled to be used (step c2-1: No), processing proceeds to step c2-2.

[0069] (Step c2-2) Query execution unit 121 deletes table identifier 302 of the relevant table from the reference row of table status management array 24.

[0070] (Step c2-3) The query execution unit 121 checks whether the shared counter 306 in the referenced row is greater than or equal to "2", or is equal to "1", or is equal to "0". If the shared counter 306 in the referenced row is greater than or equal to "2", the process proceeds to step c2-4. If the shared counter 306 in the referenced row is equal to "1", the process proceeds to step c2-5. If the shared counter 306 in the referenced row is equal to "0", the process proceeds to step c4.

[0071] (Step c2-4) Query execution unit 121 subtracts 1 from shared counter 306 in the row having the same table identifier 302 as the table identifier 302 deleted in step c2-2.

[0072] (Step c2-5) Query execution unit 121 subtracts 1 from shared counter 306 in the row having the same table identifier 302 as the table identifier 302 deleted in step c2-2, changes open / close type 303 in the row to “partial open”, and changes mode 307 in the row. The value of mode 307 after the change is based on open / close type 303 “partial open” and the value of read / write 304 in the row.

[0073] (Step c3) Query execution unit 121 sets the mode according to mode 307 in the referenced row. In this step, the value of mode 307 is set in a predetermined area of ​​memory 120, and processing corresponding to the referenced row is performed. In this processing, opening and closing of the cache file is controlled according to the set mode.

[0074] (Step c4) Query execution unit 121 determines whether there is a row following the referenced row in table status management array 24. If there is a next row (Step c4: Yes), the process proceeds to Step c5. If there is no next row (Step c4: No), the process ends.

[0075] (Step c5) Query execution unit 121 refers to the row next to the reference row. The next row becomes the reference row, and the process proceeds to step c2.

[0076] According to the example shown in FIG. 6, the rows of the table status management array 24 are referenced sequentially starting from the top row, but the order in which the rows of the table status management array 24 are referenced does not have to be limited to the order starting from the top row.

[0077] FIG. 7 is a flowchart of step c3 in FIG.

[0078] (Step d1) Query execution unit 121 refers to mode 307 (set mode) of the processing block to be executed.

[0079] (Step d2) If the value of mode 307 is "mode 1", query execution unit 121 opens and closes cache file 131 in "read & write" and "all open" mode.

[0080] (Step d3) If the value of the mode 307 is "mode 2", the query execution unit 121 opens and closes the cache file 131 in "read & write" and "partial open" mode.

[0081] (Step d4) If the value of the mode 307 is "mode 3", the query execution unit 121 opens and closes the cache file 131 in "write" and "all open" modes.

[0082] (Step d5) If the value of the mode 307 is "mode 4", the query execution unit 121 opens and closes the cache file 131 in "write" and "partial open" modes.

[0083] (Step d6) If the value of the mode 307 is "mode 5", the query execution unit 121 opens and closes the cache file 131 in "read" and "all open" mode.

[0084] (Step d7) If the value of the mode 307 is "mode 6", the query execution unit 121 opens and closes the cache file 131 in "read" and "partial open" mode.

[0085] (Step d8) If the value of mode 307 is "mode 7", query execution unit 121 opens and closes cache file 131 in "read & write" and "fully open during all processing." "All processing" refers to processing of the entire query.

[0086] In this way, as a preliminary step of determining the mode for area open / close management, metadata such as the query number 301, table identifier 302, open / close type 303, and read / write 304 are collected in the table status management array 24. This allows the mode to be determined quickly.

[0087] In this way, an appropriate area opening / closing mode is assigned to each processing block, thereby enabling data processing that achieves appropriate processing speed, reliability, and memory usage.

[0088] Although one embodiment has been described above, this is merely an example for explaining the present invention, and the scope of the present invention is not limited to this embodiment. The present invention can be implemented in various other forms.

[0089] The above description can be summarized as follows. The following summary may include supplementary explanations and explanations of modifications of the above description.

[0090] The database server 100 (an example of a data processing device) has a memory 120, a high-speed storage 130 (an example of a first storage device), and a CPU 110 (an example of a processor) connected thereto. The CPU 110 executes a program (for example, software for realizing a database management system including functions such as a query execution unit 121, a data input / output unit 122, and a communication unit 123) to perform the following processes. The CPU 110 reads out an object (an example of a data area) of input / output data from the low-speed storage 150 to the high-speed storage 130. The object read out to the high-speed storage 130 is a cache file 131 (an example of a data area). The CPU 110 performs an open operation on the cache file 131 to open an open / close area (an area including an area in which all or part of the cache file 131 read out to the high-speed storage 130 is stored and in which input / output data is stored). The CPU 110 reads data from the opened open / close area into the memory 120. The CPU 110 performs input / output for data read into the memory 120 . When data in the memory 120 is updated, the CPU 110 stores the updated data in an open open area in the high-speed storage 130. The CPU 110 performs a close operation to close the open / close area in the high-speed storage 130 on the cache file 131 having the open / close area. The CPU 110 stores the closed cache file 131 (dirty cache file 131 ) in the high-speed storage 130 that includes the updated data in the low-speed storage 150 .

[0091] The CPU 110 determines, for each processing block, the type of open / close area, which indicates whether the open / close area for the processing of the first processing block, which is the processing block, is the entire area of ​​the cache file 131 or a part of the area, based on whether the cache file 131 input / output in the processing of the first processing block can also be accessed in the processing of the second processing block, which is a processing block other than the first processing block, and whether the second processing block is running (i.e., based on the determination result of step b4). This realizes data area opening / closing that is appropriate for processing speed, reliability, and memory usage. Note that an opening operation for the cache file 131 including the open / close area for the processing of the first processing block may be performed according to the open / close area type of the first processing block.

[0092] When the cache file 131 input / output during processing of the first processing block cannot be accessed during processing of the second processing block, or when the second processing block is not being executed (when the determination result in step b4 is false), the opening and closing area may be a partial area of ​​the cache file 131. In such a case, the input / output range of the cache file 131 is limited to the range of input / output in the first processing block, and therefore it is appropriate to set the opening and closing area to a partial area of ​​the cache file 131.

[0093] When the cache file 131 input / output in the processing of the first processing block can also be accessed in the processing of the second processing block, and when the second processing block is being executed (when the determination result in step b4 is true), the opening and closing area may be the entire area of ​​the cache file 131. In such a case, the input / output range of the cache file 131 needs to include the range of input / output in the second processing block in addition to the range of input / output in the first processing block, and therefore it is efficient to set the opening and closing area to be the entire area of ​​the cache file 131 so as to cover these ranges.

[0094] If the cache file 131 input / output in the processing of the first processing block can also be accessed in the processing of the second processing block, and even if the second processing block is being executed (the determination result in step b4 is true), when the scale of a query (an example of a request) for which a process including the first processing block is executed and a query for which a process including the second processing block is executed exceeds a scale threshold determined based on the specifications of the hardware including the CPU 110 and the memory 120 (when the determination result in step b5 is true), the opening and closing area may be a partial area of ​​the cache file 131. This makes it possible to suppress the amount of data read into the memory 120 even if the scale of the query is large.

[0095] For example, in step c3, when the processing of the first processing block is completed and there is no second processing block being executed (for example, when the value of the shared counter 306 corresponding to the first processing block is zero), the CPU 110 may perform a close operation on the cache file 131 including the open / close region for the processing of the first processing block in accordance with the open / close region type of the first processing block. This allows the cache file 131 to be closed at an appropriate time, and therefore is expected to improve reliability.

[0096] When the first processing block is not executed, the CPU 110 may perform the following process. That is, when the open / close region type of the second processing block is to set the open / close region in the second processing block to the entire region of the cache file 131, and the open / close region for processing blocks other than the second processing block is not included in the cache file 131 ("=1" in step c2-3), the CPU 110 may change the open / close region type of the second processing block to set the open / close region to a partial region of the cache file 131 (step c2-5). This allows the cache file 131 to be opened and closed appropriately in the process when the second processing block will soon become the first processing block.

[0097] The mode affecting the opening operation for the cache file 131 including the opening and closing area for the first processing block may depend on the opening and closing area type of the first processing block and the type of input and output in the first processing block. This is expected to result in more appropriate opening and closing of the cache file.

[0098] As modes that affect the open operation on the cache file 131 including the open / close area for the first processing block, there are modes 1 to 6 that depend on the open / close area type of the first processing block and the type of input / output in the first processing block, as well as mode 7 that corresponds to the type of query for which processing including the first processing block is performed. Mode 7 is a mode in which, if the query is an update query, all areas of all cache files 131 input / output by the query are kept open throughout the processing of the query. This is expected to appropriately maintain processing speed, reliability, and memory usage in the processing of the update query. [Explanation of symbols]

[0099] 100: database server, 110: CPU, 120: memory, 121: query execution unit, 122: data input / output unit, 123: communication unit, 130: high-speed storage, 131: cache file, 150: low-speed storage

Claims

1. A data processing device that inputs and outputs data in response to a request, Memory, A first storage device; an interface device connected to a second storage device having a data input / output speed slower than that of the first storage device and storing a plurality of data areas; a processor connected to the first storage device and the interface device; Equipped with The processor, reading a data area of ​​data to be input / output from the second storage device to the first storage device; performing an open operation for opening an open / close area, which is an area including an area in which input / output data is stored, which is all or a part of the data area read out to the first storage device; reading data from the open opening and closing region into the memory; Input and output are performed for the data read into the memory; When the data in the memory is updated, the updated data is stored in the open open area in the first storage device; performing a close operation for closing the open / close area in the first storage device on a data area having the open / close area; storing the closed data area in the first storage device, the closed data area including the updated data, in the second storage device; It is as follows. the processor determines, for each processing block, a type of opening and closing region, i.e., whether the opening and closing region for the processing of a first processing block, which is the processing block, is the entire region of the data region or a part of the region, based on whether a data region input / output in the processing of the first processing block can also be accessed in the processing of a second processing block, which is a processing block other than the first processing block, and based on whether the second processing block is being executed. Data processing device.

2. When a data area input / output in the processing of the first processing block cannot be accessed in the processing of the second processing block, or when the second processing block is not being executed, the open / close area is a part of the data area.

2. A data processing apparatus according to claim 1.

3. a data area input / output in the processing of the first processing block can also be accessed in the processing of the second processing block, and when the second processing block is being executed, the open / closed area is the entire area of ​​the data area; 2. A data processing apparatus according to claim 1.

4. When a data area input / output in the processing of the first processing block can also be accessed in the processing of the second processing block, and when a scale of a request for executing a processing including the first processing block and a request for executing a processing including the second processing block exceeds a scale threshold determined based on the specifications of hardware including the processor and the memory, even if the second processing block is being executed, the open / closed area is a part of the data area.

4. A data processing apparatus according to claim 3.

5. the processor performs a close operation on a data area including an open / close area for the processing of the first processing block in accordance with a type of the open / close area of ​​the first processing block when the processing of the first processing block is completed and there is no second processing block being executed; 2. A data processing apparatus according to claim 1.

6. If the first processing block is not executed, the processor: When the type of the opening and closing region of the second processing block is to set the opening and closing region in the second processing block to the entire region of the data region, and the opening and closing regions of processing blocks other than the second processing block are not included in the data region, the type of the opening and closing region of the second processing block is changed to set the opening and closing region to a part of the data region.

4. A data processing apparatus according to claim 3.

7. a mode affecting an open operation on a data area including an open / close area for the first processing block depends on a type of the open / close area of ​​the first processing block and a type of input / output in the first processing block; 2. A data processing apparatus according to claim 1.

8. A mode that affects an opening operation on a data area including an opening / closing area for the first processing block includes a mode that depends on a type of the opening / closing area of ​​the first processing block and a type of input / output in the first processing block, as well as a mode that corresponds to a type of a request for which processing including the first processing block is performed.

8. A data processing apparatus according to claim 7.

9. a mode corresponding to a type of request for which processing including the first processing block is performed is a mode in which, if the request is an update query, all areas of all data areas input / output by the query are kept open throughout the processing of the query; 9. A data processing apparatus according to claim 8.

10. the request is a request for input or output of data to one or more tables in a database; 2. A data processing apparatus according to claim 1.

11. 1. A data processing method performed by a data processing device including a memory and a first storage device and performing input / output of data in response to a request, comprising: The data processing device includes: reading a data area of ​​the data to be input / output from a second storage device having a data input / output speed slower than that of the first storage device and storing a plurality of data areas into the first storage device; performing an open operation for opening an open / close area, which is an area including an area in which input / output data is stored, which is all or a part of the data area read out to the first storage device; reading data from the open opening and closing region into the memory; Input and output are performed for the data read into the memory; When the data in the memory is updated, the updated data is stored in the open open area in the first storage device; performing a close operation for closing the open / close area in the first storage device on a data area having the open / close area; storing the closed data area in the first storage device, the closed data area including the updated data, in the second storage device; It is as follows. The data processing method includes: determining, for each processing block, a type of opening and closing area, i.e., whether the opening and closing area for the processing of a first processing block, which is the processing block, is the entire area of ​​the data area or a part of the area, based on whether the data area input / output in the processing of the first processing block can also be accessed in the processing of a second processing block, which is a processing block other than the first processing block, and based on whether the second processing block is being executed; Data processing methods.

Images