Storage System and Data Caching Method

Abstract

To provide a storage system for enhancing the performance of retrieval or connection in a database by reducing useless reading from a low-speed storage even while increasing the efficiency of a memory cache.SOLUTION: A storage system 1 specifies a required column to be required to execute a query, and executes the query on the basis of data of the required column read from a storage device. Reading from a storage device from the required column is preferentially read from a high-speed storage device with data of the required column existing among a cache memory 130, a high-speed storage 140 and a low-speed storage 150, stores data 311 of a second data size unit including the data of the required column used in query execution in the cache memory, and stores the data of the second data size unit in the memory and stores read data 310 of a first data size unit in the case of performing reading from the high-speed storage.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a storage system and a data caching method, and is suitable for application to a storage system and a data caching method that hierarchically manage the data cache of a database.

Background Art

[0002] In recent years, with the explosive increase in data, there has been an increasing need to speed up databases, which are key middleware for handling data, and increase the amount of data that can be handled. As a countermeasure against such problems, for example, efforts are underway to scale out database processing and speed up the processing of a huge amount of data by building a database in the cloud and implementing search processing or join processing using distributed parallel processing.

[0003] However, although the use of the cloud enables scale-out, its maintenance cost depends on the scale, and particularly when trying to persist storage, the usage amount increases according to the capacity. And, the cloud pricing tends to be higher for higher processing speeds and lower for low-speed object storage. Therefore, although cost reduction is expected by utilizing object storage, it was necessary to devise a method that does not degrade performance.

[0004] Regarding such a devise, for example, Patent Document 1 discloses a technique for hierarchically storing and managing a storage area of a database by dividing it into a plurality of areas by having a lower-speed tertiary storage (for example, tape media) under a secondary storage (for example, a hard disk drive) as a data distribution method and performing hierarchical storage management (HSM: Hierarchical Storage Management). In this data management, a time stamp is used, and when the storage area on the secondary storage becomes insufficient, the area to be rewritten to the tertiary storage is determined using the time stamp.

Prior Art Documents

Patent Documents

[0005] [Patent Document 1] U.S. Patent No. 5,822,780 [Summary of the Invention] [Problems to be Solved by the Invention]

[0006] In a cloud database, when performing read management of data by dividing the storage area as in the technology disclosed in Patent Document 1, by using the main memory etc. to hierarchically configure the cache, it becomes possible to fetch data with a faster access time of the main memory. Here, if the data size (cache size) stored in the cache is large, data areas not used when caching will also be cached, resulting in a decrease in cache efficiency. On the other hand, if the cache size is made small, when using a medium with a large latency such as object storage on the network, access becomes latency bottleneck, leading to a decrease in performance.

[0007] The above problems can be solved by handling data in units of small areas in a high-speed cache using memory etc., and handling data in units of large areas in a low-speed storage such as object storage. In this case, if cache efficiency is emphasized, a large area is read from the low-speed storage, and from the large area read, the minute area targeted for reading is taken out and cached in the high-speed memory.

[0008] However, at this time, in the memory, among the large areas read from the low-speed storage, portions other than the above-mentioned micro-areas are discarded without being cached. Therefore, when another micro-area becomes the target of reading from the same large area next time, it becomes necessary to read the large area again from the low-speed storage. That is, in the prior art, when emphasizing cache efficiency and changing the data handling unit (area) for each storage, there is a problem that the wasteful reading from the low-speed storage increases and the overall efficiency decreases.

[0009] The present invention has been made in consideration of the above points, and aims to propose a storage system and a data caching method that can reduce wasteful reading from a low-speed storage such as an object storage while increasing the efficiency of the memory cache, and improve the performance of search and join in a database.

Means for Solving the Problems

[0010] In order to solve such problems, in the present invention, there is provided a storage system for storing data of a database, comprising a database management system for managing the database by executing a program by a processor, and a storage device. The storage device includes a first storage for storing data of a table of the database, a second storage that is faster in data reading and writing than the first storage, and a memory that is faster in data reading and writing than the second storage. The first storage reads and writes data in units of a first data size, and the memory reads and writes data in units of a second data size smaller than the first data size. When the database management system executes a query that is a query to the database, the database management system interprets the query and identifies the necessary columns required for the execution of the query , special determined the Read the data of the necessary columns from the storage device, execute the query based on the data of the necessary columns read from the storage device, and when reading the data of the necessary columns from the storage device, preferentially read from the high-speed storage device where the data of the necessary columns exists among the memory, the second storage, and the first storage, store the data in units of a second data size including the data of the necessary columns used for executing the query in the memory, and when reading from the first storage, store the data in units of the second data size in the memory and store the read data in units of the first data size in the second storage. A storage system is provided which is characterized by the above.

[0011] Also, in order to solve such problems, in the present invention, there is provided a data caching method by a storage system that stores data in a database, wherein the storage system includes a database management system that manages the database by executing a program by a processor, and a storage device. The storage device includes a first storage that stores data of the entire table of the database, a second storage that is faster in data reading and writing than the first storage, and a memory that is faster in data reading and writing than the second storage. The first storage reads and writes data in units of a first data size, and the memory reads and writes data in units of a second data size smaller than the first data size. When the database management system executes a query that is a query to the database, the database management system includes a necessary column specifying step of interpreting the query and specifying necessary columns required for execution of the query, a data reading step of reading data of the necessary columns specified in the necessary column specifying step from the storage device, and a query execution step of executing the query based on the data of the necessary columns read in the data reading step. In the data reading step, the database management system preferentially reads from a high-speed storage device in which data of the necessary columns exists among the memory, the second storage, and the first storage, stores data of a second data size unit including data of the necessary columns used for execution of the query in the memory, and when reading from the first storage, stores the data of the second data size unit in the memory and stores the read data of the first data size unit in the second storage. A data caching method is provided.

Effect of the Invention

[0012] According to the present invention, while improving the efficiency of memory caching, it is possible to reduce wasted reads from a low-speed storage and improve the performance of search or join in a database.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Embodiments for Carrying Out the Invention

[0014] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0015] Note that the following description and drawings are examples for explaining the present invention, and for the sake of clarity of explanation, omissions and simplifications are made as appropriate. Also, not all combinations of features described in the embodiments are essential for the solution means of the invention. The present invention is not limited to the embodiments, and all application examples that conform to the idea of the present invention are included in the technical scope of the present invention. Those skilled in the art can make various additions and changes within the scope of the present invention. The present invention can also be implemented in various other forms. Unless otherwise particularly limited, each component may be plural or singular.

[0016] In the following description, when describing elements of the same type without distinction, reference signs or common numbers in the reference signs are used. When describing elements of the same type separately, the reference signs of those elements may be used, or the IDs assigned to those elements may be used instead of the reference signs.

[0017] In the following description, there may be cases where the processing performed by executing a program is described. However, the program is executed by at least one processor (e.g., CPU), and in order to perform the defined processing while appropriately using storage resources (e.g., memory) and / or interface devices (e.g., communication ports), etc., the processor may be regarded as the main body of the processing. Similarly, the main body of the processing performed by executing the program may be a controller, device, system, computer, node, storage system, storage device, server, management computer, client, or host having a processor. The main body of the processing performed by executing the program (e.g., processor) may include a hardware circuit that performs part or all of the processing. For example, the main body of the processing performed by executing the program may include a hardware circuit that performs encryption and decryption, or compression and decompression. The processor operates as a functional unit that realizes a predetermined function by operating according to the program. Devices and systems including the processor are devices and systems including these functional units.

[0018] The program may be installed from a program source into a device such as a computer. The program source may be, for example, a program distribution server or a computer-readable storage medium. When the program source is a program distribution server, the program distribution server includes a processor (e.g., a CPU) and storage resources, and the storage resources may further store a distribution program and a program to be distributed. Then, by the processor of the program distribution server executing the distribution program, the processor of the program distribution server may distribute the program to be distributed to other computers. Also, in the following description, two or more programs may be implemented as one program, or one program may be implemented as two or more programs.

[0019] (1) Configuration FIG. 1 is a block diagram showing a configuration example of a storage system 1 according to an embodiment of the present invention. The storage system 1 is a system that hierarchically manages the data cache of a database, and includes a server 10 to which a query 210 is input from a client 20, and a low-speed storage 150 connected to the server 10 (DBMS 100) via a network 160.

[0020] The client 20 is a client computer. The client 20 is a source of database processing such as search processing, sends a request to the server 10 in the form of a query 210, and receives the result from the server 10.

[0021] Server 10 is a database server. Server 10 is a computer that executes DBMS 100 and is equipped with high-speed storage 140. In terms of its hardware configuration, like a general database server, Server 10 includes a processor such as a CPU (Central Processing Unit), main memory (primary storage device), auxiliary storage device, and various interfaces. DBMS 100 is a software program, and its functions are realized when it is read into the main memory and executed by the processor. In FIG. 1, cache memory 130 is shown within DBMS 100 assuming it operates on the main memory, but a configuration where cache memory 130 is provided between the processor and the main memory may also be possible. High-speed storage 140 is an auxiliary storage device. Also, as an example of various interfaces, there is a communication interface having a communication function with network 160 connected to low-speed storage 150.

[0022] When Server 10 receives query 210 from client 20, it executes DBMS 100 to perform functions such as retrieval according to the content of query 210 and returns the result to client 20.

[0023] DBMS 100 is database software for implementing database functions on Server 10 and is a so-called database management system (DBMS: DataBase Management System). For example, the program of DBMS 100 is stored in the auxiliary storage device of Server 10, read into the main memory, and executed by the processor.

[0024] As functional modules, DBMS 100 incorporates query execution module 110 and data readout module 120, and realizes functions such as database retrieval by executing these modules.

[0025] Query 210 is a query that describes the processing to be performed in a database (DBMS100). Specifically, it describes specifications such as which column data to use for searching and which column data to output. Query 210 encompasses all the information necessary when performing a search by the query itself. Query 210 is described using, for example, SQL.

[0026] The query execution module 110 has the function of interpreting Query 210 to generate a column list 220, generating necessary column information 230 from the column list 220, sending the necessary column information 230 to the data reading module 120, and requesting the data reading module 120 to read the necessary column data.

[0027] The column list 220 is a list of columns (used columns) to be used in the execution of Query 210, and is generated by the query execution module 110 from Query 210.

[0028] The necessary column information 230 classifies and lists the uses, such as whether the column is used for searching or for aggregation, for the columns (necessary columns) that need to be read by the data reading module 120 among the column list 220. The necessary column information 230 is generated by the query execution module 110 and used by the data reading module 120.

[0029] The data reading module 120 has the function of reading the data (necessary columns) that need to be read in the execution of Query 210 from any of various storages (cache memory 130, high-speed storage 140, low-speed storage 150) based on the necessary column information 230. Although details will be described later, when reading the necessary columns from storage, the data reading module 120 reads them from the cache memory 130 if the necessary columns exist in the cache memory 130, checks if they exist in the high-speed storage 140 if they do not exist in the cache memory 130, reads them from the high-speed storage 140 if they exist, and reads them from the low-speed storage 150 via the network 160 if they do not exist.

[0030] Here, the low-speed storage 150 and the high-speed storage 140 store the data of the database in the form of cache storage units (data 310). As shown in FIG. 2 described later, the data 310 of the cache storage unit is composed of data 311 of one or more data access units and at least one data access unit management information 312. And the cache memory 130 stores the data in the form of data access units (data 311). Specifically, when the data read module 120 reads the data 310 including the required column from the low-speed storage 150, it stores the data 310 in the high-speed storage 140 and stores the data 311 including the required column in the data 310 in the cache memory 130.

[0031] In the following description, for simplicity, the data 310 of the cache storage unit may be referred to as "cache storage unit 310", and the data 311 of the data access unit may be referred to as "data access unit 311".

[0032] The cache storage unit 310 is a data handling unit in the low-speed storage 150 and internally includes data 311 of one or more data access units. Also, the cache storage unit 310 is used as a write and storage management handling unit in the high-speed storage 140. As described above, when the data read module 120 reads the required column based on the request of the required column information 230, it selects and reads the data access unit 311 having the required column from among the cache storage units 310 including the required column.

[0033] The data access unit 311 is the basic unit for the DBMS 100 to access data and is the unit for handling data in the cache memory 130 and the high-speed storage 140. Strictly speaking, the high-speed storage 140 writes and stores data management in the cache storage unit 310 and reads data in the data access unit 311. Also, the data sent from the data read module 120 to the query execution module 110 is also in the data access unit 311.

[0034] Figure 2 is a diagram showing an example of the structure of the cache storage unit 310. As shown in Figure 2, the cache storage unit 310 is composed of one or more data access units 311 and at least one data access unit management information 312. The data access unit management information 312 is management information for managing the data access units 311 existing inside the cache storage unit 310. The data access unit management information 312 holds information indicating at least which data and which column data for each data access unit 311. By having such data access unit management information 312 in the cache storage unit 310, the data read module 120 can identify the column data included in the cache storage unit 310 in the data access unit 311 and selectively store the identified data access unit 311 in the cache memory 130.

[0035] The cache memory 130 is a cache memory provided to improve the data access performance by the server 10's processor. Specifically, for example, it is a DRAM (Dynamic Random Access Memory). The cache memory 130 is one of the storage devices constituting the hierarchical cache in the storage system 1 and has the highest access performance (fast data processing speed) compared to the high-speed storage 140 and the low-speed storage 150 that also constitute the hierarchical cache.

[0036] When the data read module 120 reads data 310 including the required columns from the low-speed storage 150 or the high-speed storage 140 in response to the required column information 230, the cache memory 130 stores the required columns included in the data 310 in the form of data access units (data 311). In subsequent data reads, it is checked whether the cache memory 130 stores the required columns. If the corresponding data is stored in the cache memory 130, the data read module 120 reads it and sends it to the query execution module 110.

[0037] The high-speed storage 140 is an embedded storage usually installed within the server 10, specifically, for example, an SSD (Solid State Drive). The storage capacity of the high-speed storage 140 is larger than that of the cache memory 130 but smaller than that of the low-speed storage 150. The data storage unit (write unit) in the high-speed storage 140 is the same cache storage unit 310 as that of the low-speed storage 150. On the other hand, the data read unit in the high-speed storage 140 is the same data access unit 311 as that of the query execution module 110 and the cache memory 130. As described above, the high-speed storage 140 stores the cache storage unit 310 read by the data read module 120 from the low-speed storage 150. The data processing speed (e.g., read / write speed) in the high-speed storage 140 is at least higher than that of the low-speed storage 150.

[0038] When there is no space left in the high-speed storage 140 to store a new cache storage unit 310, the LRU (Least Recently Used) method or the like is used to select a cache storage unit 310 that has not been used for a long time among the deletable data and erase (invalidate) it, thereby securing a space to store a new cache storage unit 310.

[0039] The low-speed storage 150 is storage installed outside the server 10 and is connected to the server 10 via the network 160. Specifically, the low-speed storage 150 is, for example, object storage or NAS (Network Attached Storage). The data processing speed (e.g., read / write speed) in the low-speed storage 150 is lower than that of the high-speed storage 140 and the cache memory 130, but the utilization cost of the low-speed storage 150 is relatively low. In the low-speed storage 150, the data of the entire table of the database used by the storage system 1 is stored as a plurality of object files (data 310) in the form of cache storage units.

[0040] The network 160 is a network that connects the server 10 and the low-speed storage 150 installed outside it, and is used to read the necessary data from among the data 310 of the plurality of cache storage units stored in the low-speed storage 150. Specifically, the network 160 is, for example, Ethernet, but other methods may also be used.

[0041] The above is the overall configuration of the storage system 1 according to this embodiment. The storage system 1 configured in this way performs the following operations.

[0042] First, a query 210 for search is issued from the client 20 to the server 10. In the server 10, a query execution module 110 inside the DBMS 100 receives the query 210 and creates a column list 220 from the query 210. Then, the query execution module 110 creates necessary column information 230 that requests reading of the necessary columns required by the query 210 from the column list 220 and sends it to the data reading module 120. The data reading module 120 that has received the necessary column information 230 first checks the cache memory 130. If the necessary data access unit 311 is stored therein, it reads it out. If the necessary data access unit 311 is not stored in the cache memory 130, the data reading module 120 checks the high-speed storage 140. If the necessary data access unit 311 is stored therein, it reads it out. If the necessary data access unit 311 is not stored in the high-speed storage 140, the data reading module 120 reads out a cache storage unit 310 including the necessary data from the low-speed storage 150, stores it in the high-speed storage 140, and reads out the necessary data access unit 311 from among the stored cache storage units 310. Further, at this time, data access units 311 of all the columns indicated by the necessary column information 230 are extracted from the cache storage unit 310 stored in the high-speed storage 140 and stored in the cache memory 130. By operating in this way, the data (data of the entire database table) stored in the low-speed storage 150 can be hierarchically cache-managed by utilizing the high-speed storage 140 and the cache memory 130. In particular, by extracting data of the columns indicated by the necessary column information 230 from the data of a large handling unit (cache storage unit) read from the low-speed storage 150 in small handling units (data access units) and storing them in the cache memory 130, data that is likely to be read in the future can be cached in the cache memory 130 in small handling units. Therefore, it is possible to increase the cache hit rate in the memory (cache memory 130), suppress unnecessary reads from the low-speed storage 150, and speed up the execution of the query 210.

[0043] (1-1) Data reading module 120 FIG. 3 is a block diagram showing an internal configuration example of the data reading module 120. As shown in FIG. 3, the data reading module 120 includes a cache storage unit specifying unit 121, an object file reading unit 122, a temporary buffer 123, a page loading unit 124, a high-speed storage writing unit 125, a high-speed storage reading unit 126, segment management information 127, and a cache memory management unit 128.

[0044] The cache storage unit specifying unit 121 uses the necessary column information 230 issued from the query execution module 110 to specify a cache storage unit 310 that contains column data (data access unit 311 of the necessary columns) necessary for the execution of the query 210. Then, the cache storage unit specifying unit 121 creates a list of the specified cache storage units 310 and passes it to the page loading unit 124 and the object file reading unit 122.

[0045] The object file reading unit 122 is a mechanism for issuing a sequence to read a desired object file from a storage that stores the object file. In this example, the low-speed storage 150 is the storage that handles the object file. To implement the above mechanism, the object file reading unit 122 operates the network 160, connects to the low-speed storage 150 on the network 160, and has a function of reading a specific cache storage unit 310 from the low-speed storage 150. Specifically, based on the list received from the cache storage unit specifying unit 121, the object file reading unit 122 reads the cache storage unit 310 specified by the cache storage unit specifying unit 121 from the low-speed storage 150 via the network 160 and stores the read data in the temporary buffer 123.

[0046] The temporary buffer 123 is a buffer that temporarily stores cache storage units 310 read via the network 160. Specifically, the temporary buffer 123 is, for example, a memory (DRAM). The temporary buffer 123 can store a plurality of cache storage units 310, and the data access unit management information 312 included in each cache storage unit 310 is retrieved and stored as segment management information 127. Also, the temporary buffer 123 can store a certain number of cache storage units 310, but when this certain number is exceeded, the least recently used cache storage unit 310 among the deletable data is replaced with a new cache storage unit 310 using, for example, the LRU. This replacement process is controlled, for example, by the object file reading unit 122.

[0047] When the object file reading unit 122 reads an object file, if the cache storage unit 310 corresponding to the target object file is stored in the temporary buffer 123, the object file reading unit 122 reads the cache storage unit 310 stored in the temporary buffer 123 without reading from the low-speed storage 150. Also, the data (cache storage unit 310) stored in the temporary buffer 123 is written to the high-speed storage 140 in cache storage units by the high-speed storage writing unit 125.

[0048] The page reading unit 124 has a function of identifying a data access unit 311 having the required columns requested by the required column information 230, reading out this data access unit 311, and sending it to the query execution module 110.

[0049] The page loading unit 124 first matches the segment management information 127 with the necessary column information 230 to identify the data access unit 311 having the necessary columns within the cache storage unit 310. Next, the page loading unit 124 checks whether the identified data access unit 311 exists in the cache memory 130. If it exists, the page loading unit 124 uses the cache memory management unit 128 to read the data access unit 311 from the cache memory 130 and send it to the query execution module 110. If the identified data access unit 311 does not exist in the cache memory 130, the page loading unit 124 checks whether the cache storage unit 310 containing the data access unit 311 is stored in the temporary buffer 123. If the corresponding cache storage unit 310 is stored in the temporary buffer 123, the page loading unit 124 reads the data access unit 311 from the cache storage unit 310 and sends it to the query execution module 110. If the corresponding cache storage unit 310 is not stored in the temporary buffer 123, the page loading unit 124 checks whether the cache storage unit 310 containing the data access unit 311 is stored in the high-speed storage 140. If the corresponding cache storage unit 310 is stored in the high-speed storage 140, the page loading unit 124 uses the high-speed storage reading unit 126 to read the data access unit 311 from the cache storage unit 310 stored in the high-speed storage 140 and send it to the query execution module 110. If the corresponding cache storage unit 310 is not stored in the high-speed storage 140, since the cache storage unit 310 only exists in the low-speed storage 150, the page loading unit 124 requests the object file reading unit 122 to read the cache storage unit 310. In response to this request, the object file reading unit 122 reads the cache storage unit 310 from the low-speed storage 150 and stores it in the temporary buffer 123. Therefore, the page loading unit 124 extracts the data access unit 311 having the necessary columns from the cache storage unit 310 stored in the temporary buffer 123 and sends it to the query execution module 110.Also at this time, the page loading unit 124 stores the data access unit 311 having the necessary columns in the cache memory 130 using the cache memory management unit 128.

[0050] The high-speed storage writing unit 125 is a mechanism for writing data to the high-speed storage 140. The high-speed storage writing unit 125 writes (writes in) data to the high-speed storage 140 in the cache storage unit 310. When free capacity for storing the cache storage unit 310 cannot be secured in the high-speed storage 140, the high-speed storage writing unit 125 uses LRU or the like to erase the cache storage unit 310 with the oldest access among the deletable data, and secures free capacity in the high-speed storage 140 for storing a new cache storage unit 310.

[0051] The high-speed storage reading unit 126 is a mechanism for reading data from the high-speed storage 140. The high-speed storage reading unit 126 reads data from the high-speed storage 140 in the data access unit 311.

[0052] The segment management information 127 is information obtained by collecting the data access unit management information 312 existing in each of the plurality of cache storage units 310. The segment management information 127 includes at least the data access unit management information 312 of the cache storage unit 310 stored in the temporary buffer 123 and the data access unit management information 312 of the cache storage unit 310 from which the data access unit 311 has been read from the high-speed storage 140.

[0053] The cache memory management unit 128 manages the storage of data access units 311 to be cached in the cache memory 130. When there is no available space in the cache memory 130 to store a new data access unit 311, the cache memory management unit 128 uses LRU or the like to delete the data access unit 311 with the oldest access among the deletable data, and uses the increased available space to store the new data access unit 311 in the cache memory 130.

[0054] (2) Processing The execution procedure of the query in the storage system 1 according to the present embodiment will be described in detail. In the storage system 1, the server 10 executes the query execution process shown in FIGS. 4 and 5 for the query 210 issued from the client 20.

[0055] (2-1) Query execution process FIGS. 4 and 5 are flowcharts (Part 1, Part 2) showing an example of the processing procedure of the query execution process. The end points A and B shown in FIG. 4 respectively correspond to the end points A and B shown in FIG. 5.

[0056] According to FIGS. 4 and 5, first, the query execution module 110 receives the query 210 issued from the client 20 (step S101). Next, the query execution module 110 interprets the received query 210, creates a list of columns (column list 220) used in the execution of the query 210, and for each used column included in the column list 220, identifies the column to be searched (search column) and the column to be aggregated (aggregation column), and uses these identification results to create a list of necessary columns (necessary column information 230) that need to be read (step S102). The necessary column information 230 lists the necessary columns in the call order at the time of query execution. The query execution module 110 sends the created necessary column information 230 to the data reading module 120.

[0057] In the data reading module 120, the cache storage unit specifying unit 121 specifies the first read data from the list of a plurality of necessary columns described in the necessary column information 230 (step S103). This "first read data" is the data of the necessary column described first among the necessary columns that have not been read yet in the necessary column information 230 in which the necessary columns are arranged in the call order at the time of query execution. That is, when the process of step S103 is performed via NO in step S115 described later, the first read data is specified from among the necessary columns that have not been read at that time. Summarizing the above, in the necessary column information 230, the data of the necessary column with the earliest call order at the time of query execution among the data of the necessary columns that have not been read corresponds to the "first read data".

[0058] Next, the page loading unit 124 specifies the data access unit 311 in which the first read data specified in step S103 exists (step S104). Note that the process of step S104 may be executed by the cache storage unit specifying unit 121. Also, in step S107 described later, the cache storage unit specifying unit 121 specifies the cache storage unit 310 including the data access unit 311 in which the first read data exists, but the process of step S107 may be executed before the process of step S104 or may be executed by the page loading unit 124.

[0059] Next, the page loading unit 124 determines whether the data access unit 311 specified in step S104 exists in the cache memory 130 (step S105). When the data access unit 311 exists in the cache memory 130 (YES in step S105), the page loading unit 124 reads the data access unit from the cache memory 130 using the cache memory management unit 128, returns it to the query execution module 110 (step S106), and proceeds to step S112 described later. On the other hand, when the data access unit 311 does not exist in the cache memory 130 (NO in step S105), the cache storage unit specifying unit 121 (or the page loading unit 124) specifies the cache storage unit 310 including the data access unit 311 (step S107).

[0060] After the process of step S107, the page loading unit 124 determines whether the cache storage unit 310 specified in step S107 exists in the temporary buffer 123 (step S108). When the cache storage unit 310 exists in the temporary buffer 123 (YES in step S108), the object file reading unit 122 reads the cache storage unit 310 from the temporary buffer 123, and the page loading unit 124 reads the data access unit 311 (read data) having the necessary columns from the cache storage unit 310 (step S109), and returns it to the query execution module 110. After the process of step S109, it proceeds to step S112 described later. The process of step S109 may be controlled by the page loading unit 124. On the other hand, when the cache storage unit 310 does not exist in the temporary buffer 123 in step S108 (NO in step S108), the page loading unit 124 determines whether the cache storage unit 310 exists in the high-speed storage 140 (step S110).

[0061] If the target cache storage unit 310 exists in the high-speed storage 140 in step S110 (YES in step S110), the high-speed storage reading unit 126 reads a data access unit 311 (read data) having necessary columns from the cache storage unit 310 stored in the high-speed storage 140 (step S111), and the page reading unit 124 returns this to the query execution module 110. After the process of step S111, it proceeds to step S112 described later. On the other hand, if the target cache storage unit 310 does not exist in the high-speed storage 140 in step S110 (NO in step S110), it proceeds to step S201 of FIG. 5.

[0062] In step S201 of FIG. 5, the object file reading unit 122 determines whether the temporary buffer 123 has reached the storage upper limit of the cache storage unit 310. If it has reached the storage upper limit (YES in step S201), the object file reading unit 122 applies the LRU and selects the cache storage unit 310 with the oldest access among the data that can be deleted in the temporary buffer 123, and deletes this (step S202).

[0063] And if it has not reached the storage upper limit in step S201 (NO in step S201), or after step S202, the object file reading unit 122 reads the cache storage unit 310 specified in step S107 from the low-speed storage 150 and stores it in the temporary buffer 123 (step S203).

[0064] Next, the high-speed storage writing unit 125 determines whether the high-speed storage 140 has reached the storage upper limit of the cache storage unit 310 (step S204). If it has reached the storage upper limit (YES in step S204), the high-speed storage writing unit 125 applies the LRU and selects the cache storage unit 310 with the oldest access among the data that can be deleted in the high-speed storage 140, and deletes this (step S205).

[0065] If, in step S204, the storage upper limit has not been reached (NO in step S204), or after step S205, the high-speed storage writing unit 125 stores the cache storage unit 310 stored in the temporary buffer in step S203 in the high-speed storage 140 (step S206).

[0066] Next, the page reading unit 124 extracts all the column data described in the necessary column information 230 from among one or more data access units 311 included in the cache storage unit 310 stored in the high-speed storage 140 in step S206 (step S207). That is, in step S207, the page reading unit 124 extracts not only the data access unit 311 corresponding to the leading read data specified in step S104 but also the data access units 311 corresponding to other read data indicated in the necessary column information 230 from the cache storage unit 310 stored in the high-speed storage 140.

[0067] Next, the page reading unit 124 calls the cache memory management unit 128 to determine whether the cache memory 130 has reached its storage upper limit (step S208). If it has reached the storage upper limit (YES in step S208), the cache memory management unit 128 applies the LRU to select the data access unit 311 with the oldest access among the data that can be deleted in the cache memory 130 and deletes it (step S209). If a plurality of data access units 311 are extracted in step S207, in step S209, a plurality of old data access units 311 may be selected and deleted to ensure a free capacity sufficient to store these new data access units 311.

[0068] If, in step S208, the storage upper limit has not been reached (NO in step S208), or after step S209, the cache memory management unit 128 stores the data access unit 311 retrieved in step S207 in the cache memory 130 (step S210). Then, the page reading unit 124 returns the data access unit 311 retrieved in step S207 to the query execution module 110 and proceeds to step S112 in FIG. 4.

[0069] As a modification example of the query execution process in this embodiment, when the data access unit 311 corresponding to the read data is read from the temporary buffer 123 or the high-speed storage 140 (steps S108 - S109, S110 - S111 in FIG. 4), the same processing as steps S208 - S210 may be performed to store the read data in the cache memory 130.

[0070] Step S112 is a process executed after steps S106, S109, S111, and S210. Specifically, it is a process performed after the page reading unit 124 returns the data access unit 311 including the first read data specified in step S103 to the query execution module 110. In step S112, the query execution module 110 determines whether the received data access unit 311 is a search column. If the received data access unit 311 is a search column (YES in step S112), the query execution module 110 performs comparison processing using the column of the received data access unit 311 (step S113). On the other hand, if the received data access unit 311 is not a search column (NO in step S112), since the data access unit 311 is an aggregation column, the query execution module 110 performs aggregation processing using this column (step S114).

[0071] After the completion of steps S113 and S114, the query execution module 110 determines whether the data reading by the data reading module 120 has been completed for all the read data described in the necessary column information 230 created in step S102 (step S115). If all the readings have been completed (YES in step S115), the query execution process ends. If there is remaining uncompleted read data (NO in step S115), the process returns to step S103, and the processing for the next leading read data is repeated from the uncompleted read data.

[0072] As described above, in the query execution process, for the data of the necessary columns that need to be read for the execution of the query 210, when the target data is stored in a plurality of storages (cache memory 130, high-speed storage 140, temporary buffer 123) that hierarchically store the data cache, the target data can be preferentially read from the faster storage, so that high-speed data reading using the cache data can be realized. Also, the data access unit for data reading from the cache memory 130, high-speed storage 140, and temporary buffer 123 is set to a data access unit smaller than the cache storage unit handled by the low-speed storage 150. When the target data is read from the slowest object storage (low-speed storage 150), the data 311 of the data access unit having the necessary columns is extracted from the data 310 of the read cache storage unit and stored in the cache memory 130, thereby suppressing the usage capacity per data in the cache memory 130.

[0073] Furthermore, as described in steps S207 and S210, all the data 311 of the data access unit corresponding to the necessary columns described in the necessary column information 230 is extracted from the data 310 of the cache storage unit read from the low-speed storage 150, and these are stored in the cache memory 130. By doing so, it is possible to prefetch the column data that is likely to be accessed in the future and store it in the cache memory 130. As a result, when reading data in the future, cache data can be read from the cache memory 130 at high speed without reading the data 310 of the cache storage unit from the low-speed storage 150 again. Therefore, wasted reads from the low-speed storage 150 can be reduced, and the efficiency of the memory cache can be improved. In the storage system 1, since data reading of the necessary columns can be performed at high speed as described above, search and join processes in the database can be executed at high speed, and the performance can be improved.

[0074] (2-2) Column Processing FIG. 6 is a diagram for explaining an example of the processing image of columns in the query execution process.

[0075] First, an example of an SQL query 210 issued from the client 20 is shown in step S301 of FIG. 6. In the first line of this SQL query, aggregation processing is described, so the column appearing in the first line is extracted as the aggregation column. Also, the table to be used is described in the second line, and the search condition is described in the third line. Therefore, the column appearing in the third line is extracted as the search column. That is, when the DBMS 100 interprets the SQL query in step S301, the aggregation column is extracted from the first line, and the search column is extracted from the third line (step S302). The processing of steps S301 to S302 corresponds to the processing of steps S101 to S102 in FIG. 4.

[0076] Next, the DBMS 100 extracts the search columns and the aggregation columns from the database table and caches them in the cache memory 130 (step S303). Then, the DBMS 100 reads the search columns from the cache (step S304) and executes the search (step S305). Specifically, the search is executed using the conditional expression shown in step S305 to determine whether a hit occurs. Next, for those that hit in the search, the DBMS 100 reads the aggregation columns from the cache (step S306) and executes the aggregation calculation as shown in step S307.

[0077] (2-3) Cache processing FIG. 7 is a diagram for explaining an example of the cache processing image in the query execution process. In FIG. 7, a part of the configuration of the storage system 1 shown in FIGS. 1 and 3 is used to show the state in which the object file stored in the low-speed storage 150 is read out and a part of it is cached.

[0078] According to FIG. 7, first, the query execution module 110 creates the necessary column information 230 from the query 210. In this example, it is assumed that the data 311A (column A) and the data 311C (column C) are specified as the necessary columns. This necessary column information 230 is sent to the page reading unit 124 (or the cache storage unit specifying unit 121), where it is divided into the necessary columns A and C, and the data 311A and 311C of the data access unit including each necessary column are specified. In this example, it is assumed that these data 311A and 311C are found to be included in the data F1 of the cache storage unit.

[0079] Here, when the data 311A and 311C of the data access unit are specified, the cache memory 130 does not store any data. Therefore, as a result of querying the cache memory 130, the page reading unit 124 confirms that the data 311A and 311C do not exist in the cache memory 130. Next, the page reading unit 124 confirms that the data F1 of the cache storage unit also does not exist in the high-speed storage 140. Therefore, the page reading unit 124 sends the specification information of the data 311A and 311C to the object file reading unit 122.

[0080] Then, the object file reading unit 122 reads the data F1 of the cache storage unit from the low-speed storage 150 and stores it in the temporary buffer 123. Next, the object file reading unit 122 also stores the data F1 in the high-speed storage 140.

[0081] Next, the page reading unit 124 uses the data access unit management information 312 of the data F1 included in the segment management information 127 to extract the data 311A and 311C of the data access unit existing inside the data F1. Then, the page reading unit 124 sends the data 311A corresponding to the first read data to the query execution module 110 and the cache memory 130, and sends the data 311C that is not yet used at the current time to the cache memory 130.

[0082] The query execution module 110 receives the data 311A, continues the query process, and performs a search. If the data 311C is determined to be necessary as a result of this search, the page reading unit 124 reads the data 311C previously stored in the cache memory 130 and sends it to the query execution module 110. Then, the query execution module 110 that has received the data 311C executes an aggregation process.

[0083] In the above description, it is assumed that the data 311A and 311C having the necessary columns are included in the data F1 of one cache storage unit. However, in the case where the data 311 of the data access unit having the necessary columns exists in the data 310 of a plurality of cache storage units (for example, data F1 and data F2), the above processing may be sequentially performed on the corresponding data F1 and data F2.

[0084] By performing the cache processing as described above, the storage system 1 can pre-cache the data 311 of the data access unit that is highly likely to be accessed next in the cache memory 130 with many constraints on capacity (in short, the storage capacity is small compared to other storages), so that the utilization efficiency of the cache memory 130 can be increased and the processing performance of the database can be improved.

[0085] Note that the present invention is not limited to the above-described embodiments, and includes various modifications. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described. Also, it is possible to add, delete, or replace a part of the configuration of the embodiment with other configurations.

[0086] Also, in the drawings, control lines and information lines show those considered necessary for explanation, and not all control lines and information lines are necessarily shown on the product. In practice, it may be considered that almost all the components are interconnected.

Explanation of Reference Numerals

[0087] 1 Storage system 10 Server 20 Client 100 DBMS 110 Query execution module 120 Data read module 121 Cache storage unit specifying unit 122 Object file read unit 123 Temporary Buffer 124 Page Loading Section 125 High-Speed Storage Writing Section 126 High-Speed Storage Reading Section 127 Segment Management Information 128 Cache Memory Management Section 130 Cache Memory 140 High-Speed Storage 150 Low-Speed Storage 160 Network 210 Query 220 Column List 230 Required Column Information 310 Data (Cache Storage Unit) 311 Data (Data Access Unit) 312 Data Access Unit Management Information

Claims

1. A storage system for storing data in a database, comprising: A database management system for managing the database by executing a program by a processor; A storage device; Comprising: The storage device includes: A first storage for storing data of a table in the database; A second storage that is faster in data reading and writing than the first storage; A memory that is faster in data reading and writing than the second storage; Comprising: The first storage reads and writes data in units of a first data size, and the memory reads and writes data in units of a second data size smaller than the first data size; When the database management system executes a query that is a query to the database, The database management system: Interprets the query to identify necessary columns required for execution of the query; Reads data of the identified necessary columns from the storage device; Executes the query based on the data of the necessary columns read from the storage device; In reading the necessary columns from the storage device, Priority is given to reading from a high-speed storage device in which data of the necessary columns exists among the memory, the second storage, and the first storage; Stores data in units of a second data size including data of the necessary columns used for query execution in the memory; When reading from the first storage, stores data in units of the second data size in the memory and stores the read data in units of the first data size in the second storage. A storage system characterized by the above.

2. The database management system: By interpreting the query, identifies the order of calling the necessary columns during execution of the query; Reads data of the necessary columns from the memory, the second storage, or the first storage, stores data related to the read necessary columns in the memory or / and the second storage, and repeats query execution using the read necessary columns in the identified order of calling. The storage system according to claim 1, characterized by the above.

3. The data of the necessary columns includes the data of the search columns used for the search process required by the query and the data of the aggregation columns used for the aggregation process required by the query. The storage system according to claim 1, characterized in that.

4. The data in the first data size unit is composed of the data in the second data size unit and the management information of the data in the second data size unit. The database management system identifies the data in the second data size unit having the data of the necessary columns within the data in the first data size unit by referring to the management information. The storage system according to claim 1, characterized in that.

5. The database management system, When there is insufficient free space when writing data to the second storage, the data in the first data size unit that is the oldest accessed among the deletable data in the second storage is deleted, and then new data is written. When there is insufficient free space when writing data to the memory, the data in the second data size unit that is the oldest accessed in the memory is deleted, and then new data is written. The storage system according to claim 1, characterized in that.

6. The first storage is an object storage installed separately from the server on which the database management system operates, and object files are transmitted and received between the server and the first storage unit via a network. The storage system according to claim 1, characterized in that.

7. A data caching method by a storage system for storing data in a database, The storage system includes a database management system for managing the database by executing a program by a processor and a storage device. The storage device, A first storage for storing the data of the entire table of the database, A second storage that is faster in reading and writing data than the first storage, A memory that is faster in reading and writing data than the second storage, Having, The first storage reads and writes data in a first data size unit, and the memory reads and writes data in a second data size unit smaller than the first data size unit. When the database management system executes a query that is a query to the database, a necessary column specifying step in which the database management system interprets the query and specifies necessary columns required for execution of the query; a data reading step in which the database management system reads data of the necessary columns specified in the necessary column specifying step from the storage device; a query execution step in which the database management system executes the query based on the data of the necessary columns read in the data reading step; comprising: In the data reading step, the database management system prioritizes reading from a high-speed storage device in which data of the necessary columns exists among the memory, the second storage, and the first storage, stores data in units of a second data size including data of the necessary columns used for execution of the query in the memory, when reading from the first storage, stores data in units of the second data size in the memory and stores the read data in units of the first data size in the second storage A data caching method characterized by the above.

Images