Data storage and query method and device of HTAP database

Abstract

The invention discloses a data storage and query method and device of an HTAP database, and relates to the field of data processing, and the data storage and query method comprises the following steps: storing a primary key of a data file in the HTAP database in the form of a primary key index list, and storing the data file in a local disk. When data are queried, whether the target primary key exists in the primary key index list in the HTAP database or not is queried firstly, and if yes, the data file corresponding to the target primary key is obtained from the local disk. On the basis, the memory occupation of the HTAP database is reduced, and the query mode reduces the same data files in the HTAP database, so that the data redundancy in the HTAP database can be effectively relieved.

Description

Technical Field

[0001] The present application relates to the field of database technology, and more specifically, to a data storage and query method and device for a HTAP database. Background Art

[0002] HTAP (Hybrid transaction and analytical processing) refers to a hybrid transaction and analytical processing system. The HTAP database built based on HTAP is a hybrid relational database that can provide mixed load processing of OLTP (On-line Transaction Processing) and OLAP (On-Line Analytical Processing) at the same time.

[0003] To support mixed load processing of OLTP and OLAP, the HTAP database uses an OLTP and OLAP storage isolation mode to store data, that is, the same data file is stored in two copies, one for OLTP and one for OLAP. For example, based on the OLTP distributed database, the HTAP database introduces a dedicated node (called learner) in the distributed system consensus protocol. If there is a query or analysis task for OLAP, the learner asynchronously copies the transaction log from the leader node and constructs a new column format copy for the OLAP query. Since each query requires copying data or directly storing two data files, the database load pressure is high, and there is a lot of identical data in the database, resulting in data redundancy, affecting the storage capacity and query performance of the HTAP database.

[0004] Therefore, there is an urgent need for a data storage method for an HTAP database that can support mixed load processing of OLTP and OLAP while alleviating data redundancy in the HTAP database. Summary of the invention

[0005] In view of the above problems, the present application provides a data storage and query method and device for an HTAP database to achieve the purpose of alleviating data redundancy in the HTAP database. The specific solution is as follows:

[0006] In a first aspect, the present application provides a data storage method of an HTAP database, comprising:

[0007] Constructing a distributed database in the HTAP database, the distributed database comprising: at least one node corresponding to a processing data processor, and a primary key index list corresponding to each of the nodes;

[0008] Determine the primary key and node corresponding to the data file stored in the HTAP database;

[0009] Storing the primary key of the stored data file in the primary key index list corresponding to the node;

[0010] The primary key index list and the stored data file corresponding to each of the nodes are stored in the local disk of the processor corresponding to the node.

[0011] In a possible implementation, the method further includes:

[0012] In response to receiving a data storage request, the data storage request at least includes: a data file to be stored and a primary key to be stored and a node to be stored corresponding to the data file to be stored;

[0013] From the primary key index list corresponding to the to-be-stored node pre-stored in the HTAP database, query whether there is a primary key identical to the to-be-stored primary key;

[0014] If there is no primary key identical to the primary key to be stored in the primary key index list corresponding to the node to be stored, the primary key to be stored is stored in the primary key index list, and the data file to be stored and the primary key to be stored are stored correspondingly in the local disk corresponding to the node to be stored.

[0015] In a possible implementation, storing the to-be-stored data file and the to-be-stored primary key in correspondence to a local disk of the processor corresponding to the to-be-stored node includes:

[0016] The data file to be stored and the primary key to be stored are stored in the virtual memory disk corresponding to the node to be stored in the HTAP database;

[0017] When the storage capacity of the virtual memory disk reaches a preset threshold,

[0018] Compressing the data file to be stored and the primary key to be stored in the virtual memory disk to obtain a compressed file named after the node to be stored;

[0019] The compressed file is stored in the local disk corresponding to the node to be stored.

[0020] In a possible implementation, the method further includes:

[0021] In response to receiving a data editing request, the data editing request includes at least: a target primary key of a target editing data file, and an editing operation instruction for the target editing data file, the editing operation instruction includes at least: an update operation instruction and a delete operation instruction;

[0022] An edit list corresponding to the target edit data file is established, wherein the edit list is used to store the edit operation instructions for the target edit data file.

[0023] In a possible implementation, the method further includes:

[0024] In response to a read request for the target edit data file, reading all the edit operation instructions in the edit list corresponding to the target edit data file;

[0025] An editing operation corresponding to the editing operation instruction is performed on the target editing data file to obtain an edited target editing data file.

[0026] In a possible implementation, the method further includes:

[0027] When the amount of data in the partition of the distributed database for storing the primary key index list corresponding to any of the nodes reaches a preset threshold, splitting the primary key index list to obtain a primary key index list to be migrated;

[0028] Establishing a new node corresponding to the node in the distributed database;

[0029] Migrate the to-be-migrated primary key index list to the primary key index list of the new node.

[0030] A second aspect of the present application provides a data query method for an HTAP database, comprising:

[0031] In response to receiving a data query request, identifying a query type of the data query request;

[0032] When the query type is point query, obtaining a target query primary key corresponding to the target query data file included in the data query request;

[0033] Querying whether there is a primary key identical to the target query primary key from a distributed database pre-built in the HTAP database, wherein the distributed database includes: at least one node corresponding to a processor for processing data, and a primary key index list corresponding to each of the nodes, wherein the primary key index list stores primary keys corresponding to stored data files;

[0034] In the case where there is a primary key identical to the target query primary key in the distributed database, determining a target node corresponding to the target query primary key;

[0035] The target data file corresponding to the target query primary key is obtained from the local disk corresponding to the target node, and the target data file is used as the query result corresponding to the data query request.

[0036] In a possible implementation, the method further includes:

[0037] When the query type of the data query request is a file content query, reading the data file stored in the local disk corresponding to each node of the HTAP database;

[0038] The content of the data file is analyzed to obtain a query result corresponding to the data query request.

[0039] A third aspect of the present application provides a data storage device for an HTAP database, comprising:

[0040] A database construction unit, configured to construct a distributed database in the HTAP database, wherein the distributed database includes: at least one node corresponding to a processing data processor, and a primary key index list corresponding to each of the nodes;

[0041] A primary key node statistics unit, used to determine the primary key and node corresponding to the data file stored in the HTAP database;

[0042] An index list creating unit, used for storing the primary key of the stored data file into the primary key index list corresponding to the node;

[0043] The file storage unit is used to store the primary key index list and the stored data file corresponding to each of the nodes in the local disk of the processor corresponding to the node.

[0044] A fourth aspect of the present application provides a data query device for an HTAP database, comprising:

[0045] a query response unit, configured to identify a query type of the data query request in response to receiving the data query request;

[0046] A primary key acquisition unit, configured to acquire a target query primary key corresponding to a target query data file included in the data query request when the query type is a point query;

[0047] A primary key query unit, configured to query whether there is a primary key identical to the target query primary key in a distributed database pre-built in the HTAP database, wherein the distributed database includes: at least one node corresponding to a processor for processing data, and a primary key index list corresponding to each of the nodes, wherein the primary key index list stores primary keys corresponding to stored data files;

[0048] A node determination unit, configured to determine a target node corresponding to the target query primary key if a primary key identical to the target query primary key exists in the distributed database;

[0049] The query result determination unit is used to obtain the target data file corresponding to the target query primary key from the local disk corresponding to the target node, and use the target data file as the query result corresponding to the data query request.

[0050] Beneficial Effects

[0051] It can be seen from the above technical solutions that the data storage and query method of the HTAP database provided by the embodiment of the present application adopts a storage sharing mode in the HTAP database, and only stores the primary key of each data file in the HTAP database, and independently stores the data file in the local disk, which first reduces the memory usage of the HTAP database and reduces the load of the HTAP database. In addition, each processor can access the corresponding local disk, so when receiving a query request, it can access the local disk according to the primary key to obtain the data file. Compared with the query method of asynchronously replicating transaction logs in the prior art, it reduces the same data files in the HTAP database, and can effectively alleviate data redundancy in the HTAP database. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0053] Figure 1 A schematic diagram of a flow chart of a method for implementing data storage in an HTAP database provided in an embodiment of the present application;

[0054] Figure 2 A flowchart of a method for implementing a data query of an HTAP database provided in an embodiment of the present application;

[0055] Figure 3 A schematic diagram of the structure of a data storage device for an HTAP database provided in an embodiment of the present application;

[0056] Figure 4 A schematic diagram of the structure of a data query device for an HTAP database provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0058] In order to solve the problems raised by the above background technology, the present application embodiment provides a data storage method of an HTAP database. Figure 1 , a flow chart of a method for implementing data storage of an HTAP database provided in an embodiment of the present application, which illustrates the process steps of the method for implementing data storage of an HTAP database, may specifically include:

[0059] Step S110: construct a distributed database in the HTAP database, where the distributed database includes: at least one node corresponding to a processing data processor, and a primary key index list corresponding to each node.

[0060] It is understandable that the HTAP database stores data of multiple processors. If all the data is mixed and stored together, there will be problems with low data management and query efficiency. Therefore, the embodiment of the present application constructs a distributed database in the HTAP database to achieve partitioned storage of primary key data of multiple processing nodes. Each node corresponds to a processor, and the primary key stored under the node is the primary key corresponding to the data of the processor.

[0061] Optionally, the distributed database can be built based on TiDB, CockroachDB, OceanBase, Spanner, etc., where the Multi-Raft protocol and the storage structure of the LSM tree can be used to refine the distributed database. Multi-Raft is used to implement distributed consistency partitioned multi-copy data storage to ensure the scalability and high availability of the distributed database. The data storage uses the LSM storage structure. Since the LSM tree storage structure is ordered by primary key, it is very suitable for constructing a primary key index list in the embodiment of the present application. It is understandable that the distributed database can also adopt other similar distributed system consensus protocols or storage structures, and the above is only an example.

[0062] It can be understood that step S110 can be regarded as a transformation of the original HTAP database in the isolated storage mode, or as creating a new HTAP database. Therefore, the content of the primary key index list corresponding to each node is empty. After the content of the original HTAP database in the isolated storage mode is migrated to the HTAP database in the shared storage model created in the embodiment of the present application, the blank primary key index list is filled with the migrated data content.

[0063] Step S120, determining the primary key and node corresponding to the data file stored in the HTAP database.

[0064] Step S130, storing the primary key of the stored data file into the primary key index list corresponding to the node.

[0065] Traverse each data file stored in the HTAP database in the original isolated storage mode, extract the primary key of each data file and the corresponding node. Use the node as the classification basis, determine the primary key corresponding to each node, and store the primary keys belonging to the same node in the same primary key index list.

[0066] The primary key index list can refer to Table 1, where the primary key index list contains at least three columns of data, namely: sequence number, primary key and node ID. The sequence number is an increasing sequence number to facilitate orderly storage of primary keys.

[0067] Table 1 Primary key index list

[0068]

[0069] Step S140 , storing the primary key index list and stored data files corresponding to each node to the local disk of the processor corresponding to the node.

[0070] The embodiment of the present application migrates and stores the data files corresponding to each node to a local disk independent of the HTAP database, and the data files corresponding to each node are stored on the local disk of the processor, thereby isolating and storing the data files of each node. Compared with uniform storage on a local disk, the storage pressure of the local disk can be reduced, thereby improving the query performance when querying data files. At the same time, the data files are separated from the HTAP database to reduce the storage pressure of the HTAP database. Only the primary key of the data file needs to be stored, which alleviates the data redundancy problem of HTAP.

[0071] In addition, the embodiment of the present application also stores the primary key index list corresponding to each node in the local disk synchronously, which is understandable to facilitate querying data files in the local disk. Optionally, a directory for storing data files can also be constructed based on the primary key index list. Taking Table 1 as an example, a file named "1-key1-A" is created on the local disk, in which the data file corresponding to the primary key key1 is stored. As a result, each data file on the local disk is stored in the order of the primary keys in the primary key index list, which is convenient for querying.

[0072] The data storage method of the HTAP database described above changes the isolated storage mode of the original HTAP database, and the data query path of the HTAP database changes. Therefore, the data query method of the HTAP database corresponding to the data storage method of the HTAP database described above can be referred to. Figure 2 , the flowchart of the method for implementing the data query of the HTAP database provided in the embodiment of the present application is described, which may specifically include:

[0073] Step S210: in response to receiving a data query request, identifying a query type of the data query request.

[0074] It is understandable that the HTAP database can provide mixed load processing of OLTP and OLAP at the same time, wherein the query mode of OLTP is point query, that is, a single or a small number of data records are retrieved from the HTAP database according to a specific key (Key) or condition. In the embodiment of the present application, the data record is a data file. OLAP is usually used for data analysis, data mining, etc. Therefore, obtaining data files only through primary key query cannot support the functions of OLAP. Therefore, the query mode of OLAP is to query the file content, that is, the local disk can be directly accessed to analyze and query the data in the data files stored on the local disk.

[0075] Therefore, after receiving a data query request, the query type of the data query request should be identified first, and the query steps to be performed subsequently should be determined to achieve accurate query.

[0076] Step S220: when the query type is point query, obtain the target query primary key corresponding to the target query data file included in the data query request.

[0077] Step S230, query whether there is a primary key identical to the target query primary key in the distributed database pre-built in the HTAP database. If yes, execute step S240; if not, return the query result of failed production information.

[0078] Step S240, determining the target node corresponding to the target query primary key.

[0079] Step S250: Acquire a target data file corresponding to the target query primary key from the local disk corresponding to the target node, and use the target data file as a query result corresponding to the data query request.

[0080] After confirming the point query, the target query primary key to be queried is obtained from the data query request, and the target query primary key is used as the query basis. By retrieving each primary key index list in the HTAP database, it is determined whether the target query primary key is stored in the HTAP database. If the target query primary key is not retrieved, it proves that the data corresponding to the target query primary key is not stored in the HTAP database and the local disk. Therefore, the query failure result can be returned.

[0081] If the target query primary key is found in the primary key index list of the HTAP database, it proves that the corresponding data file is stored in the local disk, and the query result is not empty. Further, according to the target node corresponding to the target query primary key in the primary key index list, the local disk corresponding to the node is accessed, and the data file corresponding to the target query primary key is queried from the local disk, and the data file is returned as the query result. Optionally, if the directory of the data file in the local disk is created according to the primary key index list, it is also possible to determine the corresponding position of the target data file in the directory of the data file according to the target node, primary key, and serial number corresponding to the target query primary key in the primary key index list, so as to accurately obtain the target data file, shorten the query time, and improve the query efficiency.

[0082] In a possible implementation, when the query type of the data query request is a file content query, the data file stored in the local disk corresponding to each node of the HTAP database is read; the content of the data file is analyzed to obtain a query result corresponding to the data query request.

[0083] It is understandable that OLAP query does not use primary key index list query, and can directly scan data files stored in the local disk, and determine the query result of the data query request by querying and analyzing the content of the data file.

[0084] In summary, the data storage and query method of the HTAP database provided by the embodiment of the present application adopts a storage sharing mode in the HTAP database, only the primary key of each data file is stored in the HTAP database, and the data file is independently stored in the local disk, which first reduces the memory usage of the HTAP database and reduces the load of the HTAP database. In addition, each processor can access the corresponding local disk, so when receiving a query request, it can access the local disk according to the primary key to obtain the data file. Compared with the query method of asynchronously replicating transaction logs in the prior art, the number of identical data files in the HTAP database is reduced, which can effectively alleviate data redundancy in the HTAP database.

[0085] Next, the embodiments of the present application describe a data storage method of an HTAP database, a data query method of an HTAP database, and other possible implementations.

[0086] In a possible implementation, according to the HTAP database constructed above, the process of storing data files may include: in response to receiving a data storage request, the data storage request includes at least: a data file to be stored and a primary key to be stored and a node to be stored corresponding to the data file to be stored; querying whether there is a primary key identical to the primary key to be stored in a primary key index list corresponding to the node to be stored pre-stored in the HTAP database; if there is no primary key identical to the primary key to be stored in the primary key index list corresponding to the node to be stored, storing the primary key to be stored in the primary key index list, and storing the data file to be stored and the primary key to be stored in correspondence to the local disk corresponding to the node to be stored.

[0087] When a data storage request is received, first obtain the data file to be stored in the data storage request and the primary key and node corresponding to the data file to be stored, that is, the primary key to be stored and the node to be stored. Further determine whether the data file to be stored has been stored in the HTAP database to avoid duplicate storage. Optionally, query from the HTAP database whether there is a node to be stored. If there is a node to be stored, query whether the primary key to be stored is stored in the primary key index list corresponding to the node to be stored. Query from the stored primary key index list whether the same primary key as the primary key to be stored is stored. If so, it proves that the data file to be stored has been stored in the local disk. To avoid duplicate storage, the response to the data storage request can be canceled, and a prompt message indicating that the storage failed or the data already exists in the database is returned.

[0088] If the primary key that is the same as the primary key to be stored is not stored in the stored primary key index list, it proves that the data file to be stored is stored in the HTAP database for the first time. Therefore, the primary key to be stored is stored in the corresponding primary key index list, and can be inserted into the primary key index list in ascending sequence with reference to Table 1. At the same time, the data file to be stored is stored in the local disk corresponding to the node to be stored. It can be understood that the data files and directories of the data files in the local disk are stored according to the primary key index list. Therefore, the data file to be stored is also inserted into the local disk in ascending sequence relative to the storage of the primary key index list.

[0089] In another possible implementation, the data stored in the HTAP database needs to be modified, deleted, or other editing operations, and the process may include: responding to receiving a data editing request, the data editing request at least including: a target primary key of a target editing data file, and editing operation instructions for the target editing data file, the editing operation instructions at least including: an update operation instruction and a delete operation instruction; establishing an editing list corresponding to the target editing data file, the editing list being used to store the editing operation instructions for the target editing data file.

[0090] In the local disk, each data file corresponds to an edit list, which is used to store the edit instructions for the corresponding data file. For example, if the data edit request requires the deletion of the Nth row of data in data file A, the delete operation instruction, such as "delete the Nth row of data", or the result of the deletion operation, such as "the Nth row of data has been deleted", is recorded in the edit list. In the column storage model, the edit list is equivalent to a column of data. When updating data, you only need to refresh the edit list, which has almost no damage to the query performance of the HTAP database and the local disk, greatly reducing the IO overhead.

[0091] Optionally, the data file is composed of multiple lines of data, and the edit list can be designed as an edit list as shown in Table 2, which includes two columns of data: sequence number and edit instruction. The sequence number is an increasing sequence number, and each sequence number corresponds to a row of data in the data file, that is, sequence number 2 corresponds to the second row of data in the data file, and the edit instruction corresponding to the second row represents the edit instruction for the second row of data in the data file. As shown in Table 2, the edit instruction corresponding to sequence number 2 is delete, which means deleting the second row of data in the data file, and the edit instruction of the first row in Table 2 is empty, which means that the first row of data maintains the original data and does not need to be modified. It can be understood that the edit list can also include edit instructions such as insert, modify, and update, which are not listed here one by one.

[0092] Table 2 Edit List

[0093]

[0094] In an embodiment of the present application, the editing object of the above-mentioned editing operation is the data in the data file, and the editing object of the editing operation may also be a data file. For example, the data file a corresponding to the primary key key2 in node A is replaced with the data file b, and the primary key key2 is synchronously updated to keyn. Based on this, the editing operation of the data file in the database is often accompanied by the overall change of the primary key, serial number, etc. Therefore, in response to the editing request for the data file in the HTAP database, the primary key index list in the HTAP database is updated while the data file stored on the local disk is edited. In the case of inserting or adding a new primary key to the primary key index list, the serial number corresponding to each primary key in the serial number column is updated according to the arrangement order of the primary key after the operation of inserting or adding the new primary key.

[0095] Based on this, in response to a read request for the target editing data file, all editing operation instructions in the editing list corresponding to the target editing data file are read; the editing operation corresponding to the editing operation instruction is performed on the target editing data file to obtain the edited target editing data file.

[0096] It is understandable that the embodiment of the present application records the editing operation of the data file in the form of an edit list without changing the original data file. Therefore, when the data file is subsequently read, in order to obtain the latest data file, it is necessary to read each editing operation instruction in the edit list at the same time, and perform the editing operation on the original data file according to the editing operation instruction to obtain the edited data file. For example, when reading the data file and the edit list corresponding to the data file, the data in the data file marked with the deletion operation is ignored.

[0097] In another possible implementation, step S140 includes: storing the data file to be stored and the primary key to be stored in a virtual memory disk corresponding to the node to be stored in the HTAP database; when the storage capacity of the virtual memory disk reaches a preset threshold, compressing the data file to be stored and the primary key to be stored in the virtual memory disk to obtain a compressed file named after the node to be stored; and storing the compressed file in a local disk corresponding to the node to be stored.

[0098] In an embodiment of the present application, a data file is first written into a virtual memory disk (ramdisk) of the HTAP database file. After the amount of data in the data file stored in the virtual memory disk reaches a threshold, the data file in the virtual memory disk is compressed in a columnar storage manner, and the compressed file is stored in a local file. At the same time, in order to ensure the consistency of data in the HTAP database and the local disk, the primary key index list in the HTAP database is synchronously stored in the local disk.

[0099] The embodiment of the present application uses nodes as the partition basis of the distributed database, and the primary key index list of a node corresponds to a ramdisk. The ramdisk stores data files belonging to the same node, which can ensure that the compressed files can be directly stored on the local disk, without the need to classify and compress data based on nodes, thereby improving the efficiency of data storage. In addition, compressing a large number of data files and writing them to the local disk at one time can reduce the writing frequency of the local disk, avoiding the problem of local disk downtime caused by excessive local disk I / O pressure when storing a large number of data files at one time, thereby improving the security of data in the local disk.

[0100] In the embodiment of the present application, when the number of data files or the amount of data stored on the local disk reaches a threshold or the storage space of the local disk is occupied, the data files stored on the local disk are merged and uploaded to the virtual distributed storage system, thereby clearing the local disk and providing storage space for subsequent data files to be stored. The merged file is no larger than the size of a slice in the distributed storage system, for example, no larger than the size of a block in HDFS, so as to minimize the amount of storage space occupied by data files and increase the amount of data that can be stored in the HTAP database.

[0101] The virtual distributed storage system is usually HDFS or object storage, which is used to store the data files after the database nodes are merged. It can be built through the open source software Alluxio. For example, a table directory is created for each primary key index list on Alluxio, and a HTAP database node directory is created under the table directory. The data file directory of the HTAP database node is mounted to the directory. A distributed storage file directory is created under the table directory, and the distributed file system is mounted to the directory. Based on this, the computing engine (Spark, Flink, Presto) can access the entire heterogeneous table file by accessing the table directory of Alluxio. It can be understood that the virtual distributed storage system is equivalent to a database system that is independent of the HTAP database and has a distributed database structure similar to that of the HTAP database, except that each node in the virtual distributed storage system also stores the data files corresponding to each primary key in the primary key index list.

[0102] It is understandable that after uploading the compressed and merged files to the next storage space, the virtual memory disk (ramdisk) and the local disk need to delete the uploaded data files to achieve the purpose of clearing the storage space.

[0103] In the embodiment of the present application, data files and data files after compression and merging are operated on the data files in units of nodes, and it is difficult to represent each data file with a merged name after compression or merging. Therefore, in the embodiment of the present application, whether it is a data file before or after compression or merging, the prefix of the file name is the node ID, so as to distinguish the correspondence between the data file and the node.

[0104] In another possible implementation, when the amount of data in a partition used to store the primary key index list corresponding to any node in a distributed database reaches a preset threshold, the primary key index list is split to obtain a primary key index list to be migrated; a new node corresponding to the node is established in the distributed database; and the primary key index list to be migrated is migrated to the primary key index list of the new node.

[0105] In the embodiment of the present application, a partition corresponds to a Raft Group in Multi-Raft. When the amount of data in a partition exceeds the threshold, it will be split into two partitions. The primary key index data of LSM is divided into a memory part and a disk part, and the data of the data file is also divided into a memory part and a disk part. LSM and data files are saved synchronously from memory to disk.

[0106] When the primary key index list and related data stored in a partition of a node in the HTAP database reaches the storage threshold of the partition, the partition is split. First, before splitting the data, the primary key index list, data files and other related data of the partition are persisted to the local disk. According to the amount of data stored in the partition, determine the primary key that can split the data in the partition into two parts as evenly as possible. Split the primary key index list according to the primary key to obtain two new primary key index lists, create a partition of a new node in the distributed database, and migrate any new primary key index list to the partition of the new node. It can be understood that the node ID of the new node is the same as that of the node to be split, but the data of the same node is stored in two locations.

[0107] In addition, although the primary key list is split into two, the corresponding data files still belong to the same node and need to be stored in the same local disk. Therefore, the data files in the local disk do not need to be split, and since the data files, primary key index lists, etc. in the local disk are all changed, in order to maintain the consistency between the local disk and the HTAP database, the serial numbers in the two new primary key index lists obtained by splitting still maintain the original serial numbers before splitting, that is, the serial number of the primary key key2 in the primary key index list before splitting is 5. After splitting, although the primary key key2 is ranked first in the new primary key index list, its serial number is still 5. Since the data file is uploaded to the distributed file system, the node only retains the incremental serial number column, and it is stored in column compression, so the amount of data is not large. If the data file is to be split, then the data files that have been uploaded to the distributed file system are split, which generates unnecessary performance consumption. Therefore, the processing flow of this patent combines the ID information of the processing node stored in the primary key. When the partition is split, only the primary key is split, and the data is not split, which greatly improves the efficiency of partition splitting.

[0108] Optionally, for single data query (point query) of OLTP, the OLTP and OLAP storage sharing mode of this patent only queries the index after querying LSM, and needs to query the data in the columnar file according to the index, which is lower than the performance of TiDB and CockroachDB. To solve this problem, the external solution can be redis cache, and the internal solution can be solved to a certain extent through mature technologies such as vectorization engine, predicate pushdown, column pruning, etc.

[0109] In summary, the data storage method of the HTAP database provided in the embodiment of the present application and the proposed OLTP and OLAP storage sharing mode greatly save the storage space of the database because the LSM does not save data but only saves the primary key index information. In addition, a small amount of LSM index data can greatly reduce the LSM merge (compaction) overhead and improve the overall LSM write and query performance. In the index data stored in the embodiment of the present application, the primary key is stored together with the node. When the partition is split, only the primary key is split, and the data is not split, which greatly improves the efficiency of the partition split. Furthermore, based on the data storage method of the above-mentioned HTAP database, when performing data query, compared with the prior art, there is no need to copy the data file before querying, which almost eliminates the delay of OLAP query. What's more, the embodiment of the present application can easily use the complete functions of a mature distributed database to store the primary key index to build an HTAP database without paying attention to the concurrency of database transactions and the characteristics of ACID.

[0110] In view of the respective characteristics of the two modes of OLTP and OLAP storage isolation and OLTP and OLAP storage sharing proposed in this patent, and because the OLTP and OLAP storage isolation and sharing modes can be derived from the distributed database of Multi-Raft and LSM technology, it is very convenient to integrate and use them together as two different modes in the table of HTAP database. For data with a small amount of single data and a high frequency of query, use the OLTP and OLAP storage isolation mode, and for data with a large amount of single data and a relatively low frequency of query, use the OLTP and OLAP storage sharing mode to achieve the best storage effect.

[0111] For example, in an HTAP database, when designing a primary key index list, the data table that is frequently checked and modified is designed to be very narrow, that is, the field or data volume of a data is very small. For a row of data with a small amount of data and frequent check, such as bank customer account information, the OLTP and OLAP storage isolation mode is used for storage; while for a row of data with a large amount of data and infrequent query, such as customer information, the above-mentioned HTAP database storage method, that is, the OLTP and OLAP storage sharing mode is used for storage.

[0112] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0113] The following describes a data storage device for an HTAP database and a data query device for an HTAP database provided in an embodiment of the present application. The data storage device for an HTAP database and the data query device for an HTAP database described below can correspond to the data storage method for an HTAP database and the data query method for an HTAP database described above.

[0114] First, combine Figure 3 , the data storage device of the HTAP database is introduced, such as Figure 3 As shown, the data storage device of the HTAP database may include:

[0115] The database construction unit 1100 is used to construct a distributed database in the HTAP database, wherein the distributed database includes: at least one node corresponding to a processing data processor, and a primary key index list corresponding to each of the nodes;

[0116] A primary key node statistics unit 1200 is used to determine the primary key and node corresponding to the data file stored in the HTAP database;

[0117] An index list creation unit 1300, configured to store the primary key of the stored data file into the primary key index list corresponding to the node;

[0118] The file storage unit 1400 is used to store the primary key index list and the stored data file corresponding to each of the nodes in the local disk of the processor corresponding to the node.

[0119] In a possible implementation, the method further includes:

[0120] A storage request response unit, configured to respond to a received data storage request, wherein the data storage request includes at least: a data file to be stored and a primary key to be stored and a node to be stored corresponding to the data file to be stored;

[0121] A list query unit, configured to query whether there is a primary key identical to the primary key to be stored from the primary key index list corresponding to the node to be stored pre-stored in the HTAP database;

[0122] A storage unit is used to store the primary key to be stored in the primary key index list if there is no primary key identical to the primary key to be stored in the primary key index list corresponding to the node to be stored, and to store the data file to be stored and the primary key to be stored in correspondence with each other in the local disk corresponding to the node to be stored.

[0123] In a possible implementation, the storage unit includes:

[0124] A memory storage subunit, configured to store the data file to be stored and the primary key to be stored in a virtual memory disk corresponding to the node to be stored in the HTAP database;

[0125] A file compression subunit, configured to compress the data file to be stored and the primary key to be stored in the virtual memory disk to obtain a compressed file named after the node to be stored when the storage capacity of the virtual memory disk reaches a preset threshold;

[0126] The compressed file storage subunit is used to store the compressed file in the local disk corresponding to the node to be stored.

[0127] In a possible implementation, the method further includes:

[0128] An edit request response unit, configured to respond to a received data edit request, wherein the data edit request includes at least: a target primary key of a target edit data file, and an edit operation instruction for the target edit data file, wherein the edit operation instruction includes at least: an update operation instruction and a delete operation instruction;

[0129] The instruction storage unit is used to establish an editing list corresponding to the target editing data file, and the editing list is used to store the editing operation instructions for the target editing data file.

[0130] In a possible implementation, the method further includes:

[0131] a read request response unit, configured to respond to a read request for the target edit data file and read all the edit operation instructions in the edit list corresponding to the target edit data file;

[0132] The target file determination unit is used to perform an editing operation corresponding to the editing operation instruction on the target editing data file to obtain an edited target editing data file.

[0133] In a possible implementation, the method further includes:

[0134] A primary key splitting unit, configured to split the primary key index list to be migrated when the amount of data in the partition for storing the primary key index list corresponding to any of the nodes in the distributed database reaches a preset threshold;

[0135] A node creation unit, used to establish a new node corresponding to the node in the distributed database;

[0136] The list migration unit is used to migrate the primary key index list to be migrated to the primary key index list of the new node.

[0137] Combination Figure 4 , the data query device of HTAP database is introduced, such as Figure 4 As shown, the data query device of the HTAP database may include:

[0138] The query response unit 2100 is configured to identify the query type of the data query request in response to receiving the data query request;

[0139] A primary key acquisition unit 2200 is configured to acquire a target query primary key corresponding to a target query data file included in the data query request when the query type is point query;

[0140] A primary key query unit 2300 is used to query whether there is a primary key identical to the target query primary key in a distributed database pre-built in the HTAP database, wherein the distributed database includes: at least one node corresponding to a processor for processing data, and a primary key index list corresponding to each of the nodes, wherein the primary key index list stores primary keys corresponding to stored data files;

[0141] A node determination unit 2400 is configured to determine a target node corresponding to the target query primary key if a primary key identical to the target query primary key exists in the distributed database;

[0142] The query result determination unit 2500 is configured to obtain the target data file corresponding to the target query primary key from the local disk corresponding to the target node, and use the target data file as the query result corresponding to the data query request.

[0143] In a possible implementation, the method further includes:

[0144] A file reading unit, configured to read the data file stored in the local disk corresponding to each node of the HTAP database when the query type of the data query request is a file content query;

[0145] The analysis result determination unit is used to analyze the content of the data file to obtain a query result corresponding to the data query request.

[0146] In summary, the data storage and query device of the HTAP database provided by the embodiment of the present application adopts a storage sharing mode in the HTAP database, and only stores the primary key of each data file in the HTAP database, and independently stores the data file in the local disk, which first reduces the memory usage of the HTAP database and reduces the load of the HTAP database. In addition, each processor can access the corresponding local disk, so when receiving a query request, it can access the local disk according to the primary key to obtain the data file. Compared with the query method of asynchronously replicating transaction logs in the prior art, the number of identical data files in the HTAP database is reduced, which can effectively alleviate data redundancy in the HTAP database.

[0147] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0148] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0149] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data storage method for an HTAP database, characterized in that: include: Constructing a distributed database in the HTAP database, the distributed database comprising: at least one node corresponding to a processor for processing data, and a primary key index list corresponding to each of the nodes; Determine the primary key and node corresponding to the data file stored in the HTAP database; Storing the primary key of the stored data file in the primary key index list corresponding to the node; The primary key index list and the stored data file corresponding to each of the nodes are stored in the local disk of the processor corresponding to the node.

2. The data storage method of the HTAP database according to claim 1, characterized in that: Also includes: In response to receiving a data storage request, the data storage request includes at least: a data file to be stored and a primary key to be stored and a node to be stored corresponding to the data file to be stored; From the primary key index list corresponding to the to-be-stored node pre-stored in the HTAP database, query whether there is a primary key identical to the to-be-stored primary key; If there is no primary key identical to the primary key to be stored in the primary key index list corresponding to the node to be stored, the primary key to be stored is stored in the primary key index list, and the data file to be stored and the primary key to be stored are stored correspondingly in the local disk corresponding to the node to be stored.

3. The data storage method of the HTAP database according to claim 2, characterized in that: The storing the data file to be stored and the primary key to be stored in correspondence with each other in the local disk corresponding to the node to be stored includes: The data file to be stored and the primary key to be stored are stored in the virtual memory disk corresponding to the node to be stored in the HTAP database; When the storage capacity of the virtual memory disk reaches a preset threshold, Compressing the data file to be stored and the primary key to be stored in the virtual memory disk to obtain a compressed file named after the node to be stored; The compressed file is stored in the local disk corresponding to the node to be stored.

4. The data storage method of the HTAP database according to claim 1, characterized in that: Also includes: In response to receiving a data editing request, the data editing request includes at least: a target primary key of a target editing data file, and an editing operation instruction for the target editing data file, the editing operation instruction includes at least: an update operation instruction and a delete operation instruction; An edit list corresponding to the target edit data file is established, wherein the edit list is used to store the edit operation instructions for the target edit data file.

5. The data storage method of the HTAP database according to claim 4, characterized in that: Also includes: In response to a read request for the target edit data file, reading all the edit operation instructions in the edit list corresponding to the target edit data file; An editing operation corresponding to the editing operation instruction is performed on the target editing data file to obtain an edited target editing data file.

6. The data storage method of the HTAP database according to claim 1, characterized in that: Also includes: When the amount of data in the partition of the distributed database for storing the primary key index list corresponding to any of the nodes reaches a preset threshold, splitting the primary key index list to obtain a primary key index list to be migrated; Establishing a new node corresponding to the node in the distributed database; Migrate the to-be-migrated primary key index list to the primary key index list of the new node.

7. A data query method for an HTAP database, characterized in that: include: In response to receiving a data query request, identifying a query type of the data query request; When the query type is point query, obtaining a target query primary key corresponding to the target query data file included in the data query request; Querying whether there is a primary key identical to the target query primary key from a distributed database pre-built in the HTAP database, wherein the distributed database includes: at least one node corresponding to a processor for processing data, and a primary key index list corresponding to each of the nodes, wherein the primary key index list stores primary keys corresponding to stored data files; In the case where there is a primary key identical to the target query primary key in the distributed database, determining a target node corresponding to the target query primary key; A target data file corresponding to the target query primary key is obtained from a local disk corresponding to the target node, and the target data file is used as a query result corresponding to the data query request.

8. The data query method of the HTAP database according to claim 7, characterized in that: Also includes: When the query type of the data query request is a file content query, reading the data file stored in the local disk corresponding to each node of the HTAP database; The content of the data file is analyzed to obtain a query result corresponding to the data query request.

9. A data storage device for an HTAP database, characterized in that: include: A database construction unit, configured to construct a distributed database in the HTAP database, wherein the distributed database includes: at least one node corresponding to a processing data processor, and a primary key index list corresponding to each of the nodes; A primary key node statistics unit, used to determine the primary key and node corresponding to the data file stored in the HTAP database; An index list creating unit, used for storing the primary key of the stored data file into the primary key index list corresponding to the node; The file storage unit is used to store the primary key index list and the stored data file corresponding to each of the nodes in the local disk of the processor corresponding to the node.

10. A data query device for an HTAP database, characterized in that: include: a query response unit, configured to identify a query type of the data query request in response to receiving the data query request; A primary key acquisition unit, configured to acquire a target query primary key corresponding to a target query data file included in the data query request when the query type is a point query; A primary key query unit, configured to query whether there is a primary key identical to the target query primary key in a distributed database pre-built in the HTAP database, wherein the distributed database includes: at least one node corresponding to a processor for processing data, and a primary key index list corresponding to each of the nodes, wherein the primary key index list stores primary keys corresponding to stored data files; A node determination unit, configured to determine a target node corresponding to the target query primary key if a primary key identical to the target query primary key exists in the distributed database; The query result determination unit is used to obtain a target data file corresponding to the target query primary key from a local disk corresponding to the target node, and use the target data file as a query result corresponding to the data query request.

Images