Adaptive Method, Device, and Computer Equipment for Database Storage Mode

Abstract

The present application relates to an adaptive method, device, computer device, storage medium, and computer program product for a database storage mode. The method is applied to the fields of cloud storage and intelligent transportation. The method includes: obtaining the operation parameters of the workload and the read / write operation performance parameters of each key-value pair; the operation parameters include the read / write operation ratio, query statement, and data distribution; determining a first space size of the space occupied by the key-value pair according to the read / write operation ratio and the read / write operation performance parameters; determining the partitioning method of the storage area according to the semantic information of the query statement; determining the data density under the candidate grouping range according to the data distribution; and determining the storage mode of the data table in the database when running the workload based on the partitioning method, the candidate grouping range, the data density, and the first space size. Using this method can effectively improve the overall performance of the database system.

Description

Technical Field

[0001] The present application relates to the field of computer technologies, and in particular, to an adaptive method, apparatus, computer device, storage medium, and computer program product for a database storage mode. Background Art

[0002] With the development of computer technologies and Internet technologies, data has become the core asset of enterprises, and protecting data is one of the effective means to protect enterprise assets. When storing data, a row storage or column storage mode is usually adopted so that these storage modes can improve the read / write (I / O) performance of the database when processing queries under a certain workload.

[0003] However, in the current adaptive methods for database storage modes, since the storage mode is limited to row storage and column storage, the adjustment of parameters mostly depends on the hardware configuration and the characteristics of queries, and it is impossible to adjust the underlying basic configuration such as the storage mode. Therefore, the finally selected storage mode is often not the best, which may cause relatively large storage overhead and even space waste, resulting in poor overall performance of the database system. Summary of the Invention

[0004] Based on this, in view of the above technical problems, it is necessary to provide an adaptive method, apparatus, computer device, computer-readable storage medium, and computer program product for a database storage mode, which can effectively improve the overall performance of the database system.

[0005] In a first aspect, the present application provides an adaptive method for a database storage mode. The method includes: obtaining the operation parameters of the workload and the read / write operation performance parameters of each key-value pair; the operation parameters include the read / write operation ratio, query statement, and data distribution; determining a first space size of the space occupied by the key-value pair according to the read / write operation ratio and the read / write operation performance parameters; determining the partitioning method of the storage area according to the semantic information of the query statement; determining the data density of the data under the candidate grouping range according to the data distribution; and determining the storage mode of the data table in the database when running the workload based on the partitioning method, the candidate grouping range, the data density, and the first space size.

[0006] In a second aspect, the present application also provides an adaptive device for a database storage mode. The device includes: an acquisition module, configured to acquire the running parameters of the workload and the read / write operation performance parameters of each key-value pair; the running parameters include the read / write operation ratio, query statements, and data distribution; a determination module, configured to determine a first space size of the space occupied by the key-value pair according to the read / write operation ratio and the read / write operation performance parameters; determine a partitioning method of the storage area according to the semantic information of the query statement; determine the data density under the candidate grouping range according to the data distribution; and determine a storage mode of the data table in the database when running the workload based on the partitioning method, the candidate grouping range, the data density, and the first space size.

[0007] In a third aspect, the present application also provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented: acquiring the running parameters of the workload and the read / write operation performance parameters of each key-value pair; the running parameters include the read / write operation ratio, query statements, and data distribution; determining a first space size of the space occupied by the key-value pair according to the read / write operation ratio and the read / write operation performance parameters; determining a partitioning method of the storage area according to the semantic information of the query statement; determining the data density under the candidate grouping range according to the data distribution; and determining a storage mode of the data table in the database when running the workload based on the partitioning method, the candidate grouping range, the data density, and the first space size.

[0008] In a fourth aspect, the present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, and when the computer program is executed by a processor, the following steps are implemented: acquiring the running parameters of the workload and the read / write operation performance parameters of each key-value pair; the running parameters include the read / write operation ratio, query statements, and data distribution; determining a first space size of the space occupied by the key-value pair according to the read / write operation ratio and the read / write operation performance parameters; determining a partitioning method of the storage area according to the semantic information of the query statement; determining the data density under the candidate grouping range according to the data distribution; and determining a storage mode of the data table in the database when running the workload based on the partitioning method, the candidate grouping range, the data density, and the first space size.

[0009] In a fifth aspect, the present application also provides a computer program product. The computer program product includes a computer program which, when executed by a processor, implements the following steps: obtaining the running parameters of a workload and the read / write operation performance parameters of each key-value pair; the running parameters including the read / write operation ratio, query statements, and data distribution; determining a first space size of the space occupied by the key-value pair according to the read / write operation ratio and the read / write operation performance parameters; determining a partitioning method of a storage area according to the semantic information of the query statements; determining the data density under a candidate grouping range according to the data distribution; and determining a storage mode of a data table in a database when running the workload based on the partitioning method, the candidate grouping range, the data density, and the first space size.

[0010] The above-mentioned adaptive method, device, computer device, storage medium, and computer program product for the database storage mode obtain the running parameters of a workload and the read / write operation performance parameters of each key-value pair; the running parameters include the read / write operation ratio, query statements, and data distribution; determine a first space size of the space occupied by the key-value pair according to the read / write operation ratio and the read / write operation performance parameters; determine a partitioning method of a storage area according to the semantic information of the query statements; determine the data density under a candidate grouping range according to the data distribution; and determine a storage mode of a data table in a database when running the workload based on the partitioning method, the candidate grouping range, the data density, and the first space size. Since the read / write operation performance parameters of each key-value pair are obtained through pre-testing, it is possible to determine an optimal value range of the space size occupied by the key-value pair based on the read / write operation ratio in the running parameters of different workloads and the read / write operation performance parameters of each key-value pair obtained through pre-testing, so that the server can determine an optimal storage mode of a data table in a database when running different workloads based on the partitioning method, the candidate grouping range, the data density, and the optimal value range of the space size occupied by the key-value pair, solving the problem in the traditional method that the optimal storage mode cannot be matched according to different loads, effectively improving the I / O efficiency of the system, and thus effectively improving the overall performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is an application environment diagram of an adaptive method for a database storage mode in an embodiment;

[0012] Figure 2 It is a flowchart of an adaptive method for a database storage mode in an embodiment;

[0013] Figure 3 It is a schematic diagram of the change of a data table structure in an embodiment;

[0014] Figure 4 It is a schematic diagram of the conversion of a storage mode in an embodiment;

[0015] Figure 5 Schematic diagram of column segment storage mode in one embodiment;

[0016] Figure 6 Schematic diagram of column family segment storage mode in one embodiment;

[0017] Figure 7 Schematic diagram of whole table storage mode in one embodiment;

[0018] Figure 8 System overall architecture diagram in one embodiment;

[0019] Figure 9 Schematic diagram of cell row storage in one embodiment;

[0020] Figure 10 Schematic diagram of cell column storage in one embodiment;

[0021] Figure 11 Schematic diagram of multi - row storage in one embodiment;

[0022] Figure 12 Schematic diagram of column family storage format in one embodiment;

[0023] Figure 13 Schematic diagram of encoding scheme for diverse storage modes in one embodiment;

[0024] Figure 14 Flowchart of storage mode adaptive processing in one embodiment;

[0025] Figure 15 Structural block diagram of adaptive device for database storage mode in one embodiment;

[0026] Figure 16 Internal structure diagram of a computer device in one embodiment. Detailed implementation manners

[0027] In order to make the objectives, technical solutions and advantages of the present application clearer and more understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0028] The adaptive method for database storage mode provided by the embodiments of the present application can be applied to, for example Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or can be placed on the cloud or other servers. The server 104 can obtain the running parameters of the workload and the read-write operation performance parameters of each key-value pair from the terminal 102. The running parameters include the read-write operation ratio, query statements, and data distribution. The server 104 determines the first space size of the space occupied by the key-value pair according to the read-write operation ratio and the read-write operation performance parameters. The server 104 determines the partitioning method of the storage area according to the semantic information of the query statement. Further, the server 104 determines the data density of the data under the candidate grouping range according to the data distribution, and determines the storage mode of the data table in the database when running the workload based on the partitioning method, candidate grouping range, data density, and the first space size.

[0029] Among them, the terminal 102 can be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. The portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0030] It can be understood that the server 104 provided in the embodiments of the present application can also be a service node in a blockchain system. The service nodes in the blockchain system form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP).

[0031] Cloud technology is the general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, be used on demand, and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites, and more portal websites. With the highly developed application of the Internet industry, in the future, each item may have its own identification mark and needs to be transmitted to the background system for logical processing. Data at different levels will be processed separately, and various industry data requires a powerful system support, which can only be achieved through cloud computing.

[0032] Cloud computing is a computing model that distributes computing tasks on a resource pool composed of a large number of computers, allowing various application systems to obtain computing power, storage space and information services as needed. The network that provides resources is called a "cloud". From the user's perspective, the resources in the "cloud" are infinitely scalable and can be obtained at any time, used on demand, expanded at any time, and paid for by use.

[0033] As a provider of basic cloud computing capabilities, a cloud computing resource pool (referred to as a cloud platform, generally referred to as an IaaS (Infrastructure as a Service) platform) will be established, and various types of virtual resources will be deployed in the resource pool for external customers to choose to use. The cloud computing resource pool mainly includes: computing devices (virtualized machines, including operating systems), storage devices, and network devices.

[0034] According to the logical function division, the PaaS (Platform as a Service) layer can be deployed on the IaaS (Infrastructure as a Service) layer, and the SaaS (Software as a Service) layer can be deployed on the PaaS layer. SaaS can also be deployed directly on IaaS. PaaS is a platform for software operation, such as databases, web containers, etc. SaaS is a variety of business software, such as web portals, SMS mass senders, etc. Generally speaking, SaaS and PaaS are upper layers relative to IaaS.

[0035] Cloud storage is a new concept that extends and develops from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of different types of storage devices (storage devices are also called storage nodes) in the network through application software or application interfaces to work together and provide external data storage and business access functions.

[0036] Currently, the storage method of the storage system is as follows: create a logical volume. When creating a logical volume, physical storage space is allocated for each logical volume. This physical storage space may be a certain storage device or the disks of several storage devices. The client stores data on a certain logical volume, that is, stores the data on the file system. The file system divides the data into many parts, and each part is an object. The object not only contains data but also contains additional information such as data identifiers (ID, ID entity). The file system writes each object into the physical storage space of the logical volume respectively, and the file system will record the storage location information of each object. Thus, when the client requests to access data, the file system can enable the client to access the data according to the storage location information of each object.

[0037] The process of the storage system allocating physical storage space for a logical volume is specifically as follows: according to the capacity estimation of the objects stored in the logical volume (this estimation usually has a large margin relative to the actual capacity of the objects to be stored) and the group of the redundant array of independent disks (RAID, Redundant Array of Independent Disk), the physical storage space is pre-divided into stripes. A logical volume can be understood as a stripe, thus allocating physical storage space for the logical volume.

[0038] With the research and progress of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields. For example, common ones include smart home, smart wearable devices, virtual assistants, smart speakers, smart marketing, driverless, autonomous driving, drones, robots, smart healthcare, smart customer service, vehicle networking, autonomous driving, intelligent transportation, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0039] In one embodiment, as Figure 2 shown, an adaptive method for a database storage mode is provided. Taking the server in Figure 1 as an example for illustration, it includes the following steps:

[0040] Step 202, obtain the running parameters of the workload and the read and write operation performance parameters of each key-value pair; the running parameters include the read and write operation ratio, query statements, and data distribution.

[0041] Among them, the workload refers to each workload running on top of the database. The database in this application can be a relational database, and the structures of each data table in the relational database are pre-configured. The structure of the database can be a distributed database. The workloads in this application can include different types of workloads. For example, OLTP (On-Line Transaction Processing) workloads have many write operations on data, and OLAP (On-Line Analytical Processing) operations generally have many range query operations. OLTP (On-Line Transaction Processing) is one of the ways to quickly respond to user operations. Its basic feature is that the user data received by the front end can be immediately transmitted to the computing center for processing and the processing results can be given within a short time. OLAP (On-Line Analytical Processing) is a software mechanism for performing high-speed multidimensional analysis on a large amount of data from data warehouses, data marts, or other unified centralized data storages.

[0042] The running parameters of the workload refer to the parameters when the workload runs normally on top of the database. For example, the running parameters can include the proportion of each I / O operation in the workload, all the query statements involved in the workload, and the data distribution situation corresponding to the workload.

[0043] The key-value (abbreviated as KV) refers to that in this application, data is organized, indexed, and stored in the form of key-value pairs. That is, in the storage engine, key-value pairs are used as the basic unit of storage.

[0044] The read-write operation performance parameters refer to the performance parameters corresponding to each I / O operation. For example, for the point query operation in the I / O operation, the latency time of the 16B-sized KV pair is 0.01ms, and the latency time of the 32B-sized KV pair is 0.005ms. The latency time is the performance parameter corresponding to the point query operation.

[0045] The read-write operation proportion refers to the proportion of each I / O operation in the workload. For example, the corresponding proportions of the single-point query operation, update operation, insert operation, delete operation, and range query operation of workload A within a preset time period.

[0046] The query statements refer to all the query statements involved in the workload. For example, the running parameters can include the proportion of the query statements and the meta-information involved in each query statement.

[0047] Data distribution refers to the data distribution in the workload, for example, the range of the primary key and the corresponding proportion. Specifically, the server can obtain the running parameters of a certain workload during normal operation in the database within a preset time period, and the storage mode in the database at this time can be any storage mode. Generally, the row storage mode can be selected as the benchmark mode for testing. In this step, the running parameters to be collected include the following three types: First, the proportion of each I / O operation in the workload, such as the corresponding proportions of single-point query operations, update operations, insert operations, delete operations, and range query operations; Second, the proportion of all query statements involved in the workload, and the meta-information corresponding to the columns involved; Third, the data distribution in the workload, such as the range of the primary key and the corresponding proportion. Further, the server can obtain the read and write operation performance parameters of key-value pairs occupying different space sizes from a pre-established performance model. Among them, the performance parameters can include at least one of latency, bandwidth, or IOPS (Input / Output Operations Per Second).

[0048] For example, assume that the current workload A includes 1000 point query operations, 10 full table scan operations involving 100,000 rows, 1000 update operations, 1000 insert operations, and 10 delete operations. The server obtains the running parameters of workload A within a preset time period, and the running parameters include 1000 point query operations, 10 full table scan operations involving 100,000 rows, 1000 update operations, 1000 insert operations, and 10 delete operations. Further, the server can obtain the performance of key-value pairs of each size under different I / O operations from a pre-established performance model. For example, the server obtains that under the point query operation, the latency of the 16B-sized key-value pair is 0.01ms, the latency of the 32B-sized key-value pair is 0.005ms, and the latency of the 48B-sized key-value pair is 0.009ms.

[0049] Step 204, determine the first space size of the space occupied by the key-value pair according to the read and write operation ratio and the read and write operation performance parameters.

[0050] Among them, the first space size refers to the value range of the space occupied by the key-value pair with the optimal overall performance of the selected workload. For example, assume that it is estimated that the overall performance of the current workload is the highest under the 16B-sized key-value pair. Therefore, the first space size of the space occupied by the key-value pair can be determined to be 16B.

[0051] Specifically, after the server obtains the running parameters of the workload and the read / write operation performance parameters of each key-value pair, the server can estimate the overall performance of the current workload under each KV pair size based on the read / write operation ratios included in the running parameters and the read / write operation performance parameters of each key-value pair. Furthermore, the server can select the KV pair size with the highest overall performance from them as the first space size occupied by the key-value pairs set by the system. Since the size of the space occupied by the key-value pairs has a great impact on the performance of the storage system, it is necessary to estimate the most suitable KV pair size according to the proportion of each I / O operation in the current workload. Generally, large-grained KV pairs are more suitable for processing full-table scan operations, and small-grained KV pairs are more suitable for processing single-point queries, updates, insertions, deletions, etc.

[0052] For example, assume that the current workload A includes 1,000 point query operations, 10 full-table scan operations involving 100,000 rows, 1,000 update operations, 0 insertion operations, and 0 deletion operations. The server obtains the running parameters of workload A within a preset time period, and the running parameters include 1,000 point query operations, 10 full-table scan operations involving 100,000 rows, 1,000 update operations, 0 insertion operations, and 0 deletion operations. Further, the server can obtain the performance of each size of KV pair under different I / O operations from the pre-established performance model. For example, the server obtains that in the point query operation, the latency time of the 16B-sized KV pair is 0.01 ms, the latency time of the 32B-sized KV pair is 0.005 ms, and the latency time of the 48B-sized KV pair is 0.009 ms. Then the server can estimate that in the current workload A, the time consumption t a1 of the 16B KV pair in the point query operation is 1,000 * 0.01 ms, and the time consumption t a2 of the 32B KV pair in the point query operation is 1,000 * 0.05 ms, and the time consumption t a3 of the 48B KV pair in the point query operation is 1,000 * 0.09 ms. By analogy, the server can estimate the time consumptions of the 16B KV pair in the full-table scan operation and the update operation as t b1 and t c1 respectively. After the server sums up the time consumptions of the 16B KV pair in the point query operation, the full-table scan operation, and the update operation, the total time consumption T1 = t a1 + t b1 + t c1 . Similarly, for KV pairs of other sizes, the server can also calculate an estimated total time consumption. Assume that after the server sums up the time consumptions of the 32B KV pair in the point query operation, the full-table scan operation, and the update operation, the total time consumption T2 = t a2 + t b2 + t c2The server sums up the time consumption of the 48B KV pairs in the point query operation, full table scan operation, and update operation, and the total time consumption T3 = t a3 +t b3 +t c3 Since T3 < T1 < T2, the server can select the KV pair size corresponding to the shortest estimated total time consumption T3, which is 48B, as the optimal KV pair size. That is, the server determines that the optimal value of the space occupied by the key-value pair is 48B according to the read-write operation ratio in the current load A and the read-write operation performance parameters of different KV pairs.

[0053] Step 206: Determine the partitioning method of the storage area according to the semantic information of the query statement.

[0054] The semantic information of the query statement refers to the semantic information obtained by parsing the syntax in the query statement. For example, for each query, its syntax can be parsed, and the column information involved can be extracted and saved from the projection clause and the WHERE condition clause.

[0055] The storage area refers to the area in the data table where data is stored. For example, if the structure of a data table is predefined as a 5-row * 8-column data table, then the 5-row * 8-column area in this data table is the storage area.

[0056] Partitioning means dividing the attributes of a certain data table in the database into several groups. The partitioning method refers to the method of partitioning a data table vertically, that is, the method of dividing a data table into several subsets vertically. For example, partitioning the columns of a data table to obtain the corresponding column partitioning scheme.

[0057] Specifically, after the server determines the first space size of the space occupied by the key-value pair according to the read-write operation ratio and the read-write operation performance parameters, the server can determine the partitioning method of the storage area according to the semantic information of the query statement in the running parameters. That is, the server can extract the information of relevant columns according to the semantic information of the query statement to deduce the candidate column partitioning scheme. In actual storage, after the server divides the attributes of a certain data table in the database into several groups according to the partitioning method, the server can store data according to these groups.

[0058] For example, as Figure 3 shown, it is a schematic diagram of the change in the data table structure. As Figure 3As shown in (1), partitioning divides a data table into several subsets in the vertical direction. First, the server can obtain all the query statements and their corresponding proportions involved in the data table. For each query statement, the server can parse its syntax, extract and save the column information involved. Further, the server can sort the query statements according to their proportions, and select the top n query statements with the highest proportions as target query statements. The server can start from the target query statement with the highest proportion, partition according to the column information involved in the target query statement, then extract the target query statement with the second highest proportion, and repeat the partitioning operation until all attributes have been partitioned.

[0059] Step 208: Determine the data density under the candidate grouping range according to the data distribution.

[0060] Among them, the candidate grouping range refers to a set of pre-set grouping ranges. For example, when pre-defining a set of grouping ranges, to facilitate the implementation of the grouping algorithm and reduce the computational complexity, powers of 2 can be used as the grouping ranges, that is, the candidate values of the pre-set candidate grouping range are {2 1 、2 2 and 2 3}. The grouping range refers to the grouping range of the primary key values.

[0061] The data density refers to the average data density under each candidate grouping range. For example, when the candidate grouping range is set to 8, the data table can be divided into 8 groups, the data density within each group is determined respectively, and then the average value P of the data densities within the 8 groups is used as the data density under this candidate grouping range, that is, the data density when the candidate grouping range is 8 is P. Among them, when determining the data density corresponding to each group, assume that the grouping range is set to 8, and the data with primary key values from 1 to 7 in the data table does not exist, and only the data with the primary key value of 0 is assigned to this group. Therefore, the positions of the primary key values from 1 to 7 are empty, and at this time, the data density within this group is 1 / 8.

[0062] Specifically, after the server determines the partitioning method of the storage area according to the semantic information of the query statement, the server can determine the data density under each candidate grouping range according to the data distribution and a pre-set set of candidate grouping ranges. That is, the server can calculate the data density under each candidate grouping range according to the data distribution situation during the operation of the current workload. Grouping aggregation can concentrate multiple rows of data into one KV pair, improving the efficiency of reading and writing data. And the grouping strategy and grouping range need to be determined according to the data distribution situation of the current workload. Such as Figure 3As shown in (2), grouping divides a data table into several subsets horizontally. The grouping strategy in this application may include hash grouping and sequential grouping. Hash grouping means that data with the same hash value is stored in the same data group through a certain hash function. Sequential grouping means that a series of continuously arriving data is stored in the same data group according to the arrival order of the data. It can be understood that the grouping strategy in this application includes, but is not limited to, hash grouping and sequential grouping, and can also be other custom grouping strategies.

[0063] For example, assume that the adopted grouping strategy is hash grouping, that is, the grouping function is set as a hash function, and a predefined set of candidate grouping ranges is {2 1 , 2 2 , 2 4}. Assume that the primary key of the first row of data in data table A is of int type and the value is 32, and the primary key value of the second row of data is 31. At this time, based on the hash function, that is, the division function, when the candidate grouping range is 2 4 , for the first row of data with key 32, its grouping serial number = 32 divided by 16 = 2, and it should be grouped into the 2nd group; for the second row of data with key 31, its grouping serial number = 31 divided by 16 = 1, and it should be grouped into the 1st group. Assume that the server divides data table A into 2 groups, namely the 1st group and the 2nd group. Then the server determines the data density within each group respectively, and then takes the average value P of the data densities within the 2 groups as the data density under the candidate grouping range of 2 4 , that is, the data density under the candidate grouping range of 16 is P

[0064] Step 210, based on the partitioning method, candidate grouping range, data density, and the first space size, determine the storage mode of the data table in the database when running the workload.

[0065] Among them, the data table refers to different data tables with pre-set data table structures in the database.

[0066] The storage mode refers to the storage method of the data table in the database. The storage modes in this application include at least one of row storage mode, column storage mode, cell storage mode, multi-row storage mode, column segment storage mode, column family storage mode, column family segment storage mode, or whole table storage mode.

[0067] Specifically, after the server determines the data density within the candidate grouping range based on the data distribution, the server can determine the optimal storage mode of the data tables in the database when running the workload based on the partitioning method, the candidate grouping range, the data density, and the first space size. That is, the server can estimate the second space size occupied by the key-value pairs within each candidate grouping range according to the partitioning method, each candidate grouping range, and the data density within each candidate grouping range. The server can compare the second space size occupied by the key-value pairs within each candidate grouping range with the first space size of the key-value pairs determined in step 204. When the difference between the second space size and the first space size meets the preset difference condition, the server obtains the partitioning method and the candidate grouping range corresponding to the second space size, and determines the optimal storage mode of the data tables in the database when running the current workload based on the partitioning method and the candidate grouping range corresponding to the second space size.

[0068] In addition, after the server determines the optimal storage mode of the data tables in the database when running the current workload, when the server is in an idle state, the server can re-encode and store the data in the data tables according to the encoding method corresponding to the optimal storage mode.

[0069] In the above adaptive method for the database storage mode, by obtaining the running parameters of the workload and the read / write operation performance parameters of each key-value pair; the running parameters include the read / write operation ratio, the query statement, and the data distribution; according to the read / write operation ratio and the read / write operation performance parameters, determine the first space size occupied by the key-value pair; according to the semantic information of the query statement, determine the partitioning method of the storage area; according to the data distribution, determine the data density within the candidate grouping range; based on the partitioning method, the candidate grouping range, the data density, and the first space size, determine the storage mode of the data tables in the database when running the workload. Since the read / write operation performance parameters of each key-value pair are obtained through pre-testing, therefore, based on the read / write operation ratio in the running parameters of different workloads and the read / write operation performance parameters of each key-value pair obtained through pre-testing, the optimal value range of the space size occupied by the key-value pair can be determined, so that the server can determine the optimal storage mode of the data tables in the database when running different workloads based on the partitioning method, the candidate grouping range, the data density, and the optimal value range of the space size occupied by the key-value pair, solving the problem in the traditional method that the optimal storage mode cannot be matched according to different loads, effectively improving the I / O efficiency of the system, and thus effectively improving the overall performance of the system.

[0070] In one embodiment, before obtaining the running parameters of the workload and the read / write operation performance parameters of each key-value pair, the method further includes:

[0071] Obtain the space size occupied by the key-value pair;

[0072] Determine the read and write operation performance parameters of key-value pairs under the occupied space size;

[0073] Establish a performance model of key-value pairs based on the read and write operation performance parameters;

[0074] The obtaining of the running parameters of the workload and the read and write operation performance parameters of each key-value pair includes:

[0075] Obtain the running parameters of the workload and the read and write operation performance parameters of key-value pairs based on the performance model.

[0076] Among them, the occupied space size of the key-value pair refers to the space size occupied by storing the key-value pair. For example, a set of pre-set occupied space sizes of key-value pairs is: {16B, 32B, 48B, 64B}.

[0077] The performance model refers to the read and write operation performance parameters corresponding to different key-value pairs. For example, in the point query operation, the latency time of a 16B-sized KV pair is 0.01ms, and the latency time of a 32B-sized KV pair is 0.005ms, that is, the read and write operation performance parameters corresponding to key-value pairs with different occupied space sizes are also different.

[0078] Specifically, before the server obtains the running parameters of the workload and the read and write operation performance parameters of each key-value pair, the server can pre-establish a performance model for different KV pair sizes. That is, the server can obtain a set of pre-set occupied space sizes of key-value pairs, and under the condition that the key-value pairs occupy different space sizes, determine the read and write operation performance parameters of key-value pairs with different space sizes, so that the server can establish a performance model of key-value pairs with different space sizes based on the read and write operation performance parameters, that is, the server establishes a mapping relationship between key-value pairs with different space sizes and read and write operation performance parameters, so that the server can directly obtain the read and write operation performance parameters corresponding to key-value pairs with different space sizes based on the performance model in the future.

[0079] For example, assume that the set of occupied space sizes of a pre-set group of key-value pairs is: {16B, 32B, 48B, 64B}. Then, through a series of prior experiments, the performance of each size of key-value pair under various typical I / O operations can be measured. The typical I / O operations include at least one of update operation, insert operation, delete operation, point query operation, or range query operation. That is, measure the performance parameters of the 16B-sized key-value pair in update operation, insert operation, delete operation, point query operation, and range query operation respectively, the performance parameters of the 32B-sized key-value pair in update operation, insert operation, delete operation, point query operation, and range query operation, the performance parameters of the 48B-sized key-value pair in update operation, insert operation, delete operation, point query operation, and range query operation, and the performance parameters of the 64B-sized key-value pair in update operation, insert operation, delete operation, point query operation, and range query operation. These test results will be used as the basis for determination in subsequent steps. Further, the server can establish a performance model for key-value pairs with space sizes of 16B, 32B, 48B, and 64B based on the performance parameters corresponding to the above various typical I / O operations. That is, the server establishes a mapping relationship between key-value pairs of different space sizes and the performance parameters of each read-write operation. These test results will be used as the basis for determination in subsequent steps.

[0080] In this embodiment, through a series of prior experiments, the performance of each size of key-value pair under various typical I / O operations can be measured, enabling the subsequent server to directly obtain the performance parameters of read-write operations corresponding to key-value pairs of different space sizes based on the performance model, and combined with the running parameters of different workloads, quickly and accurately estimate the optimal range of the space occupied by key-value pairs, improving the computational efficiency in the data processing process.

[0081] In one embodiment, the step of determining the first space size of the space occupied by the key-value pair according to the read-write operation ratio and the read-write operation performance parameters includes:

[0082] Determine the overall performance parameter of the workload when the key-value pair occupies the space according to the read-write operation ratio and the read-write operation performance parameters;

[0083] Based on the overall performance parameter, determine the first space size of the space occupied by the key-value pair.

[0084] Among them, the overall performance parameter refers to the overall performance parameter obtained by overall evaluating the performance parameters of each read-write operation. For example, the server can respectively obtain the time-consuming of the 16B key-value pair in full table scan operation and update operation as T1 and T2 from the performance model, and the total time-consuming T = T1 + T2 after summing T1 and T2. The total time-consuming can be used as the overall performance parameter corresponding to the 16B key-value pair.

[0085] The first space size refers to the optimal range of the space occupied by key-value pairs, that is, the first space size can be an interval range or an optimal target value. For example, in workload A, the first space size of the space occupied by key-value pairs is determined to be 16B, and in workload B, the first space size of the space occupied by key-value pairs is determined to be 16B - 17B.

[0086] Specifically, after the server obtains the running parameters of the workload and the read and write operation performance parameters of each key-value pair, the server can determine the overall performance parameters when determining the space occupied by key-value pairs under the current workload according to the read and write operation ratio in the running parameters and the read and write operation performance parameters of each key-value pair in the pre-established performance model, and determine the optimal range of the space occupied by key-value pairs under the current workload based on the overall performance parameters. Among them, the parameters used to characterize the overall performance parameters in this application may include at least one of: latency, bandwidth, or IOPS (Input / Output Operations Per Second). In this application, when determining the overall performance parameters corresponding to key-value pairs of different space sizes, the overall performance of the current workload under each KV pair size can be estimated by the method of weighted average, and then the KV pair size with the highest overall performance can be selected as the optimal value set by the system.

[0087] For example, assume that the current workload B includes 1,000 point query operations, 10 full table scan operations involving 100,000 rows, 1,000 update operations, 0 insert operations, and 0 delete operations. The server obtains the running parameters of workload B within a preset time period, and the running parameters include 1,000 point query operations, 10 full table scan operations involving 100,000 rows, 1,000 update operations, 0 insert operations, and 0 delete operations. Further, the server can obtain the performance parameters of KV pairs of each size under different I / O operations from the pre-established performance model. Assume that the server obtains that in the point query operation, the IOPS (i.e., the number of read and write operations per second) corresponding to the 16B-sized KV pair is I a1 = 850, the IOPS of the 32B-sized KV pair is I a2 = 750, and the IOPS of the 48B-sized KV pair is I a3 = 950. And so on, the server can estimate that the number of read and write operations per second of the 16B KV pair in the full table scan operation and the update operation is I b1 and I c1 respectively. The server sums up the IOPS of the 16B KV pair in the point query operation, the full table scan operation, and the update operation to obtain the total IOPS as I 总1 = I a1 + I b1 + I c1Similarly, for KV pairs of other sizes, the server can also calculate an estimated total IOPS. Assume that the server sums up the IOPS of 32B KV pairs in point query operations, full table scan operations, and update operations to obtain the total IOPS as I 总2 = I a2 + I b2 + I c2 The server sums up the IOPS of 48B KV pairs in point query operations, full table scan operations, and update operations to obtain the total IOPS as I 总3 = I a3 + I b3 + I c3 Since I 总1 < I 总3 < I 总2 , therefore, the server can select the KV pair size corresponding to the maximum value I 总2 of the estimated total IOPS, which is 32B, as the optimal KV pair size. That is, based on the read-write operation ratios in the current workload B and the read-write operation performance parameters of different KV pairs, the server determines that the optimal value range of the space occupied by the key-value pairs under the current workload B is 32B. Thus, the server can directly obtain the read-write operation performance parameters corresponding to key-value pairs of different space sizes based on the performance model, and combine with the operation parameters of different workloads to quickly and accurately estimate the optimal range of the space occupied by the key-value pairs, improving the calculation efficiency in the data processing process.

[0088] In one embodiment, the step of determining the first space size of the space occupied by the key-value pair according to the read-write operation ratio and the read-write operation performance parameter includes:

[0089] When the overall performance parameter is the overall time consumption, select the overall time consumption that meets the time consumption condition among the overall time consumptions as the target overall time consumption;

[0090] Take the space size of the key-value pair corresponding to the target overall time consumption as the first space size.

[0091] Among them, the time consumption condition refers to a preset condition for the overall time consumption. For example, the time consumption condition can be preset to be less than a certain threshold. For example, less than threshold 15. The time consumption condition can also be set to automatically select the minimum value among multiple overall time consumptions.

[0092] Specifically, after the server obtains the running parameters of the workload and the read and write operation performance parameters of each key-value pair, the server can determine the overall performance parameters when determining the space occupied by the key-value pairs under the current workload according to the read-write operation ratio in the running parameters and the read-write operation performance parameters of each key-value pair in the pre-established performance model. When the overall performance parameter is the overall time consumption, the server can select the overall time consumption that meets the time consumption condition from each overall time consumption as the target overall time consumption, and use the space size occupied by the key-value pair corresponding to the target overall time consumption as the optimal range of the space occupied by the key-value pair under the current workload.

[0093] For example, taking the overall performance parameter as the overall time consumption as an example for illustration. Suppose the current workload A includes 1,000 point query operations, 10 full table scan operations involving 100,000 rows, 1,000 update operations, 0 insert operations, and 0 delete operations. The server obtains the running parameters of workload A within a preset time period, and the running parameters include 1,000 point query operations, 10 full table scan operations involving 100,000 rows, 1,000 update operations, 0 insert operations, and 0 delete operations. Further, the server can obtain the performance of each size of KV pair under different I / O operations from the pre-established performance model. For example, the server obtains that in the point query operation, the latency time of the 16B-sized KV pair is 0.01 ms, the latency time of the 32B-sized KV pair is 0.005 ms, and the latency time of the 48B-sized KV pair is 0.009 ms. Then the server can estimate that in the current workload A, the time consumption t a1 of the 16B KV pair in the point query operation is 1,000 * 0.01 ms, and the time consumption t a2 of the 32B KV pair in the point query operation is 1,000 * 0.05 ms, and the time consumption t a3 of the 48B KV pair in the point query operation is 1,000 * 0.09 ms. And so on, the server can estimate the time consumption of the 16B KV pair in the full table scan operation and the update operation as t b1 and t c1 respectively. The server sums up the time consumption of the 16B KV pair in the point query operation, the full table scan operation, and the update operation, and the total time consumption T1 = t a1 + t b1 + t c1 . Similarly, for KV pairs of other sizes, the server can also calculate an estimated total time consumption. Suppose the server sums up the time consumption of the 32B KV pair in the point query operation, the full table scan operation, and the update operation, and the total time consumption T2 = t a2 + t b2 + t c2The server sums up the time taken for the 48B KV pairs in point query operations, full table scan operations, and update operations, and the total time taken is T3 = t a3 +t b3 +t c3 Since T3 < T1 < T2, the server can select T3 with the shortest estimated total time as the target overall time, and take the KV pair size corresponding to the target overall time T3, which is 48B, as the optimal KV pair size. That is, based on the read-write operation ratios in the current workload A and the read-write operation performance parameters of different KV pairs, the server determines that the optimal value range for the space occupied by the key-value pairs under the current workload A is 48B. This enables the server to directly obtain the read-write operation performance parameters corresponding to key-value pairs of different space sizes based on the performance model, and in combination with the operating parameters of different workloads, quickly and accurately estimate the optimal range of the space occupied by the key-value pairs, improving the computational efficiency in the data processing process.

[0094] In one embodiment, the steps of determining the partitioning method of the storage area according to the semantic information of the query statement include:

[0095] Obtain the query statements involved in the data table and the total number of query statements;

[0096] Determine the ratio between the number of each query statement and the total number of query statements;

[0097] Based on the ratio, determine the target query statement among the involved query statements;

[0098] Based on the column information in the target query statement, determine the partitioning method of the storage area.

[0099] Among them, the total number of query statements refers to the number of all query statements involved in the current workload. For example, assume that the current workload A includes 1000 point query operations, 10 full table scan operations involving 100,000 rows, 1000 update operations, 0 insert operations, and 0 delete operations. Then the total number of query statements involved in the current workload A is S = 1000 + 10 + 1000 = 2010.

[0100] Specifically, after the server determines the first space size of the space occupied by the key-value pairs according to the read-write operation ratio and the read-write operation performance parameters, the server can obtain the query statements involved in the data table and the total number of query statements, determine the ratio between the number of each query statement and the total number of query statements, and based on the ratio, determine the target query statement among the involved query statements. Further, the server can determine the partitioning method of the storage area based on the column information in the target query statement. That is, the server can extract the information of relevant columns according to the semantic information of the query statement to derive the candidate schemes for column partitioning. For example, Figure 3As shown in (1), column partitioning means dividing a data table into several subsets in the vertical direction.

[0101] For example, take data table A in the database as an example for illustration. Suppose there are 100 query statements related to data table A in workload A. Among them, there are 10 query statements for full table scan operations, 50 for the first type of query statements, and 40 for the second type of query statements. Then the server can obtain all the query statements and the total number of query statements involved in data table A, which is 100. For each query statement, the server can parse its syntax, extract and save the column information involved. The server can determine the ratio between the number of each query statement and the total number of query statements, that is, the proportion of query statements for full table scan operations is S1 = 10 / 100 = 0.1, the proportion of the first type of query statements is S2 = 50 / 100 = 0.5, and the proportion of the second type of query statements is S3 = 40 / 100 = 0.4. Then the server can sort each query statement according to the proportion of the above various query statements, and take the top n query statements with the highest proportion as the target query statements. Suppose the query statement A with the highest proportion is: query the data in columns 10 and 11. Then the server can start from the query statement A with the highest proportion and partition according to the relevant columns corresponding to the query statement A, that is, the server can aggregate columns 10 and 11 in data table A into a column group. And so on. Then the server takes out the query statement B with the second highest proportion and partitions according to the relevant columns corresponding to the query statement B, repeating the above column partitioning operation until all the attributes in data table A have been partitioned.

[0102] In addition, when the partitioning algorithm processes the same column in multiple queries, different strategies can be adopted. When the same column is involved in multiple query statements, this column is called a "conflicting column". The processing strategies for the conflicting column in this application can include: The first strategy is that the grouping of high-priority queries exclusively occupies the conflicting column. High-priority queries refer to query statements with a higher sorting position. Under this strategy, the more important query, that is, the query with a higher priority or a greater weight, adds the conflicting column to its own grouping, while other groupings do not contain this column. The second strategy is to merge these groupings. Under this strategy, the groupings containing the conflicting column will be merged into a larger grouping. Each of the two strategies has its own advantages. The first strategy is more inclined to conform to the characteristics of high-priority queries and is more suitable for the situation where the proportion gap between different queries is relatively large; the second strategy is more suitable for the situation where the proportion gap between different queries is not large. Thus, it is possible to determine the partitioning method of the data table in the vertical direction based on the proportion of query statements in different workload operation parameters, so as to solve the problem that the optimal storage mode cannot be matched according to different loads in the traditional method, effectively improving the I / O efficiency of the system, and thus effectively improving the overall performance of the system.

[0103] In one embodiment, the data density includes a first data density; and the step of determining the data density in the candidate grouping range according to the data distribution includes:

[0104] Determine the scope of candidate groups;

[0105] Determine data grouping based on the primary key value of the data table and the candidate grouping range;

[0106] determining a second data density within each data group according to the data distribution;

[0107] The mean value of the second data density is used as the first data density in the candidate grouping range.

[0108] Among them, the first data density refers to the average data density under each candidate grouping range. For example, assuming that the grouping range is 8, the data in data table A is divided into 4 data groups. The server can take the average data density of the 4 data groups and obtain the average data density S1 as the first data density corresponding to the grouping range of 8.

[0109] Specifically, after the server determines the partitioning method of the storage area according to the semantic information of the query statement, the server can determine the candidate grouping range. For example, the candidate grouping range can be a set of candidate grouping ranges pre-set in the system {2 1 , 2 2 and 2 3}. Further, the server can be based on the primary key value of the data table and the candidate grouping range {2 1 , 2 2 and 2 3}Determine the data grouping under each candidate grouping range, and determine the second data density within each data grouping according to the data distribution under the current workload, and the server uses the average of the second data density as the first data density under each candidate grouping range.

[0110] For example, suppose the candidate group range pre-set in the system is {2 1 , 2 2 and 2 3}, if there are two rows of data in data table A, after the server obtains the above candidate grouping range, the server can determine the data grouping based on the primary key value of data table A and the above candidate grouping range. That is, when the grouping range is 2, the primary key of the first row of data in data table A is of int type and the value is 32; the primary key value of the second row of data in data table A is 31. Assume that the preset grouping strategy in the system is to use a hash function. Taking the hash function as the division function as an example, that is, the grouping function is y = primary key value ÷ grouping range, where y represents the grouping serial number, that is, grouping serial number = primary key value ÷ grouping range. Then the server can determine the first row of data with a primary key value of 32 based on the above grouping function, and its grouping serial number = 32 ÷ 2 = 16, and it should be grouped into the 16th group; for the second row of data with a primary key value of 31, its grouping serial number = 31 ÷ 2 = 15, and it should be grouped into the 15th group. And so on, when the grouping range is 4, the server can determine the first row of data with a primary key value of 32 based on the above grouping function, and its grouping serial number = 32 ÷ 4 = 8, and it should be grouped into the 8th group; for the second row of data with a primary key value of 31, its grouping serial number = 31 ÷ 4 = 7, and it should be grouped into the 7th group. When the grouping range is 8, the server can determine the first row of data with a primary key value of 32 based on the above grouping function, and its grouping serial number = 32 ÷ 8 = 4, and it should be grouped into the 4th group; for the second row of data with a primary key value of 31, its grouping serial number = 31 ÷ 8 = 3, and it should be grouped into the 3rd group.

[0111] Further, for the candidate grouping range {2 1 , 2 2 and 2 3For each candidate grouping range in}, calculate the average data density under each candidate grouping range. Since the distribution of the primary key values is not necessarily strictly increasing, it is possible that the actual number of elements in a data grouping does not reach the grouping range. For example, taking the grouping range of 8 as an example, when the grouping range is 8, the data in data table A is divided into 4 data groupings. The server can separately calculate the data densities within group 1, group 2, group 3, and group 4. If the primary key values from 1 to 7 in data table A do not exist and only the primary key value of 0 is assigned to group 1, then the positions with primary key values from 1 to 7 in this grouping are empty. At this time, the server can calculate that the data density of this data grouping, i.e., group 1, is 1 / 8. The server successively calculates the data densities within each data grouping when the candidate grouping range is 8 and obtains that the data densities of group 1, group 2, group 3, and group 4 are S1 = 1 / 8, S2, S3, and S4 respectively. The server can calculate the average value S of the data densities of all the data groupings obtained from the statistics. The calculated average value S is the first data density S8 corresponding to the grouping range of 8, i.e., S8 = (S1 + S2 + S3 + S4) ÷ 4 = S. By analogy, the server can separately calculate the first data densities S4 and S2 corresponding to the grouping ranges of 4 and 2 respectively. That is, the server finally obtains the average data densities S8, S4, and S2 under each candidate grouping range. Thus, based on the data distribution in different workload operation parameters, the data density under each candidate grouping range can be determined, solving the problem in the traditional method that the optimal storage mode cannot be matched according to different loads, effectively improving the I / O efficiency of the system, and thus effectively improving the overall performance of the system.

[0112] It can be understood that the grouping strategy in this application includes, but is not limited to, the grouping method using a hash function, and can also be other grouping strategies. For example, the sequential grouping strategy. Sequential grouping means that a series of continuously arriving data is stored in the same data grouping according to the arrival order of the data. For the sequential grouping strategy, the corresponding data density can also be calculated using the above method.

[0113] In one embodiment, the steps of determining the storage mode of a data table in a database when running a workload based on the partitioning method, candidate grouping range, data density, and first space size include:

[0114] Based on the partitioning method, candidate grouping range, and the first data density under the candidate grouping range, determine the second space size occupied by the key-value pairs under the candidate grouping range;

[0115] When the difference between the second space size and the first space size meets the preset difference condition, determine the storage mode of the data table in the database when running the workload based on the partitioning method and candidate grouping range corresponding to the second space size.

[0116] Among them, the second space size refers to the space size occupied by key-value pairs calculated according to the partitioning method, candidate grouping range, and the first data density under the candidate grouping range. For example, numerically, grouping range * data density * partitioning size = space size occupied by KV pairs. The server can calculate the second space size occupied by key-value pairs under different candidate grouping ranges based on the above calculation method.

[0117] Specifically, after the server determines the data density under each candidate grouping range according to the data distribution, the server can determine the second space size occupied by key-value pairs under each candidate grouping range based on the partitioning method, candidate grouping range, and the first data density under the candidate grouping range; when the difference between the second space size and the first space size meets the preset difference condition, the server can determine the optimal storage mode of the data table in the database when running the workload based on the partitioning method and candidate grouping range corresponding to the second space size. As Figure 3 shown, the server will horizontally and vertically divide the data table according to the partitioning method and grouping range to obtain the final storage mode. That is, the server can calculate the size of the KV pairs based on the partitioning method determined in step 206, each candidate grouping range determined in step 208, and the data density. Numerically, when the difference between the second space size of the KV pairs obtained by multiplying a certain candidate grouping range A by the data density S under this candidate grouping range and the partitioning size B and the optimal first space size determined in step 204 meets the preset difference threshold, the server can determine the optimal storage mode of the data table in the database when running the current workload based on the partitioning method and candidate grouping range corresponding to the above second space size. In addition, after the server determines the optimal storage mode of the data table in the database when running the current workload, when it is in the idle state, the server can re-encode and store the data of the data table in the database according to the obtained optimal storage mode.

[0118] For example, assume that the candidate grouping ranges preset in the system are {2 1 、2 2 and 2 3}, there are two rows of data in data table A. After the server calculates the first data densities corresponding to the grouping ranges of 8, 4, and 2 as S8 = 1 / 8, S4 = 1 / 4, and S2 = 3 / 4 respectively, the server can determine the second space sizes occupied by the key-value pairs under each candidate grouping range based on the partitioning method, each candidate grouping range, and the first data density under each candidate grouping range. Assume that partitioning method A is to aggregate the first cell and the second cell in each column of data in data table A into a column group, and each cell stores 4 bytes of data, then the size of each column group is 8 bytes. Then the server can calculate the second space sizes occupied by the key-value pairs under each candidate grouping range based on the following calculation function: grouping range * data density * partitioning size = space size occupied by the KV pair. That is, when the grouping range is 8, the second space size L8 of the space occupied by the KV pair calculated by the server is L8 = grouping range * data density * partitioning size = 8 * 1 / 8 * 8 = 1; when the grouping range is 4, the second space size L4 of the space occupied by the KV pair calculated by the server is L4 = grouping range * data density * partitioning size = 4 * 1 / 4 * 8 = 8; when the grouping range is 2, the second space size L2 of the space occupied by the KV pair calculated by the server is L2 = grouping range * data density * partitioning size = 2 * 3 / 4 * 8 = 12. Assume that the server determines in step 204 that the first space size L of the space occupied by the key-value pair is 16B based on the read-write operation ratio in the current workload and the read-write operation performance parameters in the performance model, and the preset difference condition is less than the difference threshold 5. Then the server can calculate the absolute values of the differences between the second space sizes L8, L4, L2 occupied by the key-value pairs under each candidate grouping range and the first space size L respectively, and get |L8 - L| = |1 - 16| = 15, |L4 - L| = |8 - 16| = 8, |L2 - L| = |12 - 16| = 4. Since |L2 - L| = |12 - 16| = 4 is less than the difference threshold 5, that is, when the difference between the second space size L2 and the first space size L satisfies the preset difference condition, the server can determine the optimal storage mode of the data table in the database when running the current workload based on the partitioning method A and the candidate grouping range 2 corresponding to the second space size L2. That is, the server can determine that the optimal storage mode of the data table in the database when running the current workload is: the server horizontally partitions data table A according to the grouping range of 2, and vertically partitions it according to partitioning method A. That is, the server aggregates every two cell data in each column of data in data table A into a column group, and at the same time groups each row of data in data table A according to the grouping range of 2, then the schematic diagram of the optimal storage mode as shown in Figure 3 (3) can be obtained. Thus, the problem that the optimal storage mode cannot be matched according to different loads in the traditional method is solved, the I / O efficiency of the system is effectively improved, and thus the overall performance of the system is effectively improved.

[0119] In one embodiment, the method further includes:

[0120] When the workload is replaced with another workload, determining a target storage mode of the other workload in the database;

[0121] When in an idle state, converting the storage mode to the target storage mode.

[0122] Wherein, the other workload is used to distinguish the current workload. For example, when workload A is replaced with workload B, workload B is the other workload.

[0123] The target storage mode refers to the storage mode of data tables in the database when running other workloads. For example, the server can determine that the optimal storage mode of data tables in the database when running workload A is the multi-row storage mode according to the adaptive method of the database storage mode provided in this application. When workload A is replaced with workload B, the server can re-determine the optimal storage mode of data tables in the database when running workload B according to the adaptive method of the database storage mode provided in this application. It can be understood that the target storage mode can be the same as or different from the current storage mode.

[0124] Specifically, in actual operation, the server system can automatically switch the storage mode of replica nodes according to different workloads. When the adaptive module in the server senses a change in the load, it will re-evaluate the optimal storage mode and the corresponding conversion cost. During a relatively idle period of the system, the server can perform the work of converting the storage mode, converting the current storage mode to the target storage mode, that is, converting to the new optimal storage mode.

[0125] For example, assume that there are 3 service nodes in a distributed server cluster, namely service node A, service node B, and service node C. The storage mode corresponding to service node A is the row storage mode, the storage mode corresponding to service node B is the multi-row storage mode, and the storage mode corresponding to service node C is the column family segment storage mode. At a certain moment, workload A is replaced with workload B. When the adaptive model in the server determines that the storage model required by the replaced workload B is column storage, and there is no column storage mode among the 3 server nodes in the current service cluster, it is necessary to select one of the service nodes for storage mode conversion, which may be a conversion from row storage to column storage. In addition, when performing the conversion, the load of the service node needs to be considered, and the node with a smaller load is preferably selected. The storage modes in this application include at least one of row storage mode, column storage mode, cell storage mode, multi-row storage mode, column segment storage mode, column family storage mode, column family segment storage mode, or whole table storage mode. As Figure 4 shown, it is a schematic diagram of the conversion of the storage mode.Figure 4 Eight different storage modes are considered, corresponding to a total of 7 * 8 = 56 conversion methods. Thus, the problem that the optimal storage mode cannot be matched according to different loads in the traditional method is solved. On the basis of the diversity of storage modes, a multi-copy heterogeneous collaborative computing solution is proposed, which can store data in different modes on different replicas in the same consensus group and can select the most suitable storage node for different workloads. Compared with the traditional distributed system, the method in this embodiment can select storage nodes with the optimal storage mode for both AP and TP at the same time, so as to better adapt to the HTAP scenario. At the same time, the method in this embodiment takes into account the load balancing problem between different service nodes. Therefore, it can effectively improve the utilization rate of each node, thereby improving the overall performance of the system.

[0126] In one embodiment, after converting the storage mode to the target storage mode, the method further includes:

[0127] Re-encode the data in the data table according to the encoding method corresponding to the target storage mode to obtain the encoded data;

[0128] Store the encoded data.

[0129] Specifically, when the workload is changed to other workloads, the server determines the target storage mode of the other workloads in the database. When it is in the idle state, the server can convert the storage mode to the target storage mode. That is, the server can re-encode the data in the data table according to the encoding method corresponding to the target storage mode to obtain the encoded data, and store the encoded data.

[0130] For example, assume that the current storage mode is row storage and the target storage mode is column storage. Row storage means that data is stored as a key-value pair with one row in the underlying storage engine, and column storage means that data is stored as a key-value pair with one column in the underlying storage engine. Since the encoding method corresponding to the row storage mode is: using the table identifier tableA of data table A and the row identifier rowID of each row of data in data table A as keys, and storing the data in each row as values. When the server is idle, the server can re-encode the data in data table A according to the encoding method corresponding to the target storage mode, i.e., column storage. That is, the server reads the data in data table A, uses the table identifier tableA of data table A and the column identifier columnID of each column of data in data table A as keys, and stores the data in each column as values, to obtain the data composed of key-value pairs, and stores the data composed of the encoded key-value pairs. Thus, the storage mode of the replica can be automatically switched according to different workloads. When the adaptive module senses a change in the load, it will evaluate the optimal storage mode and the corresponding conversion cost, solving the problem in the traditional method that the optimal storage mode cannot be matched according to different loads, effectively improving the I / O efficiency of the system, and thus effectively improving the overall performance of the system.

[0131] In one embodiment, the method further includes:

[0132] When the storage mode is row storage mode and the target storage mode is cell storage mode, read the data in the data table;

[0133] Use the table identifier of the data table, the row identifier of the target row in the data table, and the column identifier of the target column in the data table as keys, and use the data in the cell where the target row and the target column intersect as values, to obtain the cell data composed of keys and values;

[0134] Store the cell data composed of keys and values.

[0135] Among them, cell storage means storing cells in a row or a column as key-value pairs in the underlying storage engine. Cell storage breaks the storage mode of the entire row or column, making the granularity of key-value units smaller, and showing better performance advantages in some workloads that only want to operate on a single cell. In the traditional row storage and column storage modes, when a certain cell needs to be updated, the server must first read out the data of the entire row or column, modify it and then write it back, which causes serious read-write amplification. In this case, using cell storage can well eliminate this write amplification. The cell storage mode in this application can be further divided into cell row storage and cell column storage. Cell row storage means organizing cells in row format to keep the overall storage continuous in rows; cell column storage means organizing cells in column format to keep the overall storage continuous in columns.

[0136] Specifically, due to the large number of conversion types, generally speaking, the conversion storage mode can be abstracted into two types of conversions, namely splitting and recombination. The application scenario of splitting is mainly to convert data from a larger key-value pair into several smaller key-value pairs. That is, when data is written in row storage mode, the encoding form of its key is: table identifier + row identifier, and the value is the data in each row. In some scenarios, when it is necessary to convert from row storage mode to cell storage mode, the server needs to re-encode the key-value pair, and set a corresponding cell identifier for each attribute. That is, when data is written in cell mode, the encoding form of its key is: table identifier + row identifier + column identifier, or table identifier + column identifier + row identifier, and the value is the data in each cell, thus splitting the data from row storage mode into cell storage mode.

[0137] In this embodiment, the conversion from single-row storage to cell storage is taken as an example for detailed description. If the current storage mode is row storage mode and the target storage mode is cell storage mode, the server can read the data in the data table according to the encoding method corresponding to the target storage mode, that is, cell storage mode, and use the table identifier of the data table, the row identifier of the target row in the data table, and the column identifier of the target column in the data table as the key, and use the data in the cell where the target row and the target column intersect as the value, to obtain the cell data composed of the key and the value, and store the cell data composed of the key and the value, then the conversion of the storage mode can be completed.

[0138] For example, assume that the structure of data table A is a 2*2 two-dimensional table. Then the server can read the data in data table A and use the table identifier tableA of data table A, the row identifier row1 of the target row (i.e., the first row) in data table A, and the column identifier column1 of the target column (i.e., the first column) in data table A as the key, that is, the key is: tableA+row1+column1. Use the data in the cell where the target row (i.e., the first row) and the target column (i.e., the second row) intersect as the value to obtain the first cell data composed of key-value, and store the first cell data composed of key-value. And so on, the server can encode all the data in data table A according to the above encoding method to obtain 4 key-value pairs composed of key-value for storage, and the 4 key-value pairs respectively correspond to 4 cell data. It can be understood that the above encoded key: tableA+row1+column1 means storing cells by row. If the encoded key is: tableA+column1+row1, it means storing cells by column. Thus, according to the different workloads, the storage mode of the replica can be automatically switched. When the adaptive module senses a change in the load, it will evaluate the optimal storage mode and the corresponding conversion cost, solving the problem in the traditional method that the optimal storage mode cannot be matched according to different loads, effectively improving the I / O efficiency of the system, and thus effectively improving the overall performance of the system.

[0139] In one embodiment, the method further includes:

[0140] When the storage mode is the cell storage mode and the target storage mode is the column storage mode, write the cells in the cell storage mode into the buffer.

[0141] When the number of cells in the buffer meets the number required by the column storage mode, use the table identifier of the data table and the column identifier of the target column in the data table as the key, and use the data in the target column as the value to obtain the column data composed of the key and the value.

[0142] Store the column data composed of the key and the value.

[0143] Specifically, when converting the storage mode, the main application scenarios for reorganization are to recombine several smaller KVs into a larger KV. For example: the conversion from single-row storage to multi-row storage, the conversion from cell storage to row storage, etc. In this embodiment, the conversion from cell storage to column storage is taken as an example for detailed description.

[0144] Suppose the data on a certain replica node starts to be stored in cell storage. As the load changes, it is necessary to convert the storage mode of this replica node to column storage. However, there are not enough cells required for column storage yet. The server of this replica node can temporarily write these cells into the buffer, and then perform re-encoding and conversion when the number of cells accumulated in the buffer meets the number required for conversion to column storage. That is, when the storage mode is cell storage mode and the target storage mode is column storage mode, the server can write the cells in cell storage mode into the buffer; when the number of cells in the buffer meets the number required for column storage mode, the server can use the table identifier of the data table and the column identifier of the target column in the data table as keys, and the data in the target column as values, to obtain the column data composed of keys and values, and store the column data composed of keys and values, thus completing the conversion of the storage mode. Therefore, according to different workloads, the storage mode of the replica can be automatically switched. When the adaptive module senses a change in the load, it will evaluate the optimal storage mode and the corresponding conversion cost, solving the problem in the traditional method that the optimal storage mode cannot be matched according to different loads, effectively improving the I / O efficiency of the system, and thus effectively improving the overall performance of the system.

[0145] In one embodiment, the method further includes:

[0146] When the storage mode is column segment storage mode, at least two cells in the target column are aggregated into a group;

[0147] Stored with the table identifier of the data table, the column identifier of the target column, and the group identifier of at least two cells as keys, and the data in at least two cells in the target column as values.

[0148] Among them, column segment storage means aggregating at least two cell data in a column into a KV pair for storage. This mechanism can be understood as the vertical partitioning of MySQL. The reason for introducing this mechanism is that queries may not need to return all the column data in an entire column. For certain specific workloads, the performance will be improved to a certain extent after introducing column segments.

[0149] Specifically, when the server determines that the storage mode of the data table in the database is column segment storage mode when running the current workload, when the server is in an idle state, the server can aggregate at least two cells in each column of the data table into a group, and use the table identifier of the data table, the column identifier of each column, and the group identifier of at least two cells as keys, and the data in at least two cells in each column as values for storage.

[0150] For example, as Figure 5 shown, it is a schematic diagram of column segment storage mode. Figure 5The first cell in the 6th column of the first row in

[0151] In one embodiment, the method further includes:

[0152] When the storage mode is the column family segment storage mode, at least two cells in each column in the first direction in the data table are aggregated into a column family;

[0153] At least two cells in each row in the second direction in the data table are aggregated into a data group;

[0154] Using the table identifier of the data table, the family identifier of the column family, and the group identifier of the data group as keys, and storing the data in at least two cells in each column and the data in at least two cells in each row as values.

[0155] Among them, column family storage means that several attributes in one row are aggregated into a key-value pair, and column family segment storage means that several attributes in multiple rows are aggregated into a key-value pair, thereby improving the throughput rate of the system in the AP scenario, that is, it can be understood as increasing the number of rows of column family storage.

[0156] The first direction may be the vertical direction, and the second direction may be the horizontal direction.

[0157] Specifically, as Figure 6 shown, it is a schematic diagram of the column family segment storage mode. Figure 6The first cell in the first row of the first column in [[ID=]], row1co1, represents the first row and the first column. The key is: tableID + CFID + groupID, and its meaning is: table identifier + column family identifier + grouping identifier. When the server determines that the storage mode of the data table in the database during the current workload operation is the column family segment storage mode, when the server is in the idle state, the server can aggregate the first two cells in each row in the horizontal direction in data table A into one data grouping, and the last four cells into one data grouping, thus obtaining 12 data groupings. Further, the server can aggregate the 6 data groupings on the left in the vertical direction in data table A into one column family, and the 6 data groupings on the right into one column family, and use the table identifier tableA of data table A, the family identifier column family 1 of the column family, and the grouping identifier group1-6 of the data grouping as the key of the first key-value pair, and store the data in the 12 cells on the left as the value of the first key-value pair; at the same time, use the table identifier tableA of data table A, the family identifier column family 2 of the column family, and the grouping identifier group7-12 of the data grouping as the key of the second key-value pair, and store the data in the 24 cells on the right as the value of the second key-value pair. Thus, according to the different data granularities and sorting methods, more storage modes can be expanded, enabling the system to obtain support for diverse storage modes.

[0158] In one embodiment, the method further includes:

[0159] When the storage mode is the whole table storage mode, store with the table identifier of the data table as the key and the data in the data table as the value.

[0160] Among them, whole table storage means storing the entire table as a single KV. Whole table storage significantly improves performance under the query condition of full table scan.

[0161] Specifically, when the server determines that the storage mode of the data table in the database during the current workload operation is the whole table storage mode, when the server is in the idle state, the server stores with the table identifier of the data table as the key and the data in the data table as the value.

[0162] For example, as Figure 7As shown, it is a schematic diagram of the whole-table storage mode. When the server is in an idle state, the server can use the table identifier tableA of the data table A as the key and store all the data in the data table A as the value. Thus, a key-value pair stored in the whole-table storage mode can be obtained, that is, the key of the key-value pair is: tableA, and the corresponding value is: all the data in the data table A. Therefore, according to the different data granularities and sorting methods, more storage modes can be extended, enabling the system to support diverse storage modes.

[0163] In one embodiment, the storage mode is the first storage mode; after determining the storage mode of the data table in the database when running the workload, the method further includes:

[0164] Receiving different types of query requests;

[0165] Determining the execution time, waiting time, and transmission time of the query request;

[0166] Based on the execution time, waiting time, transmission time, and a preset weight, determining the second storage mode corresponding to the query request; the first storage mode and the second storage mode are different storage modes;

[0167] Routing the query request to the replica node corresponding to the second storage mode, so that the replica node performs data query based on the query request.

[0168] Wherein, the first storage mode refers to the optimal storage mode of the data table in the database when running a certain workload, and the second storage mode refers to the optimal storage mode of the data table in the database corresponding to processing a certain query request.

[0169] Specifically, the query cost is an important indicator for formulating a query plan, which consists of three parts: the execution time T proc , the waiting time T wait and the transmission time T trans , that is, the total time T for processing the query request = T proc + T wait + T trans , and the storage mode of the data table in the database mainly affects the execution time T proc, therefore, in this embodiment, the impact of multi-copy nodes on query plan customization is considered. That is, after the server determines that the storage mode of the data table in the database is the multi-row storage mode when running the current workload, when the server receives different types of query requests, the server can determine the execution time, waiting time, and transmission time of each query request, and determine the second storage mode corresponding to each query request based on the execution time, waiting time, transmission time, and a preset weight. Among them, when the first storage mode and the second storage mode are different storage modes, the server can, based on a query strategy, for example, the query strategy includes giving priority to the replicas where the low-load nodes are located, the server can route the query request to a certain replica node with a relatively small load corresponding to the second storage mode, so that the replica node performs data query based on the query request.

[0170] For example, assume that there are 3 service nodes in a distributed service cluster, namely the A service node, the B service node, and the C service node. The storage mode corresponding to the A service node is the row storage mode, the storage mode corresponding to the B service node is the multi-row storage mode, and the storage mode corresponding to the C service node is the column family segment storage mode. When the B service node receives query request A, the server of the B service node determines that the execution time of query request A is T proc and the waiting time is T wait and the transmission time is T trans . The server can, based on the above execution time T proc , waiting time T wait , transmission time T trans and the preset weight, determine that the second storage mode corresponding to query request A is the row storage mode. Since the current storage mode of the B service node is the multi-row storage mode and the determined second storage mode corresponding to query request A is the row storage mode, therefore, the server of the B service node can route query request A to the replica node corresponding to the row storage mode, that is, the A service node, so that the A service node performs data query based on query request A.

[0171] In this embodiment, due to the use of the multi-copy heterogeneous scheme based on diversified storage, under the mixed load, different types of requests can better utilize the disk reading advantage on replicas with different storage modes and reduce the occupancy of network bandwidth.

[0172] In one embodiment, the determining the execution time, waiting time, and transmission time of the query request includes:

[0173] Obtain the search time, processing time, and tuple construction time of the query request, and determine the execution time of the query request based on the search time, read / write data volume processing time, and tuple construction time;

[0174] Obtain the request queue time, machine load latency time, and slave node data synchronization time of the query request, and determine the waiting time of the query request based on the request queue time, machine load latency time, and slave node data synchronization time;

[0175] Determine the transmission time of the query request.

[0176] Among them, the processing time refers to the time for reading or writing data.

[0177] Specifically, when the server receives different types of query requests, the server can obtain the search time, processing time, and tuple construction time of each query request, and determine the execution time of each query request based on the search time, read / write data volume processing time, and tuple construction time of each query request; further, the server can obtain the request queue time, machine load latency time, and slave node data synchronization time of each query request, and determine the waiting time of each query request based on the request queue time, machine load latency time, and slave node data synchronization time of each query request; the server can determine the transmission time of each query request. In this embodiment, due to the expansion of the underlying storage mode, query optimization can obtain a better configuration scheme, and the database can speed up the processing speed of user requests. Therefore, the user's satisfaction with the system will also increase.

[0178] In one embodiment, the storage mode is the first storage mode; the method further includes:

[0179] Receive a range query request;

[0180] Determine the total number of columns in the query table corresponding to the range query request, and determine the number of columns of the target columns required by the range query request; the target columns are the data columns in the query table;

[0181] Based on the number of columns, the total number of columns, and a preset threshold, determine the third storage mode corresponding to the range query request; the first storage mode and the third storage mode are different storage modes;

[0182] Route the range query request to the replica node corresponding to the third storage mode, so that the replica node performs data query based on the range query request.

[0183] Specifically, when the read / write operation request received by the server is a range query request, the server can determine the total number of columns in the query table corresponding to the range query request, and determine the number of columns of the target columns required for the range query request, where the target columns are the data columns in the query table; further, the server can determine the third storage mode corresponding to the range query request based on the number of columns, the total number of columns, and a preset threshold. When the first storage mode and the third storage mode are different storage modes, the server can route the range query request to the replica node corresponding to the third storage mode, so that the replica node performs data query based on the range query request.

[0184] For example, assume that after the server determines that the storage mode of the data table in the database is the multi-row storage mode when running the current workload, when the server receives the range query request A, the server can determine that the total number of columns in the query table A corresponding to the range query request A is Col_num, and determine that the number of columns of the target columns required for the range query request A is Col, where the target columns are the data columns in the query table A; further, the server can determine the third storage mode corresponding to the range query request A based on the number of columns Col, the total number of columns Col_num, and a preset threshold α. For example, when Col / Col_num < α, it can be determined that the column storage mode is the best storage mode; when Col / Col_num ≥ α, it indicates that the range query request A needs to read a relatively large number of columns. Generally, the row storage or multi-row storage mode can be adopted. For example, the range query request A needs to read 19 columns out of 20 columns. When reading in the column storage mode, 19 iterators need to be created, while only one iterator needs to be created in the row storage or multi-row storage, and the overall overhead will be greatly reduced.

[0185] Assume that the server determines that the third storage mode corresponding to the range query request A is the column storage mode. Since the storage mode of the data table in the current server database is the multi-row storage mode, the server can route the range query request A to the replica node corresponding to the multi-row storage mode, so that other replica nodes perform data query based on the range query request A. In this embodiment, the original system is supported with diversified storage. At the same time, due to the expansion of the underlying storage mode, better tuning parameter solutions can be obtained for the upper-layer query optimization. Therefore, developers can save more computing and storage overhead.

[0186] This application also provides an application scenario that applies the above adaptive method for the database storage mode. Specifically, the application of the adaptive method for the database storage mode in this application scenario is as follows:

[0187] When it is necessary to decide the most suitable storage structure according to the characteristics of different workloads, the above-mentioned adaptive method of the database storage mode can be adopted. That is, when running a certain workload, for each data table in the database, the above-mentioned adaptive method of the database storage mode can be used to decide the optimal storage scheme.

[0188] The method provided by the embodiments of this application can be applied to any scenario where the workload runs normally. Taking the optimization scenario of the distributed database storage mode based on the LSM-Tree KV storage engine as an example, the adaptive method of the database storage mode provided by the embodiments of this application will be described below.

[0189] In the traditional method, the row storage or column storage mode is usually adopted so that these storage modes can improve the read and write (I / O) performance of the database when processing queries under a certain workload. For example, OLTP-type workloads have many write operations on data, and OLAP operations generally have many range query operations. In the design of traditional databases, OLTP and OLAP databases are generally separated into independent products. Among them, OLTP databases basically adopt row storage as the storage mode, which is convenient for write operations and point query operations; while OLAP databases generally adopt column storage as the storage mode to accelerate the range query of continuous access, especially the scan operation on one or several columns. Obviously, the selection of storage modes for OLTP and OLAP is contradictory. Therefore, the selection and optimization of storage modes in HTAP (Hybrid Transactional / Analytical Processing) databases is a very challenging problem. HTAP is a new type of application program architecture that can well support transaction processing and analysis. Specifically, in the new distributed database system based on LSM-Tree KV, the problem of storage mode becomes more complex. Since the storage mode is limited to row storage and column storage, the adjustment of parameters mostly depends on the hardware configuration and query characteristics, and the underlying basic configuration such as the storage mode cannot be adjusted. Therefore, the finally selected storage mode is often not the best, which may cause relatively large storage overhead and even space waste, resulting in poor overall performance of the database system.

[0190] Therefore, to solve the above problems, the present application provides an optimization method for a distributed database storage mode based on the LSM-Tree KV storage engine to solve the problem of poor overall performance of the database system. In this method, according to the characteristics of the LSM-Tree KV storage engine, the design space of the database storage mode is expanded, and through the statistical information and cost evaluation function of the database, the optimal storage scheme for the current load can be determined to meet the different needs of different enterprises' businesses. As a result, the most suitable storage mode can be selected according to the user access load. Compared with the traditional method, the method in this embodiment can find the storage structure that best adapts to the query load within a larger range, and can store data in different modes on different replicas in the same consensus group, and can select the most suitable storage node for different workloads. Compared with the traditional distributed system, this method can select storage nodes with the optimal storage mode for both AP and TP type loads, so as to better adapt to the HTAP scenario. At the same time, the problem of load balancing between the master and slave nodes is also considered in this method, which can improve the utilization rate of each node, thereby enhancing the overall performance of the system.

[0191] On the product side, the method provided by the present application can adapt to the user's query request and select a suitable storage mode for different workloads. For traditional row-store and column-store databases, there are too few storage modes to choose from, and the optimal storage mode cannot be provided for different query requests, which may lead to waste of space and loss of performance. In this embodiment, a distributed database system based on a diversified storage mode is proposed, which expands the types of storage modes, improves the disk I / O efficiency, reduces space waste, ensures data consistency between different replica nodes, increases the concurrency performance of the system, and improves the performance of the system under mixed loads.

[0192] The features on the product side are as follows:

[0193] This product has storage and computing functions. At the storage end, data will be sharded, and each data shard has several replicas, and each replica provides an access function. The storage modes corresponding to different replicas of the data shard can be the same or different. At the computing layer, according to the analysis and statistics, the type of the request is judged, the replica with better performance is determined, and considering the load balance, the request is sent to the selected replica for processing. After the node where the replica is located executes the request, the data is returned to the computing layer, and then returned to the client through the computing layer. For upper-layer users, they will not perceive the diversification of the underlying storage mode, but the database product itself completes the management of the diversified storage mode. At the same time, for developers, they only need to modify the KV mapping rules and transaction processing procedures to directly be compatible with the changes of this method, so that the system can obtain support for the diversified storage mode.

[0194] On the technical side, such as Figure 8 shown, it is the overall architecture diagram of the system.

[0195] Section 1 - Overall Description.

[0196] This method is applicable to databases that adopt LSM-Tree KV as the underlying storage engine. The system can adopt the storage-computation separation architecture commonly used in distributed service systems to separate the storage layer and the computation layer. Such as Figure 8 shown, SQLENGINE in the computation layer is the SQL engine in the database system. The SQL engine is an important part of the database system, and its main responsibility is to generate an efficient execution plan for the SQL statements input by the application program under the current load scenario, and plays an important role in the efficient execution of SQL statements. Such as Figure 8 shown, in the storage layer, Consensus Group 1 includes multiple replica nodes, namely Storage Node 1, Storage Node 2, and Storage Node 3. The server forms a consensus group of multiple replicas through the consistency protocol layer so that the data stored in each storage node in the consensus group is the same, improving the fault tolerance of the system. And the storage modes of the replica nodes in each consensus group can be different or the same.

[0197] For example, assume that the current load is Load A. When the current Load A runs in the scenario of the database, the server of Storage Node 1 can decide the optimal storage solution for the current Load A based on the optimization method of the distributed database storage mode of the LSM-Tree KV storage engine proposed in the embodiments of the present application. Such as Figure 8The storage mode of the state machine in storage node 1 in consensus group 1 shown in the figure is row storage, that is, the storage mode in the database at this time is based on the row storage mode. The server can obtain the running parameters of the current load A, and the running parameters include the read-write operation ratio, query statements, and data distribution. The server can determine the first space size of the space occupied by the key-value pair according to the read-write operation ratio in the running parameters of the current load A and the read-write operation performance parameters of each key-value pair, and determine the partitioning method of the storage area according to the semantic information of the query statements in the running parameters of the current load A. Further, the server determines the data density under the candidate grouping range according to the data distribution in the running parameters of the current load A, and based on the partitioning method, candidate grouping range, data density, and the first space size, determines that the optimal storage mode of the data table in the database when running the current load A is the multi-row storage mode. Then, when the server is in the idle state, the server can convert the storage mode of the state machine in storage node 1 corresponding to the current load A from the row storage mode to the multi-row storage mode, that is, the server can re-encode the data in the data table according to the encoding method corresponding to the multi-row storage mode and store it, so that the data in the data table is stored in the multi-row storage mode. In addition, when the load A is replaced with another load B, the server can re-determine the target storage mode of the other load B in the database based on the optimization method of the distributed database storage mode of the LSM-Tree KV storage engine proposed in the embodiments of the present application, and convert the current storage mode to the target storage mode when the server is in the idle state.

[0198] Further, in the scenario where the current load A is running in the database, when the server determines that the optimal storage mode of the data table in the database when running the current load A is the multi-row storage mode, and after converting the storage mode of the state machine in storage node 1 corresponding to the current load A to the multi-row storage mode when the server is in the idle state, when the server receives different types of query requests, first, Figure 8 the SQL engine in the computing layer as shown in the figure interacts with the user's input, that is, the SQL engine in the computing layer is responsible for processing the SQL statements corresponding to different query requests initiated by the user, parsing the SQL statements and formulating a query plan. That is, in the computing layer, the server can determine the execution time, waiting time, and transmission time of each query request, and determine the optimal storage mode corresponding to each query request based on the execution time, waiting time, transmission time, and preset weights.

[0199] For example, Figure 8 there are 3 storage nodes in consensus group 1 as shown in the figure, namely storage node 1, storage node 2, and storage node 3. Figure 8The storage mode corresponding to storage node 1 shown is the row storage mode, the storage mode corresponding to storage node 2 is the column segment storage mode, and the storage mode corresponding to storage node 3 is the row storage mode. When the server of storage node 1 receives query request A, the server determines the execution time, waiting time, and transmission time of query request A, and based on the execution time, waiting time, transmission time, and a preset weight, determines that the optimal storage mode corresponding to query request A is the column segment storage mode. Since the current storage mode corresponding to storage node 1 is the row storage mode, the server can route query request A to the replica node corresponding to the column segment storage mode, that is, storage node 2, so that storage node 2 performs data query processing based on query request A. If the server determines that the optimal storage mode corresponding to query request A is the row storage mode, and since the current storage mode corresponding to storage node 1 is the row storage mode, there is no need to route it to other replica nodes for processing. If the current storage node 1 fails and becomes unavailable, the server can route query request A to other replica nodes corresponding to the row storage mode, that is, storage node 3, so that storage node 3 performs data query processing based on query request A.

[0200] In the embodiments of the present application, for the two major modules of the computing layer and the storage layer, their specific functions are as follows:

[0201] I. Computing layer:

[0202] Through this computing layer, the optimal storage mode can be determined, which can refer to step 210 in the above Figure 2 embodiment. Specifically, the specific functions of this computing layer are as follows:

[0203] 1. Interact with the user's input.

[0204] 2. Be responsible for processing the SQL statements of the business, parsing them and formulating a query plan, that is, the server can determine the execution time, waiting time, and transmission time of each query request according to different types of query requests received, and based on the execution time, waiting time, transmission time, and a preset weight, determine the optimal storage mode corresponding to each query request, and route each query request to the replica node corresponding to the optimal storage mode, so that the replica node performs data query based on each query request.

[0205] 3. For different workloads, determine the storage mode of the data tables in the database based on the partitioning method, candidate grouping range, data density, and the first space size, and route them to different nodes for execution.

[0206] 4. When the metadata changes, the computing layer reselects a suitable storage mode.

[0207] In this embodiment, the improvement in the computing layer mainly lies in the formulation of query plans. That is, the computing layer will analyze and process according to the current system status, select appropriate storage modes for different query requests, and the introduction of various storage modes will be specifically elaborated in Section 2; the situation of metadata changes will be discussed in Section 12.

[0208] II. Storage layer: Specifically, the storage layer is further divided into a transaction processing layer, a distributed consistency protocol layer, and a state machine. The functions of each module are as follows:

[0209] 1. Transaction processing layer:

[0210] The transaction processing layer is responsible for: 1) receiving transaction processing requests sent by the computing layer; 2) coordinating the processing of 2PC transactions.

[0211] In this embodiment, the main change in the transaction processing layer is that for different storage modes, the lock granularity will change during transaction processing, and the specific transaction processing process will be specifically elaborated in Section 5.

[0212] 2. Distributed consistency protocol layer:

[0213] The distributed consistency protocol layer is responsible for: 1) controlling data synchronization to ensure data consistency between each replica; 2) receiving and processing read and write requests from the computing layer and controlling access to data in the storage layer; 3) leader election.

[0214] The Raft protocol is adopted in the distributed consistency protocol layer, which is divided into two major modules: leader election (i.e., the main node election) and log replication. Raft is a strong leader consistency protocol. When writing data, it must pass through the main node and then be replicated to other secondary nodes to ensure data consistency between each replica. The working process of the Raft protocol will be elaborated in detail in Section 10. The storage modes used by each replica can be the same or different. In this method, heterogeneity between multiple replicas is achieved using Raft, and the specific solution will be elaborated in detail in Section 8. At the same time, each replica can switch the storage mode according to different loads during operation, and the specific content will be elaborated in detail in Section 4. In the actual operation of the distributed system, the configuration of replicas, the splitting and aggregation of data on replicas, etc. are all controlled by the multi-replica manager, and the management strategy of the multi-replica manager will be elaborated in detail in Section 9.

[0215] 3. State machine:

[0216] The state machine is responsible for: 1) receiving and processing access requests from the distributed consistency protocol layer; 2) managing multiple replicas of data to ensure data consistency between each replica.

[0217] In this embodiment, the state machine uses RocksDB, which is a storage engine based on LSM-Tree KV. It uses key-value pairs as the basic unit of storage, aggregates data into SSTables and stores them in a hierarchical structure. The introduction of LSM-Tree will be elaborated in detail in Section 11.

[0218] The focus in this embodiment is on the encoding part at the storage layer interface. Different encoding schemes are used to implement diverse storage structures, and an adaptive algorithm is used to formulate an optimal scheme for loads with different access characteristics. The specific implementation of the encoding scheme is elaborated in Section 3. The processing flow of the adaptive algorithm is elaborated in detail in Section 7.

[0219] Section 2 - Expansion of Storage Modes

[0220] In a storage engine based on LSM-Tree KV, the order outside the KV cannot be guaranteed, only the order inside the KV can be guaranteed not to be destroyed. Therefore, what data is mapped into a KV pair has a great impact on the overall read and write performance of the system. To ensure data order, the data is sorted before being mapped into KV, and then the resulting result is mapped as a whole into a KV, that is, a larger amount of data can be placed in a KV pair. Since the KV pair is the basic unit in the KV storage system, it can ensure that the internal data will be accessed sequentially. The method provided in this application can obtain more storage modes according to the data granularity and different sorting methods. Specifically as follows:

[0221] 1. Row storage:

[0222] The data is stored as a KV pair with one row in the underlying storage engine. Row storage, as the most intuitive and common storage method, has relatively balanced read and write performance. Row storage is very common in single-machine databases, such as MySQL and PostgreSQL both use row storage. Row storage is widely used in distributed databases such as Spanner and TiDB. When processing TP-type transactions, row storage often achieves better performance because the operation granularity of TP-type transactions is mainly based on rows. However, when processing AP-type queries, row storage will expose problems such as low effective data volume and low total bandwidth.

[0223] 2. Column storage:

[0224] Data is stored as a KV pair with one column in the underlying storage engine. Columnar storage, like row storage, is a relatively common storage mode. Columnar storage often has better performance for AP-type queries. When considering physical implementation, the size of KV cannot increase infinitely. In the case of large amounts of data, columnar storage in the storage mode will show huge performance disadvantages or induce other engineering problems. Therefore, in the method of this application, it is only recommended to use this storage mode for tables with a small amount of data, such as the Warehouse table in TPC-H.

[0225] 3. Cell storage:

[0226] Storing by cell means storing the cells in a row or a column as KV pairs in the underlying storage engine. Cell storage breaks the storage mode of the entire row and column, making the granularity of the KV unit smaller. In some workloads that only operate on a single cell, it will show relatively good performance advantages. In the traditional row and column storage modes, when a certain cell needs to be updated, the server must first read out the entire row and column data, modify it and then write it back, which causes serious read and write amplification. Storing by cell can well eliminate this write amplification. Among them, cell storage can be specifically divided into: 1) Cell row storage and 2) Cell column storage, as follows:

[0227] 1) Cell row storage: Cell row storage means organizing the cells in the format of rows, making the overall storage continuous on the row. As Figure 9 shown, it is a schematic diagram of cell row storage. Figure 9 The first cell in the first row of the first column in is row1co1, indicating the first row and the first column. Figure 9 The key in is: tableID + rowID + columnID, and the meaning it represents is: table identifier + row identifier + column identifier.

[0228] 2) Cell column storage: Cell column storage means organizing the cells in the format of columns, maintaining continuity on the column. As Figure 10 shown, it is a schematic diagram of cell column storage. Figure 10 The first cell in the first row of the first column in is row1co1, indicating the first row and the first column. Figure 10 The key in is: tableID + columnID + rowID, and the meaning it represents is: table identifier + column identifier + row identifier.

[0229] 4. Multiple-row storage:

[0230] Multi-line aggregation means storing multiple lines as a key-value pair in the underlying storage engine. Compared with storing a single line as a key-value pair, the granularity of key-value storage becomes larger. For scan operations, the amount of data read each time has a great impact on performance. Therefore, compared with single-line key-value pairs, the performance of scan operations will be greatly improved in the case of multi-line aggregation. At the same time, problems also occur during updates in multi-line aggregation. Updating a single line among multiple lines means performing a larger-granularity read-modify-write (RMW). To address this problem, the Merge scheme is proposed, that is, the line to be updated is first stored in the Memtable and then merged into the row group when Rocksdb performs compaction. As Figure 11 shown, it is a schematic diagram of multi-line storage. Figure 11 In the first cell of the first row in the first column, row1co1 represents the first row and the first column. Figure 11 In it, the key is: tableID + groupID, and its meaning is: table identifier + row grouping identifier.

[0231] 5. Column segment storage:

[0232] Column segment storage means aggregating at least two cells in a column into a key-value pair for storage. This mechanism can be understood as the vertical partitioning of MySQL. The reason for introducing this mechanism is that queries may not need to return all the column data in an entire column. For certain specific workloads, performance will be improved to a certain extent after introducing column segments. As Figure 5 shown, it is a schematic diagram of the column segment storage format.

[0233] 6. Column family storage:

[0234] Data is aggregated in the form of column families. A column family represents a group of attributes in a data table, that is, at least two cells in a row are aggregated into a key-value pair for storage. In actual use, columns that are frequently read / written together are often set as a column family to improve the throughput of the system. When data is aggregated by column family, optimization for specific queries can be achieved. Taking the column family as a key-value pair can effectively improve the performance of such queries. As Figure 12 shown, it is a schematic diagram of the column family storage format. Figure 12 In the first cell of the first row in the first column, row1co1 represents the first row and the first column. Figure 12 In it, the key is: tableID + CF ID + rowID, and its meaning is: table identifier + column family identifier + row identifier.

[0235] 7. Column family segment storage:

[0236] In column family storage, several attributes in a row are aggregated into a key-value pair. Column family segment storage means aggregating several attributes in multiple rows into a key-value pair, thereby improving the system throughput rate in AP scenarios, which can be understood as increasing the number of rows in column family storage. As Figure 6 shown, it is a schematic diagram of column family segment storage.

[0237] 8. Whole table storage: Whole table storage means storing the entire table as a key-value pair. Whole table storage significantly improves performance under the query condition of full table scan. As Figure 7 shown, it is a schematic diagram of whole table storage.

[0238] Section 3 - Encoding Scheme

[0239] This section details the encoding schemes for various storage modes. As Figure 13 shown, it is a schematic diagram of the encoding schemes for diverse storage modes.

[0240] 1. Row storage encoding scheme:

[0241] Row storage takes the data of a row in the data table as a key-value pair. In encoding, the table identifier, i.e., TableID, and the row identifier, i.e., RowID, are used as the key, and the data in this row is stored as the value. In implementation, the key can include other information such as timestamp, Hash value, etc.

[0242] 2. Column storage encoding scheme:

[0243] Column storage takes the data of a column in the data table as a key-value pair. In encoding, the table identifier, i.e., TableID, and the column identifier, i.e., ColumnID, are used as the key, and the data in this entire column is stored as the value. In implementation, the data volume of an entire column may be very large, so this encoding method may generate very large key-value pairs.

[0244] 3. Cell storage encoding scheme:

[0245] Cell storage takes a cell in the data table as a key-value pair. There are two encoding methods: row-based encoding and column-based encoding. For example, encoding TableID, RowID, and ColumnID in sequence as the key, and storing the data of this cell as the value. Such an encoding method will make the key-value pairs arranged in the SST according to the same RowID, which belongs to row-based storage. Similarly, if encoding TableID, ColumnID, and RowID in sequence as the key, making the key-value pairs arranged in the SST according to the same ColumnID, it belongs to column-based storage.

[0246] 4. Multi-row storage encoding scheme:

[0247] For multi-row storage, several rows in the data table are grouped together and stored in the same key-value pair. In terms of encoding, the TableID and the grouping identifier, i.e., GroupID, are used as the key, and the data in this grouping is stored as the value.

[0248] Among them, the GroupID is determined by the grouping method. Common grouping methods include index grouping and hash grouping. Index grouping groups according to the order of data insertion. Consecutive data tuples will be stored in the same grouping until the grouping is full of data. Hash grouping is based on a certain hash function, such as the integer division function. Based on the hash function, the RowID of the data tuple is mapped to the GroupID and stored in this grouping.

[0249] 5. Column family storage encoding scheme:

[0250] Column family storage aggregates several attributes in a row of the data table into a grouping. In terms of encoding, the TableID, the grouping identifier, i.e., GroupID, and the row identifier corresponding to this row, i.e., RowID, are used as the key, and several attributes in this row are stored as the value.

[0251] 6. Column family segment storage encoding scheme:

[0252] Column family segment storage aggregates an attribute group in a row of the data table into a column family, and then aggregates several consecutive column families. Among them, the way of attribute aggregation is the same as that of the column family, and the grouping method is the same as that of multi-row storage. That is, at least two cells in each row in the horizontal direction of the data table are aggregated into a data grouping, and at least two cells in each column in the vertical direction of the data table are aggregated into a column grouping; in terms of encoding, the TableID, the identifier of the column grouping, i.e., CFID, and the group identifier of the data grouping, i.e., GroupID, are used as the key, and the data in the column grouping and the data grouping are stored as the value.

[0253] 7. Column segment storage encoding scheme:

[0254] Column segment storage aggregates several consecutive cells in the same column of the data table into a key-value pair. In terms of encoding, the TableID, ColumnID, and the column grouping CFID are used as the key, and the data in the column grouping of this column is stored as the value. Column segment storage also needs to determine the grouping method. The grouping method is the same as that of multi-row storage. Compared with storing cells by column, when processing a full-table scan operation, since the granularity of the column segment is larger, the number of key-value pairs to be read can be reduced, thereby improving the overall performance. Compared with column storage, column storage aggregates all the data in a column into a key-value pair. It can be considered that column segment storage decomposes a key-value pair of column storage into several small key-value pairs. The size of column segment storage is between single-cell storage and column storage, and can balance the read and write performance.

[0255] 8. Whole-Table Storage Encoding Scheme:

[0256] In whole-table storage, the data of the entire data table is aggregated into a single KV pair. In terms of encoding, the TableID is used as the Key, and all the data in the data table is stored as the Value. Whole-table storage needs to determine the internal data arrangement of the KV pair, and the options include by row and by column. Whole-table storage will generate huge KV pairs, so in practice it is only suitable for relatively small tables.

[0257] Section 4 - Mutual Conversion of Storage Modes

[0258] This section mainly discusses the conversion issues between different storage modes.

[0259] During actual operation, the system can automatically switch the storage mode of replicas according to different workloads. When the adaptive module senses a change in the load, it will evaluate the optimal storage mode and the corresponding conversion cost. During relatively idle periods of the system, the work of storage mode conversion can be carried out to convert the current storage mode to the new optimal storage mode. For example, when the adaptive model determines that the storage model required by the current workload is column storage, and there is no column storage mode in the current node, then replica nodes need to be selected for storage mode conversion, which may be a conversion from row storage to column storage. When converting, the load of the nodes needs to be considered, and nodes with less load are preferably selected. The specific adaptive conversion method is introduced in detail in Section 7 Query Adaptation. By statistically analyzing the running parameters of the current load, combined with the structure and data distribution of the current data table, the optimal storage mode can be solved through the adaptive model. Considering that a total of 8 different storage modes are discussed in Section 3, there are a total of 7 * 8 = 56 conversion methods, specifically as Figure 4 shown.

[0260] Due to the large number of conversion types, generally speaking, we abstract them into two types of conversions, namely splitting and recombination, as follows:

[0261] 1. Splitting: The application scenario of splitting is mainly when data changes from a larger KV to several smaller KVs. Here, the conversion from single-row storage to cell storage is taken as an example for detailed explanation:

[0262] When data is written in row storage mode, the encoding form of its key is TableID + RowID. In some scenarios, the server needs to convert the row storage mode to cell storage mode, so the server needs to re-encode each attribute in the Value and set a corresponding CeilID (TableID + RowID + ColumnID) for each attribute, splitting the data from row storage mode into Ceil (cell storage) mode.

[0263] 2. Reorganization: The main application scenarios of reorganization are to recombine data from several smaller KVs into a larger KV. For example, the conversion from single-row storage to multi-row storage, the conversion from cell storage to row storage, etc. Here, the conversion from cell storage to column storage is taken as an example for detailed description:

[0264] Suppose the data on a certain replica starts to be stored in cell storage. As the load changes, we need to convert the storage mode of the replica to column storage. However, the cells required for column storage are not enough yet. We temporarily write these cells into the buffer. When the number of cells accumulated in the buffer is sufficient for the column storage conversion, we will perform re-encoding and conversion.

[0265] Section 5 - Impact of the New Storage Mode on Transactions

[0266] This section mainly introduces the impact of the new storage mode on transactions in the database:

[0267] The transaction concurrency control mechanisms of mainstream NewSQL databases (CockRoachDB, TiDB, TDSQL 3.0) are all bound to KV. Taking locks as an example, the traditional one-row-one-KV storage mode corresponds to row locks in traditional databases. When different storage modes are adopted, such as splitting into KVs at the cell granularity, it corresponds to smaller-granularity locks, while aggregating into KVs at the multi-row or column-segment granularity corresponds to larger-granularity locks. For the former, as long as it is ensured that all relevant KV locks are acquired for each read and write, the correctness of the corresponding transaction can be guaranteed; if the locks are acquired in the order defined by the fields corresponding to the cells for each write, the circular waiting in the deadlock condition can be broken, and no more deadlocks will be caused. That is, in the embodiments of the present application, in order to eliminate deadlocks, the method of ordering the locking units can be adopted. Deadlocks may occur when there is no ordering. For example, a row of data contains two fields, id and name. Transaction A and Transaction B access this row of data simultaneously. Transaction A locks id first, and Transaction B locks name first. At this time, Transaction A waits for Transaction B to release the lock on name, and Transaction B waits for Transaction A to release the lock on id, waiting for each other to release resources, that is, a deadlock occurs. After adopting the ordering strategy, this phenomenon can be avoided: it is stipulated that the lock on id must be acquired first, and then the lock on name can be acquired. At this time, after Transaction A locks id, Transaction B attempts to lock id and finds that it is already occupied, so it enters the waiting queue. At this time, no deadlock will occur.

[0268] Section 6 - Query Optimization

[0269] This section mainly introduces how the computing layer optimizes the query plan according to information such as metadata, query characteristics, and storage mode, and improves the query performance of the database system based on diverse storage modes.

[0270] The query cost is an important metric for formulating a query plan, which consists of three parts: the execution time T proc , the waiting time T wait , and the transmission time T trans , that is, T = T proc + T wait + T trans . Among these three parts, the execution time consists of the search time T search , the processing time T io , and the tuple construction time T cons , that is, T proc = T search + T io + T cons ; the waiting time consists of the request queue time T queue , the machine load delay T load , and the slave node data synchronization time T sync , that is, T wait = T queue + T load + T sync ; the transmission time is the time of network transmission. In this solution, the impact of multiple replicas on query plan formulation is mainly considered. Therefore, the influencing factors of the execution time are mainly considered. Here, several strategies are considered:

[0271] 1. For wide tables with few columns, the column store mode is preferred. The storage mode mainly affects the execution time of the query. Directly selecting the column store mode may reduce T search and T io , but since the result needs to be reorganized, it may cause an increase in T cons . The more columns required for the result, the larger T cons . For analytical query requests, count the number of columns Col required for the request and the total number of columns Col_num in the table. If Col / Col_num < α, where the threshold α can be adjusted according to the actual situation, then the column store mode is considered the best storage mode. If Col / Col_num ≥ α, it means that more columns need to be read. Generally, the row store or multi-row store storage mode can be adopted. For example, if 19 columns out of 20 columns need to be read, when reading in the column store storage mode, 19 iterators need to be created, while only one iterator needs to be created in the row store or multi-row store, and the overall overhead will be reduced.

[0272] 2. For Scan scans, multi-row storage is preferred. Since the scan operation needs to scan multiple rows of data or even the entire table in order, storing the data together in a multi-row aggregated storage mode can effectively reduce T search and T io , and the upper-layer result does not need to be reorganized, which has no impact on T consThere is no impact, so the multi-line storage mode can be used as the best storage mode for the Scan operation.

[0273] 3. For point queries, row storage is preferred. The query target of a point query is to obtain a certain piece of data that meets the expectations. Whether it is the column storage mode or multi-line aggregation, data needs to be split at the upper layer, which will cause an increase in T. cons With row storage, T can be guaranteed while increasing. search T io When T is not affected cons reaches the optimum. Therefore, it can be considered that row storage is the best storage mode for point queries.

[0274] 4. For update operations, cell storage is preferred. Update operations often update a certain attribute in a row. Selecting row storage requires reading the entire row of data, modifying it, and then writing the entire row, resulting in read-write amplification. Selecting column storage will also have the same impact, which has an adverse effect on T. io and T cons Logically speaking, cell storage can exactly match the granularity of update operations. Therefore, it can be considered that cell storage is the best storage mode for update operations.

[0275] At the same time, we also discuss other factors that affect the query plan as follows:

[0276] 5. The replica on the low-load node is preferred. The load of the node where the replica is located mainly affects T queue and T load in the waiting time of the query. The lower the node load, the smaller these two times. The load situation of the nodes where each replica is located can be obtained from the metadata manager, and the replica with the lowest load on the node is selected as the best replica.

[0277] 6. The replica close to the computing node is preferred. The physical distance from the node where the replica is located affects the transmission time T trans , and the closer the distance, the smaller T trans . Multiple replicas of data sharding may be scattered in different computer rooms or even different data centers. Selecting the node where the replica close to the computing node is located can reduce the network transmission overhead.

[0278] 7. The replica with the latest data synchronization status is preferred. Read requests need to wait for the node logs to be synchronized to the latest. The synchronization time T sync affects the waiting time T wait , so the newer the data synchronization status of the replica, the shorter the waiting time.

[0279] The above several strategies can work independently or be combined through weights, etc. The formula for the total cost of the query is as follows:

[0280] C = a1(t search+t io ) + 1 / a1(t cons ) + a2(t queue +t load ) + a3(T trans ) + a4(t sync )

[0281] Among them, a1, a2, a3, and a4 represent the weights of each strategy. Specifically, a1 represents the weight of the storage mode selection strategy, and a2 and a3 represent the weights of other strategies. The selection of strategies needs to consider the actual situation of the system. In actual deployment, due to factors such as node performance and network speed, T proc , T wait and T trans have different magnitudes. The weight settings should ensure that the part with a longer total time is accelerated as much as possible.

[0282] Since t search , t io , t cons are all related to the selection of the storage model. When the selection of the storage model is more beneficial to t search and t io , it will deviate from the storage model of the tuple, resulting in an increase in the tuple construction time (t cons ), and the two are inversely proportional. Therefore, in the total query cost, when we select a storage model that is more beneficial to the query request to shorten t search and t io , its weight a1 increases, which also means that the weight of t cons decreases, and the weights of the two are inversely proportional. Therefore, the weight of t cons is set to 1 / a1.

[0283] Section 7 - Storage Mode Adaptive Adjustment Method

[0284] This section will continue the discussion of multiple storage structures in Section 2 and further discuss how to decide the most suitable storage structure for different load characteristics in the actual system of the database. For each data table, we can use an adaptive storage mode to decide its optimal storage scheme.

[0285] As Figure 14 shown, it is the flowchart of the adaptive processing of the storage mode. The method of adaptive storage mode is divided into the following 6 steps:

[0286] 1. Establish a performance model for different KV pair sizes.

[0287] 2. Collect the running parameters of the load.

[0288] 3. Calculate the optimal range of the KV pair size according to the proportion of load I / O operations. This optimal range can be the first space size occupied by the key-value pairs in the above embodiments.

[0289] 4. Determine the candidate solutions for column partitioning (i.e., the partitioning methods in the above embodiments) according to the query statement information of the load.

[0290] 5. Determine the candidate grouping ranges and data densities according to the data distribution of the load.

[0291] 6. Considering comprehensively the partitioning methods, candidate grouping ranges, data densities, and the first space size obtained in steps 2, 3, and 4, decide on the optimal storage mode.

[0292] In step 1, through a series of prior experiments, measure the performance of KV pairs of various sizes under each typical I / O operation (such as update, insert, point query, delete, range query, etc.). These test results will be used as the basis for judgment in the subsequent steps.

[0293] In step 2, it is necessary to collect the parameters of the database when it is running under normal load for use as the basis for decision-making in the subsequent steps. The database at this time can be any storage mode. Generally, row storage can be selected as the benchmark mode for testing. In this step, the information to be collected includes the following three types. One is the proportion of each I / O operation in the load, such as the corresponding proportions of single-point query, update, insert, delete, and full-table scan. The second is the proportion of all query statements involved in the load, as well as the meta-information corresponding to the columns involved. The third is the data distribution situation in the load, such as the range of the primary key and the corresponding proportion.

[0294] In step 3, calculate the suitable KV pair size according to the proportion of each I / O operation. The KV pairs in the key-value pair storage system have a great impact on performance. Generally, large-grained KV pairs are more suitable for processing full-table scan operations, and small-grained KV pairs are more suitable for processing single-point query, update, insert, delete, etc. operations. According to the proportion of various I / O operations statistically obtained in step 2 and the performance of various I / O operations for each possible KV pair size in step 1, through the method of weighted average, the overall performance of the current workload under each KV pair size can be estimated, and then the KV pair size with the highest overall performance can be selected as the system setting.

[0295] For example, the current workload includes 1000 point queries, 10 full table scans involving 100,000 rows, and 1000 update operations, insertions, and deletions. Combined with the performance of KV pairs of various sizes collected in step 1 under different I / O operations, such as (in the point query operation, the delay of the 16B KV pair is 0.01ms, and the delay of the 32B is 0.005ms), it can be estimated that the time consumed by the 16B KV pair in the point query is 1000*0.01ms. Similarly, the time consumed by the 16B KV pair in the full table scan and update can be calculated, and the sum is the total time consumed. The weight of the weighted average mentioned in this paragraph is the total number of rows involved in each operation in this example. Similarly, for KV pairs of other sizes, an estimated total time consumption can also be calculated. Finally, the KV pair size with the shortest estimated total time consumption is the optimal KV pair size.

[0296] Step 4: Based on the semantic information of the query statement and the information of the relevant columns, candidate solutions for column partitioning are derived. Figure 3 As shown in (1), partitioning divides a data table into several subsets in the vertical direction. First, the server obtains all queries involved in the data table and the corresponding proportions. For each query, the server can parse its syntax, extract and save the column information involved from the projection clause and the WHERE condition clause. After that, the server can sort the query statements according to the proportion of various query statements faced by the application, and extract the top n query statements with the highest proportion. Next, the server can start with the query with the highest proportion and partition according to the relevant columns corresponding to the query. The server then extracts the query with the second highest proportion and repeats the partitioning operation until all attributes have been partitioned.

[0297] In addition, when the partitioning algorithm processes the same column in multiple queries, these columns are called "conflict columns" and different strategies can be adopted: the first strategy is that the high-priority query, that is, the top-ranked group, exclusively occupies the conflict column. Under this strategy, more important queries, that is, queries with higher priority or greater weight, add the conflict column to their own group, while other groups do not contain this column. The second strategy is to merge these groups. Under this strategy, the groups containing conflicting columns will be merged into a larger group. Both strategies have their own advantages. The first strategy tends to conform to the characteristics of high-priority queries and is more suitable for situations where the proportion difference between different queries is large; the second strategy is between multiple query features and is more suitable for situations where the proportion difference between different queries is not large.

[0298] Step 5: Calculate the candidate grouping range and data density based on the data distribution of the load. Grouping can concentrate multiple rows of data into one KV pair to improve the efficiency of reading and writing data. The grouping strategy and grouping size need to be determined based on the data distribution of the current load. Figure 3As shown in (1), grouping divides a data table into several subsets in the horizontal direction. The grouping strategies include hash grouping and sequential grouping. Hash grouping saves data with the same hash value in the same data group through a certain hash function. Sequential grouping stores a series of continuously arriving data in the same data group in the order of data arrival. A conventional and feasible way to determine the optimal grouping range and data density is as follows:

[0299] i) The grouping function uses a hash function. For example, divide the grouping range by the value of the primary key, and the result of the division is the grouping serial number. For non-integer data, if the grouping range is a power of 2, bitwise right shift can be used for grouping. For example, the primary key of a row of data is of int type and the value is 32. The primary key value of another row of data is 31. At this time, using the hash function, the grouping serial number = key divided by 16. For the data with key = 32, its grouping serial number = 32 divided by 16 = 2, and it should be grouped into the 2nd group; for the data with key = 31, its grouping serial number = 31 divided by 16 = 1, and it should be grouped into the 1st group.

[0300] ii) Pre-define a set of grouping ranges. The values of the grouping ranges can be any integer, but in practice, for the convenience of implementing the grouping algorithm and reducing the calculation amount, powers of 2 can be taken as the grouping ranges.

[0301] iii) For each grouping range, calculate the average data density. Since the distribution of the primary keys is not necessarily strictly increasing, there may be a situation where the actual number of elements in a certain group does not reach the grouping range. For example, the grouping range is set to 8, and the data with primary keys from 1 to 7 do not exist in the data table, and only the data with primary key 0 is grouped into this group. Therefore, these positions are empty, and the density of this group is 1 / 8 at this time. The average of the densities of all groups is the average data density under this grouping range.

[0302] The grouping range and average data density obtained in this step will be comprehensively decided in step 6. For the sequential grouping algorithm, the data density can also be calculated in the same way, and the corresponding parameters are submitted to step 6 for the final decision. And the size of the group is determined by the range and data distribution. For example, when performing hash grouping according to the primary key value, if some primary key values are not used, these positions will be empty. A set of grouping ranges can be pre-defined, scan the data distribution, and count its actual group size.

[0303] Step 6 comprehensively considers the above three steps and obtains the optimal storage mode. Such as Figure 3As shown in (3), the data table will be horizontally and vertically partitioned to obtain the final storage mode. According to the partitioning scheme given in step 4, the grouping range and data density obtained in step 5, the size of the KV pair can be calculated. Numerically, the grouping range multiplied by the data density multiplied by the partition size should be equal to the size of the KV pair. When the size of the KV pair corresponding to a certain storage mode is equal to or closest to the KV pair size given in step 3, this storage mode is the optimal storage mode.

[0304] Section 8 - Heterogeneity of Storage Modes

[0305] To meet the needs of different environments, on the basis of diverse storage modes, we apply a multi-copy heterogeneous collaborative computing scheme. After applying multi-copy heterogeneity, problems such as how to convert the storage mode between different copies, the consensus problem of heterogeneous copies, and the transaction problems brought by heterogeneous copies will be introduced, and the following will be discussed in detail:

[0306] 1. Conversion between heterogeneous storage modes: Suppose there are three members in our consensus group, storing copies in S1 mode, S2 mode, and S3 mode respectively. From the perspective of the consensus protocol layer, the primary node will still send the same log to each node. When the state machine in the node applies the log, it will convert the data format into the corresponding storage mode according to the log translation technology in the node, and then write it layer by layer.

[0307] 2. Consensus problem between heterogeneous storage modes: In the conversion scheme between heterogeneous copies, changes in consensus group members, data transfer between nodes, and snapshot persistence to the state machine can all be achieved through the original process of the consensus protocol, which can ensure the correctness of the system. Since the storage mode switch involves reading and writing of the state machine, during the storage mode switch process, the choice of the switching mode can be determined according to the node load: when the load of the node where the S1 mode copy is located is small and the available space of the node is large, choose to switch directly. When the node load is large or the space is insufficient, a re-selection should be made. In practice, when the leader node of the row store in the system is offline and has not been successfully reconnected, a copy of another storage mode can be elected as the leader node as a temporary option, and then converted back to the leader node after recovery.

[0308] 3. Transaction problems of heterogeneous storage modes:

[0309] There are mainly three problems in the implementation of heterogeneous multi-copies: providing a read interface from the replica (i.e., Follower Read), reading transactions from the replica, and concurrent control and log translation problems of write transactions on the primary replica.

[0310] Currently, most NewSQL databases (CockRoachDB and TiDB) adopt the Lease Read mechanism for replica reads. Since Spanner uses Paxos for replica synchronization, its implementation method is that the master node maintains the latest maximum timestamp that has been applied and synchronizes it to the slave nodes, that is, to ensure that the data before this timestamp has been replicated and is up-to-date. If the read request timestamp is greater than this timestamp, it means that there may still be data that has not been replicated, and then it is necessary to block and wait.

[0311] For the concurrent control of replica read transactions and write transactions on the primary replica, there are mainly two solutions: distributed locks and snapshot read safe timestamps.

[0312] The distributed lock mechanism enables both the master node and the slave nodes to access the locks related to the current existing transactions by replicating locks among multiple nodes, so as to perform cross-node concurrent control. Taking TiDB as an example, in the Percolator transaction model of TiDB, three types of data need to be persisted, including the Lock (lock) required for transaction concurrent control. As a distributed database, TiDB itself provides a multi-replica mechanism for data. Therefore, the Lock itself is also synchronized among multiple replicas as part of the multi-replica replicated data, thus naturally providing a distributed lock. However, due to the need to replicate a large amount of data across the network, the distributed lock itself is still a relatively costly concurrent control method.

[0313] The secure timestamp mechanism, namely the master-slave node synchronous secure timestamp mechanism, must satisfy that no new transactions can be submitted before it. Therefore, at this time, snapshot read requests with timestamps less than this timestamp can be safely satisfied. At the same time, only snapshot reads with timestamps less than this timestamp are allowed to be provided externally on the slave node. Taking Spanner as an example, in addition to the timestamp for ensuring data freshness mentioned above, a secure transaction conflict timestamp is also maintained simultaneously (in fact, the minimum value of the two timestamps is presented externally as one timestamp). One implementation is to record the Prepare timestamp and Commit timestamp of each distributed transaction. At this time, the minimum Prepare timestamp of all write transactions on the master node can be used as the secure timestamp, which can ensure that all transactions that could be submitted before this timestamp have been submitted. For ongoing transactions, even if they may be submitted, the commit timestamp must be after the secure timestamp. At this time, there must be no concurrent write transactions that will cause conflicts for read transactions with timestamps less than this timestamp on the secondary node. Although this eliminates the cost of maintaining distributed locks, the above timestamp itself represents the time when all data of a certain shard is securely readable as a whole. If there is no conflict between the data ranges of the read transaction on the secondary node and the concurrent write transactions on the master node at this time, the secondary node should actually only execute the read transaction after waiting for the multi-copy data synchronization to complete. However, due to the existence of the secure timestamp of the transaction at this time, the secondary node only knows that there are concurrent write transactions on the leader and does not know whether their data ranges conflict, so it can only choose to block the read transaction, thus "misfiring" some read transactions. Since it is difficult to confirm the write set of the transaction, it is also difficult to reduce the granularity of the secure timestamp. Although CockroachDB also has a mechanism similar to TiDB in its transaction mechanism, and the lock (Write Intent) is replicated as persistent data for multiple copies, thus naturally providing a distributed lock, it still chooses the implementation method of the secure timestamp.

[0314] In a Raft system with heterogeneous replicas, replicas with non-row storage can be elected as the Leader node, but it may reduce the overall system performance. More generally, replicas of any storage mode can be used as the Leader node, and at this time, the data of this storage mode is recorded in the Raft log. Some storage modes, such as row storage and column family storage, have better write performance and are more suitable for row storage; while some storage modes, such as column segment storage, have poor write performance and are not suitable for row storage. For example, when column segment storage is the Leader, all write operations will be converted into multiple write operations, resulting in a decrease in the overall system write performance. Therefore, in general, it is not recommended that replicas with poor write performance be elected as the Leader. In practice, when the Leader node with good write performance in the system goes offline and has not been successfully reconnected, replicas of other storage modes can be elected as the Leader node as a temporary option and then converted back to the Leader node after recovery.

[0315] Section 9 - Multi-copy Management Strategy

[0316] This section mainly introduces the management strategy of multi-copies in the storage layer.

[0317] 1. Replica Storage Mode Switching: For a certain replica in the storage layer, its storage mode can be dynamically adjusted during the actual operation of the system. The solution is as follows:

[0318] Create a new replica of this data shard on the current node. This replica is not included in the consensus group members (that is, it does not participate in all processes of consensus and only synchronizes data). Read the data of all S1-mode replicas, generate new data with mode S2 through a format converter, re-persist it to the state machine through the new temporary replica, and synchronize the logs that have not been persisted by the S1-mode replicas to the new replica. After synchronization is completed, destroy the S1-mode replicas and add the S2 mode as a new member to the consensus group.

[0319] 2. Replica Configuration Strategy: For each data shard, the replica configuration strategy is not fixed. During the operation of the system, the number of replicas in different storage modes is adjusted accordingly according to the change of data access mode (change of load). If the system receives more requests routed to the nodes where column-store replicas are located and the load of column-store replicas is relatively large during a certain period, it is judged that the current load on this data shard is of the column-store type. Select the replica in the storage type with fewer processed requests (representing low load of this type) during this period and switch it to a column-store replica.

[0320] 3. Data Splitting: When a data shard is too large or overheated, in order to balance the load, the system performs a re-splitting operation on the data shard, splitting one data shard into two data shards. The replica configuration of the newly generated data shards can either be selected to be the same as before the data splitting or be re-decided and configured through the policy manager.

[0321] For example: When splitting a row-store mode replica, only part of the data needs to be migrated respectively; when splitting a column-store mode replica, since it is costly to reorganize all rows in the column storage format, directly split from the row-store mode replica and convert it to the column storage format. After completion, destroy the split column-store replica.

[0322] 4. Error Handling: When one or several replicas of a data shard encounter errors and cannot work properly, new replicas should be created immediately to ensure that the total number of replicas matches the expected configuration. The storage mode of the newly launched replicas is determined by the current system policy configuration and does not necessarily need to be consistent with the state before the error occurred. For the newly launched replicas, if there are other replicas with the same storage mode, synchronize data from this replica (this replica needs to synchronize with the master node first). If not, use the solution for reselecting nodes in the mode conversion described above to synchronize data from the master node. The availability of the data shard and the correctness of the replica data are guaranteed by the consensus protocol during the error occurrence of the replica, the process of creating new replicas, and after the new replicas are created.

[0323] Section 10 - Raft Working Process

[0324] Raft is a consensus protocol that is easy to understand and build into an actual system. It decomposes the consensus algorithm into three major modules: leader election, log replication, and security. By first electing a leader, who is responsible for log management to achieve consistency, it greatly simplifies the states to be considered, enhances understandability, and reduces the difficulty of engineering practice.

[0325] 1. Node States

[0326] There are three types of states for nodes in a Raft cluster, namely Leader, Follower, and Candidate. There is only one Leader node, and all other nodes are Followers. Raft will first elect a Leader, and the Leader fully manages the replica logs. The Leader is responsible for receiving all client log entries and replicating them to other Follower nodes, and when it is safe, notifies each Follower to apply these log entries to their respective state machines. If the Leader fails, the Followers will re-elect a new Leader. The Follower itself does not send any requests and only responds to requests from the Leader and Candidate. If a Follower cannot receive messages, it will convert to a Candidate and initiate a Leader election. The Candidate that obtains the majority of votes in the cluster will become the Leader.

[0327] Raft divides time into terms of arbitrary length, each with a term number. Each time a leader election is held, a new term is started. If a leader or candidate discovers that its term number is out of date, it immediately switches to the follower state. If a node receives a request with an out-of-date term number, it rejects the request. The conversion relationships among followers, candidates, and leaders are shown in the figure:

[0328] 2. Leader Election

[0329] Raft triggers leader elections through a heartbeat mechanism. In the initial state, each node is a follower. Follower nodes maintain continuity with the leader node through the heartbeat mechanism. If no messages are received for a period of time, the follower assumes that there is no available leader in the system and initiates a leader election.

[0330] The follower node that initiates the election increments its local current term number, switches from the follower state to the candidate state, then votes for itself and sends vote requests to other follower nodes. Each follower node may receive multiple vote requests, but it can only cast one vote on a first-come, first-served basis, and the log information of the candidate node that receives the vote cannot be older than its own.

[0331] The candidate waits for vote responses from other follower nodes. The responses received may result in the following three outcomes:

[0332] The candidate node receives more than half of the votes, wins the election, and becomes the leader. At this time, it sends heartbeat messages to other nodes to maintain its leader status and prevent new elections from occurring.

[0333] The candidate node receives a message with a larger term number from another node, indicating that another node has been elected as the leader. At this time, the candidate switches to the follower state. If the candidate receives a smaller term number, the node rejects the request and continues to remain in the candidate state.

[0334] If none of the Candidate nodes obtains more than half of the votes, the election times out. At this time, each Candidate node starts a new election by incrementing the current term number. To prevent multiple election timeouts, Raft uses an algorithm with random election timeout times. When each Candidate starts an election, it sets a random election timeout time to prevent multiple Candidate nodes from timing out simultaneously and starting the next round of elections at the same time, thereby reducing the possibility of votes being split in the new election.

[0335] In a Raft system with heterogeneous replicas, different from traditional Raft, the storage mode of the Leader has a great impact on the performance of the overall system. Therefore, the weights of different storage modes in Leader election should also be different. For example, the row storage mode is more suitable for handling write operations, so its weight in the election should be higher; the column segment storage is not suitable for handling write operations, so its weight in the election should be lower.

[0336] 3. Log Replication

[0337] Each server node has a replicated state machine implemented based on replicated logs. If the initial states of the state machines are the same and the order of execution instructions obtained from the logs is the same, the final states of the state machines will also be the same.

[0338] After the Leader is elected, the system provides services externally. The Leader receives requests sent by clients, and each request contains a command that acts on the replicated state machine. The Leader encapsulates each request into a log entry, appends it to the end of the log, and at the same time sends these log entries to Followers in parallel in order. Each log entry contains a state machine command, the current term number when the Leader received the request, and in addition, the position index of the log entry in the log file. When the log entry is safely replicated to a majority of nodes, the log entry is called Committed. The Leader returns success to the client and notifies each node to apply the state machine commands in the log entry to the replicated state machine in the same order. At this time, the log is called Applied.

[0339] In a Raft system with heterogeneous replicas, the same log synchronization process as traditional Raft is adopted. After the logs are synchronized, the information in the logs needs to go through log replay to be transformed into tabular structure data. At this time, since the structures of the Leader and Followers may be different, the process of log replay is more complex than that of traditional Raft. When a Follower performs log replay, it can usually adopt a batch processing method to convert a batch of data into the corresponding storage mode at the same time to reduce the additional overhead caused by log replay.

[0340] It can be seen that Raft log replication is a Quorum process that can tolerate the failure of n / 2 - 1 replicas. The Leader will complete the logs for the lagging replicas in the background.

[0341] To ensure that the logs of Followers are consistent with those of the Leader, the Leader needs to find the index position where the Follower's log is consistent with its own, and let the Follower delete the log entries after that position, and then send the entries after that index position of its own to the Follower. In addition, the Leader maintains a NextIndex for each Follower, which represents the index of the next log entry that the Leader will send to that Follower. When a Leader starts its term, it initializes NextIndex to its latest log entry index + 1. If a Follower finds that its log index is inconsistent with the Leader's during the consistency check, it will reject the acceptance of that log entry. After receiving the response, the Leader decrements NextIndex and then retries until NextIndex reaches a position where the Leader's and Follower's logs are consistent. At this time, the log entries from the Leader will be successfully appended, and the logs of the Leader and Follower will reach consistency. Therefore, the Raft log replication mechanism has the following characteristics:

[0342] If two log entries in different logs have the same log index and term number, then these two logs store the same state machine commands.

[0343] If two log entries in different logs have the same log index and term number, then all the log entries before them are also the same.

[0344] To prevent committed logs from being overwritten, Raft requires that Candidates need to have all committed log entries. If a node is newly elected as the Leader, it can only commit the logs of the current Term that have been replicated to the majority of nodes. Logs of old term numbers cannot be directly committed by the current Leader even if they have been replicated to the majority of nodes, but need to be indirectly committed through log matching when the Leader commits the logs of the current term number.

[0345] Section 11 - LSM - Tree

[0346] The LSM-Tree, short for Log-Structured Merge-Tree, was proposed by Professor Patrick O’Neil in the paper "The Log-Structured Merge-Tree" in 1996. The name of the Log-Structured Merge-Tree is taken from the log-structured file system. The implementation of the LSM-tree is like a log file system. It is based on an immutable storage method, using buffering and append-only writes to implement sequential write operations, avoiding most of the random write operations in the mutable storage structure, reducing the impact of multiple random I / Os caused by write operations on performance, and improving the utilization rate of data space on the disk. It ensures the orderliness of disk data storage. The immutable disk storage structure is conducive to sequential writing. Data can be written to the disk at one time and exists in an append-only form on the disk, which also makes the immutable storage structure have a higher data density and avoids the generation of external fragmentation.

[0347] Since the file is immutable, write operations, insert operations, and update operations do not need to locate the data position in advance, greatly reducing the impact caused by random I / O and significantly improving the write performance and throughput. However, for large immutable files, duplication is allowed. As the appended data increases continuously, the number of disk-resident tables grows, and the problem of file duplication during reading needs to be solved. The LSM-Tree can be maintained by triggering merge operations.

[0348] In the LSM-Tree, data exists in the form of a Sorted String Table (SSTable). The SSTable usually consists of two components, namely the index file and the data file. The index file stores the key and its offset in the data file. The data file is composed of concatenated key-value pairs. Each SSTable consists of multiple pages. When querying a piece of data, instead of directly locating the page where the data is located like a B+ tree, it first locates the SSTable and then finds the page corresponding to the data according to the index file in the SSTable.

[0349] 1. LSM-Tree Structure

[0350] The overall architecture of the LSM-tree is shown in the figure, which includes a memory-resident component and a disk-resident component. When a write request is executed to write data, the operation is first recorded on the disk Commit Log for fault recovery, and then the record is written into the variable memory component (Memtable). When the Memtable reaches a certain threshold, it will be transformed into an immutable memory-resident component (Immemtable), and the data will be flushed to the disk in the background. For the disk-resident component, the written data is divided into multiple levels. The data input from the Immentable will first enter the Level 0 layer and generate the corresponding SSTable. When the Level 0 layer reaches a certain threshold, the SSTables in the Level 0 layer will be merged into the Level 1 layer in a certain way, and merged layer by layer downward in this way.

[0351] Memory-resident component: The memory-resident component consists of Memtable and Immemtable. Data is usually stored in an ordered skip list structure in the Memtable to ensure the orderliness of disk data. The Memtable is responsible for buffering data records and serving as the primary target for read and write operations. The Immemtable completes the operation of flushing data to the disk.

[0352] Disk-resident component: The disk-resident component is composed of WAL and SSTable. Since the Memtable exists in memory, to prevent the loss of data in memory that has not been written to the disk due to system failures, before writing data to the Memtable, the operation record needs to be written to the WAL to ensure the persistence of data records. The SSTable is constructed from the data records flushed from the Immetable to the disk. The SSTable is immutable and can only be used for read merging and deletion operations.

[0353] 2. Updates and Deletions in LSM-Tree

[0354] Since the LSM-Tree is based on an immutable storage structure, update operations cannot directly modify the original data. Instead, a new data entry can only be inserted with a timestamp as a marker. Therefore, it is not possible to clearly distinguish between insert operations and update operations in the LSM-Tree. Similarly, for deletion operations, they can be achieved by inserting a special deletion marker. This entry indicates that the data record corresponding to this key has been deleted.

[0355] 3. Searching in LSM-Tree

[0356] The access order for searching for a piece of data in the LSM-Tree is as follows:

[0357] Access the variable memory-resident component.

[0358] Access immutable memory-resident components.

[0359] Access disk-resident components, starting from Level 0 and accessing them in sequence. Since the data at lower levels is newer, when the data to be searched is found, it is immediately returned to obtain the latest value of the data.

[0360] In addition, there are some optimizations for read operations. Bloom filters are often used to determine whether an SSTable contains a specific key. The lower layer of a Bloom filter is a bitmap structure used to represent a set and can determine whether an element belongs to this set. Applying a Bloom filter can greatly reduce the number of disk accesses. However, it also has a certain false positive rate. Since its bitmap values are determined based on hash functions, there will eventually be hash collision problems for multiple values. When determining whether an element belongs to a certain set, an element that does not belong to this set may be misjudged as belonging to this set, that is, the Bloom filter has false positives. At the same time, determining an element in the set requires multiple bit values in the bitmap to be 1, which also fundamentally determines that the Bloom filter has no false negatives. That is to say, it may have the following two situations:

[0361] If the Bloom filter determines that an element is not in the set, then this element must not be in it.

[0362] If the Bloom filter determines that an element is in the set, then this element may or may not be in it.

[0363] 4. Compaction Strategies of LSM-Tree

[0364] In LSM-Tree, as the data in disk-resident tables continues to increase, duplicate data can be reduced through periodic compaction operations. There are two basic compaction strategies, namely Tiered Compaction and Leveled Compaction.

[0365] (1) Specific Forms of the Two Compaction Strategies

[0366] Tiered Compaction: The maximum number of SSTable files allowed in each layer is determined by the same threshold. As the Immemtable is continuously flushed into SSTables, when the number of SSTables in a certain layer reaches the threshold, all the SSTables in that layer are merged into a large new SSTable and placed in a higher layer. Its advantage is simple implementation, and its disadvantages are that the space amplification problem during compaction is relatively serious, and the SSTables in higher layers are larger, resulting in a more serious read amplification problem.

[0367] Leveled Compaction: The size of each SSTable is fixed, defaulting to 2MB, and the maximum number of SSTables in each layer is N times that of its first layer.

[0368] (2) Process of the merge strategy

[0369] When layer L0 is full, all SSTables in layer L0 are merged with all SSTables in layer L1, and duplicate key values are removed. Based on the SSTable size limit, multiple SSTable files will be merged and classified into layer L1.

[0370] When layers L1 - LN are full, one SSTable from the full layer is selected for merging with the next layer.

[0371] Its advantage is reduced space amplification, but the disadvantage is that it will cause serious write amplification problems during merging.

[0372] 5. Read amplification, write amplification, and space amplification

[0373] The LSM - Tree based on the immutable storage structure will have the problem of read amplification, just as it is inevitable for the B + - tree based on the mutable storage structure to have the problem of write amplification. However, different merge strategies in the LSM - Tree will bring new problems. In the distributed field, the famous CAP theorem proves that a distributed system can at most satisfy two of the three items: consistency, availability, and partition tolerance simultaneously. Similarly, Manos Athanassoulis et al. proposed the RUM conjecture in 2016, which states that for any data structure, at most two of read amplification, write amplification, and space amplification can be optimized simultaneously, and it is necessary to sacrifice the other item as the price. Generally speaking, the LSM - Tree based on the immutable storage structure will face the following three problems when storing data.

[0374] Read amplification. To retrieve data, it is necessary to search layer by layer, causing additional disk I / O operations. Especially during range queries, the phenomenon of read amplification is obvious.

[0375] Write amplification. During the merge process, it is constantly rewritten into new files, resulting in write amplification.

[0376] Space amplification. Since duplicates are allowed and expired data is not immediately cleared, this will lead to space amplification.

[0377] Since these two merging strategies are implemented differently, they will respectively lead to the problems of space amplification and write amplification.

[0378] The merging process of Tiered Compaction will cause the SSTable files in the higher levels to be very large. When performing the merge operation, in order to ensure fault tolerance, the original SSTable files will not be deleted until the merge operation is completed. This will cause the data volume to nearly double in the short term. After the merge operation is completed, the old data will be deleted, and the data volume will return to normal. Although it is only temporary, this is still a serious problem of space amplification.

[0379] Leveled Compaction fixes the size of the SSTable, but its merging strategy is not to merge all the SSTables in the whole layer with the next layer. Instead, it selects an SSTable with the same key value as the next layer for merging, reducing the problem of space amplification. However, since one SSTable may have duplicate key values with 10 SSTables in the next layer during the merge process, 10 SSTable files need to be rewritten at this time, which will lead to serious write amplification.

[0380] Section 12 - The Impact of Metadata Changes on the Storage Mode

[0381] In a diverse storage mode, changes in metadata may affect the selection of the storage mode, and may even convert the data already stored on the replicas into another storage mode. For the situation where metadata changes, this solution provides two solutions, which will be discussed in detail below. When metadata changes, some storage modes will be affected. For example, if a column is added to the table structure, the columnar storage mode only needs to add a corresponding column;

[0382] The row storage and multi-row storage modes are more affected, and the data in this column needs to be added to the row storage / multi-row storage data one by one; in the Cell storage mode, since the key of the Cell is composed of the combination of (tableid, rowid, cloumnid), when adding a column, we only need to split it according to the same rule, so the cell row storage and cell column storage will not cause additional overhead.

[0383] For column family storage and column family segment storage, there are two ways to change the storage mode as follows.

[0384] 1. Select the optimal storage mode: When the table structure changes, the data distribution under the current workload will change accordingly, which will affect the grouping range in the adaptive query. In this case, the current storage mode may no longer be the optimal solution for the current workload. Therefore, we need to re-perform query adaptation to select the optimal storage mode.

[0385] 2. Select the storage mode with the least change: In Solution 1, we reselected the storage mode. Although we found the optimal storage mode, the cost of this solution is too high. We need to re-convert the storage mode of all the data in the table. To avoid such a large cost, we can choose the storage mode with the least change. For example, when the storage mode of the replica is column family storage and a new column is added to the table structure, we can choose column family storage for the newly added column to adapt to the metadata change without affecting other data in the table.

[0386] For column segment storage, it may be necessary to aggregate this column into a certain column segment, which will cause some additional combination costs.

[0387] When inserting this column into the entire table, some necessary insertion costs will be generated, which will not have too much impact on the system performance.

[0388] The beneficial effects generated by the method provided in the embodiments of the present application include:

[0389] 1. The diverse storage modes proposed in this solution are hierarchical and logically clear. There is no need to modify any query plans or consensus protocol processing flows, and there is no need to perceive changes in the underlying storage engine. Only a small amount of code needs to be modified to change the KV mapping rules to directly be compatible with the changes in this solution, enabling the original system to obtain support for diverse storage. At the same time, due to the expansion of the underlying storage mode, better tuning schemes can be obtained for upper-layer query optimization. Therefore, developers can save more computing and storage costs.

[0390] 2. Due to the expansion of the underlying storage mode, better configuration schemes can be obtained for query optimization, and the database can speed up the processing of user requests. Therefore, the user satisfaction with the system will also increase.

[0391] 3. In this solution, a multi-copy heterogeneous scheme based on diverse storage is used, so that under mixed loads, different types of requests can better utilize the disk reading advantages on replicas with different storage modes and reduce the occupancy of network bandwidth.

[0392] 4. Adopting a computing and storage separation architecture facilitates changing the distribution of storage nodes;

[0393] 5. Generally speaking, this solution expands the storage mode, solves the problem that query requests cannot well match the storage mode, saves disk storage space, improves the I / O efficiency of the system, and ultimately achieves the problem of improving the overall performance of the system.

[0394] In addition, for the management method of diverse storage modes, Raft is used as the narrative object in this solution. Raft is a distributed protocol with a strong leader. However, a leaderless replication distributed protocol can also work. For example, Dynamo is a leaderless replication distributed protocol. Compared with Raft, which must have a leader and the log must first pass through the leader and then be synchronized among the followers, waiting for more than half of the members to approve and then commit, the central strategy of Dynamo is to ensure that the number of write replicas + the number of read replicas > the total number of replicas, so that it can be ensured that each read operation can obtain the latest data. On this distributed protocol, we can also support diverse storage modes to improve the performance of the system.

[0395] For the heterogeneous multi-replica solution of diverse storage, other distributed replica management protocols can also be used, such as Paxos, Multi-Paxos, and Quorum mechanisms. As long as the KV mapping rules are reasonably designed and reasonable log translation technology is used between different replicas, heterogeneous multi-replicas for diverse storage can be achieved.

[0396] In summary, this solution has realized the diversification of storage modes in the storage engine based on the LSM Tree and basically includes all KV storage modes. Even if new storage modes appear in the future, they will only be some modifications to the storage modes proposed in this application and will not exceed the scope discussed in this solution.

[0397] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are shown in sequence according to the arrows, these steps do not necessarily need to be executed in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages do not necessarily need to be executed at the same time, but can be executed at different times. The execution order of these steps or stages does not necessarily need to be sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0398] Based on the same inventive concept, an embodiment of the present application further provides an adaptive device for a database storage mode for implementing the adaptive method of the above-mentioned database storage mode. The solution provided by this device for solving problems is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the adaptive device for the database storage mode provided below can refer to the limitations on the adaptive method of the database storage mode in the above text, and will not be repeated here.

[0399] In one embodiment, as Figure 15 shown, an adaptive device for a database storage mode is provided, including: an acquisition module 1502 and a determination module 1504, where:

[0400] The acquisition module 1502 is configured to acquire the running parameters of the workload and the read / write operation performance parameters of each key-value pair; the running parameters include the read / write operation ratio, query statements, and data distribution.

[0401] The determination module 1504 is configured to determine a first space size of the space occupied by the key-value pair according to the read / write operation ratio and the read / write operation performance parameters; determine the partitioning method of the storage area according to the semantic information of the query statement; determine the data density of the data under the candidate grouping range according to the data distribution; and determine the storage mode of the data table in the database when running the workload based on the partitioning method, the candidate grouping range, the data density, and the first space size.

[0402] In one embodiment, the device further includes: the acquisition module is further configured to acquire the space size occupied by the key-value pair; the determination module is further configured to determine the read / write operation performance parameters of the key-value pair when occupying the space size; and a building module, configured to build a performance model of the key-value pair based on the read / write operation performance parameters.

[0403] In one embodiment, the determination module is further configured to determine the overall performance parameter of the workload when the key-value pair occupies the space size according to the read / write operation ratio and the read / write operation performance parameters; and determine the first space size of the space occupied by the key-value pair based on the overall performance parameter.

[0404] In one embodiment, the device further includes: a selection module, configured to select, when the overall performance parameter is the overall time consumption, the overall time consumption that satisfies the time consumption condition among the overall time consumptions as the target overall time consumption; and use the space size occupied by the key-value pair corresponding to the target overall time consumption as the first space size.

[0405] In one embodiment, the obtaining module is further configured to obtain the query statements and the total amount of query statements involved in the data table; the determining module is further configured to determine the ratio between the number of each query statement and the total amount of query statements; among the query statements involved, determine a target query statement based on the ratio; and determine the partitioning method of the storage area based on the column information in the target query statement.

[0406] In one embodiment, the determining module is further configured to determine a candidate grouping range; determine data grouping based on the primary key value of the data table and the candidate grouping range; determine the second data density within each data grouping according to the data distribution; and use the average value of the second data density as the first data density under the candidate grouping range.

[0407] In one embodiment, the determining module is further configured to determine the second space size occupied by the key-value pairs under the candidate grouping range based on the partitioning method, the candidate grouping range, and the first data density under the candidate grouping range; when the difference between the second space size and the first space size satisfies a preset difference condition, determine the storage mode of the data table in the database when running the workload based on the partitioning method and the candidate grouping range corresponding to the second space size.

[0408] In one embodiment, the apparatus further includes: the determining module is further configured to determine the target storage mode of the other workload in the database when the workload is changed to another workload; and a conversion module, configured to convert the storage mode to the target storage mode when in an idle state.

[0409] In one embodiment, the apparatus further includes: an encoding module, configured to re-encode the data in the data table according to the encoding method corresponding to the target storage mode to obtain encoded data; and a storage module, configured to store the encoded data.

[0410] In one embodiment, the apparatus further includes: a reading module, configured to read the data in the data table when the storage mode is a row storage mode and the target storage mode is a cell storage mode; the storage module is further configured to use the table identifier of the data table, the row identifier of the target row in the data table, and the column identifier of the target column in the data table as keys, and use the data in the cell where the target row and the target column intersect as values to obtain the cell data composed of the keys and the values; and store the cell data composed of the keys and the values.

[0411] In one embodiment, the apparatus further includes: a writing module, configured to write cells in the cell storage mode into a buffer when the storage mode is the cell storage mode and the target storage mode is the column storage mode; the storage module is further configured to, when the number of cells in the buffer meets the number required by the column storage mode, use the table identifier of the data table and the column identifier of the target column in the data table as keys, and use the data in the target column as values, to obtain column data composed of the keys and the values; and store the column data composed of the keys and the values.

[0412] In one embodiment, the apparatus further includes: an aggregation module, configured to aggregate at least two cells in a target column into a group when the storage mode is the column segment storage mode; the storage module is further configured to use the table identifier of the data table, the column identifier of the target column, and the group identifier of the at least two cells as keys, and use the data in the at least two cells in the target column as values for storage.

[0413] In one embodiment, the aggregation module is further configured to, when the storage mode is the column family segment storage mode, aggregate at least two cells in each column in the first direction in the data table into a column family; and aggregate at least two cells in each row in the second direction in the data table into a data group; the storage module is further configured to use the table identifier of the data table, the family identifier of the column family, and the group identifier of the data group as keys, and use the data in the at least two cells in each column and the data in the at least two cells in each row as values for storage.

[0414] In one embodiment, the storage module is further configured to, when the storage mode is the whole table storage mode, use the table identifier of the data table as a key and use the data in the data table as a value for storage.

[0415] In one embodiment, the apparatus further includes: a receiving module, configured to receive different types of query requests; a determining module is further configured to determine the execution time, waiting time, and transmission time of the query request; based on the execution time, the waiting time, the transmission time, and a preset weight, determine a second storage mode corresponding to the query request; the first storage mode and the second storage mode are different storage modes; a routing module, configured to route the query request to a replica node corresponding to the second storage mode, so that the replica node performs data query based on the query request.

[0416] In one embodiment, the obtaining module is further configured to obtain the lookup time, processing time, and tuple construction time of the query request, and the determining module is further configured to determine the execution time of the query request based on the lookup time, the read / write data volume processing time, and the tuple construction time; the obtaining module is further configured to obtain the request queue time, machine load delay time, and slave node data synchronization time of the query request, and the determining module is further configured to determine the waiting time of the query request based on the request queue time, the machine load delay time, and the slave node data synchronization time; and determine the transmission time of the query request.

[0417] In one embodiment, the receiving module is further configured to receive a range query request;

[0418] Determine the total number of columns in the query table corresponding to the range query request, and determine the number of columns of the target columns required by the range query request; the target columns are data columns in the query table; the determining module is further configured to determine the third storage mode corresponding to the range query request based on the number of columns, the total number of columns, and a preset threshold; the first storage mode and the third storage mode are different storage modes; the routing module is further configured to route the range query request to the replica node corresponding to the third storage mode, so that the replica node performs data query based on the range query request.

[0419] Each module in the above adaptive device for database storage mode can be implemented in whole or in part by software, hardware, and their combination. Each of the above modules can be embedded in the processor of the computer device in hardware form or be independent of it, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each of the above modules.

[0420] In one embodiment, a computer device is provided. The computer device can be a server, and its internal structure diagram can be as Figure 16As shown in the figure. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store adaptive data in the database storage mode. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements an adaptive method for a database storage mode.

[0421] Those skilled in the art can understand that Figure 16 the structure shown in the figure is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0422] In one embodiment, a computer device is further provided, including a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, the steps in the above method embodiments are implemented.

[0423] In one embodiment, a computer-readable storage medium is provided, storing a computer program. When the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0424] In one embodiment, a computer program product or a computer program is provided. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the steps in the above method embodiments.

[0425] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. And the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions.

[0426] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memories can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memories can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logics, data processing logics based on quantum computing, etc., without limitation.

[0427] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as within the scope described in this specification.

[0428] The above-described embodiments merely represent several implementation manners of the present application. Their descriptions are relatively specific and detailed, but they should not be construed as limiting the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. An adaptive method for a database storage mode, characterized in that, The method includes: Obtaining the running parameters of the workload and the read and write operation performance parameters of each key-value pair; the running parameters include the read-write operation ratio, query statements, and data distribution; wherein, the read and write operation performance parameters of each key-value pair refer to the read and write operation performance parameters corresponding to key-value pairs occupying different space sizes; Determining a first space size of the space occupied by the key-value pair according to the read-write operation ratio and the read and write operation performance parameters; wherein, the first space size is determined according to the overall performance parameters of the workload when the key-value pair occupies different space sizes; Obtaining the query statements and the total number of query statements involved in the data table in the database when running the workload; Determining the ratio between the number of each query statement and the total number of query statements; Based on the ratio, determining a target query statement among the involved query statements; Based on the column information in the target query statement, determining the partitioning method of the storage area; Determining the data density under the candidate grouping range according to the data distribution, the data density includes a first data density, including: determining the candidate grouping range; determining data grouping based on the primary key value of the data table and the candidate grouping range; determining the second data density within each data grouping according to the data distribution; taking the average value of the second data density as the first data density under the candidate grouping range; Based on the partitioning method, the candidate grouping range, the data density, and the first space size, determining the storage mode of the data table in the database when running the workload.

2. The method according to claim 1, characterized in that, Before obtaining the running parameters of the workload and the read and write operation performance parameters of each key-value pair, the method further includes: Obtaining the space size occupied by the key-value pair; Determining the read and write operation performance parameters of the key-value pair in the case of occupying the space size; Based on the read and write operation performance parameters, establishing a performance model of the key-value pair; The obtaining the running parameters of the workload and the read and write operation performance parameters of each key-value pair includes: Obtaining the running parameters of the workload, and obtaining the read and write operation performance parameters of the key-value pair based on the performance model.

3. The method according to claim 2, characterized in that, The determining a first space size of the space occupied by the key-value pair according to the read-write operation ratio and the read and write operation performance parameters includes: Determining the overall performance parameters of the workload when the key-value pair occupies the space size according to the read-write operation ratio and the read and write operation performance parameters; Based on the overall performance parameters, determining a first space size of the space occupied by the key-value pair.

4. The method according to claim 3, characterized in that, The determining a first space size of the space occupied by the key-value pair according to the read-write operation ratio and the read and write operation performance parameters includes: When the overall performance parameter is the overall time consumption, selecting the overall time consumption that satisfies the time consumption condition among each overall time consumption as the target overall time consumption; Taking the space size occupied by the key-value pair corresponding to the target overall time consumption as the first space size.

5. The method according to claim 1, characterized in that, Determining the storage mode of data tables in the database when running the workload based on the partitioning method, the candidate grouping range, the data density, and the first space size includes: Based on the partitioning method, the candidate grouping range, and the first data density under the candidate grouping range, determining the second space size occupied by the key-value pairs under the candidate grouping range; When the difference between the second space size and the first space size satisfies a preset difference condition, determining the storage mode of data tables in the database when running the workload based on the partitioning method and the candidate grouping range corresponding to the second space size.

6. The method according to claim 1, characterized in that, The method further includes: When the workload is changed to another workload, determining the target storage mode of the other workload in the database; When in an idle state, converting the storage mode to the target storage mode.

7. The method according to claim 6, characterized in that, After converting the storage mode to the target storage mode, the method further includes: Re-encoding the data in the data table according to the encoding method corresponding to the target storage mode to obtain encoded data; Storing the encoded data.

8. The method according to claim 7, characterized in that, The method further includes: When the storage mode is a row storage mode and the target storage mode is a cell storage mode, reading the data in the data table; Using the table identifier of the data table, the row identifier of the target row in the data table, and the column identifier of the target column in the data table as keys, and using the data in the cell where the target row and the target column intersect as values to obtain cell data composed of the keys and the values; Storing the cell data composed of the keys and the values.

9. The method according to claim 7, wherein The method further includes: When the storage mode is a cell storage mode and the target storage mode is a column storage mode, writing the cells in the cell storage mode into a buffer; When the number of cells in the buffer meets the number required for the column storage mode, using the table identifier of the data table and the column identifier of the target column in the data table as keys, and using the data in the target column as values to obtain column data composed of the keys and the values; Storing the column data composed of the keys and the values.

10. The method according to claim 1, wherein The method further includes: When the storage mode is a column segment storage mode, aggregating at least two cells in a target column into a group; Storing with the table identifier of the data table, the column identifier of the target column, and the group identifier of the at least two cells as keys, and the data in at least two cells in the target column as values.

11. The method according to claim 1, wherein The method further includes: When the storage mode is a column family segment storage mode, aggregating at least two cells in each column in the first direction in the data table into a column family; Aggregating at least two cells in each row in the second direction in the data table into a data group; Storing with the table identifier of the data table, the family identifier of the column family, and the group identifier of the data group as keys, and the data in at least two cells in each column and the data in at least two cells in each row as values.

12. The method according to claim 1, wherein The method further includes: When the storage mode is the whole-table storage mode, store with the table identifier of the data table as the key and the data in the data table as the value.

13. The method according to claim 1, wherein The storage mode is the first storage mode; after determining the storage mode of the data table in the database when running the workload, the method further includes: Receiving query requests of different types; Determining the execution time, waiting time, and transmission time of the query request; Based on the execution time, the waiting time, the transmission time, and a preset weight, determining the second storage mode corresponding to the query request; the first storage mode and the second storage mode are different storage modes; Routing the query request to the replica node corresponding to the second storage mode, so that the replica node performs data query based on the query request.

14. The method according to claim 13, wherein The determining the execution time, waiting time, and transmission time of the query request includes: Obtaining the lookup time, processing time, and tuple construction time of the query request, and determining the execution time of the query request based on the lookup time, the processing time, and the tuple construction time; Obtaining the request queue time, machine load delay time, and slave node data synchronization time of the query request, and determining the waiting time of the query request based on the request queue time, the machine load delay time, and the slave node data synchronization time; Determining the transmission time of the query request.

15. The method according to claim 1, wherein The storage mode is the first storage mode; the method further includes: Receiving a range query request; Determining the total number of columns in the query table corresponding to the range query request, and determining the number of columns of the target columns required by the range query request; the target columns are the data columns in the query table; Based on the number of columns, the total number of columns, and a preset threshold, determining the third storage mode corresponding to the range query request; the first storage mode and the third storage mode are different storage modes; Routing the range query request to the replica node corresponding to the third storage mode, so that the replica node performs data query based on the range query request.

16. An adaptive device for a database storage mode, wherein The device includes: An acquisition module, configured to acquire the operation parameters of the workload and the read / write operation performance parameters of each key-value pair; the operation parameters include the read / write operation ratio, query statements, and data distribution; wherein, the read / write operation performance parameters of each key-value pair refer to the read / write operation performance parameters corresponding to the key-value pairs occupying different space sizes; A determination module, configured to determine a first space size of the space occupied by the key-value pair according to the read-write operation ratio and the read-write operation performance parameter; wherein, the first space size is determined according to the overall performance parameter of the workload when the key-value pair occupies different space sizes; obtain the query statements and the total number of query statements involved in the data table in the database when running the workload; determine the ratio between the number of each query statement and the total number of query statements; in the involved query statements, determine a target query statement based on the ratio; based on the column information in the target query statement, determine the partitioning method of the storage area; determine the data density under the candidate grouping range according to the data distribution, the data density includes a first data density, including: determining a candidate grouping range; determining data grouping based on the primary key value of the data table and the candidate grouping range; determining a second data density within each data grouping according to the data distribution; taking the mean value of the second data density as the first data density under the candidate grouping range; based on the partitioning method, the candidate grouping range, the data density and the first space size, determine the storage mode of the data table in the database when running the workload.

17. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 15 are implemented.

18. A computer-readable storage medium, having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 15 are implemented.

Images