Method, apparatus and device for storing multiple copies of data, and medium

Abstract

One or more embodiments of the specification provide a multi-copy data storage method, device, equipment and medium. The method comprises: obtaining a pre-set copy deployment mode in a target data table in a database; in the case where the copy deployment mode is not set, obtaining a copy deployment mode indicated by a configuration file of the database; setting storage formats of multiple nodes according to the copy deployment mode; and storing data in the target data table in the nodes according to the storage formats of the nodes. By setting the row storage and column storage nodes in the same set of database architectures, different requirements for the storage formats under different query load scenarios can be met, and compared with setting the row storage and column storage in different databases, the maintenance of the database can be realized by using a set of software systems, and the nodes with different storage formats do not need to rely on data synchronization tools outside the database, which is easy to monitor and maintain.

Description

Technical Field

[0001] This specification relates to the field of data processing technology in one or more embodiments, and in particular to a method, apparatus, device and medium for storing multiple copies of data. Background Technology

[0002] Distributed databases typically use a replica mechanism, storing multiple copies of data on different nodes so that if one node fails, other nodes can continue to provide service. Node data storage formats include row-based and column-based formats, each with its own advantages in data storage, query performance, and data analysis.

[0003] Currently, row-oriented data and column-oriented data are usually stored in two different distributed database systems. Therefore, two software systems are needed to maintain databases with different storage formats, and the synchronization between data with different storage formats depends on data synchronization tools outside the database system. Summary of the Invention

[0004] In view of the above, one or more embodiments of this specification provide a method, apparatus, device and medium for storing multiple copies of data.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of one or more embodiments of this specification, a multi-replica data storage method is proposed, applied to a distributed database, the database comprising multiple nodes, the method comprising:

[0007] Obtain the pre-set replica deployment mode from the target data table in the database, wherein the replica deployment mode indicates the storage format information of the node;

[0008] If the target data table does not have a replica deployment mode set, obtain the replica deployment mode indicated by the database configuration file;

[0009] The storage format of the multiple nodes is set according to the replica deployment mode, and the storage format includes row storage and column storage formats;

[0010] The data in the target data table is stored in the node according to the storage format of the node.

[0011] In some embodiments, the replica deployment mode includes a row-store replica count and a column-store replica count, and setting the storage format of the plurality of nodes according to the replica deployment mode includes:

[0012] Set the node with the specified number of row-store replicas among the multiple nodes to row-store format;

[0013] Set the node with the number of columnar replicas among the multiple nodes to a columnar format.

[0014] In some embodiments, the plurality of nodes includes a master node and at least one replica node, and storing the data in the target data table in the node according to the node's storage format includes:

[0015] The data is stored in the master node so that the master node synchronizes the data to the replica node in the form of a log. The log is in row-based storage format, and the replica node stores the row data in the received log according to the storage format.

[0016] In some embodiments, where the database is based on a log structure merge tree, the method further includes:

[0017] The replica node writes the rows of data received from the log into a memory table in memory;

[0018] When the amount of data in the memory table reaches a set threshold, if the replica node is in row storage format, the row data in the memory table is written to the sorted string table on the disk; if the replica node is in column storage format, the row data in the memory table is split by column, and each column data is stored in an independent data block within the sorted string table.

[0019] In some embodiments, the method further includes:

[0020] When all replica nodes in the database are in columnar storage format, each replica node calculates the check value of each column of data and writes the check value of each column of data into the check value table.

[0021] If a row-based replica node exists in the database, the row-based replica node will split the row data by column, calculate the check value of each column, and write the check value of each column into a check value table.

[0022] In some embodiments, the method further includes:

[0023] The master node obtains the verification value table of each replica node;

[0024] The data consistency of the replica node is confirmed based on the verification value of each column in the verification value table.

[0025] In some embodiments, when performing node completion, the method further includes:

[0026] If the storage format of the plurality of nodes is the same, the nodes are supplemented by copying the data stored in the first node, wherein the first node is any one of the plurality of nodes;

[0027] If the storage format of the first node is different from that of the node to be filled, the data stored in the first node is converted into a different format and then stored in the node to be filled.

[0028] In some embodiments, the step of converting the format of the data stored in the first node and storing it in the node to be completed includes:

[0029] When the first node is in row storage format and the node to be filled is in column storage format, each row of data in the first node is read out and each row of data is stored in a separate data block in the node to be filled;

[0030] When the first node is in columnar storage format and the node to be filled is in row storage format, each column of data in the first node is read out, and the data in each column is concatenated into a row in order and stored in the node to be filled.

[0031] According to a second aspect of one or more embodiments of this specification, a multi-replica data storage device is provided for use in a distributed database, the database comprising multiple nodes, the device comprising:

[0032] The first acquisition unit is used to acquire the pre-set replica deployment mode in the target data table of the database, wherein the replica deployment mode indicates the storage format information of the node;

[0033] The second acquisition unit is used to acquire the replica deployment mode indicated by the configuration file of the database when the target data table is not set to a replica deployment mode.

[0034] The setting unit is used to set the storage format of the multiple nodes according to the replica deployment mode, wherein the storage format includes row storage and column storage formats;

[0035] A storage unit is used to store the data in the target data table in the node according to the storage format of the node.

[0036] In some embodiments, the replica deployment mode includes a row-store replica count and a column-store replica count, and the second acquisition unit is specifically used for:

[0037] Set the node with the specified number of row-store replicas among the plurality of nodes to row-store format;

[0038] Set the node with the number of columnar replicas among the multiple nodes to a columnar format.

[0039] In some embodiments, the plurality of nodes includes a master node and at least one replica node, and the storage unit is specifically used for:

[0040] The data is stored in the master node so that the master node synchronizes the data to the replica node in the form of a log. The log is in row-based storage format, and the replica node stores the row data in the received log according to the storage format.

[0041] In some embodiments, where the database is based on a log structure merge tree, the apparatus further includes a synchronization unit for:

[0042] The replica node writes the rows of data received from the log into a memory table in memory;

[0043] When the amount of data in the memory table reaches a set threshold, if the replica node is in row storage format, the row data in the memory table is written to the sorted string table on the disk; if the replica node is in column storage format, the row data in the memory table is split by column, and each column data is stored in an independent data block within the sorted string table.

[0044] In some embodiments, the apparatus further includes a verification value writing unit, configured to:

[0045] When all replica nodes in the database are in columnar storage format, each replica node calculates the check value of each column of data and writes the check value of each column of data into the check value table.

[0046] If a row-based replica node exists in the database, the row-based replica node will split the row data by column, calculate the check value of each column, and write the check value of each column into a check value table.

[0047] In some embodiments, the apparatus further includes a verification unit, configured to:

[0048] The master node obtains the verification value table of each replica node;

[0049] The data consistency of the replica node is confirmed based on the verification value of each column in the verification value table.

[0050] In some embodiments, when performing node completion, the apparatus further includes a completion unit for:

[0051] If the storage format of the plurality of nodes is the same, the nodes are supplemented by copying the data stored in the first node, wherein the first node is any one of the plurality of nodes;

[0052] If the storage format of the first node is different from that of the node to be filled, the data stored in the first node is converted into a different format and then stored in the node to be filled.

[0053] In some embodiments, when the padding unit is used to convert the format of the data stored in the first node and then store it in the node to be padded, it is specifically used for:

[0054] When the first node is in row storage format and the node to be filled is in column storage format, each row of data in the first node is read out and each row of data is stored in a separate data block in the node to be filled;

[0055] When the first node is in columnar storage format and the node to be filled is in row storage format, each column of data in the first node is read out, and the data in each column is concatenated into a row in order and stored in the node to be filled.

[0056] According to a third aspect of one or more embodiments of this specification, an electronic device is provided, comprising:

[0057] processor;

[0058] Memory used to store processor-executable instructions;

[0059] The processor implements the methods proposed in one or more embodiments of this specification by running the executable instructions.

[0060] According to a fourth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method proposed in one or more embodiments of this specification.

[0061] In the embodiments of this specification, a pre-set replica deployment mode is obtained from the target data table in the database; if no replica deployment mode is set, the replica deployment mode indicated by the database configuration file is obtained; the storage format of multiple nodes is set according to the replica deployment mode; and the data in the target data table is stored in the nodes according to the node storage format. By setting row-store and column-store nodes in the same database architecture, different storage format requirements under different query load scenarios can be met. Compared with setting row-store and column-store in different databases, a single software system can be used to maintain the database, and nodes with different storage formats do not need to rely on external data synchronization tools, making them easy to monitor and maintain. Attached Figure Description

[0062] Figure 1 This is a schematic diagram of an application environment for a multi-copy data storage method provided in an exemplary embodiment.

[0063] Figure 2 This is a flowchart of a multi-copy data storage method provided in an exemplary embodiment.

[0064] Figure 3A A schematic diagram illustrating the line storage format is shown.

[0065] Figure 3B A schematic diagram of the columnar storage format is shown.

[0066] Figure 4 This is a schematic diagram illustrating storage format settings based on a replica deployment mode, provided as an exemplary embodiment.

[0067] Figure 5 This is a block diagram of a multi-copy storage device provided in an exemplary embodiment.

[0068] Figure 6 This is a schematic diagram of the structure of a device provided in an exemplary embodiment. Detailed Implementation

[0069] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0070] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0071] To aid in understanding this specification, we will first define the technical terms associated with it.

[0072] Distributed databases, also known as distributed clusters, are databases that run and store data across multiple computers. These databases distribute data across multiple nodes, which can be located in the same geographical location or distributed across multiple geographical locations. This storage method improves data reliability, availability, and performance.

[0073] Replication mechanism: Storing multiple copies of data on different nodes so that if one node fails, other nodes can continue to provide services.

[0074] LSM (Log Structured Merge-Tree) is a hierarchical, ordered, disk-oriented data structure. In databases using LSM as their storage engine, each data shard of a table corresponds to one LSM tree structure. An LSM tree consists of a memory table (Memtable) in memory and a sorted string table (SSTable) on disk. Data stored in the database is divided into baseline data and incremental data. The baseline data is stored on disk. When writing data, incremental data is cached in memory. When the incremental data reaches a certain threshold, it is flushed to disk. When the incremental data on disk reaches a certain threshold, the incremental data on disk and the baseline data are merged. The core design idea of ​​LSM is to keep incremental data modifications in memory and write these modifications to disk in batches after reaching a specified size limit, thereby significantly improving performance.

[0075] Distributed databases typically use a multi-replica mechanism. The data storage format of nodes in the database can include row-based storage and column-based storage, which have different advantages in data storage, query performance, and data analysis.

[0076] Specifically, in row storage format, each data block contains all the data fields of that row, so query operations can be processed row by row, providing efficient transaction processing and low-latency query performance. Row storage format is typically used in online transaction processing (OLTP) applications. In contrast, in column storage format, each data block contains only one column of data, so query operations can be processed column by column, thereby improving batch query performance and data analysis capabilities. Furthermore, because of column-based storage, adjacent data have higher similarity, which is more conducive to improving data compression ratio and saving storage space. Column storage format is typically used in big data warehouse and analytical processing (OLAP) applications.

[0077] Currently, row-oriented data and column-oriented data are typically stored in two different distributed database systems.

[0078] For example, combining the row-oriented database Aurora and the column-oriented database RedShift, and using a data migration service to synchronize data from Aurora to RedShift, leverages their respective advantages to achieve a hybrid storage solution. In this solution, frequently used, low-latency transactional data uses Aurora's row storage, while large-volume data analysis data uses Redshift's column storage.

[0079] This hybrid storage solution has the following problems:

[0080] (1) Row-stored / column-stored data are stored in two different database systems, requiring maintenance of both software systems;

[0081] (2) Relying on data synchronization tools outside the database system to synchronize data in two databases with different storage formats makes it difficult to monitor and maintain;

[0082] (3) Once the table structure of a data table in the row-oriented database Aurora changes, it can no longer be synchronized to the column-oriented database RedShift, and the data table needs to be recreated.

[0083] In view of this, one or more embodiments of this specification provide a multi-replica data storage scheme, which sets the storage format of multiple nodes according to the replica deployment mode indicated by the target data table in the database or the replica deployment mode of the database configuration file, and stores the data of the target data table according to the set storage format, thereby realizing the setting of row storage and column storage nodes in the same database architecture.

[0084] To better understand the multi-copy storage method, apparatus, device, and medium provided in the embodiments of this specification, the application environment applicable to the embodiments of this specification is described below. Please refer to... Figure 1 , Figure 1 This diagram illustrates an application environment for a multi-replica data storage method provided in one embodiment of this specification. As one implementation method, the multi-replica data storage method provided in this embodiment is applied to a distributed database, such as OceanBase. Figure 1 The system architecture shown includes multiple database nodes, such as node 1, node 2, ..., node N, each of which can be used to store replica data.

[0085] Figure 1 The multiple database nodes shown are merely exemplary nodes. In actual applications, the number of nodes can be set according to actual needs, and no specific limit is made here. Figure 1The client in the illustrated system architecture can run upper-layer applications corresponding to the distributed database. Specifically, this could be the database management system software for the distributed database, as well as applications performing functions such as retrieval, creation, deletion, and modification. In one example, this multi-replica data storage method can be executed by the master node in the distributed database, which can be... Figure 1 Any one of the multiple nodes in the process.

[0086] Figure 2 This is a flowchart illustrating an exemplary embodiment of a multi-copy data storage method, which includes steps 201 to 204.

[0087] In step 201, the pre-set replica deployment mode in the target data table of the database is obtained.

[0088] The target data table can be any database table in the database.

[0089] When creating the target data table, a replica deployment mode can be specified. This mode indicates the storage format information of multiple nodes in the database. This storage format information can specify the storage format of each node, i.e., whether each node is in row-based or column-based storage; it can also indicate the number of row-based replicas and column-based replicas included in the multiple nodes, where the number of row-based replicas refers to the number of nodes in row-based storage and the number of column-based replicas refers to the number of nodes in column-based storage.

[0090] When a replica deployment mode is specified for the target data table, all partitions of that table use that replica deployment mode. In the examples in this specification, the replica deployment mode specified for the target data table can be referred to as a table-level replica configuration.

[0091] In step 202, if the target data table does not have a replica deployment mode set, the replica deployment mode indicated by the database configuration file is obtained.

[0092] If no replica deployment mode is specified when creating the target data table, the replica deployment mode indicated by the database configuration file will be used. The replica deployment mode indicated by the database configuration file can be referred to as the cluster-level replica configuration.

[0093] In the embodiments of this specification, it is stipulated that cluster-level replica configuration must be set when the cluster is created. Therefore, if table-level replica configuration is not specified, cluster-level replica configuration can be obtained.

[0094] As can be seen, if a table-level replica configuration is specified when creating the target data table, then that table-level replica configuration will be used as the replica deployment mode, and all partitions of the table will use that table-level replica configuration; if no table-level replica configuration is specified when creating the data table, then the table will use the cluster-level replica configuration by default.

[0095] In step 203, the storage format of the multiple nodes is set according to the replica deployment mode, and the storage format includes row storage and column storage formats.

[0096] As described in steps 201 and 202, if a table-level replica configuration is specified for the target data table, then the table-level replica configuration is used as the replica configuration mode to set whether multiple nodes are in row-based or column-based storage format; if no table-level replica configuration is specified, then the cluster-level replica configuration is used as the replica configuration mode to set the storage format of the nodes.

[0097] Specifically, setting the node's storage format to row-based storage means that when storing data, the node stores an entire row of data in a single data block, such as... Figure 3A As shown, each data block contains all the data fields of that row. Setting the node's storage format to columnar storage means that when storing data, the node stores all values ​​of each column as a separate data block, such as... Figure 3B As shown, each data block contains only one column of data.

[0098] In step 204, the data in the target data table is stored in the node according to the storage format of the node.

[0099] When writing data from the target data table to multiple nodes in the database, the data can be written according to the storage format of each node set in step 203, thereby enabling multiple nodes in the database to store multiple copies of data in different storage formats.

[0100] In the embodiments of this specification, a pre-set replica deployment mode is obtained from the target data table in the database; if no replica deployment mode is set, the replica deployment mode indicated by the database configuration file is obtained; the storage format of multiple nodes is set according to the replica deployment mode; and the data in the target data table is stored in the nodes according to the node storage format. By setting row-store and column-store nodes in the same database architecture, different storage format requirements under different query load scenarios can be met. Compared with setting row-store and column-store in different databases, a single software system can be used to maintain the database, and nodes with different storage formats do not need to rely on external data synchronization tools, making them easy to monitor and maintain.

[0101] When the replica deployment mode specifies the number of row-store replicas and the number of column-store replicas, the node with the most row-store replicas in the database can be set to row-store format, and the node with the most column-store replicas in the database can be set to column-store format. See also Figure 4 The exemplary embodiment shown illustrates the storage format settings based on the replica deployment mode.

[0102] The configuration file for this distributed database cluster indicates that the replica deployment mode is row-based with 3 replicas. Figure 4 As shown, the cluster is configured with three row-store replicas.

[0103] When data table A (hereinafter referred to as table A) is created, a replica deployment mode is specified, specifically: the number of row-store replicas is 2 and the number of column-store replicas is 1. Then, the nodes in the database use the table-level replica configuration specified by table A to set the storage format, setting two of the three nodes in the database to row-store format and setting one node to column-store format.

[0104] When creating data table B (hereinafter referred to as table B), a replica deployment mode is also specified, specifically: the number of row-store replicas is 1 and the number of column-store replicas is 2. Then, the nodes in the database use the table-level replica configuration specified by table B to set the storage format, setting one of the three nodes in the database to row-store format and setting the other two nodes to column-store format.

[0105] If table C (hereinafter referred to as table C) does not specify a replica deployment mode during its creation, that is, if no table-level replica configuration is specified, then the cluster configuration will be used by default to deploy the replicas. Specifically, the three nodes will be set as the three row-level replicas indicated by the cluster configuration.

[0106] In the embodiments of this specification, since both row-based and column-based nodes exist in the same database system, that is, all replicas share a single data table structure information, when the data table structure changes, all replicas will synchronously complete the table structure change without any additional operations.

[0107] In a distributed database, data in a table can be synchronously sent to all nodes, or it can be sent to the master node among multiple nodes, which then synchronizes it to the other replica nodes (hereinafter also referred to as replicas). The master node can be any of the multiple nodes.

[0108] In some embodiments, after receiving data from the target database and storing it according to the set storage format, the master node can synchronize the data to the replica nodes in the form of a log. The log is stored in row-based format. Upon receiving the log, the replica nodes store the row data in the log according to the set storage format.

[0109] In a distributed database based on an LSM tree, replica nodes write the row data received from the logs into a memory table called Memtable. When the amount of data in Memtable reaches a set threshold, if the replica node is in row-oriented storage format, the row data in Memtable is written into a sorted string table (SSTable) on disk, resulting in a row-oriented SSTable. If the replica node is in column-oriented storage format, the row data in Memtable is split by column, and each column of data is stored in an independent data block within Memtable, resulting in a column-oriented SSTable.

[0110] The above methods can be used to achieve data synchronization between copies of different storage formats.

[0111] In related technologies, row-stored data and column-stored data are stored in two different database systems, making it impossible to perform data consistency verification between them. This specification proposes a method for performing data consistency verification between replicas of different storage formats within the same database system.

[0112] Data consistency verification can be performed by the primary node. Replica nodes (other than the primary node) first persist data snapshots, then calculate checksums and write them to a checksum table. The primary node can obtain checksum information from all replicas by polling their checksum tables, thus confirming data consistency. If inconsistency occurs, an inconsistency error message can be written to the checksum table to notify the user.

[0113] When all replica nodes in the database use the same storage format, each replica node performs a data snapshot and persists it at the same time point. The checksum of the data can then be calculated using the binary file of the data blocks. Since the data format is identical across multiple replicas, data consistency can be verified by directly comparing the checksums of the data blocks.

[0114] Specifically, assuming all replica nodes are in columnar storage format, each replica node calculates a checksum for each column of data and writes the checksum to a checksum table. After obtaining the checksum table, the master node confirms the data consistency of the replica nodes by comparing the checksums column by column.

[0115] When row-formatted replica nodes exist in the database, these nodes calculate checksums column-wise. Specifically, the row-formatted replica nodes split the row data into columns, calculate the checksum for each column, and write the checksum for each column into a checksum table. After obtaining the checksum table, the master node confirms the data consistency of the replica nodes by comparing the checksums column-wise.

[0116] In the embodiments described in this specification, both row-stored data and column-stored data have their check values ​​calculated column-wise, and the master node performs data consistency verification to ensure data security.

[0117] The table-level replica configuration of a data table can be changed after the table is created. For the replicas that have changed, adjustments can be made by deleting or adding replicas.

[0118] Specifically, if the number of row-stored copies or column-stored copies decreases, the corresponding number of copies are deleted; if the number of row-stored copies or column-stored copies increases, the corresponding number of copies are replenished.

[0119] For example, if the table-level replica configuration specifies that the number of row replicas is M and the number of column replicas is N (M and N are positive integers), if the number of row replicas changes by (M+1), then one row replica is added to make up the difference; if the number of column replicas changes by (N-1), then one column replica is deleted.

[0120] When supplementing replicas, if the multiple replicas have the same storage format, the supplementary node can be completed by copying the data stored in the first node, where the first node can be any one of the multiple nodes. When the storage formats of the multiple replicas are inconsistent, the replica data format needs to be converted. Specifically, if the storage formats of the first node used for supplementation are different from those of the node to be supplemented, the data stored in the first node is converted and then stored in the node to be supplemented.

[0121] When the first node is in row storage format and the node to be filled is in column storage format, the column storage copy can be filled by reading each row of data in the first node and storing each row of data in a separate data block in the node to be filled.

[0122] When the first node is in columnar storage format and the node to be filled is in row storage format, the row storage copy can be filled by reading each column of data in the first node, concatenating the data of each column in order into a row, and storing it in the node to be filled.

[0123] See Figure 5 , Figure 5This is an exemplary embodiment of a multi-replica data storage device applied to a distributed database, the database including multiple nodes, the device comprising:

[0124] The first acquisition unit 501 is used to acquire the replica deployment mode preset in the target data table of the database, wherein the replica deployment mode indicates the storage format information of the node;

[0125] The second acquisition unit 502 is used to acquire the replica deployment mode indicated by the configuration file of the database when the target data table is not set to a replica deployment mode.

[0126] Setting unit 503 is used to set the storage format of the multiple nodes according to the replica deployment mode, the storage format including row storage and column storage formats;

[0127] Storage unit 504 is used to store the data in the target data table in the node according to the storage format of the node.

[0128] In some embodiments, the replica deployment mode includes a row-store replica count and a column-store replica count, and the second acquisition unit is specifically used for:

[0129] Set the node with the specified number of row-store replicas among the plurality of nodes to row-store format;

[0130] Set the node with the number of columnar replicas among the multiple nodes to a columnar format.

[0131] In some embodiments, the plurality of nodes includes a master node and at least one replica node, and the storage unit is specifically used for:

[0132] The data is stored in the master node so that the master node synchronizes the data to the replica node in the form of a log. The log is in row-based storage format, and the replica node stores the row data in the received log according to the storage format.

[0133] In some embodiments, where the database is based on a log structure merge tree, the apparatus further includes a synchronization unit for:

[0134] The replica node writes the received rows of data from the log into a memory table in memory;

[0135] When the amount of data in the memory table reaches a set threshold, if the replica node is in row storage format, the row data in the memory table is written to the sorted string table on the disk; if the replica node is in column storage format, the row data in the memory table is split by column, and each column data is stored in an independent data block within the sorted string table.

[0136] In some embodiments, the apparatus further includes a verification value writing unit, configured to:

[0137] When all replica nodes in the database are in columnar storage format, each replica node calculates the check value of each column of data and writes the check value of each column of data into the check value table.

[0138] If a row-based replica node exists in the database, the row-based replica node will split the row data by column, calculate the check value of each column, and write the check value of each column into a check value table.

[0139] In some embodiments, the apparatus further includes a verification unit, configured to:

[0140] The master node obtains the verification value table of each replica node;

[0141] The data consistency of the replica node is confirmed based on the verification value of each column in the verification value table.

[0142] In some embodiments, when performing node completion, the apparatus further includes a completion unit for:

[0143] If the storage format of the plurality of nodes is the same, the nodes are supplemented by copying the data stored in the first node, wherein the first node is any one of the plurality of nodes;

[0144] If the storage format of the first node is different from that of the node to be filled, the data stored in the first node is converted into a different format and then stored in the node to be filled.

[0145] In some embodiments, when the padding unit is used to convert the format of the data stored in the first node and then store it in the node to be padded, it is specifically used for:

[0146] When the first node is in row storage format and the node to be filled is in column storage format, each row of data in the first node is read out and each row of data is stored in a separate data block in the node to be filled;

[0147] When the first node is in columnar storage format and the node to be filled is in row storage format, each column of data in the first node is read out, and the data in each column is concatenated into a row in order and stored in the node to be filled.

[0148] Figure 6 This is a schematic structural diagram of a device provided in an exemplary embodiment. Please refer to... Figure 6At the hardware level, the device includes a processor 602, an internal bus 604, a network interface 606, memory 608, and non-volatile memory 610, and may also include other hardware required for services. One or more embodiments of this specification can be implemented in software, such as the processor 602 reading the corresponding computer program from the non-volatile memory 610 into memory 608 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0149] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0150] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0151] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0152] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0153] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0154] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0155] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this specification. The singular forms “a,” “described,” and “the” used in one or more embodiments of this specification and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0156] It should be understood that although the terms first, second, third, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of one or more embodiments of this specification, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "in response to a determination," or "when," or "in the event of a determination."

[0157] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the protection scope of one or more embodiments of this specification.

Claims

1. A multi-replica data storage method applied to a distributed database, the database comprising multiple nodes, the multiple nodes including a master node and at least one replica node, the method comprising: Obtain the pre-set replica deployment mode from the target data table in the database, wherein the replica deployment mode indicates the storage format information of the node; If the target data table does not have a replica deployment mode set, obtain the replica deployment mode indicated by the database configuration file; The storage format of the multiple nodes is set according to the replica deployment mode, and the storage format includes row storage and column storage formats; The data is stored in the master node so that the master node synchronizes the data to the replica node in the form of a log. The log is in row-based storage format, and the replica node stores the row data in the received log according to the storage format.

2. The method according to claim 1, wherein the replica deployment mode includes a row-based replica count and a column-based replica count, and setting the storage format of the plurality of nodes according to the replica deployment mode includes: Set the node with the specified number of row-store replicas among the multiple nodes to row-store format; Set the node with the number of columnar replicas among the multiple nodes to a columnar format.

3. The method according to claim 1, wherein when the database is based on a log structure merge tree, the method further includes: The replica node writes the rows of data received from the log into a memory table in memory; If the amount of data in the memory table reaches a set threshold, and if the replica node is in row storage format, the row data in the memory table will be written to a sorted string table on the disk. If the replica node is in columnar storage format, the row data in the memory table is split by column, and each column data is stored in an independent data block within the sorted string table.

4. The method according to claim 1, further comprising: When all replica nodes in the database are in columnar storage format, each replica node calculates the check value of each column of data and writes the check value of each column of data into the check value table. If a row-based replica node exists in the database, the row-based replica node will split the row data by column, calculate the check value of each column, and write the check value of each column into a check value table.

5. The method according to claim 4, further comprising: The master node obtains the verification value table of each replica node; The data consistency of the replica node is confirmed based on the verification value of each column in the verification value table.

6. The method according to claim 1, wherein when performing node completion, the method further includes: If the storage format of the plurality of nodes is the same, the nodes are supplemented by copying the data stored in the first node, wherein the first node is any one of the plurality of nodes; If the storage format of the first node is different from that of the node to be filled, the data stored in the first node is converted into a different format and then stored in the node to be filled.

7. The method according to claim 6, wherein the step of converting the format of the data stored in the first node and storing it in the node to be supplemented includes: When the first node is in row storage format and the node to be filled is in column storage format, each row of data in the first node is read out and each row of data is stored in a separate data block in the node to be filled; When the first node is in columnar storage format and the node to be filled is in row storage format, each column of data in the first node is read out, and the data in each column is concatenated into a row in order and stored in the node to be filled.

8. A multi-replica data storage device applied to a distributed database, the database comprising multiple nodes, the multiple nodes including a master node and at least one replica node, the device comprising: The first acquisition unit is used to acquire the pre-set replica deployment mode in the target data table of the database, wherein the replica deployment mode indicates the storage format information of the node; The second acquisition unit is used to acquire the replica deployment mode indicated by the configuration file of the database when the target data table is not set to a replica deployment mode. The setting unit is used to set the storage format of the multiple nodes according to the replica deployment mode, wherein the storage format includes row storage and column storage formats; A storage unit is used to store the data in the master node so that the master node can synchronize the data to the replica node in the form of a log, wherein the log is in row storage format, and the replica node is used to store the row data in the received log according to the storage format.

9. An electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor implements the method as described in any one of claims 1-7 by executing the executable instructions.

10. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the steps of the method as claimed in any one of claims 1-7.

Images