Method for realizing adjustable consistency of storage system adopting Raft protocol

Abstract

The invention relates to a method for realizing adjustable consistency of a storage system adopting a Raft protocol, and belongs to the technical field of storage systems. The method comprises the steps that a write-in return point and a read return point are introduced for each db, the write-in return point controls the position, returned to a user, of a write-in command in the whole execution process, when write-in operation is executed to the position corresponding to the write-in return point, a state machine is notified to execute through prepply, and an execution result is returned to the user after execution is completed; the reading return point controls the reading command to read from the target copy and the reading mode; the consistency of all read-write commands on the db is controlled by combining values of different write-in return points and values of different read return points; the data applied in advance is rolled back when conflicting so as to ensure that the system data is finally recovered to be consistent. According to the method and the device, the difficulty of realizing the adjustable consistency under the condition of using the Raft protocol is reduced, the additional prepply is introduced, the adjustable consistency is realized, the retention of the onpply also ensures the correctness of the Raft, and the proof of the final consistency is simplified.

Description

TECHNICAL FIELD

[0001] The present disclosure belongs to the technical field of storage systems, and particularly relates to a method for realizing adjustable consistency of a storage system adopting a Raft protocol. BACKGROUND

[0002] Distributed storage systems need to trade off between strong consistency and high availability while providing high availability, which is determined by the CAP theorem in distributed systems. MongoDB, as a popular NoSQL database, allows clients to choose various trade-offs between consistency and latency at the operation level to meet the needs of different application scenarios. MongoDB allows explicit specification of trade-offs for data consistency and latency at the operation level by introducing the concepts of write and read parameters (writeConcern and readConcern) for each user request, providing fine-grained control for developers. For write operations, the persistence guarantee level can be specified, such as w:1, which means that the client receives confirmation as long as the primary node locally commits the write operation, w:N, which means that at least N replica set nodes receive and locally commit the write operation before confirmation, and w:majority, which means that the write operation is confirmed only after it is committed on a majority of replica set nodes, ensuring that the write operation can resist any temporary or permanent failure in the case of replica set failure. The "j" parameter can also be specified to determine whether data must be logged on replica set nodes before confirmation, and the "tag set" can be specified to specify that the write operation needs to be replicated to specific tag nodes. At the same time, write operations can have different response types, divided into satisfied and unsatisfied, satisfied means that the required conditions are met or exceeded, and unsatisfied does not mean that the write operation fails, which may be due to fewer replica nodes than required or not timely notification of the primary node. For read operations, the persistence and consistency of returned data can be specified, such as the local level, which means returning the local state data of the replica set node when the read operation is executed, without majority commitment guarantee, and the latest local committed data of a specific node can be read, while the majority level only returns majority committed data, without strict guarantee of the freshness of returned data. If the read operation is specified as linearizable level, it can be combined with the write operation specified as w:majority to provide the strongest consistency guarantee, which can return the data effect before the completion of the latest majority write operation, collectively satisfying the linear consistency condition. In addition, available, snapshot levels and causal consistency reading capabilities are also provided. MongoDB replication protocol is similar to Raft consensus algorithm, but unlike Raft, which does not apply log entries to the state machine before committing the operation, MongoDB replica set nodes apply log entries immediately after receiving them, allowing the provision of "local" read consistency levels. This model makes it necessary to implement data rollback protocols.When a node detects that its log is inconsistent with the log of a synchronization source node, a rollback process is started, including two algorithms of restoring to a timestamp algorithm (RTT) and a refetch-based rollback algorithm, the RTT uses a storage engine to roll back a database to a historical timestamp and then apply logs to reach consistency, and the refetch-based rollback algorithm restores consistency by obtaining data from other nodes, which is more complex, the RTT simplifies the rollback logic, and the system can better implement various consistency levels.

[0003] The adjustable consistency model of MongoDB has the following disadvantages: high implementation complexity, the replication protocol allows nodes to apply logs immediately after receiving the logs, and a complex rollback mechanism (such as the restoring to timestamp algorithm and the refetch-based rollback algorithm) is needed to handle data inconsistency, which increases the difficulty of system implementation. At the same time, it provides multiple consistency levels (such as linear consistency and causal consistency), which increases the complexity of developer selection and management, the complex write concern and read concern design further increases the complexity of system design, and the consistency guarantee is difficult to verify; it cannot be used on the Raft system, the replication protocol of MongoDB is different from the Raft consensus algorithm, Raft requires that the log entry is not applied to the state machine before the operation is committed, while MongoDB allows the node to commit immediately after the log is applied, resulting in protocol incompatibility. SUMMARY

[0004] The present disclosure proposes a method for implementing adjustable consistency of a storage system using a Raft protocol to solve the above technical problems.

[0005] A method for implementing adjustable consistency of a storage system using a Raft protocol, the method comprising: introducing a write return point and a read return point for each db, wherein the write return point controls the position of the write command returned to the user in the entire execution process, and the write operation is executed by pre_apply to notify the state machine when it is executed to the position corresponding to the write return point, and the execution result is returned to the user after the execution is completed; the read return point controls the read command from the target copy and the read mode; by combining the values of different write return points and read return points, the consistency of all read and write commands on the db is controlled; the data applied in advance is rolled back in conflict to ensure that the system data is finally restored to consistency.

[0006] In some embodiments, if the value of the write return point is 0, the write operation is executed directly in the TabletX state machine after being submitted to the Raft assignment engine without waiting for any Raft log persistence; if the value of the write return point is less than the number of the current majority in the Raft group to which the Tablet belongs, the Raft log entry corresponding to the write operation is applied to the state machine through pre apply after being written in the WRP nodes; if the value of the write return point is greater than or equal to the number of the current majority in the Raft group to which the Tablet belongs, the Raft log entry is executed when it is written in the majority nodes.

[0007] In some embodiments, when the value of the write return point is greater than 0 and less than the number of the current majority in the Raft group to which the Tablet belongs, the state machine is waited for application when a read operation is executed.

[0008] In some embodiments, the data applied in advance is rolled back in the event of a conflict to ensure that the system data is eventually restored to consistency, including: when a write operation is executed, an additional pre-apply plus multi-version data submission and rollback method is used in TRaft to ensure that the system data is eventually restored to consistency; the additional pre-apply plus multi-version data submission and rollback method used in TRaft to ensure that the system data is eventually restored to consistency includes: immediately notifying the upper state machine to execute through pre_apply after single-node persistence is completed, without removing from the Raft log storage; applying the pre_apply command Cmd1 on the Raft Leader, applying the normal Raft log entry after the majority is persisted, and applying on the Follower in the Raft group according to the log arrangement order; when the state machine fails, the rollback is executed after the failure is recovered, where the rollback is processed through a second on_apply;

[0009] The rollback is processed through a second on_apply, including: the state machine additionally records the information of the log entry in the pre_applied CF of the data storage after the winning log is executed.

[0010] In some embodiments, if the state machine does not fail, at the actual on_apply, the execution is performed to the entry that has been pre-applied, and it is directly detected whether there is a record of the log entry in the pre-applied CF; if not, the normal process is performed in the state machine; if yes, the log is compared to see whether they are the same; if yes, the pre_apply has been executed, and there is no need to execute, and the committed version of the multi-version metadata information of the corresponding key is updated to the current version number; if not, the current log has changed, rollback is executed, the pre_apply mechanism is temporarily closed, and the log and all pre-applied keys and versions after the log are read from the pre-applied CF, and all data is deleted from the data storage, and after the rollback is completed, the pre_apply mechanism is restarted.

[0011] In some embodiments, if the value of the write return point is 0, the write operation does not wait for any persistence, is executed after the log is submitted to the Raft replication engine, and the read operation directly reads the state machine on any replica.

[0012] In some embodiments, if the value of the write return point is -1 or quorum, all write requests are executed on the Raft leader, and are executed according to the Raft protocol, the operation log is executed after the majority is persisted, all read operations are performed on the Raft leader, and the Raft Lease Read is used.

[0013] In some embodiments, if the value of the write return point is -1 or quorum, all write commands are executed on the Raft leader, and are executed according to the Raft constraint, and the read command is executed on any replica, wherein the state machine of the replica to be read has been applied to the log index number required by the read command.

[0014] In some embodiments, if the value of the write return point is 1, the execution does not have linear consistency, and the write operation is not guaranteed to be persisted.

[0015] In some embodiments, the value of the read return point is composed of two fields, one of which is a position field, and the other is a read mode; when the read mode is Lease Read, the position field is Leader, and the read command is read from the TabletX state machine of the current Leader copy after checking the validity or renewal success of the Lease on the Leader; when the read mode is Wait Index Read, the position field is Leader or Follower, and the TabletX state machine has applied a log index greater than the index required by the read command itself, the read is performed in TabletX, otherwise, the state machine waits to apply the corresponding target log entry; when the read mode is Direct Read, no checking and special operation is performed, and the read is performed in the TabletX state machine.

[0016] By adopting the technical solutions described above, the embodiments of the present disclosure can achieve the following beneficial technical effects: Tunable Raft (TRaft) is introduced based on the Raft protocol, an additional pre_apply mechanism is introduced, users can configure the consistency of read and write operations in units of logical entities (such as DB), taking write as an example, users can configure the Write Return Point (WRP) parameter to be less than the Quorum of the Raft group, such as WRP = 1, which means that as long as one node is successfully persisted, the state machine is notified to execute immediately, and WRP = 0 means that the log is submitted to the Raft replication engine without waiting for any persistence to directly notify the state machine to execute. However, the cost of not waiting for the majority to persist is that when the Leader switches or fails, some logs that have been notified to the state machine to apply will be overwritten, at which time rollback is needed. Therefore, TRaft still retains the normal on_apply after the majority of the persistent is completed, which means that there will be two Apply at this time, on_apply will check the information of the log saved in the state machine, if it does not match, it means that the log is overwritten and needs to be rolled back all the logs that have been pre-executed after this entry. The TRaft scheme ensures the safety of the Raft protocol without being broken by retaining on_apply, and the system can still guarantee the eventual consistency of data, but at the time of failure, users may read inconsistent values.

[0017] Compared with the complex adjustable consistency model designed in Monogodb, the disclosure introduces a simple and easy-to-use adjustable consistency model for a storage system using the Raft protocol (which implements the method). The disclosure reduces the difficulty of implementing adjustable consistency under the use of the Raft protocol, and by introducing an additional pre_apply, not only does it achieve adjustable consistency, but the preservation of on_apply also guarantees the correctness of Raft, simplifying the proof of eventual consistency. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings, which constitute a part of this specification, illustrate embodiments of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0019] The disclosure can be more clearly understood and appreciated from the following detailed description, with reference to the following drawings:

[0020] Figure 1 is a schematic diagram illustrating that different WRP values correspond to different positions in the execution flow according to some embodiments of the disclosure.

[0021] Figure 2 is a schematic diagram illustrating that RRP is controlled by two fields according to some embodiments of the disclosure.

[0022] Figure 3 is a schematic diagram illustrating adjustable consistency Raft-TRaft according to some embodiments of the disclosure. DETAILED DESCRIPTION

[0023] Various exemplary embodiments of the disclosure will now be described in detail with reference to the accompanying drawings. Note that the relative arrangement, numerical expressions, and numerical values of components and steps set forth in these embodiments do not limit the scope of the disclosure unless otherwise specifically stated.

[0024] At the same time, it should be understood that the sizes of the various parts shown in the drawings are not drawn in accordance with the actual proportional relationship for the sake of description.

[0025] The following description of at least one example embodiment is merely exemplary in nature and is in no way intended to limit the disclosure, its application, or uses.

[0026] Techniques, methods, and devices known to those of ordinary skill in the relevant art can not be discussed in detail herein, but should be considered part of the specification where appropriate.

[0027] In all examples shown and discussed herein, any specific values should be understood to be exemplary only and not limiting. Thus, other examples of the example embodiments can have different values.

[0028] It should be noted that like reference numerals and characters refer to like items throughout the several views, and once an item is defined in one view, it should not have to be discussed further in subsequent views.

[0029] In Mongodb, the consistency and performance of the system is controlled by giving each operation an additional WriteConcern and ReadConcern parameter to control the consistency of each request. In the current popular distributed storage system, the Raft protocol is used for data replication and fault tolerance. The Raft protocol has a fixed log replication and application order, which is completely different from Mongodb. At the same time, in the actual scene, the consistency level is usually defined based on an abstract storage entity, such as a database DB, so in this disclosure, the request-level consistency control is changed to DB-level consistency control, that is, the consistency level of a certain DB can be dynamically set.

[0030] A Write Return Point (WRPoint) and a Read Return Point (RRPoint) configuration parameter are introduced for each DB to control the persistence of all write commands in the DB and the read return position of the read operation. By setting different combinations of WRPoint and RRPoint, the consistency and performance of the command execution on the DB can be adjusted.

[0031] The Write Return Point (WRPoint, WRP) controls the position of the write command returned to the user in the entire execution process, as shown in FIG. 1, different WRP values correspond to different positions in the execution process, when a specific WRP value is set for the DB, the write operation is executed to the position corresponding to the WRP value, and the state machine is notified in advance through pre_apply to perform the execution, and the execution result is returned to the user after the execution is completed. Figure 1

[0032] The WRP value and its meaning are shown in the following table.

[0033] WRP value Return position explanation 0 Immediately after submitting to Raft replication engine wrp <quorum Immediately after Raft log is persisted in wrp nodes Wrp >= quorum After Raft log is persisted in majority -1 Same meaning as wrp = quorum, simplify configuration

[0034] ​The WRP value strictly represents not only the corresponding return position, but also implies the persistence requirement for the current write operation. For example, if the WRP value is 0, the write operation will not wait for any Raft log persistence after being submitted to the Raft replication engine, but will be directly executed in the TabletX state machine. If the WRP value is less than a quorum (the quorum refers to the majority number of the Raft group to which the Tablet currently belongs), the write operation will be executed by notifying the state machine through pre_apply after the corresponding Raft log entry is stored in the WRP nodes. If the WRP value is greater than or equal to the quorum, the Raft log entry will be executed after being stored in the majority nodes, which is consistent with the execution time of on_apply in the general Raft replication engine. Of course, even if the WRP value is greater than the quorum, the write operation can be executed only after being stored in the majority nodes. It is worth noting that all write operations must be executed on nodes with a Leader role, and the WRP value only controls the return position and execution process of the write operation on the Leader, and does not allow multiple master writes.

[0035] As can be seen, when the WRP value is between 0 and the quorum, the state machine is directly notified to execute without waiting for the Raft log to be persisted in the majority nodes. Although this greatly improves the throughput and reduces the delay of the system, it also means that the Raft constraints are violated, and certain fault tolerance and consistency are lost. Therefore, in order to ensure that the storage system still has the ability of final consistency and fault handling after introducing adjustable consistency, additional processes and mechanisms need to be introduced based on the Raft protocol. Before explaining the correctness of the adjustable consistency modification of Raft, the meaning of the read return point is further explained.

[0036] The read return point (RRPoint, RRP) is used to control from which replica the read command is read and the read mode. As shown in Table 1, the RRP parameter and the WRP use a single integer value to control the difference. The value of the RRP is composed of two fields. The first field is the location field (loc) which indicates the allowed replica location, which can be the Leader (i.e., the Raft Leader) or Any, indicating that the read can be performed on any replica. The second field is the read mode (rmode) which defines the read process of the read command. The possible values and their meanings are shown in the following table. Figure 2

[0037]

[0038] ​When rmode is Lease Read, this time read is Lease Read using Raft, the first field loc can only be Leader, this time read command directly read from TabletX state machine in current Leader replica after checking the Lease is valid or renewal is successful, if WRP is set to -1, linearly consistent read and write can be achieved. When rmode is Wait Index Read, as long as TabletX state machine on the current replica has applied log index greater than the required index of the read command, read can be directly performed on TabletX, otherwise it needs to wait for the state machine to apply the corresponding target log entry. In this way, although the global constraint of linearly consistent read is broken, only single-client-centered consistent read can be provided, but at this time it is not required to read the Leader replica, and the processing capacity of read request increases linearly with the increase of replicas. This way, the client has certain requirements, and needs to save some state information of the current link. When rmode is set to Direct Read, no checking and special operation is performed, and read is directly performed from TableX state machine. At this time, if 0 <= WRP < quorum is configured, high-performance consistent read and write can be achieved when no fault occurs in the cluster, of course, when a fault occurs in the cluster, it cannot guarantee that the data returned successfully written is not lost, and global linearly consistent read cannot be guaranteed.

[0039] By freely combining different values of WRP and RRP, the consistency of all read and write commands on a certain db can be easily controlled to obtain different combinations of performance and reliability. When the system is fault-free, a low-consistency configuration can obtain higher performance, but when a fault occurs, the final consistency of all replicas can be guaranteed by losing some data that has been promised to be written. Although in a large-scale distributed system, faults occur frequently, the fault-free state still occupies the majority, so adjustable consistency can not only meet the needs of consistency and performance in different scenarios, but also make full use of cluster hardware resources to improve throughput to a certain extent.

[0040] Although various consistency and performance combinations can be generated by freely combining different values of WRP and RRP to meet user needs, for most users, it is obviously easier to directly use some commonly used consistency levels than to set their own values after understanding the meaning of WRP and RRP. At the same time, the WRP and RRP of the preset consistency level still have reference value for users to set specific values to meet their special needs. A total of four consistency levels are preset, as shown in the following table.

[0041] Consistency level WRP RRP Explanation linearizable -1 or quorum Leader: Lease Read Linearly consistent read-write client -1 or quorum Any: Wait Index Read Client-centric consistent read-write balance 1 Any: Wait Index Read Monotonic read single 0 Any: Direct Read Single-replica read-write

[0042] The consistency gradually decreases from top to bottom. Linearizable has the highest consistency level. All write requests are executed on the Raft Leader and strictly follow the Raft protocol. The operation log must be persisted in the majority before being actually executed. At this level, all read operations are also performed on the Raft Leader and use strict Raft Lease Read. That is, the read operation can be performed only after the current Raft Leader's lease or re-rental. At the linearizable level, global linear consistent read and write are achieved. The client level is a client-centered consistency level. All write commands and linerizable must be executed on the Raft Leader and strictly follow the Raft constraints, but the operation consistency constraint is lower than that of linerizable. At this time, the read command can be executed on any replica, and only needs to wait for the state machine of the replica to be applied to the log index number required by the read command. At the client level, all write commands of a client still have global linear consistency, but the read command only satisfies the read your write (RYW) and monotone read (MW), and does not guarantee that the current client can read the latest write in the system. Of course, an empty write operation can be performed before reading to refresh the log index of the current client, so that all the latest values at the time of executing the empty write operation can be read. The balance level relaxes the constraints on write operations based on the client level. At this time, the write operation does not have linear consistency, and no persistence is guaranteed. However, the probability of losing a successful write operation is still small. The sigle level is the weakest consistency level and also the best level for system performance. Its WRP is 0, meaning that the write operation does not wait for any persistence and is executed after the log is submitted to the Raft replication engine. The read operation can also be directly read from the state machine on any replica. At this level, no consistency guarantee is made for the read and write observations. However, if the distributed system does not fail and the read operation only reads the value on the Leader, the read and write effects are the same as single-machine read and write and linear consistency.

[0043] In order to make the DRedis system using the Raft protocol support tunable consistency without losing the ease of use and simplicity of Raft itself, the present disclosure designs a simple and easy-to-use tunable consistency Raft (TRaft) based on Raft combined with the DRedis architecture. For write operations, changing its behavior according to RRP is relatively simple, and does not need to be invasive to the Raft protocol itself and can be directly implemented in the state machine, such as a waiting state machine application. For write operations, the situation is much more complicated. When WRP >= quorum, this is the normal Raft usage process, and no additional process is needed. However, when 0 <= WRP < quorum, it cannot be directly returned after the Raft replication engine persists the corresponding number. Only when the write operation is actually executed in the state machine, the write is persisted in the underlying storage and is visible to the outside, and then the wide constraint read request can read the actual value, which is the meaningful tunable consistency. However, without being persisted in the majority, the pre_apply is applied to the state machine, which obviously destroys the constraints of the Raft protocol. The pre-applied data must be rolled back in a certain way in the event of a conflict to ensure that the system data eventually recovers to consistency.

[0044] In TRaft, the additional pre_apply and multi-version data submission and rollback are used to ensure the fault tolerance and eventual consistency of the algorithm, as shown in Figure 3 WRP = 1 is taken as an example in the figure, that is, when a node completes log persistence, the pre_apply is used to notify the state machine to apply the corresponding log in advance. For 0 <= WRP < quorum, the pre_apply is used to notify the state machine to execute in advance after the corresponding number of persistence is completed. In a storage system using Raft, a Raft group may be responsible for data replication of multiple logical entities (such as multiple physical shards), so there are both ordinary Raft logs such as the gray entries in the figure and db log entries with WRP = 1 such as the green entries in the figure in the Raft replication engine. It is worth noting that these green entries will immediately notify the upper state machine to execute through pre_apply after single-node persistence is completed, but will not be removed from the Raft log storage. The pre_apply is an additional notification and will not affect the re-application of the entry when the majority persistence is successful on_apply. By using this additional notification, the impact on the Raft protocol itself can be minimized, and the correctness of Raft election and log replication is still guaranteed. At this time, the complexity of recovering consistency in the event of a failure is transferred to the state machine.

[0045] The correctness of TRaft, in addition to the recognition that the log replication, election constraints of Raft are not changed, another focus is that the ordinary Raft log entries (gray entries in the figure) and the log entries of 0 <= WRP < quorum are irrelevant, that is, the green entries are the other gray entries of db1 and the other db, and the application of the state machine on different nodes in the order of the green entries and the gray entries can be inconsistent, without affecting the determination of the consistency of the state machine. For example, on the Raft leader in the figure, the command Cmd1 is applied in advance through pre_apply, and then the gray entries are applied after the majority persistence, and on the follower in the Raft group, the application order is according to the log arrangement order, but due to the isolation of the db, the inconsistency does not affect. But for the same db, such as db1 here, even if it is applied in advance, it must ensure that their relative application order is the same on all replicas, which can be easily guaranteed in the implementation of the AppendEntry RPC of Raft.

[0046] The remaining work is to handle the final consistency of the state machine when fault handling. When a fault occurs, as WRP = 1 in the figure, some entries have been executed by pre_apply in the state machine, but they have not completed the majority persistence, according to the Raft protocol, these entries are not committed (Commited), and may be overwritten in the subsequent process. At this time, after the fault is recovered, it is necessary to perform rollback on these commands executed in pre_apply. These rollbacks can be handled by secondary on_apply. For example, taking a distributed storage system using Rocksdb as the underlying storage engine as an example, as shown in the figure, after the state machine executes the corresponding log, it will additionally record the information of the log entry in the pre-applied CF of the data storage. The state machine must ensure that the insertion of pre-applied CF and the writing of actual data are in the same WriteBatch to avoid inconsistency when a fault occurs, resulting in the failure of the rollback mechanism. If no fault occurs, when actually on_apply, the entry that has been pre-applied is executed, it will directly check whether there is a record of the log entry in the pre-applied CF. If there is no record, it will be executed in the state machine according to the normal process. If there is a record, it will compare whether the log is the same (by checking whether the term is the same). If it is the same, it means that it has been executed by pre_apply. At this time, it is not necessary to execute, and only the committed version of the multi-version metadata information of the corresponding key needs to be updated to the current version number. If the term is different, it means that the current log has changed, and needs to be rolled back. At this time, the pre_apply mechanism needs to be temporarily closed, and the log and all pre-applied keys and versions after the log are read out from the pre-applied CF. All these data are deleted from the data storage. After the rollback is completed, the pre_apply mechanism is restarted. The checks made by on_apply are performed in the same process in the leader and follower, so that all state machines will eventually be restored to consistency.

[0047] In the Raft adjustable consistency modification, multiple versions of all key values must be stored in the data storage due to possible rollback operations, usually requiring the upper state machine to store multiple records with different version numbers in the storage engine. The version number is denoted as version, which is spliced by term and timestamp (ts) of the type uint64. Since all write operations are completed on the Raft leader, and the term of Raft is monotonically increasing with the switching of the leader, different versions can be distinguished by combining the term and the physical clock timestamp of the current node. The current strong leader submission has an updated version number. The version number is generated by the Raft replication engine when the log is submitted to the Raft replication engine, and the command executed by pre_apply, and the version number is used for the impact and new key.

[0048] Unlike the visibility rule in MVCC, the result of pre_apply execution is naturally visible, and all read operations can directly read the key value with the maximum version number as the result. Even so, with the execution of a large number of weak consistency write operations, there will be a large amount of version data in the data storage. In order to determine which is the obsolete version, additional version metadata information is saved for each key value in the mv-meta CF, where max_version represents the maximum version number in the version chain of the key, min_version represents the minimum version number, and commited_version represents the version number that has been persisted in the majority. The maximum and minimum version numbers stored therein are used for decision-making in garbage collection. In the on_apply process, the commited_version field is modified, and the version data less than commited_version is obsolete data.

[0049] In addition, TRaft needs to store log entry records in the pre-applied CF because the rollback is below the state machine itself, and the stored key value needs to record the pre-applied log information.

[0050] Compared with the complex adjustable consistency model designed in Monogodb, the present disclosure introduces a simple and easy-to-use adjustable consistency model for storage systems using the Raft protocol. The present disclosure reduces the difficulty of implementing adjustable consistency under the Raft protocol, and by introducing an additional pre_apply, not only does it achieve adjustable consistency, but the on_apply reservation also guarantees the correctness of Raft and simplifies the proof of eventual consistency.

[0051] So far, the method for achieving adjustable consistency of a storage system adopting a Raft protocol according to the present disclosure has been described in detail. In order to avoid this, some details known in the art are not described. Those skilled in the art can fully understand how to implement the technical solutions disclosed herein according to the above description.

[0052] Although some specific embodiments of the present disclosure have been described in detail through examples, those skilled in the art should understand that the above examples are only for illustration, not for limiting the scope of the present disclosure. Those skilled in the art should understand that the above embodiments can be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A method for achieving tunable consistency in a storage system employing a Raft protocol, the method comprising: The method comprises: ​ Introducing a write return point and a read return point for each db, wherein the write return point controls the position of the write command returned to the user throughout the execution process, the write operation is executed by pre_apply to notify the state machine when it is executed to the position corresponding to the write return point, and the execution result is returned to the user after the execution is completed; the read return point controls the read from the target copy and the read mode; By combining the values of different write return points and read return points, the consistency of all read and write commands on the db is controlled; The pre-applied data is rolled back in conflict to ensure that the system data is eventually consistent.

2. The method for achieving tunable consistency of a storage system employing Raft protocol according to claim 1, wherein, If the value of the write return point is 0, the write operation is submitted to the Raft assignment engine, without waiting for any Raft log to be persisted, and is directly executed in the TabletX state machine; If the value of the write return point is less than the majority number in the Raft group to which the current Tablet belongs, the Raft log entry corresponding to the write operation is recorded in the WRP nodes, and the state machine is notified for execution through pre_apply; If the value of the write return point is greater than or equal to the majority number in the Raft group to which the current Tablet belongs, the Raft log entry is executed when it is recorded in the majority nodes. 3.The method for achieving tunable consistency of a storage system employing Raft protocol according to claim 2, characterized in that, When the value of the write return point is greater than 0 and less than the majority number in the Raft group to which the current Tablet belongs, the state machine is waited for application when the read operation is executed.

4. The method for achieving tunable consistency of a storage system employing Raft protocol according to claim 3, wherein, The pre-applied data is rolled back in conflict to ensure that the system data is eventually consistent, comprising: When the write operation is executed, an additional pre-application plus multi-version data submission and rollback method is adopted in TRaft to ensure that the system data is eventually consistent; The additional pre-application plus multi-version data submission and rollback method is adopted in TRaft to ensure that the system data is eventually consistent, comprising: After single-node persistence is completed, the upper state machine is immediately notified for execution through pre_apply, and is not removed from the Raft log storage; On the Raft Leader, the command Cmd1 is applied in advance through pre_apply, and the normal Raft log entry is applied after the majority is persisted, and on the Follower in the Raft group, the log is applied in the order; When the state machine fails, rollback is performed on the command executed in pre_apply after the failure is recovered, wherein the rollback is processed through secondary on_apply; The rollback is processed through secondary on_apply, comprising: The state machine additionally records the information of the log entry in the pre_applied CF after the log is executed successfully.

5. The method for achieving tunable consistency of a storage system employing Raft protocol according to claim 4, wherein, Comprising: If the state machine does not fail, the entry that has been pre-applied is executed at the actual on_apply, and it is directly detected whether the record of the log entry exists in the pre_applied CF; If it does not exist, the state machine is executed in the normal process; If it exists, it is compared whether the log is the same; If the same, the pre_apply has been executed, no need to execute, update the committed version of the multi-version metadata information corresponding to the key to the current version number; If not the same, the current log has changed, execute rollback, temporarily close the pre_apply mechanism, and read the log and all pre-applied keys and versions from the pre_appliedCF. Delete all data from the data storage. After the rollback is completed, the pre_apply mechanism is restarted.

6. The method for achieving tunable consistency of a storage system employing Raft protocol according to claim 1, wherein, If the value of the write return point is 0, the write operation does not wait for any persistence and is executed after the log is submitted to the Raft replication engine. The read operation directly reads the state machine on any replica.

7. The method for achieving tunable consistency of a storage system employing Raft protocol according to claim 6, wherein, If the value of the write return point is -1 or quorum, all write requests are executed on the Raft leader and are executed according to the Raft protocol. The operation log is executed after being persisted in the majority. All read operations are performed on the Raft leader and use the Raft Lease Read.

8. The method for achieving tunable consistency of a storage system employing Raft protocol according to claim 6, wherein, If the value of the write return point is -1 or quorum, all write commands are executed on the Raft leader and are executed according to the Raft constraint. The read command is executed on any replica, and the state machine of the replica to be read has been applied to the log index number required by the read command.

9. The method for achieving tunable consistency of a storage system employing Raft protocol according to claim 7 or 8, characterized in that, If the value of the write return point is 1, the write operation is executed without linear consistency and without guarantee of persistence.

10. The method for achieving tunable consistency of a storage system employing Raft protocol according to claim 1, wherein, The value of the read return point consists of two fields, one of which is a position field and the other of which is a read mode; When the read mode is Lease Read and the position field is Leader, the read command is read from the TabletX state machine of the current Leader replica after checking that the Lease on the Leader is valid or has been successfully renewed. When the read mode is Wait Index Read and the position field is Leader or Follower, the TabletX state machine has applied the log index greater than the index required by the read command itself, and the read is performed on TabletX. Otherwise, wait for the state machine to apply the corresponding target log entry. When the read mode is Direct Read, no checking or special operation is performed, and the read is performed from the TabletX state machine.

Images