A scalable, consistent, and highly available distributed data system

Abstract

This invention discloses a scalable, consistent, and highly available distributed data system. The distributed data system includes clients, storage units, branch nodes, and a master node. The clients are used to write data, marking the data with a consistency level and attaching data information, where the levels are strong consistency, high availability, strong consistency priority, and extremely high availability. The storage units are used to write extremely high availability data to a local cache when the client experiences communication failures with the cluster. This scalable, consistent, and highly available distributed data system adds the Raft algorithm to the master node. After a branch node sends a write request to the current master node, if no response is received within a predetermined time, it degrades to high availability write. This allows the strong consistency priority mode to automatically degrade to high availability read / write when strong consistency cannot be maintained, reducing upper-layer application intervention and enabling automatic degradation.

Description

Technical Field

[0001] This invention relates to the field of distributed data system technology, and in particular to a distributed data system with scalability, consistency, and extreme high availability. Background Technology

[0002] Data systems categorize data into different consistency levels: strong consistency, high availability, strong consistency priority, and extreme high availability. This necessitates data designers accurately identifying data types and storing them in the appropriate CP or AP system at the right time. Existing system improvements, such as adjustable read / write consistency (allowing users to specify the consistency level during read / write operations), essentially integrate CP / AP into the same system. However, users still have to select a fixed consistency level during operations, resulting in the inability to automatically degrade. Summary of the Invention

[0003] The purpose of this invention is to provide a scalable, consistent, and highly available distributed data system to solve the problems mentioned in the background art.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a scalable, consistent, and highly available distributed data system, the distributed data system comprising a client, a storage unit, a support node, and a master node;

[0005] The client is used for writing data, marking the data with a consistency level, and attaching data information to the data. The levels are divided into strong consistency level, high availability level, strong consistency priority, and extreme high availability.

[0006] The storage unit is used to write highly available data to the local cache when the client experiences communication failure with the cluster.

[0007] The branch node is used to identify the consistency level, and when the data is at a high availability level, it executes the Gossip algorithm on the node to propagate the data and returns a successful data write to the client; when the data is at a high availability level and strong consistency is preferred, the branch node forwards the request to the master node for processing.

[0008] The master node uses the Raft algorithm to downgrade data with a strong consistency priority.

[0009] Preferably, the client includes a consistency level marking module, a packaging module, and an extreme high availability discrimination module.

[0010] Preferably, the consistency level marking module is used to distinguish the consistency level of data and to mark the data with a level, the packaging module is used to carry the IP address and local timestamp of the data, and the extreme high availability discrimination module is used to identify the data marked with the level and extract the extreme high availability data.

[0011] Preferably, the storage unit includes a receiving module, a local cache module, and a backup module.

[0012] Preferably, the receiving module is used to receive data information sent by the client, the local caching module is used to cache the highly available data locally, and the backup module is used to back up the data stored in the host.

[0013] Preferably, the branch node includes a consistency level identification module, a data differentiation module, a propagation module, a result response module, and a request module.

[0014] Preferably, the consistency level identification module is used to identify the data consistency level by marking the data entering and leaving the client branch node. The data differentiation module includes a high availability level section, a strong consistency level section, and a strong consistency priority section. The data differentiation module is used to differentiate the data of different identified levels according to the results.

[0015] Preferably, the propagation module is used to propagate data with a high availability consistency level on the node using the Gossip algorithm, the result reply module is used to send a message indicating that the data writing is complete, and the request module is used to forward data with a strong consistency level and strong consistency priority to the master node for processing.

[0016] Preferably, the master node includes a strong consistency priority processing module, a return module, a strong consistency data processing module, and a response module.

[0017] Preferably, the strong consistency priority processing module is used for the branch node to request Raft consensus from the master node first, and after timeout, it is downgraded to high availability write. The backhaul module is used to backhaul the strong consistency priority data downgraded to high availability level to the branch node. The strong consistency data processing module performs data consistency consensus through the Raft algorithm. If more than half of the consensuses are consistent, the data write is considered successful. The reply module is used to restore the data write completion information to the client.

[0018] The technical effects and advantages of this invention are as follows:

[0019] This scalable, consistent, and highly available distributed data system adds the Raft algorithm to the master node. When a branch node sends a write request to the current master node and does not receive a response within a predetermined time, it degrades to high-availability write. This allows the strong consistency priority mode to automatically degrade to high-availability read-write when strong consistency cannot be maintained, reducing upper-layer application intervention and enabling automatic degradation. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the distributed data system process of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0022] This invention provides, for example Figure 1 The diagram illustrates a scalable, consistent, and highly available distributed data system, which includes clients, storage units, branch nodes, and a master node.

[0023] The client is used by clients to write data, mark the data with a consistency level, and attach data information to the data. The levels are divided into strong consistency level, high availability level, strong consistency priority, and extreme high availability.

[0024] The storage unit is used to write highly available data to the local cache when the client experiences communication failure with the cluster;

[0025] The branch node is used to identify the consistency level. When the data is at a high availability level, it executes the Gossip algorithm on the node to propagate the data and returns a successful data write to the client. When the data is at a high availability level and strong consistency is preferred, the branch node forwards the request to the master node for processing.

[0026] The master node uses the Raft algorithm to demote data with a strong consistency priority.

[0027] It should be noted that in a distributed data system, data is stored in a distributed manner on different nodes, which can be physical servers, virtual machines, or containers. Distributed data systems provide mechanisms to manage the distribution and replication of data to ensure data consistency, reliability, and availability. Clients can access services from any node, thus enabling them to initiate data write and read requests. If, after evaluation, it is determined that data read and write operations use an ultra-high availability mode, the client registers with the cluster, and the cluster nodes use the Gossip algorithm to synchronize the latest data to the client's local cache.

[0028] Furthermore, a data write request is a request sent to a database or distributed system to write new data into the database; a data read request is a request sent to a database or distributed system to retrieve data and return it to the requester. Read requests are typically used to query data in a database to obtain the required information.

[0029] Specifically, the client includes a consistency level marking module, a packaging module, and an extreme high availability determination module.

[0030] Specifically, the consistency level marking module is used to distinguish the consistency level of data and mark the data with the level, the packaging module is used to carry the IP address and local timestamp of the data, and the extreme high availability discrimination module is used to identify the data marked with the level and extract the extreme high availability data.

[0031] It should be noted that the consistency level marking module is used to distinguish data consistency levels, which are divided into strong consistency, high availability, strong consistency priority, and extreme high availability. Different consistency levels are marked with characteristic identifiers to differentiate data of different consistency levels, thus enabling clients to mark data consistency levels and flexibly provide different data service levels within the same system. The packaging module carries the IP address and local timestamp of the data. The IP address refers to the IP address of the machine where the data was generated or sent. By carrying the local IP address, the source of the data can be determined, helping to identify which device generated the data. The local timestamp refers to the local time when the data was generated or sent, which is used to mark the exact time of data generation or transmission, helping to determine the time order and processing order of the data. The extreme high availability discrimination module is used to distinguish data as extremely high availability and extract it. If communication with the cluster fails, the data level is extremely high availability, and it is written to the local cache. When communication with the cluster is restored, the Gossip algorithm is executed to write the data to the cluster, which consists of multiple nodes.

[0032] Furthermore, when a client initiates a data write request, it first identifies the data consistency level, then packages the data's IP address and local timestamp. If the data consistency level is extremely high availability and the client-cluster communication is abnormal, the data level is extremely high availability, then the data is written to the local cache. After the client-cluster communication is restored, the Gossip algorithm is executed to write the data to the cluster.

[0033] Specifically, the storage unit includes a receiving module, a local cache module, and a backup module.

[0034] Specifically, the receiving module is used to receive data information sent by the client, the local caching module is used to cache the highly available data locally, and the backup module is used to back up the data stored in the host.

[0035] It should be noted that the storage unit is configured to receive data with an extremely high consistency level through the receiving module when communication between the client and the cluster is abnormal. The data is then stored through the local caching module. When communication between the client and the cluster resumes, the data is written to the cluster using the Gossip algorithm. This enables the client to establish a Gossip network with the cluster in the extremely high availability mode, allowing reading and writing to the local cache when communication with the cluster is unavailable. The backup module is used to back up the data stored locally and copy it to another device. Data backup is an important means of preventing data loss or damage. When the original data fails or is accidentally deleted, the backup data can be used to quickly find and restore the data, thereby avoiding data loss.

[0036] Specifically, the support nodes include a consistency level identification module, a data differentiation module, a propagation module, a result response module, and a request module.

[0037] Specifically, the consistency level identification module is used to identify the data consistency level by marking the data entering and leaving the client node. The data differentiation module includes a high availability level section, a strong consistency level section, and a strong consistency priority section. The data differentiation module is used to differentiate the data of different identified levels according to the results.

[0038] It should be noted that the consistency level identification module is used to identify the data consistency level marked by the consistency level marking module, and the data partitioning module is used to distinguish the data consistency level.

[0039] Specifically, the propagation module is used to propagate data with a high availability consistency level on the nodes using the Gossip algorithm, the result reply module is used to send a message indicating that the data writing is complete, and the request module is used to forward data with a strong consistency level and data with strong consistency priority to the master node for processing.

[0040] It should be noted that the Gossip algorithm is a distributed computing algorithm used for message passing and data synchronization. It is mainly used to handle large-scale data distribution and cluster management. The core idea of ​​this algorithm is to randomly select some nodes to pass information, and then have these nodes randomly select the next node to pass information within a specific time interval. In the end, all nodes will receive the same information, thereby achieving the purpose of data synchronization and consistency. This enables the propagation of highly available data with a consistency level to the branch nodes through the Gossip algorithm. After the propagation is completed, the result reply module sends a message to the user terminal that the data was successfully written.

[0041] Furthermore, during use, the branch node identifies the consistency level of the received data through the consistency level identification module. When the data consistency level is high availability, the node executes the Gossip algorithm for propagation. The propagation method has the following steps: First, the branch node determines the data to be propagated; second, it sends the selected data to other branch nodes; third, the branch node merges the received information with its own information and updates its own data. The data propagation is completed through these three steps. After the data propagation is completed, a write success message is sent to the client through the result reply module.

[0042] Specifically, the master node includes a strong consistency priority processing module, a backhaul module, a strong consistency data processing module, and a response module.

[0043] Specifically, the strong consistency priority processing module is used for the branch node to request Raft consensus from the master node first. After timeout, it is downgraded to high availability write. The backhaul module is used to backhaul the strong consistency preferred data downgraded to high availability to the branch node. The strong consistency data processing module performs data consistency consensus through the Raft algorithm. If more than half of them are consistent, the data is considered to have been written successfully. The reply module is used to restore the data write completion information to the client.

[0044] It should be noted that when the Raft algorithm is added to the master node, if a branch node does not receive a response within a predetermined time after sending a write request to the current master node, it will be downgraded to high-availability write. This allows the strong consistency priority mode to automatically downgrade to high-availability read-write when strong consistency cannot be maintained. The downgraded strong consistency priority data is then sent back to the branch node, where it is classified as high-availability data for further processing. The node forwards the request to the master node for processing. The master node performs data consistency consensus through the Raft algorithm. After achieving majority consensus, it returns a successful data write to the client.

[0045] Furthermore, when reading data, if the data consistency level is extremely high availability and the client communication with the cluster is abnormal, the data is read from the local cache.

[0046] When the data consistency level is high availability, the branch node queries the local data and returns the result.

[0047] When the data consistency level is strong consistency, the branch node will forward the request to the master node for processing, and the master node will query the data and return the result.

[0048] When the data consistency level is set to strong consistency priority, nodes prioritize querying data from the master node, and after a timeout, they are downgraded to high availability read.

[0049] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A scalable, consistent, and highly available distributed data system, characterized in that, The distributed data system includes a client, a storage unit, a branch node, and a master node; The client is used for writing data, marking the data with a consistency level, and attaching data information to the data. The levels are divided into strong consistency level, high availability level, strong consistency priority, and extreme high availability. The storage unit is used to write highly available data to the local cache when the client experiences communication failure with the cluster. The branch node is used to identify the consistency level, and when the data is at a high availability level, it executes the Gossip algorithm on the node to propagate the data and returns a successful data write to the client; when the data is at a high availability level and strong consistency is preferred, the branch node forwards the request to the master node for processing. The master node uses the Raft algorithm to downgrade data with a strong consistency priority. The master node includes a strong consistency priority processing module, a backhaul module, a strong consistency data processing module, and a reply module. The strong consistency priority processing module is used for branch nodes to prioritize requesting Raft consensus from the master node. After a timeout, the data is downgraded to high availability write. The backhaul module is used to backhaul the strongly consistent preferred data that has been downgraded to high availability back to the branch nodes. The strong consistency data processing module performs data consistency consensus through the Raft algorithm. Data is considered successfully written if more than half of the data are consistent. The reply module is used to restore the data write completion information to the client.

2. The scalable, consistent, and highly available distributed data system according to claim 1, characterized in that, The client includes a consistency level marking module, a packaging module, and an extreme high availability discrimination module.

3. The scalable, consistent, and highly available distributed data system according to claim 2, characterized in that, The consistency level marking module is used to distinguish the consistency level of data and mark the data with a level. The packaging module is used to carry the IP address and local timestamp of the data. The extreme high availability discrimination module is used to identify the data marked with the level and extract the extreme high availability data.

4. The scalable, consistent, and highly available distributed data system according to claim 1, characterized in that, The storage unit includes a receiving module, a local cache module, and a backup module.

5. A scalable, consistent, and highly available distributed data system according to claim 4, characterized in that, The receiving module is used to receive data information sent by the client, the local caching module is used to cache the highly available data locally, and the backup module is used to back up the data stored in the host.

6. The scalable, consistent, and highly available distributed data system according to claim 1, characterized in that, The branch nodes include a consistency level identification module, a data differentiation module, a propagation module, a result response module, and a request module.

7. A scalable, consistent, and highly available distributed data system according to claim 6, characterized in that, The consistency level identification module is used to identify the data consistency level by marking the data entering and leaving the client node. The data differentiation module includes a high availability level section, a strong consistency level section, and a strong consistency priority section. The data differentiation module is used to differentiate the data of different identified levels according to the results.

8. A scalable, consistent, and highly available distributed data system according to claim 7, characterized in that, The propagation module is used to propagate data with a high availability consistency level on the nodes using the Gossip algorithm. The result reply module is used to send a message indicating that the data writing is complete. After the data propagation is complete, the result reply module sends a write success message to the client. The request module is used to forward data with a strong consistency level and strong consistency priority to the master node for processing.

Images