A data processing method, device and storage medium

Abstract

The present application discloses a data processing method, which is applied to a data storage system, wherein the data storage system includes: at least two storage nodes, different storage nodes storing the same data; a leader node receiving a first processing request from a client; the leader node is any storage node among the at least two storage nodes; the leader node creates a log corresponding to the first processing request, and sends the log to a follower node, the log is used by the follower node to process the data in the follower node corresponding to the first processing request, the follower node is a storage node among the at least two storage nodes other than the leader node; the leader node uses a first storage engine to process the data stored in the leader node corresponding to the first processing request; the first storage engine is a storage engine in the leader node. The present application also discloses a device and a storage medium.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and relate to but are not limited to a data processing method and system, and a storage medium. Background Art

[0002] The HBase cluster intelligent switching component is used for automatic switching of HBase master and standby clusters and high-speed access to cluster metadata. The component can be deployed independently on multiple nodes, provides a Representational State Transfer REST (Application Programming Interface, API) interface, and combines with front-end load balancing to provide intelligent and fast switching services for large-scale HBase clusters.

[0003] In the related technology, the metadata access solutions adopted by the Hbase cluster intelligent switching component include: access based on the relational database MySQL, access based on the middleware distributed application coordination service (Zookeeper), and access based on the local file system, etc. However, the above solutions have problems such as difficult maintenance, single point failure, performance bottleneck, and the inability to guarantee data consistency. Summary of the invention

[0004] In view of this, the embodiments of the present application provide a data processing method and system, and a storage medium to solve at least one problem existing in the related art, which can avoid single point failures, facilitate maintenance, and ensure data consistency.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a data processing method, which is applied to a data storage system, wherein the data storage system comprises: at least two storage nodes, and the data stored in each node in the storage cluster is the same, and the method comprises:

[0007] A leader node receives a first processing request from a client; the leader node is any storage node among the at least two storage nodes;

[0008] The leader node creates a log corresponding to the first processing request, and sends the log to a follower node, wherein the log is used by the follower node to process the data in the follower node corresponding to the first processing request, and the follower node is a storage node other than the leader node among the at least two storage nodes;

[0009] The leader node uses a first storage engine to process the data stored in the leader node corresponding to the first processing request; the first storage engine is a storage engine in the leader node.

[0010] In a second aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned data processing method when executing the computer program.

[0011] In a third aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the steps in the above-mentioned data processing method when executed by a processor.

[0012] In an embodiment of the present application, a data processing method and device, and a storage medium are provided, which are applied to a data storage system, wherein the data storage system includes: at least two storage nodes, different storage nodes storing the same data; a leader node receiving a first processing request from a client; the leader node is any one of the at least two storage nodes; the leader node creates a log corresponding to the first processing request and sends the log to a follower node, the log is used by the follower node to perform processing corresponding to the first processing request on the data in the follower node, and the follower node is a storage node other than the leader node among the at least two storage nodes; the leader node uses a first storage engine to perform processing corresponding to the first processing request on the data stored in the leader node; the first storage engine is a storage engine in the leader node, and while processing the first processing request from the client, it ensures data consistency of the data storage system, thereby avoiding single point failures and improving the convenience of maintenance of the data storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a schematic diagram of an optional architecture of a data processing system according to an embodiment of the present application;

[0014] Figure 2 This is a schematic diagram of an optional architecture of a data processing system according to an embodiment of the present application;

[0015] Figure 3A This is a schematic diagram of an optional architecture of a data processing system according to an embodiment of the present application;

[0016] Figure 3B This is a schematic diagram of an optional architecture of a data processing system according to an embodiment of the present application;

[0017] Figure 4 An optional flowchart of a data processing method provided in an embodiment of the present application;

[0018] Figure 5 Schematic diagram of optional state transition of a storage node provided in an embodiment of the present application

[0019] Figure 6 An optional flowchart of a data processing method provided in an embodiment of the present application;

[0020] Figure 7 An optional flowchart of a data processing method provided in an embodiment of the present application;

[0021] Figure 8 An optional page diagram of a maintenance page provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the specific technical solution of the application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0023] The embodiments of the present application can be provided as a data processing method, system and storage medium. In practical applications, the data processing method can be implemented by a data processing system, and each functional entity in the data processing system can be collaboratively implemented by hardware resources of a computer device (such as a terminal device or a server), such as computing resources such as a processor, and communication resources (such as for supporting various communication methods such as optical cables and cellular).

[0024] Before describing the data processing method provided in the embodiments of the present application, some terms are explained first.

[0025] 1) Data, data stored in the data storage system, including: user data, metadata, and storage characteristics, etc.; user data includes: Hbase user data, Hbase client reported data; metadata, also known as intermediary data, relay data, is data about data, mainly describing data properties, used to support functions such as indicating storage location, historical data, resource search, file records, etc. Metadata includes: primary cluster name, backup cluster name, cluster configuration, zookeeper root node, zookeeper address, etc.; storage characteristics are used to describe the storage status of the data storage system, including: cluster health data, monitoring indicators, etc. Here, the storage nodes in the data storage system constitute a cluster.

[0026] 2) Data consistency: The cluster as a whole can provide the outside world with a unique and up-to-date data view like a single-point system. At this time, the data copies are strongly consistent, and the client returns the same result when accessing different nodes.

[0027] 3) Linear consistent read: After the update operation is completed, any subsequent request at any time point will obtain the latest updated value.

[0028] 4) Elastic expansion: When the number of nodes in the cluster increases, the cluster expansion can be completed by modifying the configuration files of each node in the cluster. In one example, 3 nodes are deployed to form a JRaftKV distributed storage cluster, and then 2 nodes are added. You only need to modify the configuration files of 3+2 nodes, and include the address and port configuration of the new node into the cluster communication address, and then restart the 5 nodes to complete the cluster expansion.

[0029] 5) Leader node: A storage node in the leader state. The leader node is elected from multiple storage nodes in a cluster through a consensus algorithm. If multiple shards (groups) are not deployed in the cluster, the default shard (group) is used. At this time, there is only one leader node in the cluster. If multiple shards (groups) are deployed in the cluster, there is only one node in each shard (group) as the leader node of the shard (group). The leader nodes of different shards (groups) can be the same or different.

[0030] The leader node is responsible for actively communicating with all follower nodes in the shard (group), sending "proposals" and receiving responses from the majority of follower nodes; the leader node needs to actively send heartbeats to all follower nodes to maintain its leadership position; the leader node is responsible for processing client read and write requests and replicating logs to all follower nodes.

[0031] The relationship between shards or shard groups and leader nodes is as follows:

[0032] Here we assume that a cluster consists of 3 nodes, and 10 shards (groups) are custom deployed, so that each shard (group) spans 3 nodes. Each shard (group) elects a node as the leader node through the consensus algorithm. In this way, there will be 10 leader nodes in the cluster at the same time. The leader node is generally load-balanced to 3 nodes, and of course, all loads can be placed on a designated node.

[0033] 6) Follower nodes: storage nodes in the follower state. Nodes other than the leader node in the cluster are follower nodes, or storage nodes where shards other than the leader node in a shard group are located are follower nodes. Follower nodes are passive. They do not actively send messages, but only respond to messages from storage nodes in the leader or candidate state, or forward client requests to the leader node. Here, storage nodes in the follower state are called follower nodes or follower nodes corresponding to the shard group.

[0034] 7) Candidate is a temporary state that can initiate voting. When a node receives enough votes, it is converted to leader. When it is determined that a cluster or a shard group already includes a leader, it is converted to follower.

[0035] The data processing method of the embodiment of the present application can be applied to Figure 1 The data processing system shown includes: a client 10 and a data storage system 20, wherein the client can interact with the user based on an input device, wherein the input device includes: a display, a mouse, a keyboard and other devices that can receive user input information, and the data storage system 20 includes at least two storage nodes 201, each storage node 201 stores data, and the data stored in each storage node 201 is the same, and each storage node stores data in key-value pairs, wherein all storage nodes in the data storage system 20 constitute a first storage cluster.

[0036] Among them, the client 10 receives the user's input operation and generates a data processing instruction based on the input operation; the client 10 calls the first interface corresponding to the data processing instruction provided by the data storage system 20; the client 10 sends the data processing instruction to the data storage system 20 through the called first interface; the data storage system 20 receives the data processing instruction.

[0037] In a data storage system, a leader node receives a first processing request from a client; the leader node is any one of the at least two storage nodes; the leader node creates a log corresponding to the first processing request and sends the log to a follower node, the log is used by the follower node to perform processing corresponding to the first processing request on data in the follower node, and the follower node is a storage node other than the leader node among the at least two storage nodes; the leader node uses a first storage engine to perform processing corresponding to the first processing request on data stored in the leader node; the first storage engine is the storage engine in the leader node.

[0038] In an embodiment of the present application, the data storage system stores data in a key-value pair format and in a table format. The table consists of rows and columns, and each row has a corresponding row key, i.e., a key, which is used to retrieve data.

[0039] In one example, the data storage system is a distributed storage system, and a lightweight storage engine is embedded in the node, wherein the lightweight storage engine may be RocksDB.

[0040] In one example, if Figure 2As shown, the data processing system further includes: at least two second storage clusters 30, at least two second storage clusters 30 include a switching component 301, the switching component 301 can perform active / standby switching of the second storage clusters 30, and can access metadata from the data storage system 20. The second storage cluster 30 can be an Hbase cluster.

[0041] In one example, if Figure 3A As shown, the data storage system 20 further includes a single-point database 40, wherein the single-point database 40 may be a relational database. The single-point database 40 may be used to store data and keep synchronized updates with the data in the data storage system.

[0042] In one example, if Figure 3B As shown, the data storage system 20 further includes: a control node 50, wherein the control node 50 is the central master control node of the data storage system 20 and is responsible for the scheduling of the first storage cluster composed of storage nodes. The control node 50 can perform the following processing: obtain and report the metadata and monitoring indicators of the first storage cluster through the event trigger mechanism; periodically detect the status of the leader node, automatically balance the number of leader nodes, prevent the leader nodes of all shards from piling up on individual storage nodes, and evenly distribute the cluster pressure of the first storage cluster.

[0043] In the embodiment of the present application, when the functions such as automatic balancing of the leader node or automatic slicing of slices are not needed, the control node may not be started. Here, the control node may schedule the first storage cluster through a placement driver (PD) service.

[0044] In actual applications, the PD service may be provided with replicas, and the replicas of the PD service and the PD service constitute a PD cluster.

[0045] In combination with the above data processing system, this embodiment proposes a data processing method, which can avoid single point failures and ensure data consistency while being easy to maintain.

[0046] Next, combine Figure 1 , Figure 2 , Figure 3A or Figure 3B The data processing system shown illustrates various embodiments of the data processing method, system and storage medium provided in the embodiments of the present application.

[0047] This embodiment provides a data processing method, which is applied to a data storage system, which may be a distributed network composed of computer devices. The function implemented by the method can be implemented by calling program codes by a processor in the computer device, and of course the program codes can be stored in a computer storage medium. It can be seen that the computer device at least includes a processor and a storage medium.

[0048] Figure 4 A schematic diagram of the implementation flow of a data processing method according to an embodiment of the present application is shown in FIG. Figure 4 As shown, the method may include the following steps:

[0049] S401: The leader node receives a first processing request from a client.

[0050] The leader node is any storage node among the at least two storage nodes.

[0051] The client receives an input operation from a user, and generates a first processing request based on the input operation;

[0052] The client can provide a maintenance page through a maintenance application or browser that maintains the data storage system, and an operation area for data is set in the maintenance page; wherein the operation area for data can be set with: a filter information input interface and a data processing type interface. The client generates a first processing request based on the first filter information received by the filter information input interface and the data processing type determined by the data processing type interface, and sends the first processing request to the data storage system through the first interface. The first interface is an interface between the client and the first processing request. The first interface can use a remote procedure call (RPC) protocol.

[0053] The first interface may be a native interface provided by the data storage system, or an interface extended by the user according to actual needs. The first interface may include: a start interface (start), a read interface (get), a write interface (put), a delete interface (delete), a scan interface (scan), etc. The embodiment of the present application does not impose any limitation on the interface provided by the data storage system.

[0054] In an embodiment of the present application, an operation area for different operation options is provided in the maintenance page, wherein the operation options may include: data, node, leader node, synchronization, etc.

[0055] Each operation area is provided with a processing type interface for the operation option. Optionally, the operation area is also provided with an interface for receiving filtering information.

[0056] For data, the filtering information may include at least one of the following: time, primary key, and the processing type may include: write, read, scan, delete, etc. For nodes, the filtering information may include at least one of the following: slice information and node information, and the processing type may include: obtain node information associated with the node, add a node, delete a node, etc. For leader nodes, the filtering information may include at least one of the following: slice information and node information, and the processing type may include: obtain information about the leader node, leader node switching, and balance the number of leader nodes in the data storage system, etc. For synchronization, the processing type may include: start, stop, etc.

[0057] For the node, the maintenance page also includes a slice quantity input interface to receive the slice quantity, and the corresponding processing type is: slice division.

[0058] In actual application, this application does not impose any restrictions on the operation options provided by the maintenance page in the client. The maintenance page provided by the client includes one or more pages.

[0059] When the user operates the maintenance page in the client, the client receives the user's input operation on the interface of the maintenance page, and generates a business request instruction according to the interface receiving the input operation and the content of the input operation. Here, the business request may include: a first processing request for data, a second processing request for a node, a third processing request for a leading node, and a data synchronization request for synchronization.

[0060] In one example, the operation option is data, and the processing type is acquisition, then a first processing request indicating acquisition of data is generated.

[0061] In the embodiment of the present application, the format of the service request meets the format requirements of the first interface, and the embodiment of the present application does not specifically limit the format of the service request.

[0062] The data storage system receives a first processing request sent by a client based on a first interface.

[0063] In one example, when a follower node in a data storage system receives a first processing request sent by a client, it does not process the first processing request itself but sends the first processing request to a leader node.

[0064] In one example, the data storage system also includes a control node, which serves as the central control node of the data storage system, receives the first processing request, determines the leading node, and sends the identifier of the leading node to the client, so that the client sends the first processing request to the leading node.

[0065] In some embodiments, a shard group is deployed in the data storage system, in which case the storage node includes at least two shards; the shards in different storage nodes are divided in the same way; the same shards in different storage nodes constitute a shard group; wherein the leader nodes of different shard groups are independent of each other; the data storage system further includes: a control node; the data processing method provided in the embodiment of the present application, before S401, further includes: the control node receives the first processing request, determines the target shard group to which the first processing request belongs; the control node determines the leader node of the target shard group, and sends the node representation of the leader node of the target shard group to the client to indicate to the client the leader node corresponding to the first processing request. The control node sends the first processing request to the leader node of the target shard group.

[0066] In one example, the nodes in the data storage system include: node 1, node 2 and node 3. The first processing request sent by the client is a data write request. The client first accesses the control node. The control node obtains the shard (region) where the key in the data write request is located according to the stored routing table, and determines that the leader node of the region is node 3. At this time, node 3, the leader node of the region in the data storage system, receives the data write request.

[0067] S402: The leader node creates a log corresponding to the first processing request, and sends the log to the follower node.

[0068] The log is used by the follower node to process the data in the follower node corresponding to the first processing request, and the follower node is a storage node other than the leader node among the at least two storage nodes.

[0069] After receiving the first processing request, the leader node creates a log and records the first processing request in the created log.

[0070] In the embodiment of the present application, the log also includes the following additional information: index value (index) and term number (Term). The index value is used to identify the log, and is a continuous, monotonically increasing integer number. One log corresponds to one index value. The term number is the term number of the leader who created this log entry.

[0071] After the leader creates the log, it synchronizes the log to the follower nodes in the data storage system.

[0072] Here, the leader can copy the created log, obtain the log copy, and send the log copy to the follower node.

[0073] In one example, the storage nodes in the data storage system include: node A, node B, node C, node D, node E and node F, wherein node E is the leader node, node E creates a log corresponding to the received first processing request, and synchronizes the log to node A, node B, node C, node D and node F respectively, so that node F performs the processing corresponding to the first processing request.

[0074] S403: The leader node uses the first storage engine to process the data stored in the leader node corresponding to the first processing request.

[0075] Each storage node in the data storage system uses a storage engine as a lightweight embedded storage library. The storage engine in the leader node is called the first storage engine, and the storage engine in the follower node is called the second storage engine. The storage engine used in the storage node can be RocksDB.

[0076] In the embodiment of the present application, the storage nodes in the data storage system store data in the form of key-value pairs, and the primary key is different for different scenarios. Rowkey includes: shard ID, business identifier, and record primary key hash. In one example, in a scenario with more reads and fewer writes, the primary key uses a fixed prefix, and in a scenario with less reads and more writes, the primary key is determined by hash inversion.

[0077] The leader node calls the first storage engine to process the stored data according to the first processing request. In one example, the first processing request is used to request to delete the data corresponding to key1, and the leader node calls the first storage engine to delete the data corresponding to key1 in the stored data. In one example, the data request is used to request to modify the data of key2 to value2, and the leader node calls the first storage engine to modify the data value1 corresponding to key2 in the stored data to value2.

[0078] In some embodiments, the implementation of S403 includes:

[0079] The first storage engine determines first filter information and data processing type corresponding to the first processing request; the first storage engine determines first target data according to the first filter information; the first storage engine performs processing corresponding to the data processing type on the first target data.

[0080] The leader node receives the first processing request and calls the first storage engine, which processes the first processing request. The first storage engine is an embedded, persistent storage, KV type storage engine that is very suitable for fast read access, such as RocksDB.

[0081] The first storage engine determines the first filtering information and data processing type carried by the first processing request, wherein the first filtering information represents the filtering condition of the first target data to be processed, wherein the first filtering information may include at least one of the following information: start date, end date, start primary key, end primary key. The data processing types include: deserialization data acquisition, serialization data acquisition, data writing, data deletion, etc.

[0082] When the data processing type is deserialized data acquisition, the leader node reconstructs the object through deserialization according to the object characteristic information in the first target data, and sends the reconstructed object to the client, wherein the object characteristic information includes: data status and description information.

[0083] Here, deserialization is to restore the data which is a byte sequence into a Java object. In one example, the data which is a byte sequence can be a protobuf object, and then the protobuf object is converted into a simple Java object (Plain Ordinary Java Objects, POJO object) through deserialization.

[0084] In actual applications, when deserialization fails, the leader node sends the first target data to the client in the form of a byte sequence.

[0085] In one example, the first filtering information includes a start date and an end date, and the data processing type is deserialized data acquisition. The first storage engine determines the target primary key corresponding to the time period between the start date and the end date based on the date of each primary key, searches for the first target data based on the target primary key, deserializes the first target data, obtains a serialized processing result, and sends the processing result to the client.

[0086] In one example, the first processing request indicates obtaining serialized data between time 1 and time 2, then the first storage engine obtains the data between time 1 and time 2, serializes the data between time 1 and time 2, and returns the serialization result to the client.

[0087] In one example, the first processing request indicates to delete the data between time 1 and time 2, then the first storage engine obtains the data between time 1 and time 2, and deletes the data between time 1 and time 2.

[0088] In one example, the first processing request indicates to add data, and the first storage engine writes the data.

[0089] In an embodiment of the present application, when adding data, the data storage system can first obtain the stored data and return the stored data to the client. The client adds the data to be added to the data returned by the data storage system to obtain new data and write the new data to the data storage system.

[0090] In the embodiment of the present application, the execution order of S402 and S401 is not particular.

[0091] In one example, after the leader node completes processing of the first processing request, it sends the log to the follower node.

[0092] In one example, after the leader node sends the log to the follower node, the leader node processes the first processing request.

[0093] After the leader node sends the log to the follower node, in the case of processing the first processing request, after receiving the log, the follower node determines whether to agree to process the first processing request. If it agrees, it returns a successful log reception response to the leader node to indicate that the leader node agrees to process the first processing request. When the leader node receives a successful log reception response from the follower node, and the number of follower nodes corresponding to the received successful log responses reaches the specified number, it is considered that a consensus has been reached between the leader node and the follower node to process the first processing request, and the first processing request is processed.

[0094] In an embodiment of the present application, a leading node receives a first processing request from a client; the leading node creates a log corresponding to the first processing request and sends the log to a follower node, wherein the log is used by the follower node to perform processing corresponding to the first processing request on data in the follower node, and the follower node is a storage node other than the leading node among the at least two storage nodes; the leading node uses a first storage engine to perform processing corresponding to the first processing request on data stored in the leading node; the first storage engine is a storage engine in the leading node, which ensures data consistency of the data storage system while processing the first processing request of the client, thereby avoiding single point failures and improving the convenience of maintenance of the data storage system.

[0095] In some embodiments, after receiving the log, the follower node processes the first processing request carried in the log to maintain data synchronization with the leader node.

[0096] The implementation of the follower node processing the first processing request includes: the follower node receives the log and calls the second storage engine; the second storage engine is the storage engine in the follower node; the second storage engine determines the first filtering information and data processing type corresponding to the first processing request according to the log; the second storage engine determines the second target data according to the first filtering information; the second storage engine performs the processing corresponding to the data processing type on the second target data.

[0097] The follower node receives the first processing request and calls the second storage engine, which processes the first processing request. The second storage engine is an embedded, persistent storage, KV type storage engine that is very suitable for fast read access, such as RocksDB.

[0098] The second storage engine determines the first processing request carried in the log, determines the first filtering information and data processing type carried in the first processing request, wherein the first filtering information represents the filtering condition of the second target data to be processed, wherein the first filtering information may include at least one of the following information: start date, end date, start primary key, end primary key. The data processing types include: deserialized data acquisition, serialized data acquisition, data writing, data deletion, etc.

[0099] In one example, the first processing request indicates to delete the data between time 1 and time 2, then the second storage engine obtains the data between time 1 and time 2, and deletes the data between time 1 and time 2.

[0100] In one example, the first processing request indicates to add data, and the second storage engine writes the data.

[0101] In actual applications, after processing the first processing request, the leader node sends a processing confirmation notification to the follower node; the follower node receives the processing confirmation notification for the first processing request sent by the storage node, and processes the data in the follower node based on the log. The processing confirmation notification can be a heartbeat sent by the leader node to the follower node, or a message sent by the leader node carrying the next log.

[0102] In some embodiments, Figure 3B As shown, the data storage system also includes: a control node.

[0103] The method also includes: the control node receives a second processing request; the control node determines second filtering information and a node processing type according to the second processing request; the control node determines a target storage node according to the second filtering information; and the control node performs processing corresponding to the node processing type on the target storage node.

[0104] The client generates a second processing request based on the user's input operation on the node in the maintenance page, and sends the second processing request to the control node through the interface between the client and the control node.

[0105] After receiving the second processing request, the control node determines the filtering information carried in the second processing request, namely, the second filtering information and the node processing type, and determines the target storage node according to the second filtering information.

[0106] In one example, the second filtering information includes one or more slice information, and the peer node, participant node, and surviving node of the slice corresponding to the slice information are used as the target storage node. The slice information may include a slice identifier.

[0107] In one example, the second filtering information includes: one or more node information, and the node corresponding to the node information is used as the target storage node. The node information may include address information such as an IP address and a port of the node.

[0108] The control node executes the processing type carried in the second processing request on the target storage node, namely the node processing type, wherein the node processing type includes: obtaining node information associated with the node, adding a node, deleting a node, etc.

[0109] When the node processing type includes: obtaining node information associated with a node, the control node obtains node information of a node associated with the target storage node, wherein the associated node may be a peer node, an observer node, or a surviving node.

[0110] When the node processing type includes: adding a node, the control node controls adding a target storage node in the data storage system.

[0111] When the node processing type includes: deleting a node, the control node controls the deletion of a target storage node in the data storage system.

[0112] In one example, the second processing request indicates obtaining node information associated with shard 1, the control node obtains node information of the peer node, observer node (learner node) and storage node of shard 1, and sends the obtained node information to the client.

[0113] In one example, the second processing request indicates obtaining node information associated with all shards, and the control node obtains node information of the peer node, observer node (learner node) and storage node of each shard respectively, and sends the obtained node information to the client.

[0114] In one example, the second processing request indicates to delete the node with the node address A, the control node determines the storage node with the node address A, and deletes the determined storage node.

[0115] In some embodiments, the node processing type includes: shard division, and the method also includes: the control node determines the target number of shards based on the second processing request; the control node schedules the number of shards in the target storage node to be the target number of shards; the target storage node includes the at least two storage nodes.

[0116] Here, in an embodiment of the present application, when the input information of the client input operation includes the target number of shards, the shards of the target storage node can be adjusted, and the number of shards of each target storage node is adjusted to the target number of shards. In one example, the target storage node is all the storage nodes in the data storage system. In another example, the target storage node is some of the storage nodes in the data storage system.

[0117] In some embodiments, in some embodiments, as Figure 3B As shown, the data storage system further includes: a control node. The method further includes: the control node receiving a third processing request; the control node determining third filtering information and a leader node processing type according to the third processing request; the control node determining a target leader node according to the third filtering information; and the control node performing processing corresponding to the leader node processing type on the target leader node.

[0118] The client generates a third processing request based on the user's input operation on the leader node in the maintenance page, and sends the third processing request to the control node through the interface between the client and the control node.

[0119] After receiving the third processing request, the control node determines the filtering information carried in the third processing request, ie, the third filtering information and the leader node processing type, and determines the target leader node according to the third filtering information.

[0120] In one example, the second filtering information includes one or more slice information, and the leader node of the slice corresponding to the slice information is used as the target leader node. The slice information may include a slice identifier.

[0121] In one example, the second filtering information includes: one or more node information, and the node corresponding to the node information is used as the target leader node. The node information may include address information such as the IP address and port of the node.

[0122] The control node executes the processing type carried in the third processing request on the target storage leader node, namely the leader node processing type. The leader node processing type includes: obtaining leader node information, leader node switching, leader node balancing, etc.

[0123] When the leader node processing type includes: obtaining leader node information, the control node obtains node information of the target leader node.

[0124] When the leader node processing type includes: leader node switching, the control node controls the leader node to be forcibly switched to the target leader node. Here, the leader nodes of all slice groups can be forcibly switched to the target leader node.

[0125] When the leader node processing type includes: leader node balancing, the control node balances the leader nodes of all shards.

[0126] In one example, the third processing instruction instructs to obtain the node information of the leader node associated with shard 1, the control node obtains the leader node of shard 1 as the shard of the leader node, and sends the number of shards to the client.

[0127] In one example, the third processing instruction is to forcibly switch the leader nodes of all shards to designated nodes, and the control node switches the master nodes of all shards to designated nodes.

[0128] In one example, the third processing instruction is to balance the nodes of all shards. The control node obtains the number of shards and nodes in the data storage system, calculates the number of leader nodes shared by the nodes, balances the number of leader nodes undertaken by each node, and modifies the information of the leader nodes of each shard in the routing table, thereby automatically balancing the number of leader nodes, preventing the leader nodes of all shard nodes from piling up on individual nodes, and evenly distributing cluster pressure.

[0129] In one example, the third processing instruction instructs to obtain the node information of all leader nodes, control the node to count the information of the leader nodes of each current shard, and send the information of the leader nodes of each shard to the client.

[0130] In one example, the third processing instruction instructs to count the number of shards that serve as leader nodes in the current node, control the node to obtain node information of all main leader nodes, count the information of the main nodes of each current shard, and send the information of the leader node of each shard to the client.

[0131] In some embodiments, Figure 2 As shown, the data storage system further includes: a single point database; the method further includes:

[0132] The leader node receives a data synchronization request;

[0133] The leader node controls the synchronization of data with the single-point database according to the data synchronization request.

[0134] In the embodiment of the present application, the data synchronization request may include: a start synchronization request and a stop synchronization request.

[0135] When the leader node receives a start synchronization request, it starts synchronizing data with the single-point database; when the leader node receives a stop synchronization request, it stops synchronizing data with the single-point database.

[0136] In the embodiment of the present application, when the data synchronization between the leader node and the single point database fails, the dual write request can be triggered again. When the data synchronization between the leader node and the single point database is successful, a synchronization mark can be inserted to avoid repeated unnecessary synchronization.

[0137] In an embodiment of the present application, an interface for starting or stopping dual writing is provided in the maintenance page. The user can trigger a dual writing request, i.e., a data synchronization request, based on the interface provided by the maintenance page, to synchronize data between the data storage system and the single-point database, thereby being compatible with the single-point database.

[0138] In some embodiments, the data storage system elects a leader node among at least two storage nodes included in the consensus algorithm. The first storage node enters an election state and sends a voting request to the second storage node; when the first storage node receives a first number of voting confirmations from the second storage node in response to the voting request, and determines that the first number is greater than a first number threshold, the first storage node enters a leadership state, wherein the first storage node in the leadership state is a leader node, and the second number threshold is related to the number of storage nodes.

[0139] In an embodiment of the present application, the storage nodes in the data storage system constitute a data storage cluster, namely a first storage cluster. The first storage cluster selects a leader node from the included storage nodes or the storage nodes corresponding to a shard group through a consensus algorithm.

[0140] In some embodiments, the first storage node broadcasts an election completion message to the second storage node; the election completion message indicates that the first storage node is the leader node; the second storage node switches from the election state to the follower state, and the second storage node in the follower state is a follower node.

[0141] In the embodiment of the present application, the state of the leading node is leader, and the state of the follower node is follower. Among them, taking the consensus algorithm as the Raft algorithm as an example, the state of the storage node is in one of the following three states: leader, follower, candidate. Under normal conditions, only one node is in the leader state, and the states of the remaining nodes are all followers. When the state of a node is follower, the node is passive, and they will not actively send messages, but only respond to messages from nodes in the leader or candidate state, or forward client requests to nodes in the leader state. Candidate is a temporary state.

[0142] In one example, the nodes in the first cluster include 10 nodes: node 1, node 2...node 10, and node 1 is elected as the leader through the consensus algorithm, then node 1 is the primary node, and the remaining nodes are auxiliary nodes with the status of followers.

[0143] The state transition diagram of the node is as follows Figure 5 As shown, each node is initialized as follower501 when joining the cluster. When there is no node in the leader502 state in the current cluster, follower501 will try to become leader502 through election. First, the node in the follower501 state changes its state to candidate503, and then initiates a voting request to other members in the cluster. If the node as candidate503 receives the majority of votes in favor, its state will become leader502, and then a heartbeat message will be broadcast in the cluster. Follower501 or other candidate503 that receives the heartbeat message will realize that there is already a master node in leader502 at this time, and will stop its election behavior and become follower501 again, and a stable cluster state will be formed.

[0144] When the data in the node is sharded, each shard is managed by an independent shard group, and the master nodes of each shard group are independent of each other. For example, the first storage cluster includes: Node 1, Node 2 and Node 3, and the data in the node is divided into three shards, then Node 1 includes shard 11, shard 12, and shard 13, Node 2 includes shards: shard 21, shard 22, and shard 23, and Node 3 includes shards: shard 31, shard 32, and shard 33, among which shard 11, shard 21 and shard 31 belong to the same shard group: shard group 1, and the master node of shard group 1 is node 1, shard 12, shard 22 and shard 32 belong to the same shard group: shard group 2, and the master node of shard group 2 is node 3, shard 13, shard 23 and shard 33 belong to the same shard group: shard group 3, and the master node of shard group 3 is node 2.

[0145] In some embodiments, the control node generates storage characteristics according to the stored data; the control node sends the storage characteristics to the client.

[0146] Here, the client can monitor the data storage system according to the storage characteristics sent by the control node.

[0147] In an embodiment of the present application, the data storage system generates storage characteristics that characterize the storage status based on the stored data and reports them to the client, so that the client can monitor the storage status of the data storage system, wherein the storage characteristics include: metadata, cluster node status, and performance data, etc.

[0148] In the embodiment of the present application, after receiving the storage characteristics, the client may monitor all or part of the storage characteristics, or send the storage characteristics to the monitoring background to analyze the status of the data storage system.

[0149] The data processing method provided in the embodiment of the present application takes the consensus algorithm as the core to realize a set of lightweight embedded distributed KV clusters, and the lightweight embedded distributed KV cluster has the characteristics of consistent access, easy operation, automatic monitoring, etc.

[0150] Below, taking the consensus algorithm as the Raft algorithm as an example, the data processing method provided in the embodiment of the present application is further explained.

[0151] In the related technology, HBase is a distributed, column-oriented open source database. The Hbase cluster intelligent switching component is used for automatic switching of Hbase master and standby clusters and high-speed access to cluster metadata. The component can be deployed independently on multiple nodes, provides a representational state transfer (REST) ​​API interface, and combines with front-end load balancing to provide intelligent and fast switching services for large-scale Hbase clusters. The metadata access solution currently used by the Hbase cluster intelligent switching component is as follows:

[0152] (1) Based on the relational database MySQL

[0153] Use the traditional relational database MySQL to provide Hbase cluster intelligent switching component metadata read and write services.

[0154] (2) Middleware-based distributed application coordination service (Zookeeper)

[0155] Use the Zookeeper cluster to provide Hbase cluster intelligent switching component metadata consistency read and write services.

[0156] (3) Based on local file system

[0157] Based on the local file system, it provides metadata reading and writing services by directly reading and writing local files.

[0158] The metadata access solution (1) has technical problems: the MySQL database must be deployed and maintained separately. The Hbase cluster intelligent switching component belongs to a lightweight application scenario and does not consider the deployment and maintenance of the MySQL cluster. Only a single point MySQL is deployed. However, a single point MySQL has a single point of failure and performance bottleneck.

[0159] The metadata access solution (2) has technical problems: the Zookeeper cluster must be deployed and maintained separately. In addition, Zookeeper provides cluster consistency coordination services, but it is obviously incapable of intelligently switching components in the Hbase cluster to read and write metadata with strong consistency. In addition, there are node data size limitations, bloated node trees, and performance bottlenecks.

[0160] The metadata access solution (3) has technical problems: data storage uses a file reading and writing method. It is difficult to organize and establish associations for multi-dimensional business metadata. It needs to be read and parsed line by line, and the performance cannot meet the requirements. In addition, the data consistency problem needs to be solved, otherwise it cannot meet the business scenarios and needs. Taking two-dimensional data as an example, two-dimensional tables are used to organize data, but all data cannot be placed in the same table, which will cause structural confusion, poor scalability, space waste and other problems. Therefore, the table design needs to meet the paradigm and be divided into tables. However, the division of tables will cause new problems. There needs to be a connection between tables, so the association is established through constraints. It is difficult to solve these problems by reading and writing files, not to mention higher-dimensional data. If all data is stored redundantly, the cost and efficiency will not meet the requirements.

[0161] The data processing method provided in the embodiment of the present application is used to solve the single point failure, performance bottleneck problem and the problem of distributed consistent access to metadata existing in the above-mentioned solution.

[0162] The data processing method provided in the embodiment of the present application is developed based on the JRaft lib library, uses RocksDB as a lightweight embedded storage library, and implements a lightweight and efficient distributed KV storage system through the Raft algorithm node election algorithm and the RPC communication framework. It supports distributed consistent access to metadata, supports multi-shard multi-node deployment, can be elastically expanded, solves the MySQL single point problem, and solves the data consistency problem through the Raft election algorithm. At the same time, it supports JRaftKV cluster monitoring indicator reporting and cluster convenient operation.

[0163] The data processing method provided in the embodiment of the present application involves the following aspects:

[0164] 1. JRaftKV node lifecycle management, node service start and stop are compatible with the original HBase cluster intelligent switching component service. First, start and stop the JRaftKV cluster node through the original HBase component start and stop command. According to the characteristics of Raft, an odd number of such nodes are generally deployed to form a cluster service. The lifecycle includes: starting the node service and stopping the node service. Among them, the node service of the JRaftKV node is started and stopped in the HBase process.

[0165] Start command demonstration: hbase-daemon.sh start component service name

[0166] 2. Custom storage driver, through the compatible development of the external API interface of the smart switching component service, there is no need to change the original MySQL interface logic. The underlying storage is mainly realized through the native interface call of RocksDB to quickly access metadata. In one example, the interface includes: read (get), write (put), scan (scan), etc.

[0167] 3. Customize the number of shards and support multi-shard deployment. At the same time, different storage strategies can be adopted according to different scenarios. For example, in the scenario with more reads and fewer writes, Rowkey uses a fixed prefix, and in the scenario with more reads and fewer writes, Rowkey uses hash+reversal.

[0168] In one example, the Rowkey consists of the following information: shard ID: business ID: record primary key hash.

[0169] 4. Automatically report KV cluster metadata by extending the JRaft lib library serial peripheral interface (SPI) interface, and report cluster metadata and cluster health indicators to the component background and monitoring system, where the monitoring system can be the Grafana system. At the same time, implement component background KV cluster operations, including support for manually triggering KV cluster and MySQL dual-write data synchronization operations, KV cluster data operations, KV cluster node operations, and one-click switching of leader nodes and manual balancing operations.

[0170] In one example, the cluster includes three nodes: node 61, node 62, and node 63. The startup and data consistency access process of the cluster is as follows: Figure 6 As shown, including:

[0171] S601, nodes in the cluster establish connections;

[0172] Among them, node 61 sends a connection request to node 62, node 62 sends a connection request to node 63, node 63 sends a connection response to node 62 to establish a connection, and node 62 sends a connection response to node 61 to establish a connection, thereby establishing a connection between node 61, node 62 and node 63.

[0173] S602, initialization of nodes in the cluster;

[0174] The HBase component sends a startup command to the nodes in the cluster. Each node starts and is initialized as a follower.

[0175] S603: The nodes in the cluster elect a leader.

[0176] The nodes in the cluster elect a leader from nodes 61, 62, and 63 through the consensus algorithm Raft.

[0177] The state transition of the node is as follows Figure 5As shown in the figure, when a node initiates an election based on the timeout mechanism, it changes its state from follower to candidate and initiates a voting request to other nodes. When the node receives votes from other nodes and receives votes from the majority of nodes, it changes its state from candidate to leader and then broadcasts a heartbeat message in the cluster. The node that receives the heartbeat message finds that there is already a leader in the cluster through the received heartbeat message, and then stops its own election behavior and changes its state from candidate to follower.

[0178] Among them, if the follower does not receive the heartbeat message from the leader for a long time, the election time is triggered and the follower is transformed into a candidate, the Term is increased by one, and then the new Term and its latest log sequence number are broadcasted in order to obtain votes.

[0179] Other nodes that receive voting requests first filter out requests with a term smaller than their own, and then make the following judgments: 1. Whether they have voted; 2. Whether they think there is a leader now; 3. Whether the log index of the voting request is larger than their own. If all judgments are passed, they vote in favor.

[0180] The candidate that receives the approval votes from more than half of the nodes will become the leader and start sending heartbeat messages regularly to suppress the election behavior of other nodes.

[0181] The voting condition is used to ensure the correctness (uniqueness) of the leader election, and the random election time is used to ensure the efficiency of the election. Because the cluster cannot provide external services when it is in the election state, the duration of this state should be shortened as much as possible to minimize the probability of election failure.

[0182] In the embodiment of the present application, through the above election, node 62 is elected as the leader of the cluster.

[0183] S604: The leader of the cluster receives the data write request and performs the data write operation.

[0184] The leader node 62 of the cluster receives the data write request of the cluster, performs the write operation in the node 62, and generates log data.

[0185] S605: The leader node in the cluster and the follower node in the cluster perform data synchronization.

[0186] The leader in the cluster copies the log and sends the copied log to other nodes acting as followers. The follower nodes synchronize data based on the copied log, so that the data of each follower node and the leader node are consistent, thereby achieving data consistency in the cluster.

[0187] exist Figure 6 In the example, a JRaftKV storage cluster consisting of three nodes is used to describe the data processing method provided by the embodiment of the present application. After each node is started, a channel is established to communicate with each other. The initial state of the node after startup is Follower. An election is initiated through a timeout mechanism, and the election round term is recorded, where the election round term is monotonically increasing. The leader is selected through the Raft majority election algorithm, and the leader node is responsible for processing client requests and copying logs to all Followers.

[0188] The data processing system provided in the embodiment of the present application is as follows: Figure 7 As shown, including:

[0189] Client 71 , PD cluster 72 and storage cluster 73 , wherein PD cluster 72 includes: PD1 , PD2 , PD3 , and storage cluster 73 includes node 61 , node 62 and node 63 .

[0190] Client 71 sends a request to refresh the routing table to PD cluster 72. PD cluster 72 obtains the data shard 1 (region1) where the key is located according to the routing table. Assume that the leader of region 1 group is selected by the raft election algorithm as node 63, and nodes 61 and 62 are followers. Node 63 accepts and executes the put request indicating data writing. The raft algorithm realizes strong data consistency. The data is successfully synchronized from the leader to all followers before returning success.

[0191] Among them, the shard in the leader node in the same shard group in the node processes the client's request and synchronizes the request result to other shards in the shard group through log replication.

[0192] In the embodiment of the present application, the PD service provided by the PD mainly provides the following functions:

[0193] As the global central control node, it is responsible for the scheduling of the entire cluster. It does not require self-management and is independent of dependencies. That is, if functions such as automatic balancing and split are not required, the PD service does not need to be started.

[0194] Acquire and report cluster metadata and monitoring indicators through event triggering mechanisms;

[0195] Periodically detect the leader node status and automatically balance the number of leader nodes to prevent the leaders of all shard nodes from piling up on individual servers and evenly distribute cluster pressure.

[0196] The automatic balancing process includes: obtaining the number of cluster shards and nodes, and calculating the number of leaders shared by nodes. Obtaining the metadata of the current shard in the routing table, modifying the leader node in the metadata; re-refreshing the shard routing table, obtaining the shard leader node information, if it is empty (unsuccessful switching), the switching fails; calculating the number of rowkeys stored in the cluster shard, and performing a split operation on the shard (group) when the threshold is reached.

[0197] Figure 8 A schematic diagram of the page effect of the maintenance page provided in the embodiment of the present application, wherein page 800 includes areas 801, 802, 803, and 804. The operation option of area 801 is data storage system, the operation option of area 802 is data, the operation option of area 803 is node, and the operation option of area 804 is leader node.

[0198] The operation interfaces provided in area 801 include: Get, Start, Stop, among which Start / Stop: is used to trigger the synchronization job, insert a synchronization mark after the synchronization is successful to prevent repeated execution, and can be triggered again for failed jobs. Get: is used to obtain the current synchronization status.

[0199] The operation interfaces provided in area 802 include: input interfaces of parameters StartDate, EndDate, StartKey / Key, EndKey / Value, Get Deserial, Get Serial, Add Serial / Key, and Delete.

[0200] in,

[0201] Parameters StartDate and EndDate: indicate the time range and support date filtering by including timestamps in the data storage rowkey.

[0202] Parameters StartKey and EndKey: Get partition data based on rowkey, and support fuzzy matching by default.

[0203] Parameters Key, Value: support kv format data storage.

[0204] Get Deserial: Get deserialized data. If deserialization fails, serialized data is returned. Date filtering is supported.

[0205] Get Serial: Get serialized data, supporting date filtering.

[0206] Add Serial: kv data storage, corresponding to Get Serial serialized data, and also supports String type data. To prevent data misoperation, it is recommended to first obtain and save data through Get Serial, and then roll back the inserted data.

[0207] Delete: supports deletion based on the starting rowkey range, which is highly efficient but cannot be rolled back, so it is not recommended. It supports date filtering and deletion. It is less efficient and can be rolled back based on time points and log information. It is recommended to add date conditions. The log will record the records deleted at detailed time points, so as to support rollback. Deleted records cannot be rolled back without date conditions.

[0208] The operation interface provided in region 803 includes: input interface of parameters RegionId, ServerAddress (ip:port[ / learner]), Get, Add, and Remove.

[0209] Parameter RegionId: Shard ID, multiple shards separated by commas

[0210] Parameter ServerAddress(ip:port[ / learner]): node address, supports learner nodes, separated by commas

[0211] Get: Get the peer nodes, learner nodes, and surviving nodes of all shards.

[0212] Add, Remove: add or delete nodes.

[0213] The operation interfaces provided in region 804 include: parameter RegionId, ServerAddress (ip:port) input interface, Get, Transfer, Rebalance.

[0214] Parameter RegionId: Shard ID, multiple shards separated by commas

[0215] Parameter ServerAddress(ip:port): node address, separated by English commas.

[0216] Get: Get the number of leaders of the current node and the leader information of all shards, supporting conditional filtering.

[0217] Transfer: Forcefully switch (all) shard leaders to the specified node.

[0218] Rebalance: Manually balance all shard leader nodes.

[0219] In the embodiment of the present application, the HBase cluster intelligent switching component service uses a customized storage driver for dual writing (JRaftKV cluster, single-point MySQL) and customized 10 shards, and the Rowkey uses a fixed prefix to support elastic expansion of the number of deployed nodes and data consistency access, which can solve the bottleneck of the original MySQL single-point solution, and support cluster monitoring reporting and background operation functions, including data synchronization operations, KV cluster data operations, KV cluster node operations, and one-click switching of leader nodes and manual balancing operations.

[0220] Figure 1 A schematic diagram of the implementation structure of a data storage system according to an embodiment of the present application is shown in FIG. Figure 1 As shown, it includes: at least two storage nodes, different storage nodes store the same data, and the at least two storage nodes include a leader node;

[0221] The leader node receives a first processing request from a client; the leader node is any storage node among the at least two storage nodes;

[0222] The leader node creates a log corresponding to the first processing request and sends the log to a follower node, where the log is used by the follower node to process the data in the follower node corresponding to the first processing request, and the follower node is a storage node other than the leader node among the at least two storage nodes;

[0223] The leader node uses a first storage engine to process the data stored in the leader node corresponding to the first processing request; the first storage engine is a storage engine in the leader node.

[0224] In some embodiments, the storage node includes at least two shards; the shards in different storage nodes are divided in the same way; the same shards in different storage nodes constitute a shard group; wherein the leader nodes of different shard groups are independent of each other; the data storage system further includes: a control node;

[0225] The control node receives the first processing request and determines a target shard group to which the first processing request belongs;

[0226] The control node determines a leader node of the target shard group, and sends a node representation of the leader node of the target shard group to the client to indicate to the client the leader node corresponding to the first processing request.

[0227] In some embodiments, the first storage engine determines first filter information and data processing type corresponding to the first processing request; the first storage engine determines first target data based on the first filter information; and the first storage engine performs processing corresponding to the data processing type on the first target data.

[0228] In some embodiments, the follower node receives the log and calls the second storage engine; the second storage engine is a storage engine in the follower node;

[0229] The second storage engine determines, according to the log, first filtering information and a data processing type corresponding to the first processing request;

[0230] The second storage engine determines second target data according to the first filtering information;

[0231] The second storage engine performs processing corresponding to the data processing type on the second target data.

[0232] In some embodiments, the data storage system further comprises: a control node;

[0233] The control node receives a second processing request;

[0234] The control node determines second filtering information and a node processing type according to the second processing request;

[0235] The control node determines a target storage node according to the second filtering information;

[0236] The control node performs a process corresponding to the node processing type on the target storage node.

[0237] In some embodiments, when the node processing type includes: sharding;

[0238] The control node determines the target number of shards according to the second processing request;

[0239] The control node schedules the number of shards in the target storage node to be the target number of shards; the target storage node includes the at least two storage nodes.

[0240] In some embodiments, the data storage system further includes: a control node, and the method further includes:

[0241] The control node receives a third processing request;

[0242] The control node determines third filtering information and a leader node processing type according to the third processing request;

[0243] The control node determines the target leader node according to the third filtering information;

[0244] The control node performs a process corresponding to the leader node processing type on the target leader node.

[0245] In some embodiments, the data storage system further comprises: a single-site database;

[0246] The leader node receives a data synchronization request;

[0247] The leader node controls the synchronization of data with the single-point database according to the data synchronization request.

[0248] In some embodiments, the first storage node enters an election state and sends a voting request to the second storage node;

[0249] When the first storage node receives a first number of voting confirmations from the second storage node in response to the voting request, and determines that the first number is greater than a first number threshold, it enters a leadership state, wherein the first storage node in the leadership state is a leadership node, and the second number threshold is related to the number of the storage nodes.

[0250] In some embodiments, the first storage node broadcasts an election completion message to the second storage node; the election completion message indicates that the first storage node is the leader node;

[0251] The second storage node switches from the election state to the follower state, and the second storage node in the follower state is a follower node.

[0252] It should be noted that the data processing system provided in the embodiment of the present application includes various logic units included, which can be implemented by a processor in an electronic device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU, Central Processing Unit), a microprocessor (MPU, MicroProcessor Unit), a digital signal processor (DSP, Digital Signal Processor) or a field programmable gate array (FPGA, Field-Programmable Gate Array), etc.

[0253] The description of the above system embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the system embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0254] It should be noted that in the embodiment of the present application, if the above-mentioned data processing method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0255] An embodiment of the present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the above-mentioned data processing method when executing the computer program.

[0256] Correspondingly, an embodiment of the present application provides a storage medium, that is, a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the data processing method provided in the above embodiment is implemented.

[0257] It should be noted here that the description of the above storage medium embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the storage medium embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0258] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in some embodiments" appearing throughout the specification may not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The above-mentioned sequence numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments.

[0259] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0260] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0261] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0262] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0263] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0264] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0265] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data processing method, characterized in that: Applied to a data storage system, the data storage system comprises: at least two storage nodes, different storage nodes store the same data, the method comprises: A leader node receives a first processing request from a client; the leader node is any storage node among the at least two storage nodes; The leader node creates a log corresponding to the first processing request, and sends the log to a follower node, wherein the log is used by the follower node to process the data in the follower node corresponding to the first processing request, and the follower node is a storage node other than the leader node among the at least two storage nodes; The leader node uses a first storage engine to process the data stored in the leader node corresponding to the first processing request; the first storage engine is a storage engine in the leader node; The storage node includes at least two shards; the shards in different storage nodes are divided in the same way; the same shards in different storage nodes constitute a shard group; the leader nodes of different shard groups are independent of each other; the data storage system further includes: a control node; the method further includes: The control node receives the first processing request and determines a target shard group to which the first processing request belongs; The control node determines a leader node of the target shard group, and sends an identification representation of the leader node of the target shard group to the client to indicate to the client the leader node corresponding to the first processing request.

2. The method according to claim 1, characterized in that The leader node uses the first storage engine to process the data stored in the leader node corresponding to the first processing request, including: The first storage engine determines first filtering information and a data processing type corresponding to the first processing request; The first storage engine determines first target data according to the first filtering information; The first storage engine performs processing corresponding to the data processing type on the first target data.

3. The method according to claim 1, characterized in that The method further comprises: The follower node receives the log and calls a second storage engine; the second storage engine is a storage engine in the follower node; The second storage engine determines, according to the log, first filtering information and a data processing type corresponding to the first processing request; The second storage engine determines second target data according to the first filtering information; The second storage engine performs processing corresponding to the data processing type on the second target data.

4. The method according to claim 1, characterized in that: The data storage system further includes: a control node, and the method further includes: The control node receives a second processing request; The control node determines second filtering information and a node processing type according to the second processing request; The control node determines a target storage node according to the second filtering information; The control node performs a process corresponding to the node processing type on the target storage node.

5. The method according to claim 4, characterized in that When the node processing type includes: sharding, the method further includes: The control node determines the target number of shards according to the second processing request; The control node schedules the number of shards in the target storage node to be the target number of shards; the target storage node includes the at least two storage nodes.

6. The method according to claim 1, characterized in that The data storage system further includes: a control node, and the method further includes: The control node receives a third processing request; The control node determines third filtering information and a leader node processing type according to the third processing request; The control node determines the target leader node according to the third filtering information; The control node performs a process corresponding to the leader node processing type on the target leader node.

7. The method according to claim 1, characterized in that The data storage system further includes: a single point database; the method further includes: The leader node receives a data synchronization request; The leader node controls the synchronization of data with the single-point database according to the data synchronization request.

8. The method according to claim 1, characterized in that The method further comprises: The first storage node enters the election state and sends a voting request to the second storage node; When the first storage node receives a first number of voting confirmations from the second storage node in response to the voting request, and determines that the first number is greater than a first number threshold, it enters a leadership state, wherein the first storage node in the leadership state is a leadership node, and the first number threshold is related to the number of the storage nodes.

9. The method according to claim 8, characterized in that The method further comprises: The first storage node broadcasts an election completion message to the second storage node; the election completion message indicates that the first storage node is the leader node; The second storage node switches from the election state to the follower state, and the second storage node in the follower state is a follower node.

10. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the data processing method according to any one of claims 1 to 9 when executing the computer program.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data processing method according to any one of claims 1 to 9 is implemented.

Images