Distributed cluster management method, device and electronic equipment

Abstract

The embodiments of the present application provide a distributed cluster management method, device and electronic device. In this embodiment, for each service, the service cluster corresponding to the service is determined from the N control nodes of the control cluster, and the master service node and slave service node of the service are selected from the service cluster. The event permissions belonging to the service processed by the service instance corresponding to the service on the master service node are different from the event permissions belonging to the service processed by the service instance corresponding to the service on the slave service node. In this way, the service events corresponding to each service are processed by the service nodes in the service cluster corresponding to the service, avoiding the problems of high node CPU overhead and low event processing efficiency caused by the existing situation that the service events corresponding to each service are all processed by one master control node, and effectively improving the event processing efficiency.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a distributed cluster management method, device, and electronic device. Background Art

[0002] In a distributed cluster, to detect faults and collect status for services on storage nodes in the storage cluster, each storage node periodically reports service events to the control node in the control cluster. Service events are events associated with a service (such as an object storage service), and may include, for example, events corresponding to read and write requests for that service. Currently, in practical applications, a control node is typically pre-selected from the control cluster as the master control node (which processes received service events), while other nodes serve as slave control nodes (which process events for which they have event processing privileges and report events for which they do not have event processing privileges to the master control node).

[0003] However, when the scale of the above-mentioned storage cluster is large (that is, when the number of storage nodes is large), the service events received and processed by the main control node will increase linearly, which will cause the central processing unit (CPU) overhead of the main control node to increase. In this case, due to the limited CPU resources of the main control node, it is unable to process a large number of service events in a timely manner, thereby reducing the processing efficiency of service events. Summary of the Invention

[0004] In view of this, the present application provides a distributed cluster management method, apparatus, and electronic device to improve the processing efficiency of service events.

[0005] The present invention provides a distributed cluster management method, wherein the distributed cluster includes at least a storage cluster, a management cluster, and a control cluster; the management cluster includes multiple management nodes, which are used to configure and manage each node in the distributed cluster; the control cluster includes multiple control nodes, which are used to process service events reported by each storage node in the storage cluster and control the node working status of each storage node; the method is applied to any management node in the management cluster; the method includes:

[0006] Create a service instance corresponding to each service for each control node in the control cluster; a service instance of any service is used to handle events under the service;

[0007] For each service, a service cluster corresponding to the service is determined from the N control nodes in the control cluster, and the service cluster includes at least one control node; one control node is selected from the service cluster as the master service node of the service, and the remaining control nodes are used as slave service nodes of the service; wherein, the service instance corresponding to the service on the master service node and the service instance corresponding to the service on the slave service node have different permissions to process events belonging to the service; when the service instance corresponding to the service on the slave service node receives an event belonging to the service, if it does not have the permission to process the event, it reports the event to the service instance corresponding to the service on the master service node for processing.

[0008] The present application also provides a distributed cluster management device, wherein the distributed cluster includes at least a storage cluster, a management cluster, and a control cluster; the management cluster includes multiple management nodes, which are used to configure and manage each node in the distributed cluster; the control cluster includes multiple control nodes, which are used to process service events reported by each storage node in the storage cluster and control the node working status of each storage node; the device is configured at any management node in the management cluster; the device includes:

[0009] The creation module is used to create service instances corresponding to each service for each control node in the control cluster; the service instance of any service is used to handle various events under the service;

[0010] A determination module is used to determine, for each service, a service cluster corresponding to the service from N control nodes in a control cluster, wherein the service cluster includes at least one control node; select one control node from the service cluster as the master service node of the service, and the remaining control nodes as slave service nodes of the service; wherein the service instance corresponding to the service on the master service node and the service instance corresponding to the service on the slave service node have different permissions for processing events belonging to the service; when the service instance corresponding to the service on the slave service node receives an event belonging to the service, if it does not have the permission to process the event, it reports the event to the service instance corresponding to the service on the master service node for processing.

[0011] Optionally, determining a service cluster corresponding to the service from N control nodes in the control cluster includes:

[0012] For each service, a control node that meets the set requirements is selected from the N control nodes in the control cluster; the control node that meets the set requirements means that the first usage count of the control node is less than the first usage count of other control nodes in the N control nodes;

[0013] If the number X of control nodes that meet the set requirements is less than the number M of control nodes in the service cluster corresponding to the configured service, then P control nodes are selected from the remaining control nodes based on the node identifiers of the remaining control nodes; the remaining control nodes are the nodes in the N control nodes excluding the control nodes that meet the set requirements; P is the difference between M and X; the service cluster corresponding to the service is constructed based on the control nodes that meet the set requirements and the P control nodes;

[0014] If X is equal to M, a service cluster corresponding to the service is built based on the control nodes that meet the set requirements;

[0015] If X is greater than M, select M control nodes from the control nodes that meet the set requirements, and build a service cluster corresponding to the service based on the selected M control nodes;

[0016] After determining the service cluster corresponding to the service from the N control nodes in the control cluster, the determination module is further configured to:

[0017] The first usage count of the control node selected for constructing the service cluster corresponding to the service is increased by a set value.

[0018] Optionally, the node identifier is a node number;

[0019] Selecting P control nodes from the remaining control nodes according to the node identifiers of the remaining control nodes includes:

[0020] The first P remaining control nodes with the largest node numbers are selected from the remaining control nodes as the selected P control nodes.

[0021] Optionally, select a control node from the service cluster as the primary service node for the service, including:

[0022] Select the control node with the second smallest number of uses from the service cluster as the primary service node for the service;

[0023] After selecting a control node from the service cluster as the primary service node for the service, a module is determined, further configured to:

[0024] The second usage count corresponding to the control node selected as the primary service node of the service is increased by a set value.

[0025] Optionally, the event permissions for the service processed by the service instance corresponding to the service on the primary service node include: permissions for events corresponding to read requests and permissions for events corresponding to write requests under the service;

[0026] The event permissions for the service processed by the service instance corresponding to the service on the service node include: permissions for events corresponding to read requests under the service.

[0027] Optionally, when receiving an event corresponding to a write request under the service, the service instance corresponding to the service on the master service node processes the event to obtain an event processing result, and sends the event processing result to the slave service node of the service; the service instance corresponding to the service on the master service node is further configured to cache the event processing result in a preconfigured cache space;

[0028] The service instance corresponding to the service on the slave service node is used to cache the event processing result under the service into a pre-configured cache space when receiving the event processing result;

[0029] Create modules, which are further used to:

[0030] For each service, a storage instance is created on each control node in the service cluster corresponding to the service; wherein the storage instance corresponding to the service is used to persistently store the event processing results under the service.

[0031] Optionally, for each service, when there are control nodes other than the service cluster corresponding to the service in the control cluster, the service instance corresponding to the service on any control node in the service cluster corresponding to the service is further used to send the latest event processing result to one of the control nodes other than the service cluster corresponding to the service; the service instance corresponding to the service on the control node other than the service cluster corresponding to the service is used to cache the event processing result under the service in a preconfigured cache space upon receiving the event processing result;

[0032] The service instance corresponding to the service on the control node other than the service cluster corresponding to the service and the service instance corresponding to the service on the primary service node corresponding to the service have different permissions for handling events belonging to the service;

[0033] When a service instance corresponding to a service on a control node other than the service cluster corresponding to the service receives an event belonging to the service, if it does not have the authority to handle the event, it reports the event to the service instance corresponding to the service on the primary service node corresponding to the service for processing.

[0034] An embodiment of the present application further provides an electronic device, comprising:

[0035] processor; and

[0036] A memory stores computer program instructions, which, when executed by a processor, cause the processor to execute the steps of the above method.

[0037] An embodiment of the present application further provides a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the processor is caused to execute the steps in the above method.

[0038] It can be seen from the above technical solutions that the embodiment of the present application determines the service cluster corresponding to each service from the N control nodes of the control cluster, and selects the master service node and slave service node of the service from the service cluster. The event permissions belonging to the service processed by the service instance corresponding to the service on the master service node are different from the event permissions belonging to the service processed by the service instance corresponding to the service on the slave service node. In this way, the service events corresponding to each service are processed by the service nodes in the service cluster corresponding to the service, avoiding the problems of high node CPU overhead and low event processing efficiency caused by the existing situation that the service events corresponding to each service are all processed by one master control node, and effectively improving the event processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0040] Figure 1 A schematic diagram of a distributed cluster provided in an embodiment of the present application.

[0041] Figure 2 A schematic diagram of the service instance layout provided in an embodiment of the present application.

[0042] Figure 3 A flow chart of the method provided in an embodiment of the present application.

[0043] Figure 4 A schematic diagram of a control cluster provided in an embodiment of the present application.

[0044] Figure 5 A schematic diagram of a service cluster provided in an embodiment of the present application.

[0045] Figure 6 This is a diagram of the device structure provided in an embodiment of the present application.

[0046] Figure 7 A schematic diagram of the electronic device structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] To facilitate understanding of this solution, before describing this solution, we first describe the technical problems existing in existing distributed storage clusters:

[0048] See also Figure 1As shown, the distributed cluster 100 includes at least a management cluster 101, a control cluster 102, and a storage cluster 103. The management cluster includes multiple management nodes, which are used to configure and manage each node in the distributed cluster. For example, the management node can configure node configuration information (such as authentication, communication protocols, port configuration, and the service cluster configuration described below in this embodiment, etc.) for the connected control node and the storage node connected through the control node, as well as manage node resource allocation (such as CPU resources, memory resources, etc.). Optionally, the management nodes can also periodically synchronize cluster management and configuration related information.

[0049] The control cluster includes multiple control nodes, which are responsible for processing service events reported by each storage node in the storage cluster and controlling the node operating status of each storage node. Here, multiple services are deployed on any storage node, such as the Object Storage Device (OSD) service and the Metadata Server (MDS) service. Therefore, service events refer to events associated with the corresponding service, such as events indicating the service's operating status or fault status, and are not specifically limited here.

[0050] For example, as an embodiment, when the control node receives a service event reported by a node, if the service event indicates that a service of the node (such as an OSD service) has an operational failure, the control node will generate a corresponding control message based on the service event and send it to each storage node. The control message indicates that the node indicated by the service event should be isolated.

[0051] The storage cluster consists of multiple storage nodes, which are used to store data, process data read or write requests, etc. The storage nodes regularly report the service status of each service to the corresponding management node in the form of service events.

[0052] It should be noted that this embodiment does not specifically limit the work content of each node in the management cluster, control cluster, and storage cluster, and can be flexibly configured based on actual needs.

[0053] As an example, an odd number of control nodes (also called monitor nodes) form a control cluster. Within the control cluster, a corresponding election algorithm is used to select one monitor node as the master monitor node, and the others as slave monitor nodes. Several storage nodes and the control cluster together form a distributed cluster.

[0054] In order to realize fault detection and status collection of storage nodes, each storage node will periodically send its own status to any corresponding monitor node in the form of a heartbeat message (such as a TCP data packet or an IP data packet, etc.) (for example, storage node 3 can periodically send its own status to the slave monitor node 1 in the form of a heartbeat message); among them, if the message recipient is a slave monitor node, it will also forward the message to the master monitor node for processing.

[0055] When the scale of the distributed storage cluster is large, that is, when the number of storage nodes is large, the messages received and processed by the main monitor node will increase linearly, and the CPU overhead of the main monitor node will increase. However, the CPU cores allocated to a single monitor node (including the main monitor node) are limited and cannot process a large number of messages in a timely manner. This will cause problems such as control message processing timeouts and cluster status maintenance abnormalities.

[0056] See also Figure 2 As shown, from the perspective of transaction processing within the monitor node, each of the above messages is processed by the corresponding internal service monitor instance (referred to as a service instance, such as the OSD service instance, MDS service instance, etc. in the service layer 201). The service instance processes the corresponding service message (i.e., the above-mentioned service event) to generate an abstract map (i.e., a data structure used to record service information of a certain type of service, such as OSD_map, MDS_map, etc.), and persists it through the paxos (a distributed consistency algorithm) layer 202 (i.e., stored in the persistent storage medium 203). When each map is persisted through the paxos layer, the same paxos instance (i.e., storage instance) is used. Therefore, it can be understood that the processing of each message is serial, and the next message can only be processed after one message is processed and persisted. In large-scale cluster scenarios, the amount of various messages is large, and paxos persistence has a performance bottleneck.

[0057] In view of this, the present application proposes a distributed cluster management method to solve the above problems.

[0058] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, and to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application are further described in detail below with reference to the accompanying drawings.

[0059] An embodiment of the present application provides a distributed cluster management method. The distributed cluster includes at least a storage cluster, a management cluster, and a control cluster. The management cluster includes multiple management nodes, which are used to configure and manage each node in the distributed cluster. The control cluster includes multiple control nodes, which are used to process service events reported by each storage node in the storage cluster and control the node operating status of each storage node. This embodiment does not specifically limit the function of each node in the distributed cluster.

[0060] See also Figure 3 , Figure 3 This is a flowchart of a method provided in an embodiment of the present application. The method is applied to any management node in a management cluster; the method flow may include the following steps:

[0061] Step 301: Create a service instance corresponding to each service for each control node in the control cluster.

[0062] In this embodiment, each service may be a service pre-configured based on actual needs, such as an OSD service, an MDS service, etc., which is not specifically limited in this embodiment. A service instance of any service is used to handle various events under that service. As described above, events under any service here refer to events associated with that service, such as a write request event or a read request event for the status information of that service.

[0063] Step 302 : For each service, determine the service cluster corresponding to the service from the N control nodes in the control cluster, select one control node from the service cluster as the master service node of the service, and the remaining control nodes as slave service nodes of the service.

[0064] In this embodiment, the service cluster includes at least one control node. The service instance corresponding to the service on the master service node and the service instance corresponding to the service on the slave service node have different permissions for handling events belonging to the service. When a service instance corresponding to the service on the slave service node receives an event belonging to the service and does not have permission to handle the event, it reports the event to the service instance corresponding to the service on the master service node for processing. The specific details of the event permissions for any service will be described below with examples and will not be elaborated on here.

[0065] As an embodiment, the above-mentioned determination of the service cluster corresponding to the service from the N control nodes in the control cluster can be implemented in various ways. For example, for each service, a control node that meets the set requirements is first selected from the N control nodes in the control cluster. Here, a control node that meets the set requirements means that the first usage count of the control node is less than the first usage count of other control nodes in the N control nodes. Then, if the number X of control nodes that meet the set requirements is less than the number M of control nodes in the service cluster configured for the service, P control nodes are selected from the remaining control nodes based on the node identifiers of the remaining control nodes. Here, the remaining control nodes are the control nodes in the N control nodes excluding the control nodes that meet the set requirements. P is the difference between M and X. The service cluster corresponding to the service is constructed based on the control nodes that meet the set requirements and the P control nodes. If X is equal to M, the service cluster corresponding to the service is constructed based on the control nodes that meet the set requirements. If X is greater than M, M control nodes are selected from the control nodes that meet the set requirements, and the service cluster corresponding to the service is constructed based on the selected M control nodes.

[0066] Optionally, the node identifier is a node number, such as 1, 2, 3, etc. Based on this, the above selecting P control nodes from the remaining control nodes based on the node identifiers of the remaining control nodes may be specifically implemented as: selecting the first P remaining control nodes with the largest node numbers from the remaining control nodes as the selected P control nodes.

[0067] In this embodiment, if X is greater than M, M control nodes are selected from the control nodes that meet the set requirements. In a specific implementation, for example, M control nodes may be randomly selected from the control nodes that meet the set requirements; or the first M control nodes with the largest node numbers may be selected from the control nodes that meet the set requirements.

[0068] As an embodiment, after determining the service cluster corresponding to the service from the N control nodes in the control cluster, the first usage count of the control node selected for building the service cluster corresponding to the service is increased by a set value. Here, the set value can be, for example, 1, and this is not specifically limited here.

[0069] As an embodiment, the above selecting a control node from the service cluster as the primary service node for the service may be specifically implemented as: selecting the control node with the second smallest number of uses from the service cluster as the primary service node for the service.

[0070] In this embodiment, after a control node is selected from the service cluster as the primary service node for the service, the second usage count corresponding to the control node selected as the primary service node for the service may be increased by a set value.

[0071] So far, completed Figure 3 The process shown.

[0072] pass Figure 3 As can be seen from the process shown, the embodiment of the present application determines the service cluster corresponding to each service from the N control nodes of the control cluster, and selects the master service node and slave service node of the service from the service cluster. The event permissions belonging to the service processed by the service instance corresponding to the service on the master service node are different from the event permissions belonging to the service processed by the service instance corresponding to the service on the slave service node. In this way, the service events corresponding to each service are processed by the service nodes in the service cluster corresponding to the service, avoiding the problems of high node CPU overhead and low event processing efficiency caused by the existing situation that the service events corresponding to each service are all processed by one master control node, and effectively improving the event processing efficiency.

[0073] The following describes the specific content of event permissions under any service:

[0074] As an embodiment, for any service, the event permissions belonging to the service processed by the service instance corresponding to the service on the main service node corresponding to the service include: permissions for events corresponding to read requests under the service and permissions for events corresponding to write requests; the event permissions belonging to the service processed by the service instance corresponding to the service on the slave service node include: permissions for events corresponding to read requests under the service.

[0075] As an embodiment, when the service instance corresponding to the service on the master service node receives an event corresponding to a write request under the service, the service instance processes the event to obtain an event processing result, and sends the event processing result to the slave service node of the service; the service instance corresponding to the service on the master service node is also used to cache the event processing result to a preconfigured cache space. The service instance corresponding to the service on the slave service node is used to cache the event processing result to a preconfigured cache space when receiving the event processing result under the service. Here, the event processing result can be considered as the map described above, which is not specifically limited in this embodiment.

[0076] Based on the above description, for each service, a storage instance is created on each control node in the service cluster corresponding to the service; wherein, the storage instance corresponding to the service is different from the service instance corresponding to the service; the storage instance corresponding to the service is used to persistently store the event processing results under the service.

[0077] As an embodiment, for each service, if there are control nodes other than the service cluster corresponding to the service in the control cluster, the service instance corresponding to the service on any control node in the service cluster corresponding to the service is also used to send the event processing result under the service to one of the control nodes other than the service cluster corresponding to the service. The service instance corresponding to the service on the control node other than the service cluster corresponding to the service is used to cache the event processing result under the service in a preconfigured cache space upon receiving the event processing result under the service.

[0078] Optionally, the service instance corresponding to the service on the control node other than the service cluster corresponding to the service and the service instance corresponding to the service on the main service node corresponding to the service have different permissions to process events belonging to the service; when the service instance corresponding to the service on the control node other than the service cluster corresponding to the service receives an event belonging to the service, if it does not have the permission to process the event, it reports the event to the service instance corresponding to the service on the main service node corresponding to the service for processing.

[0079] In this embodiment, the event permissions belonging to the service processed by the service instance corresponding to the service on the control node other than the service cluster corresponding to the service include: permissions for events corresponding to read requests under the service.

[0080] As an embodiment, in order to enable the service instances on the control node to execute tasks in parallel and improve task processing efficiency, different threads may be configured for different service instances on the control node.

[0081] In order to facilitate understanding of the specific implementation process of the above-mentioned distributed cluster management method, a specific embodiment is described below by way of example.

[0082] The embodiment of the present application provides a method for solving large-scale cluster control, based on the multi-leader architecture of the monitor cluster (i.e., the control cluster), wherein the leader node (denoted as the decision node) refers to the working node in the monitor cluster that is responsible for coordinating and leading the entire monitor cluster, which can be considered as the above-mentioned main monitor node. The multi-leader architecture means that there are multiple leader nodes in the monitor cluster, and each service corresponds to a leader node, which increases the ability of parallel transaction processing, business isolation between different maps, and each map has its own small-scale decision quorum cluster (a cluster of nodes that need to achieve transaction consistency, i.e., a service cluster), which shortens the transaction processing time on the basis of ensuring data consistency, thereby avoiding performance hot spots. Here, the leader node processes the service message to generate a map and sends it to the corresponding follower node. The process of receiving the feedback notification of the follower node receiving the map is called a transaction.

[0083] First, we need to introduce the concept of "service quorum". The so-called "service cluster" is a subset of the monitor cluster. Figure 4 As shown, Figure 4 The mon, OSD, MDS and other instance groups in the ceph.mon refers to the monitor, and the mon instance is used to process the monitor process / node information; OSD refers to the basic building unit of ceph storage, and the OSD instance is used to process the osd service message; the MDS instance is used to process the metadata service information. Figure 4 The Paxos process (represented by the monitor process in the example) abstracts the cluster state (i.e., service messages) into different map objects. Each map object corresponds to a type of service instance, which handles read and write requests for the map object. A service instance is bound to a Paxos Service (the Paxos service layer, used to ensure the consistency of state information within the monitor cluster and coordinate state changes within the cluster). The Paxos Service is responsible for serializing its corresponding data into key-value pairs and writing them to the Paxos layer. The "service cluster" is the cluster on which Paxos operations are based.

[0084] When a distributed cluster is initialized, a dedicated service cluster is selected for each service (such as OSD, MDS, monitor, and other services). Nodes in this service cluster then enter the bootstrap state (a required step for the initial startup of a Ceph monitor or the addition of a new monitor, which guides the monitor cluster to start correctly). After detection, synchronization, election, and recovery, the service cluster enters the active state (the state after initial startup), ready to process transactions. The number of nodes in a service cluster is a subset of all monitor nodes, and only nodes within the service cluster need to process transactions using the Paxos algorithm. Nodes within the service cluster act as proposers (the nodes that initiate transactions, decide on transactions, and achieve transaction consistency) or followers (the nodes that achieve transaction consistency) for that service instance. Proposers have decision-making power over transactions (they can generate maps for messages). Nodes outside the service cluster act as learners (they do not participate in decision-making, do not persist data, and only cache maps), meaning they learn map data. For each instance, the proposer, follower, and learner nodes cache and persist a certain number of maps for business queries (such as map data read or write requests). For example, the proposer and follower nodes cache the most recent map and persist a maximum of 250 maps; the learner node caches 40 maps and does not persist any maps.

[0085] See also Figure 5 As shown in the figure, the selection algorithm for the quorum cluster (i.e., the service cluster) is based on the following rules:

[0086] 1) Select the node with the smallest Join_quorum_num;

[0087] 2) If the join_quorum_num is the same, the node with the largest rank value (i.e., the unique identifier of the monitor node in the cluster, such as node numbers 1, 2, 3, etc.) is selected;

[0088] Here, Join_quorum_num refers to the number of nodes participating in the quorum cluster (ie, the first number of times used above).

[0089] The quorum cluster configuration is generated only once. After the quorum cluster nodes are generated, they will only change in a scaling-down scenario (scaling-down scenario means removing one or more monitor nodes in the control cluster from the cluster). When mon_list (i.e., the node list indicating each monitor node in the control cluster) remains unchanged, the service quorum generated by each node in order is consistent. That is, no matter what happens (such as restart, etc.), the service cluster generated according to the above service cluster selection algorithm remains unchanged. Optionally, when the number of mon clusters (i.e., the above control clusters) is less than or equal to 3, the quorum cluster selection will not be triggered. All services corresponding to each service use the same set of service quorums (i.e., each service corresponds to a service cluster that can be used to process events under that service).

[0090] In this embodiment, after the quorum cluster is introduced, the monitor cluster does not need to achieve transaction consistency on all nodes. The transaction is considered completed only when transaction consensus is reached in the quorum cluster. The quorum cluster is a subset of the monitor cluster. For example, the monitor node supports a maximum of 128 monitor nodes. According to the original architecture, the transaction is considered completed only when transaction consensus is reached among 128 monitor nodes. The default value of the quorum cluster is 3, that is, only 3 monitor nodes are required to participate in transaction consistency processing in the new architecture. The identity of the remaining 125 monitor nodes is learner nodes. After the transaction reaches consensus, the nodes in the quorum cluster need to distribute it to the learner nodes to update the map cache. That is, the map in the nodes in the quorum cluster needs to be written to the disk to save the real value of the map, and the learner node has the cached value of the map. This architecture speeds up transaction processing and ensures the reliability of the map. There is no relationship between multiple types of quorum clusters. Their worker threads, node identities, transaction processing, and DB (database) data are all isolated. For example, there are 4 nodes in the monitor cluster, and every 3 nodes form a quorum cluster. That is, the nodes in different quorum clusters are reused. A node may simultaneously play the proposer role of the OSD service instance, the follower role of the mon service instance, and the learner role of the MDS service instance, but the services do not affect each other.

[0091] From the above analysis, we can see that different service instances are distributed on different monitor nodes, and the CPU pressure of the monitor nodes is reduced. Figure 5 As shown, taking 5 monitor nodes as an example, 5 service instances such as mon and OSD service instances are distributed on the 5 monitor nodes (such as Figure 5 Each box marked monx (x = 0, 1, 2, 3, 4) is considered a monitor node), the main monitor node (such as Figure 5 The monitor node (with a triangle inside the box, representing the primary monitor node) experiences only one-fifth the original CPU load. Each service instance has its own Paxos instance (i.e., storage instance), completely distributing service event processing and sharing the load.

[0092] This embodiment can improve the processing capacity of the monitor cluster and support large-scale cluster process control and state maintenance. This embodiment introduces a service cluster and stipulates that transaction consistency is achieved on the nodes within the service cluster. The nodes outside the service cluster serve as a cache. On the basis of ensuring the data reliability between the distributed monitor nodes, transactions are processed as quickly as possible. In order to ensure consistency, transactions are processed serially. This embodiment also divides transactions by service through different service clusters, thereby forming a concurrent effect. For example, there is only one leader node in the original framework monitor cluster, and transactions corresponding to each service, such as mds transactions and OSD transactions, are processed serially on the leader node. In this embodiment, mds transactions are processed by the master node in the service cluster corresponding to the mds service, and OSD transactions are processed by the master node in the service cluster corresponding to the OSD service. Transactions are isolated through the service cluster, but the same type of transactions are still processed serially within the service cluster.

[0093] This completes the description of the method provided in the embodiment of the present application. The following describes the device provided in the embodiment of the present application:

[0094] As an embodiment, this embodiment also provides a distributed cluster management device. The distributed cluster includes at least a storage cluster, a management cluster, and a control cluster. The management cluster includes multiple management nodes, which are used to configure and manage each node in the distributed cluster. The control cluster includes multiple control nodes, which are used to process service events reported by each storage node in the storage cluster and control the node working status of each storage node.

[0095] See also Figure 6 , Figure 6 This is a schematic diagram of the device structure provided in the embodiment of the present application. Figure 6 As shown, the device 600 is configured at any management node in the management cluster; the device includes:

[0096] The creation module 601 is used to create a service instance corresponding to each service for each control node in the control cluster; the service instance of any service is used to process various events under the service;

[0097] Determination module 602 is used to determine, for each service, a service cluster corresponding to the service from N control nodes in the control cluster, where the service cluster includes at least one control node; select one control node from the service cluster as the master service node of the service, and the remaining control nodes as slave service nodes of the service; wherein the service instance corresponding to the service on the master service node and the service instance corresponding to the service on the slave service node have different permissions for processing events belonging to the service; when the service instance corresponding to the service on the slave service node receives an event belonging to the service, if it does not have the permission to process the event, it reports the event to the service instance corresponding to the service on the master service node for processing.

[0098] As an embodiment, determining the service cluster corresponding to the service from N control nodes in the control cluster includes:

[0099] For each service, a control node that meets the set requirements is selected from the N control nodes in the control cluster; the control node that meets the set requirements means that the first usage count of the control node is less than the first usage count of other control nodes in the N control nodes;

[0100] If the number X of control nodes that meet the set requirements is less than the number M of control nodes in the service cluster corresponding to the configured service, then P control nodes are selected from the remaining control nodes based on the node identifiers of the remaining control nodes; the remaining control nodes are the nodes in the N control nodes excluding the control nodes that meet the set requirements; P is the difference between M and X; the service cluster corresponding to the service is constructed based on the control nodes that meet the set requirements and the P control nodes;

[0101] If X is equal to M, a service cluster corresponding to the service is built based on the control nodes that meet the set requirements;

[0102] If X is greater than M, select M control nodes from the control nodes that meet the set requirements, and build a service cluster corresponding to the service based on the selected M control nodes;

[0103] After determining the service cluster corresponding to the service from the N control nodes in the control cluster, the determination module 602 is further configured to:

[0104] The first usage count of the control node selected for constructing the service cluster corresponding to the service is increased by a set value.

[0105] As an embodiment, the node identifier is a node number;

[0106] Selecting P control nodes from the remaining control nodes according to the node identifiers of the remaining control nodes includes:

[0107] The first P remaining control nodes with the largest node numbers are selected from the remaining control nodes as the selected P control nodes.

[0108] As an embodiment, selecting a control node from a service cluster as a primary service node for the service includes:

[0109] Select the control node with the second smallest number of uses from the service cluster as the primary service node for the service;

[0110] After selecting a control node from the service cluster as the primary service node for the service, the determination module 602 is further configured to:

[0111] The second usage count corresponding to the control node selected as the primary service node of the service is increased by a set value.

[0112] As an embodiment, the event permissions for the service processed by the service instance corresponding to the service on the primary service node include: permissions for events corresponding to read requests and permissions for events corresponding to write requests under the service;

[0113] The event permissions for the service processed by the service instance corresponding to the service on the service node include: permissions for events corresponding to read requests under the service.

[0114] As an embodiment, when a service instance corresponding to the service on the master service node receives an event corresponding to a write request under the service, the service instance processes the event to obtain an event processing result, and sends the event processing result to the slave service node of the service; the service instance corresponding to the service on the master service node is further configured to cache the event processing result in a preconfigured cache space;

[0115] The service instance corresponding to the service on the slave service node is used to cache the event processing result under the service into a pre-configured cache space when receiving the event processing result;

[0116] Create module 601, further configured to:

[0117] For each service, a storage instance is created on each control node in the service cluster corresponding to the service; wherein the storage instance corresponding to the service is different from the service instance corresponding to the service; the storage instance corresponding to the service is used to persistently store the event processing results under the service.

[0118] As an embodiment, for each service, when there are control nodes other than the service cluster corresponding to the service in the control cluster, the service instance corresponding to the service on any control node in the service cluster corresponding to the service is further configured to send the latest event processing result to one of the control nodes other than the service cluster corresponding to the service; the service instance corresponding to the service on the control node other than the service cluster corresponding to the service is configured to cache the event processing result under the service in a preconfigured cache space upon receiving the event processing result;

[0119] The service instance corresponding to the service on the control node other than the service cluster corresponding to the service and the service instance corresponding to the service on the main service node corresponding to the service have different permissions to process events belonging to the service; when the service instance corresponding to the service on the control node other than the service cluster corresponding to the service receives an event belonging to the service, if it does not have the permission to process the event, it will report the event to the service instance corresponding to the service on the main service node corresponding to the service for processing.

[0120] So far, completed Figure 6 Structural description of the device shown.

[0121] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0122] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present application scheme. Those of ordinary skill in the art can understand and implement it without paying any creative work.

[0123] See Figure 7 , is a schematic diagram of the hardware structure of an electronic device provided by an exemplary embodiment of the present application. The electronic device may include a processor 701, a communication interface 702, a memory 703, and a communication bus 704. The processor 701, the communication interface 702, and the memory 703 communicate with each other via the communication bus 704. The memory 703 stores computer program instructions; the processor 701 can execute the steps of the method described in the above embodiment by executing the computer program instructions stored in the memory 703. The electronic device may also include other hardware according to the actual function of the electronic device, which will not be described in detail.

[0124] Correspondingly, an embodiment of the present application further provides a computer-readable storage medium, on which a number of computer program instructions are stored. When the computer program instructions are executed by a processor, the method disclosed in the above example of the present application can be implemented.

[0125] Exemplarily, the computer-readable storage medium may be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, and the like. For example, the computer-readable storage medium may be RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, a storage drive (such as a hard disk drive), a solid-state drive, any type of storage disk (such as a CD, DVD, etc.), or similar storage media, or a combination thereof. The processor and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0126] The above are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A distributed cluster management method, characterized in that: The distributed cluster includes at least a storage cluster, a management cluster and a control cluster; the management cluster includes a plurality of management nodes, and the management nodes are at least used to configure and manage each node in the distributed cluster; The control cluster includes a plurality of control nodes, and the control nodes are used at least to process service events reported by each storage node in the storage cluster and control the node working status of each storage node; The method is applied to any management node in the management cluster; the method includes: Creating a service instance corresponding to each service for each control node in the control cluster; a service instance of any service is used to process events under the service; For each service, a service cluster corresponding to the service is determined from the N control nodes in the control cluster, wherein the service cluster includes at least one control node; one control node is selected from the service cluster as the master service node of the service, and the remaining control nodes are used as slave service nodes of the service; wherein the service instance corresponding to the service on the master service node and the service instance corresponding to the service on the slave service node have different permissions for processing events belonging to the service; when the service instance corresponding to the service on the slave service node receives an event belonging to the service, if it does not have the permission to process the event, it reports the event to the service instance corresponding to the service on the master service node for processing.

2. The method according to claim 1, characterized in that The determining the service cluster corresponding to the service from the N control nodes in the control cluster includes: For each service, selecting a control node that meets set requirements from the N control nodes in the control cluster; the control node that meets the set requirements means that the first usage count of the control node is less than the first usage count of other control nodes in the N control nodes; If the number X of control nodes that meet the set requirements is less than the number M of control nodes in the service cluster corresponding to the configured service, then select P control nodes from the remaining control nodes based on the node identifiers of the remaining control nodes; the remaining control nodes are nodes in the N control nodes excluding the control nodes that meet the set requirements; P is the difference between M and X; and the service cluster corresponding to the service is established based on the control nodes that meet the set requirements and the P control nodes. If X is equal to M, a service cluster corresponding to the service is constructed based on the control nodes that meet the set requirements; If X is greater than M, M control nodes are selected from the control nodes that meet the set requirements, and a service cluster corresponding to the service is constructed based on the selected M control nodes; After determining the service cluster corresponding to the service from the N control nodes in the control cluster, the method further includes: The first usage count of the control node selected for constructing the service cluster corresponding to the service is increased by a set value.

3. The method according to claim 2, characterized in that The node identifier is a node number; The selecting P control nodes from the remaining control nodes according to the node identifiers of the remaining control nodes includes: The first P remaining control nodes with the largest node numbers are selected from the remaining control nodes as the selected P control nodes.

4. The method according to claim 1, wherein The selecting a control node from the service cluster as a primary service node for the service includes: Selecting the control node with the second smallest number of uses from the service cluster as the primary service node for the service; After selecting a control node from the service cluster as a primary service node for the service, the method further includes: The second usage count corresponding to the control node selected as the primary service node of the service is increased by a set value.

5. The method according to claim 1, wherein The event permissions for the service instance corresponding to the service on the primary service node include: permissions for events corresponding to read requests and permissions for events corresponding to write requests under the service; The permission for events belonging to the service processed by the service instance corresponding to the service on the slave service node includes: permission for events corresponding to read requests under the service.

6. The method according to claim 5, characterized in that When the service instance corresponding to the service on the master service node receives an event corresponding to a write request under the service, the service instance processes the event to obtain an event processing result, and sends the event processing result to the slave service node of the service; The service instance corresponding to the service on the primary service node is further used to cache the event processing result in a preconfigured cache space; The service instance corresponding to the service on the slave service node is used to cache the event processing result under the service into a preconfigured cache space when receiving the event processing result; The method further comprises: For each service, a storage instance is created on each control node in the service cluster corresponding to the service; wherein the storage instance corresponding to the service is used to persistently store the event processing results under the service.

7. The method according to claim 6, characterized in that For each service, if there is a control node other than the service cluster corresponding to the service in the control cluster, the service instance corresponding to the service on any control node in the service cluster corresponding to the service is also used to send the event processing result under the service to one of the control nodes other than the service cluster corresponding to the service; The service instance corresponding to the service on the control node other than the service cluster corresponding to the service is used to cache the event processing result under the service into a preconfigured cache space when receiving the event processing result; The service instance corresponding to the service on the control node other than the service cluster corresponding to the service and the service instance corresponding to the service on the primary service node corresponding to the service have different event permissions for the service; When a service instance corresponding to the service on a control node other than the service cluster corresponding to the service receives an event belonging to the service, if it does not have the authority to process the event, it reports the event to the service instance corresponding to the service on the main service node corresponding to the service for processing.

8. A distributed cluster management device, characterized in that: The distributed cluster includes at least a storage cluster, a management cluster and a control cluster; the management cluster includes a plurality of management nodes, and the management nodes are at least used to configure and manage each node in the distributed cluster; The control cluster includes a plurality of control nodes, and the control nodes are used at least to process service events reported by each storage node in the storage cluster and control the node working status of each storage node; The device is configured at any management node in the management cluster; the device includes: A creation module is used to create a service instance corresponding to each service for each control node in the control cluster; a service instance of any service is used to process various events under the service; A determination module is used to determine, for each service, a service cluster corresponding to the service from N control nodes in the control cluster, wherein the service cluster includes at least one control node; select one control node from the service cluster as the master service node of the service, and the remaining control nodes as slave service nodes of the service; wherein the service instance corresponding to the service on the master service node and the service instance corresponding to the service on the slave service node have different permissions for processing events belonging to the service; when the service instance corresponding to the service on the slave service node receives an event belonging to the service, if it does not have the permission to process the event, it reports the event to the service instance corresponding to the service on the master service node for processing.

9. An electronic device, characterized in that: The electronic device includes: processor; and A memory, wherein computer program instructions are stored in the memory, and when the computer program instructions are executed by the processor, the processor is caused to perform the steps of any one of the methods of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer program instructions, which, when executed by a processor, enable the processor to perform the steps of the method according to any one of claims 1 to 7.

Images