A large-scale storage cluster deployment grouping method and device
Through automatic grouping methods and unified management services, the problems of low deployment efficiency and poor user experience of large-scale storage clusters are solved, concise deployment and highly available grouping management are achieved, and deployment efficiency and operation and maintenance efficiency are improved.
Patent Information
- Application Number
- CN202211355996.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-01
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2042-11-01
AI Technical Summary
Large-scale storage cluster deployment is inefficient and poor user experience. The existing technology requires users to intervene in the clustering, resulting in complex deployment and increased high-reliability configurations, complex capacity reduction processes, and affecting user experience.
Provide an automatic grouping method, by analyzing management nodes and storing node information, initializing the grouping sequence, setting the upper limit value of the grouping node, automatically adding nodes to a unified management service, simplifying the grouping process, and automatically discovering and removing exceptions in failure scenarios Node, avoid user manual configuration.
It improves the deployment efficiency of large-scale storage clusters, simplifies user operations, reduces operation and maintenance complexity, ensures high availability and convenience of packets in failure scenarios, and improves user experience and operation and maintenance efficiency.
Smart Images

Figure CN116055307B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of cluster deployment, and in particular to a large-scale storage cluster deployment grouping method and device. Background Art
[0002] In a storage cluster, nodes are divided into management nodes and storage nodes according to their uses. The management node carries the management and control tasks of the entire cluster. It needs to obtain the status of each hard disk pool, node, and disk in the cluster in real time, perceive the overall operation status of the cluster, and make corresponding processing. As the scale of the storage cluster becomes larger and larger, the network pressure and business pressure of the management node increase greatly, and its response speed is greatly affected, becoming a bottleneck in large-scale scenarios.
[0003] For large-scale deployment scenarios of storage clusters, in order to reduce the pressure on management nodes, the multi-sub-cluster mode can be adopted to divide the entire cluster into multiple sub-clusters for management. Each sub-cluster has multiple sub-management nodes responsible for the management and control tasks of the sub-cluster, thereby sharing the pressure of the main management node to the sub-management nodes on each sub-cluster, reducing the pressure on the main management node. The multi-sub-cluster mode divides the entire cluster into multiple sub-clusters for management, which can solve the single-node performance problem in large-scale scenarios, but in actual applications, there are the following shortcomings: 1. The creation of sub-clusters requires user intervention. Each sub-cluster requires users to plan the nodes it contains, and the deployment efficiency is not high in large-scale scenarios; 2. The high reliability of sub-management nodes in fault scenarios requires the introduction of additional high-availability mechanisms to ensure, such as through the virtual IP mode, users are required to allocate additional IPs, and the complexity of cluster operation and maintenance and deployment configuration will increase significantly; 3. Because the sub-management node carries the management and control tasks of the sub-cluster, the node role migration needs to be performed when the node is deleted. The scaling process in this scenario is complicated, and users need to intervene to specify new sub-management nodes, which affects the user experience.
[0004] Therefore, how to provide a method to improve the deployment efficiency and user experience of large-scale storage clusters has become a technical problem that needs to be solved urgently. Summary of the invention
[0005] In view of this, the present invention mainly solves the problems of low efficiency in large-scale cluster deployment and poor user experience, and provides a large-scale storage cluster automatic grouping method to free users from the cumbersome deployment process.
[0006] In one aspect, the present invention provides a large-scale storage cluster deployment grouping method, comprising:
[0007] Parse the management node and storage node information selected by the user, initialize a group sequence, add the management node to the head of the group sequence, and add the storage nodes to the group sequence in sequence according to the IP order;
[0008] Set the upper limit of grouping nodes, compare the number of remaining nodes to be grouped in the grouping sequence with the upper limit of grouping nodes, group the remaining nodes to be grouped in the grouping sequence according to the comparison result, and add the nodes in each group to the unified management service;
[0009] Update the grouping information in the distributed configuration file.
[0010] Furthermore, the large-scale storage cluster deployment grouping method of the present invention groups the remaining nodes to be grouped in the grouping sequence according to the comparison result, including:
[0011] If the number of remaining nodes to be grouped in the grouping sequence is not less than the grouping node upper limit, select nodes in the grouping sequence in sequence according to the grouping node upper limit to form a new group, increase the group number by one, and add all nodes in the new group to the unified management service, continue to compare the number of remaining nodes to be grouped in the grouping sequence with the grouping node upper limit, and group the remaining nodes to be grouped in the grouping sequence according to the comparison result;
[0012] If the number of remaining nodes to be grouped in the grouping sequence is less than the upper limit, the remaining nodes to be grouped in the grouping sequence are formed into a new group, the group number is increased by one, and all nodes in the new group are added to the unified management service.
[0013] Furthermore, the large-scale storage cluster deployment grouping method of the present invention adds the nodes in each group to a unified management service, including: when the management node needs to obtain data from each node, it establishes communication with the unified management service and sends request data to the unified management service, and the unified management service routes the request data to a storage node in the group, which then collects information about all nodes in the group.
[0014] Furthermore, the large-scale storage cluster deployment grouping method of the present invention adds the nodes in each group to a unified management service, and also includes: when the management node needs each node to perform business, it establishes communication with the unified management service and sends request data to the unified management service, and the unified management service routes the request data to a storage node in the group, and the storage node distributes the configuration instructions.
[0015] Furthermore, the large-scale storage cluster deployment grouping method of the present invention adds the nodes in each group to a unified management service, and also includes: when an abnormality occurs in a node, the unified management service will automatically discover the abnormality and remove the node from the unified management service list, and the requested data will not be routed to the faulty node.
[0016] On the other hand, the present invention also provides a large-scale storage cluster deployment grouping device, comprising:
[0017] A grouping sequence building module is used to parse the management node and storage node information selected by the user, initialize a grouping sequence, add the management node to the head of the grouping sequence, and add the storage nodes to the grouping sequence in sequence according to the IP order;
[0018] A grouping module is used to set the upper limit of grouping nodes, compare the number of remaining nodes to be grouped in the grouping sequence with the upper limit of grouping nodes, group the remaining nodes to be grouped in the grouping sequence according to the comparison result, and add the nodes in each group to the unified management service module;
[0019] A unified management service module is used to establish communication with the management node, receive request data sent by the management node, route the request data to a storage node of the group, collect information of all nodes of the group or distribute configuration instructions through the storage node;
[0020] The update module is used to update the group information in the distributed configuration file.
[0021] Furthermore, the grouping module in the large-scale storage cluster deployment grouping device of the present invention is specifically used for:
[0022] Setting a grouping node upper limit, comparing the number of remaining nodes to be grouped in the grouping sequence with the grouping node upper limit, if the number of remaining nodes to be grouped in the grouping sequence is not less than the grouping node upper limit, selecting nodes in the grouping sequence in turn according to the grouping node upper limit to form a new group, increasing the group number by one, and adding all nodes in the new group to the unified management service module, continuing to compare the number of remaining nodes to be grouped in the grouping sequence with the grouping node upper limit, and grouping the remaining nodes to be grouped in the grouping sequence according to the comparison result;
[0023] If the number of remaining nodes to be grouped in the grouping sequence is less than the upper limit, the remaining nodes to be grouped in the grouping sequence are formed into a new group, the group number is increased by one, and all nodes in the new group are added to the unified management service module.
[0024] Furthermore, the present invention deploys a grouping device in a large-scale storage cluster. When a management node needs to obtain data from each node, it establishes communication with a unified management service module and sends request data to the unified management service module. The unified management service module routes the request data to a storage node of the group, and the storage node collects information of all nodes of the group.
[0025] Furthermore, the present invention deploys grouping devices in a large-scale storage cluster. When the management node needs each node to execute a business, it establishes communication with the unified management service module and sends request data to the unified management service module. The unified management service module routes the request data to a storage node in the group, and the storage node distributes the configuration instructions.
[0026] Furthermore, the present invention deploys grouping devices in a large-scale storage cluster. When an abnormality occurs in a node, the unified management service module will automatically discover the abnormality and remove the node from the unified management service list, and the requested data will not be routed to the faulty node.
[0027] The large-scale storage cluster deployment grouping method and device of the present invention have the following beneficial effects:
[0028] 1. Through automatic grouping strategies, users are prevented from dividing and configuring clusters, making user operations in large-scale clusters simpler and improving deployment efficiency.
[0029] 2. The management and control strategy of the management service ensures that each group can provide services to the outside world even in a fault scenario. The management and control strategy does not require user intervention for configuration, making the operation more convenient and improving the efficiency of operation and maintenance. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0031] Figure 1 It is a schematic diagram of the architecture of a large-scale storage cluster of the present invention.
[0032] Figure 2 The present invention is a flowchart of a large-scale storage cluster deployment grouping method according to an exemplary first embodiment of the present invention.
[0033] Figure 3 This is an architectural diagram of a large-scale storage cluster deployment grouping device according to the third exemplary embodiment of the present invention. DETAILED DESCRIPTION
[0034] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.
[0035] It should be noted that the following embodiments and features in the embodiments may be combined with each other in the absence of conflict; and, based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making any creative work are within the scope of protection of the present disclosure.
[0036] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.
[0037] The technical principles of the present invention are as follows:
[0038] like Figure 1 As shown, a large-scale cluster is composed of a management node group and a storage node, and the management node group is configured in a master-slave mode. The present invention automatically groups large-scale clusters, and the master group includes a unified management and control service consisting of a management node group and all storage nodes, and the slave group includes a unified management and control service consisting of all storage nodes in this group. The management node group only communicates and interacts with the unified management service of each group, and its overall pressure is greatly reduced, which can meet the original demand of reducing the pressure on the management node.
[0039] The present invention solves the problem of low deployment efficiency caused by the need for user intervention for cluster division and multiple sub-cluster deployment. In the usual strategy, users need to plan sub-clusters on the page, select nodes to join different sub-clusters, and configure sub-management nodes of sub-clusters, which brings complexity to the configuration process, and also increases the deployment time, affecting the user experience. In order to solve this problem, the present invention divides the entire cluster into multiple groups according to the number of cluster nodes and the upper limit of the grouping value. The number of nodes in each group is not greater than the set upper limit of the grouping. After the grouping, the remaining nodes are automatically grouped into a group. Because the grouping operation is an automated process, it is invisible to the user, which simplifies the user configuration process.
[0040] The present invention adds the nodes in each group to a unified management and control service for unified management. When the management node needs to obtain the data of each node or needs each node to perform business, it only needs to communicate with the unified management service. The unified management service routes the request data to a storage node in the group, and the storage node is responsible for collecting information of all nodes in the group or distributing configuration instructions. In this way, each group does not need to introduce an additional high-availability mechanism. When an abnormality occurs in a node, the unified management service will automatically discover the abnormality and remove the node from the unified management service list, and the request data will not be routed to the faulty node.
[0041] For large-scale scenarios, nodes joining a cluster need to specify the subcluster to join, which requires users to plan and configure complexly. In a scaling-down scenario, if the management node of a subcluster is deleted, another node needs to be added as a new management node to ensure high availability, which brings complexity to operation and maintenance. In the present invention, since all nodes joining the unified management service are indifferent, deleting a node only requires removing it from the unified management service, which is simple and efficient.
[0042] Figure 2 FIG. 1 is a flow chart of a large-scale storage cluster deployment grouping method according to an exemplary first embodiment of the present invention. Figure 2 As shown, the method of this embodiment includes:
[0043] Parse the management node and storage node information selected by the user, initialize a group sequence, add the management node to the head of the group sequence, and add the storage nodes to the group sequence in sequence according to the IP order;
[0044] Set the upper limit of grouping nodes, compare the number of remaining nodes to be grouped in the grouping sequence with the upper limit of grouping nodes, group the remaining nodes to be grouped in the grouping sequence according to the comparison result, and add the nodes in each group to the unified management service;
[0045] Update the grouping information in the distributed configuration file.
[0046] In the method of this embodiment, grouping the remaining nodes to be grouped in the grouping sequence according to the comparison result includes:
[0047] If the number of remaining nodes to be grouped in the grouping sequence is not less than the grouping node upper limit, select nodes in the grouping sequence in sequence according to the grouping node upper limit to form a new group, increase the group number by one, and add all nodes in the new group to the unified management service, continue to compare the number of remaining nodes to be grouped in the grouping sequence with the grouping node upper limit, and group the remaining nodes to be grouped in the grouping sequence according to the comparison result;
[0048] If the number of remaining nodes to be grouped in the grouping sequence is less than the upper limit, the remaining nodes to be grouped in the grouping sequence are formed into a new group, the group number is increased by one, and all nodes in the new group are added to the unified management service.
[0049] The second exemplary embodiment of the present invention provides a large-scale storage cluster deployment grouping method. Figure 2 The preferred embodiment of the method shown in the embodiment, in which the nodes in each group are added to the unified management service, includes:
[0050] When the management node needs to obtain data from each node, it establishes communication with the unified management service and sends request data to the unified management service. The unified management service routes the request data to a storage node in the group, and the storage node collects information from all nodes in the group.
[0051] When the management node needs each node to perform business, it establishes communication with the unified management service and sends request data to the unified management service. The unified management service routes the request data to a storage node in the group, and the storage node distributes the configuration instructions.
[0052] When an exception occurs in a node, the unified management service will automatically detect the exception and remove the node from the unified management service list. The requested data will not be routed to the faulty node.
[0053] Figure 3 FIG. 1 is an architecture diagram of a large-scale storage cluster deployment grouping device according to an exemplary third embodiment of the present invention. Figure 3 As shown, the device of this embodiment includes:
[0054] A grouping sequence building module is used to parse the management node and storage node information selected by the user, initialize a grouping sequence, add the management node to the head of the grouping sequence, and add the storage nodes to the grouping sequence in sequence according to the IP order;
[0055] A grouping module is used to set the upper limit of grouping nodes, compare the number of remaining nodes to be grouped in the grouping sequence with the upper limit of grouping nodes, group the remaining nodes to be grouped in the grouping sequence according to the comparison result, and add the nodes in each group to the unified management service module;
[0056] A unified management service module is used to establish communication with the management node, receive request data sent by the management node, route the request data to a storage node of the group, collect information of all nodes of the group or distribute configuration instructions through the storage node;
[0057] The update module is used to update the group information in the distributed configuration file.
[0058] In practical applications, the grouping module of the device of this embodiment is specifically used for:
[0059] Setting a grouping node upper limit, comparing the number of remaining nodes to be grouped in the grouping sequence with the grouping node upper limit, if the number of remaining nodes to be grouped in the grouping sequence is not less than the grouping node upper limit, selecting nodes in the grouping sequence in turn according to the grouping node upper limit to form a new group, increasing the group number by one, and adding all nodes in the new group to the unified management service module, continuing to compare the number of remaining nodes to be grouped in the grouping sequence with the grouping node upper limit, and grouping the remaining nodes to be grouped in the grouping sequence according to the comparison result;
[0060] If the number of remaining nodes to be grouped in the grouping sequence is less than the upper limit, the remaining nodes to be grouped in the grouping sequence are formed into a new group, the group number is increased by one, and all nodes in the new group are added to the unified management service module.
[0061] When the management node needs to obtain data from each node, it establishes communication with the unified management service module and sends request data to the unified management service module. The unified management service module routes the request data to a storage node in the group, and the storage node collects information about all nodes in the group.
[0062] When the management node needs each node to execute a service, it establishes communication with the unified management service module and sends request data to the unified management service module. The unified management service module routes the request data to a storage node in the group, and the storage node distributes the configuration instructions.
[0063] When an abnormality occurs in a node, the unified management service module will automatically detect the abnormality and remove the node from the unified management service list. The requested data will not be routed to the faulty node.
[0064] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims
1. A large-scale storage cluster deployment grouping method, characterized in that: The method comprises: Parse the management node and storage node information selected by the user, initialize a group sequence, add the management node to the head of the group sequence, and add the storage nodes to the group sequence in sequence according to the IP order; Set the upper limit of grouping nodes, compare the number of remaining nodes to be grouped in the grouping sequence with the upper limit of grouping nodes, and group the remaining nodes to be grouped in the grouping sequence according to the comparison result. The main group includes the unified management service composed of the management node group and all storage nodes, and the slave group includes the unified management service composed of all storage nodes in this group. The unified management service is used to establish communication with the management node, receive request data sent by the management node, route the request data to a storage node of the group, and collect information of all nodes of the group or distribute configuration instructions through the storage node; Update the grouping information in the distributed configuration file.
2. The large-scale storage cluster deployment grouping method according to claim 1, characterized in that: The remaining nodes to be grouped in the grouping sequence are grouped according to the comparison result, including: If the number of remaining nodes to be grouped in the grouping sequence is not less than the grouping node upper limit, select nodes in the grouping sequence in sequence according to the grouping node upper limit to form a new group, increase the group number by one, and add all nodes in the new group to the unified management service, continue to compare the number of remaining nodes to be grouped in the grouping sequence with the grouping node upper limit, and group the remaining nodes to be grouped in the grouping sequence according to the comparison result; If the number of remaining nodes to be grouped in the grouping sequence is less than the upper limit, the remaining nodes to be grouped in the grouping sequence are formed into a new group, the group number is increased by one, and all nodes in the new group are added to the unified management service.
3. The large-scale storage cluster deployment grouping method according to claim 1, characterized in that: Add the nodes in each group to the unified management service, including: when the management node needs to obtain the data of each node, it establishes communication with the unified management service and sends request data to the unified management service. The unified management service routes the request data to a storage node in the group, and the storage node collects the information of all nodes in the group.
4. The large-scale storage cluster deployment grouping method according to claim 1, characterized in that: Adding the nodes in each group to the unified management service also includes: when the management node needs each node to perform business, by establishing communication with the unified management service and sending request data to the unified management service, the unified management service routes the request data to a storage node in the group, and the storage node distributes the configuration instructions.
5. The large-scale storage cluster deployment grouping method according to claim 1, characterized in that: Adding the nodes in each group to the unified management service also includes: when an abnormality occurs in a node, the unified management service will automatically discover the abnormality and remove the node from the unified management service list, and the request data will not be routed to the faulty node.
6. A large-scale storage cluster deployment grouping device, characterized in that: The large-scale storage cluster grouping device comprises: A grouping sequence building module is used to parse the management node and storage node information selected by the user, initialize a grouping sequence, add the management node to the head of the grouping sequence, and add the storage nodes to the grouping sequence in sequence according to the IP order; The grouping module is used to set the upper limit of grouping nodes, compare the number of remaining nodes to be grouped in the grouping sequence with the upper limit of grouping nodes, and group the remaining nodes to be grouped in the grouping sequence according to the comparison result. The main group includes the unified management service composed of the management node group and all storage nodes, and the slave group includes the unified management service composed of all storage nodes in this group; A unified management service module is used to establish communication with the management node, receive request data sent by the management node, route the request data to a storage node of the group, collect information of all nodes of the group or distribute configuration instructions through the storage node; The update module is used to update the group information in the distributed configuration file.
7. The large-scale storage cluster deployment grouping device according to claim 6, characterized in that: Grouping module, specifically used for: Setting a grouping node upper limit, comparing the number of remaining nodes to be grouped in the grouping sequence with the grouping node upper limit, if the number of remaining nodes to be grouped in the grouping sequence is not less than the grouping node upper limit, selecting nodes in the grouping sequence in turn according to the grouping node upper limit to form a new group, increasing the group number by one, and adding all nodes in the new group to the unified management service module, continuing to compare the number of remaining nodes to be grouped in the grouping sequence with the grouping node upper limit, and grouping the remaining nodes to be grouped in the grouping sequence according to the comparison result; If the number of remaining nodes to be grouped in the grouping sequence is less than the upper limit, the remaining nodes to be grouped in the grouping sequence are formed into a new group, the group number is increased by one, and all nodes in the new group are added to the unified management service module.
8. The large-scale storage cluster deployment grouping device according to claim 6, characterized in that: When the management node needs to obtain data from each node, it establishes communication with the unified management service module and sends request data to the unified management service module. The unified management service module routes the request data to a storage node in the group, and the storage node collects information about all nodes in the group.
9. The large-scale storage cluster deployment grouping device according to claim 6, characterized in that: When the management node needs each node to execute a service, it establishes communication with the unified management service module and sends request data to the unified management service module. The unified management service module routes the request data to a storage node in the group, and the storage node distributes the configuration instructions.
10. The large-scale storage cluster deployment grouping device according to claim 6, characterized in that: When an abnormality occurs in a node, the unified management service module will automatically detect the abnormality and remove the node from the unified management service list. The requested data will not be routed to the faulty node.
