Cluster deployment method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a cluster deployment method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring cluster configuration information corresponding to a plurality of cluster categories and disk information corresponding to a plurality of node servers; on the basis of the cluster configuration information, establishing respective cluster architecture of each cluster category; performing type analysis on respective disk information of each node server, and determining respective corresponding disk type of each node server; for each node server, based on the disk type, determining a target architecture of which the cluster category is matched with the node server from each cluster architecture; and deploying each node server in the corresponding target architecture to obtain a service cluster corresponding to each cluster category. By adopting the method, the cluster building efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a cluster deployment method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the development of computer technology, cluster deployment technology has emerged. When building a cluster, technicians are required to manually configure the complete configuration information required for cluster deployment according to the on-site environment and deploy the cluster. At the same time, technicians are also required to query logs to track the startup status.

[0003] However, since the configuration information required to build a cluster is redundant, the manual method of building a cluster has the problem of low cluster building efficiency. Summary of the invention

[0004] Based on this, it is necessary to provide a cluster deployment method, apparatus, computer equipment, computer-readable storage medium and computer program product that can improve the efficiency of cluster construction in response to the above technical problems.

[0005] In a first aspect, the present application provides a cluster deployment method, comprising:

[0006] Obtain cluster configuration information corresponding to each of multiple cluster categories and disk information corresponding to each of multiple node servers;

[0007] Based on the cluster configuration information, build a cluster architecture for each cluster category;

[0008] Performing type analysis on the disk information of each of the node servers respectively to determine the disk type corresponding to each of the node servers;

[0009] For each of the node servers, based on the disk type, determining a target architecture whose cluster type matches the node server from each of the cluster architectures;

[0010] Each of the node servers is deployed in a corresponding target architecture to obtain a service cluster corresponding to each of the cluster categories.

[0011] In one embodiment, obtaining cluster configuration information corresponding to each of the plurality of cluster categories includes:

[0012] Obtain cluster identification information corresponding to each of multiple cluster categories;

[0013] For each of the cluster categories, information is collected on the cluster category based on the cluster identification information to obtain cluster configuration information of the cluster category.

[0014] In one embodiment, for each of the node servers, based on the disk type, determining a target architecture whose cluster type matches the node server from each of the cluster architectures includes:

[0015] For each of the node servers, determining the number of disk types included in the node server;

[0016] In the case that the number of the types is one, based on the disk capacity of each disk included in the node server, a target architecture whose cluster type matches the node server is determined from the cluster architectures.

[0017] In one embodiment, the method further comprises:

[0018] In the case where there are multiple types, determining the number of disks corresponding to each of the disk types;

[0019] If the number of disks corresponding to the first disk type among the disk types is one, the cluster architecture corresponding to the first disk type is determined as the target architecture that matches the node server.

[0020] In one embodiment, the method further comprises:

[0021] In response to a cluster expansion instruction for a deployed service cluster, determining a configuration file corresponding to the cluster expansion instruction;

[0022] Matching the cluster expansion information contained in the configuration file with each of the cluster configuration information, and if the match is successful, adding the expansion content contained in the configuration file to the target service cluster corresponding to the target cluster configuration information that successfully matches the cluster expansion information;

[0023] If the matching fails, a new update service cluster corresponding to the cluster expansion information is created based on the cluster expansion information.

[0024] In one embodiment, adding the expansion content included in the configuration file to the target service cluster corresponding to the target cluster configuration information that successfully matches the cluster expansion information includes:

[0025] Determining an expansion type of the expansion content;

[0026] If the expansion content is a disk, determine a target node server that matches the disk, and add the disk to the target node server;

[0027] If the expanded content is a node server, the node server is added to the target service cluster corresponding to the target cluster configuration information that successfully matches the node server.

[0028] In a second aspect, the present application also provides a cluster deployment device, including:

[0029] An information acquisition module, used to acquire cluster configuration information corresponding to each of the multiple cluster categories, and disk information corresponding to each of the multiple node servers;

[0030] A cluster architecture building module, used to build a cluster architecture of each cluster category based on the cluster configuration information;

[0031] A type analysis module, used to perform type analysis on the disk information of each of the node servers to determine the disk type corresponding to each of the node servers;

[0032] A target architecture determination module, configured to determine, for each of the node servers, a target architecture whose cluster type matches the node server from among the cluster architectures based on the disk type;

[0033] The cluster deployment module is used to deploy each of the node servers in the corresponding target architecture to obtain a service cluster corresponding to each of the cluster categories.

[0034] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the above method are implemented.

[0035] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0036] In a fifth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0037] The above-mentioned cluster deployment method, apparatus, computer equipment, computer-readable storage medium and computer program product obtain cluster configuration information corresponding to each of the multiple cluster categories and disk information corresponding to each of the multiple node servers, and can determine the basic situation of the cluster deployment. Based on the cluster configuration information, a cluster architecture of each cluster category can be built. The pre-steps of cluster deployment can be implemented first to build an initial cluster framework without server deployment, and then the type of disk information of each node server can be analyzed to determine the disk type corresponding to each node server. For each node server, based on the disk type, a target architecture that matches the cluster category and the node server is determined from each cluster architecture, and each node server is deployed in the corresponding target architecture to obtain a service cluster corresponding to each cluster category. The node server can be assigned a cluster architecture according to the disk type, and while ensuring the accuracy of cluster deployment, automated cluster deployment can be achieved to improve the efficiency of cluster deployment. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 An application environment diagram of a cluster deployment method in an embodiment;

[0040] Figure 2 A schematic diagram of a cluster deployment method in one embodiment;

[0041] Figure 3 A flowchart of cluster deployment information in one embodiment;

[0042] Figure 4 A schematic diagram of a cluster deployment method according to another embodiment;

[0043] Figure 5 A flowchart of a target architecture matching step in one embodiment;

[0044] Figure 6 A diagram showing a scenario of capacity expansion and grid connection of three types of clusters in an embodiment;

[0045] Figure 7 A schematic diagram of a flow chart of cluster expansion and grid connection steps in an embodiment;

[0046] Figure 8 A schematic diagram of a cluster deployment method in another embodiment;

[0047] Fig. 9 It is a structural block diagram of a cluster deployment device in one embodiment;

[0048] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0050] The cluster deployment method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the management terminal 102 communicates with multiple node servers 104 through the network. Among them, the management terminal 102 can be, for example, Ansible Tower (Ansible management node), which is essentially a unified management interface of Ansible, similar to the management platform in virtualization. It allows users to intuitively see the status of the managed system through a graphical interface, and at the same time provides data center automation management and control functions combined with operations, logs, audits, version control, etc. Node server 104, in short, is a server that acts as an independent computing and storage unit in a cluster environment. In a cluster, each such server is regarded as a node, and they are interconnected through a network and work together to complete large computing tasks or store massive data. Specifically, during the cluster deployment process, the management end 102 can obtain cluster configuration information corresponding to each of the multiple cluster categories, and disk information corresponding to each of the multiple node servers 104; based on the cluster configuration information, build a cluster architecture for each cluster category; perform type analysis on the disk information of each node server 104 to determine the disk type corresponding to each node server 104; for each node server 104, based on the disk type, determine from each cluster architecture the target architecture that matches the cluster category and the node server 104; deploy each node server 104 in the corresponding target architecture to obtain a service cluster corresponding to each cluster category.

[0051] In an exemplary embodiment, Figure 2 As shown, a cluster deployment method is provided, which is applied to Figure 1 The management terminal 102 in the example is used to illustrate, including the following steps S202 to S210. Among them:

[0052] Step S202, obtaining cluster configuration information corresponding to each of the plurality of cluster types and disk information corresponding to each of the plurality of node servers.

[0053] Among them, the cluster is composed of multiple node servers, which are interconnected through the network and work together to provide high-performance and scalable services. There are multiple cluster categories, such as management and control clusters and storage clusters. Cluster configuration information refers to the information required to configure the cluster deployment process, such as network configuration information of different network interfaces of node servers, system kernel architecture information, software version and disk array information, storage multipath configuration, etc. In short, a node server is a server that acts as an independent computing and storage unit in a cluster environment. In a cluster, each such server is regarded as a node, which is interconnected through the network and works together to complete large computing tasks or store massive data. Disk information refers to the information of the disk contained in the node server, for example, the disk information may include disk type, disk identification, disk address, etc.

[0054] Specifically, when the management end is performing cluster deployment, it is necessary to first determine the cluster category corresponding to the cluster, determine the cluster configuration information corresponding to each cluster category, and determine the disk information corresponding to all node servers involved in the cluster deployment process, so as to determine the initial situation of the cluster deployment, so as to subsequently deploy the cluster. Exemplarily, the operation of obtaining the cluster configuration information corresponding to each of the multiple cluster categories and the disk information corresponding to each of the multiple node servers can be actively obtained or passively received. In a specific embodiment, the management end can directly obtain the cluster identification information corresponding to each of the multiple clusters to be built, and for each cluster to be built, collect information for the cluster to be built based on the cluster identification information to obtain the cluster configuration information of the cluster to be built. In another specific embodiment, the management end can also obtain the cluster configuration information corresponding to each of the multiple cluster categories collected by the information collection device and the disk information corresponding to each of the multiple node servers from the information collection device.

[0055] Step S204: Building a cluster architecture for each cluster type based on the cluster configuration information.

[0056] The cluster architecture refers to an architectural design that combines multiple node group servers into an overall system. In this embodiment, the cluster architecture specifically refers to a cluster framework structure that has complete cluster configuration information but has not yet deployed node servers.

[0057] Specifically, after obtaining the cluster configuration information corresponding to each of the multiple cluster types, the management end can build a cluster architecture corresponding to each cluster type based on the above cluster configuration information to facilitate subsequent node server deployment.

[0058] Step S206: perform type analysis on the disk information of each node server to determine the disk type corresponding to each node server.

[0059] Among them, the disk types may include: OST (Object Storage Target): a component device used to store the actual data of the file in the Lustre file system; MDT (Metadata Target): a component device used to store the metadata of the file in the Lustre file system, such as the file size, creation time, permissions and other information; MGT (Management Target): a component device used to store the management information and configuration data of the file system in the Lustre file system.

[0060] Specifically, after building the cluster architecture for each cluster category, node servers can be deployed for each cluster architecture. First, the node servers need to be distinguished, that is, the management end needs to perform type analysis on the disk information of each node server separately to determine the disk type corresponding to each node server.

[0061] Step S208 : for each node server, based on the disk type, determine a target architecture whose cluster type matches the node server from various cluster architectures.

[0062] The target architecture is the cluster architecture selected for performing a specific operation.

[0063] Specifically, after determining the disk type corresponding to each node server, the management end can determine the cluster architecture corresponding to the node server for each node server, that is, for each node server, the management end can judge the cluster category corresponding to each cluster architecture according to the disk type of the node server, and determine the target architecture that matches the cluster category and the node server from each cluster architecture. In some specific embodiments, the management end can determine the number of types of disk types contained in the node server for each node server, and when the number of types is one, determine the target architecture that matches the cluster category and the node server from each cluster architecture based on the disk capacity of each disk contained in the node server; when the number of types is multiple, determine the number of disks corresponding to each disk type, and if the number of disks corresponding to the first disk type in each disk type is one, determine the cluster architecture corresponding to the first disk type as the target architecture that matches the node server.

[0064] Step S210 , deploying each node server in a corresponding target architecture to obtain a service cluster corresponding to each cluster category.

[0065] Among them, the service cluster is a cluster that has been fully deployed and can provide services.

[0066] Specifically, after determining the target architecture that matches each node server, the management end can deploy each node server in the corresponding target architecture, so as to obtain the service cluster corresponding to each cluster category.

[0067] The above-mentioned cluster deployment method obtains cluster configuration information corresponding to each of the multiple cluster categories and disk information corresponding to each of the multiple node servers, so as to determine the basic situation of cluster deployment. Based on the cluster configuration information, the cluster architecture of each cluster category can be built. The pre-order steps of cluster deployment can be implemented first, and the initial cluster framework without server deployment can be built. Then, the type of disk information of each node server can be analyzed to determine the disk type corresponding to each node server. For each node server, based on the disk type, the target architecture that matches the cluster category and the node server is determined from each cluster architecture, and each node server is deployed in the corresponding target architecture to obtain the service cluster corresponding to each cluster category. The node server can be assigned a cluster architecture according to the disk type, so as to realize automated cluster deployment while ensuring the accuracy of cluster deployment and improve the efficiency of cluster deployment.

[0068] In an exemplary embodiment, obtaining cluster configuration information corresponding to each of the plurality of cluster categories includes: obtaining cluster identification information corresponding to each of the plurality of cluster categories; and for each cluster category, collecting information about the cluster category based on the cluster identification information to obtain cluster configuration information of the cluster category.

[0069] The cluster identification information is specific information used to uniquely identify and distinguish a cluster.

[0070] Specifically, the management end may first obtain cluster identification information corresponding to each of the multiple cluster categories, and based on the cluster identification information, collect information for each cluster category to obtain cluster configuration information corresponding to each of the multiple cluster categories.

[0071] In a specific embodiment, the management end can first obtain cluster identification information for cluster deployment corresponding to each of multiple cluster categories. This information is relatively simple and cannot be used to build a complete cluster for the time being. Then, the new configuration file is parsed using the ansible module of python. It performs the information collection function based on the script file with the written configuration items, such as collecting network configuration information of different network interfaces of the node server, system kernel architecture information, software version and disk array information, storage multipath configuration, etc.; finally, after completing the collection and organization of information about the controlled node servers, the cluster configuration information required for building the cluster is obtained.

[0072] In this embodiment, cluster identification information corresponding to each cluster category is first obtained, and then information is collected for each cluster category based on the cluster identification information. This is different from the traditional deployment method that relies on manual configuration of cluster configuration information, and can avoid the inefficiency caused by manual configuration.

[0073] In an exemplary embodiment, for each node server, based on the disk type, a target architecture that matches the cluster category and the node server is determined from each cluster architecture, including: for each node server, determining the number of types of disk types contained in the node server; when the number of types is one, based on the disk capacity of each disk contained in the node server, determining the target architecture that matches the cluster category and the node server from each cluster architecture.

[0074] Among them, disk capacity, also commonly known as hard disk capacity, is a parameter that measures the data storage capacity of computer storage devices.

[0075] Specifically, the matching process of the cluster architecture and the node server is related to the number of types and the disk capacity of the disk type. For each node server, the management end needs to first determine the number of types of disk types contained in the node server. If the number of types is one, it means that the disks contained in the node server are all of a certain disk type. Then, it is necessary to determine the target architecture that matches the cluster category and the node server from each cluster architecture according to the disk capacity of each disk contained in the node server. In a specific embodiment, if the disk types all meet the pre-defined ost (Object Storage Target, object data storage target) specification range, then the cluster architecture oss is used as the target architecture that matches the node server; if the disk types all meet the pre-defined mdt (Metadata Target, metadata target) specification range, then the cluster architecture mds is used as the target architecture that matches the node server; and if the disk types all meet the pre-defined mgt (Management Target, management target) rule range, then the cluster architecture mgs is used as the target architecture that matches the node server.

[0076] After the judgment is completed, it will be judged again whether the number of mds and oss ​​required for this deployment meets the requirements, that is, whether there is at least a pair of mgs (Management Server) and a pair of oss for building the cluster. If not, it means that the integration is incorrect and does not meet the requirements. If it is satisfied, continue with the subsequent deployment steps.

[0077] In this embodiment, the number of types of disk types included in the node server is first determined. When the number of types is one, the target architecture is determined based on the disk capacity of each disk included in the node server, which can ensure the accuracy of the match between the cluster architecture and the node server.

[0078] In an exemplary embodiment, the cluster deployment method also includes: when the number of types is multiple, determining the number of disks corresponding to each disk type; if the number of disks corresponding to the first disk type in each disk type is one, then determining the cluster architecture corresponding to the first disk type as the target architecture that matches the node server.

[0079] In this embodiment, the first disk type refers to the disk type corresponding to mgt.

[0080] Specifically, when there are multiple types, the number of disks corresponding to each disk type is determined. If the number of disks corresponding to the first disk type (mgt) in each disk type is one, and the remaining disk types are all mdt, the cluster architecture corresponding to the first disk type is determined as the target architecture that matches the node server, which can ensure the accuracy of the match between the cluster architecture and the node server.

[0081] In an exemplary embodiment, the cluster deployment method also includes: in response to a cluster expansion instruction for a service cluster after deployment, determining a configuration file corresponding to the cluster expansion instruction; matching the cluster expansion information contained in the configuration file with each cluster configuration information, and if the match is successful, adding the expansion content contained in the configuration file to the target service cluster corresponding to the target cluster configuration information that successfully matches the cluster expansion information; if the match fails, creating a new update service cluster corresponding to the cluster expansion information based on the cluster expansion information.

[0082] The cluster expansion instruction refers to an instruction for expanding the cluster, for example, the cluster can be expanded or the node server can be expanded. The configuration file refers to a file for parameter configuration or initialization configuration corresponding to the cluster expansion. Specifically, the configuration file contains cluster expansion information. The cluster expansion information mainly indicates specific information for cluster expansion, for example, it can be the specific content of cluster expansion.

[0083] Specifically, as the number of users increases or users demand new types of storage devices and network types, when the original cluster capacity and carrying capacity become more challenging, technicians are required to expand and network the underlying storage cluster so that it can effectively operate in large-scale HPC and other application scenarios that require high throughput and low latency. Among the three types of cluster expansion and networking scenarios, the first is the linear expansion of Lustre storage components OST and MDT, which can add storage hardware media to server nodes and add new node servers to the cluster to achieve linear expansion of cluster capacity; the second is to add Lustre disk MDS (Metadata Server) and disk OSS (ObjectStorage Server) to achieve distributed expansion. The node server has a newly mapped disk array, and the new node and storage media are expanded in the original cluster to achieve distributed expansion, increase capacity and improve IO read and write carrying performance; the third is to expand new types of clusters based on the original control end, such as building a cluster of high-performance storage media or a cluster of high-performance network bandwidth.

[0084] Exemplarily, the management end can respond to a cluster expansion instruction for a service cluster after deployment, determine the configuration file corresponding to the cluster expansion instruction, and match the cluster expansion information contained in the configuration file with each cluster configuration information. If the match is successful, it means that the original cluster contains the content of the above-mentioned cluster expansion information. Therefore, the expansion content contained in the configuration file needs to be added to the target service cluster corresponding to the target cluster configuration information that successfully matches the cluster expansion information; if the match fails, then based on the cluster expansion information, a new updated service cluster corresponding to the cluster expansion information is created.

[0085] In this embodiment, after receiving the cluster expansion instruction, the cluster may be expanded according to the cluster expansion information included in the cluster expansion instruction, thereby achieving better load balancing and performance optimization.

[0086] In an exemplary embodiment, the expansion content contained in the configuration file is added to the target service cluster corresponding to the target cluster configuration information that successfully matches the cluster expansion information, including: determining the expansion type of the expansion content; if the expansion content is a disk, determining the target node server that matches the disk, and adding the disk to the target node server; if the expansion content is a node server, adding the node server to the target service cluster corresponding to the target cluster configuration information that successfully matches the node server.

[0087] Specifically, when the cluster expansion information contained in the configuration file successfully matches the cluster configuration information, expansion can be performed based on the expansion type of the expansion content. If the expansion content is a disk, the target node server that matches the disk is determined, and the disk is added to the target node server; if the expansion content is a node server, the node server is added to the target service cluster corresponding to the target cluster configuration information that successfully matches the node server.

[0088] In this embodiment, further capacity expansion can be performed based on the capacity expansion content of the capacity expansion content, thereby achieving better load balancing and performance optimization.

[0089] In a specific embodiment, a cluster deployment method in a specific application scenario is provided, including:

[0090] First, build a highly readable and easily extensible configuration file based on the inventory file.

[0091] Table 1 Example of deployment configuration in Inventory format

[0092]

[0093] Among them, as shown in Table 1, the Inventory file is a configuration file for the Ansible operation and maintenance tool to manage host information. It can easily build a host name list of various types of hosts, and specify the variable configuration of the host through the commonly used custom host type and "vars" and "children" configurations. For example, the configuration items under "all:vars" represent the public configuration information of the global node, and the configuration items under "typeX:vars" represent the configuration information of the type typeX node; the nodes under "typeX:children" represent multiple nodes of this type; as shown in Table 1, it shows the simple inventory configuration file required for deployment; cm* represents the control node, and cm:vars represents the public network configuration required for the control node;

[0094] Traditional deployment methods rely on manual configuration of complete configuration information required for one-click deployment. The configuration items are complicated and errors are difficult to troubleshoot. This method aims to collect necessary configuration items that cannot be collected and detected from the nodes, such as special configurations such as the virtual IP of keepalived, CN2 gateway, and cluster name. The process of parsing, collecting and building the cluster deployment information required for one-click deployment is as follows: Figure 3 As shown:

[0095] First, the cluster identification information for cluster deployment is obtained. This information is relatively simple and cannot be used to build a complete cluster for the time being. Then, the new configuration file is parsed using the ansible module of python. It performs the function of information collection according to the script file of the written configuration items, such as collecting network configuration information of different network interfaces of the node server, system kernel architecture information, software version and disk array information, storage multipath configuration, etc. Finally, after collecting and organizing the information about the controlled node server, the cluster configuration information required for building the cluster is obtained. Through this inventory-based approach, the cluster environment configuration that needs to be deployed can be configured, added, and modified very flexibly.

[0096] Secondly, Ansible multi-point control automation deployment strategy based on integrated features

[0097] The above describes the configuration files that need to be prepared in advance when building a cluster. They can be parsed into the configuration items required for management and backend storage. After obtaining these configuration items, they can be applied to the deployment of Lustre storage cluster and management cluster. The initial deployment process is as follows: Figure 4 The detailed steps are as follows:

[0098] S401. Determine the cluster configuration information corresponding to each cluster category:

[0099] This configuration file contains the specified information required by the cluster, such as the cluster name, lnet network type, management database type, various network devices and IP addresses, etc., for high-availability master and backup host pairs and other information; this type of configuration file is parsed by Python's Ansible module to obtain complete deployment configuration items for management and backend nodes.

[0100] S402. Implement parallel distribution of data packets and scripts for node servers to be deployed based on Ansible:

[0101] Utilize Ansible's multi-point control and data distribution capabilities to transmit the Lustre and control installation packages required for deployment, as well as the scripts required to be called during the deployment process;

[0102] S403. Determine the target architecture corresponding to each node server;

[0103] In this step, first collect the system architecture and network configuration information of the cluster in s3.1, which is used to install the installation dependency packages of the corresponding architecture and kernel version, set up firewalls, configure lnet networks, configure array multipaths, and collect features to transmit back to deployment nodes. After that, in the s3.2 deployment process, the component identity (target architecture) of the node server can be identified based on the disk and high-availability host pair information. After identification, the disk is formatted and the cluster is built.

[0104] The general process of identifying the identity of components according to integration is as follows Figure 5 The detailed steps are as follows:

[0105] Step S501, for each node server, determining the number of disk types included in the node server;

[0106] Step S502, when the number of types is one, based on the disk capacity of each disk included in the node server, determine a target architecture whose cluster type matches the node server from each cluster architecture;

[0107] Step S503, when there are multiple types, determine the number of disks corresponding to each disk type. If the number of disks corresponding to the first disk type among the disk types is one, determine the cluster architecture corresponding to the first disk type as the target architecture that matches the node server.

[0108] Specifically, the range of the Lustre components mdt, mgt and ost disks is pre-set. mdt is used to store metadata, mgt is used to store management data, and ost is a cluster data storage disk. There are obvious limits on the capacity of the three types of disks in the production environment. At the beginning, the sequence numbers of mdt and ost are both 0. The node server is traversed serially, and the disk that needs to be formatted is identified based on the storage multi-path of the disk array. The disks are traversed in turn to obtain the disk name and disk capacity. After traversing all the array disks of the storage server node, its Lustre component identity is determined according to the integration characteristics. If it meets the requirements of single mgt, If the remaining conditions are mdt, the node is identified as mgs. If the conditions of the three types of servers are met, the component identity is marked, and the serial numbers of mdt and ost are updated, and the traversal of the next pair of storage server nodes continues. On the contrary, if the conditions of the three types of storage servers are not met, it is judged that the integration configuration is wrong and does not meet the requirements, and the execution fails. After traversing all nodes, it will be judged again whether the number of mds nodes and oss ​​nodes required for this deployment meets the requirements, and at least one pair of mgs and one pair of oss are used to build the cluster. If not, it means that the integration is incorrect and does not meet the requirements. If it meets the requirements, continue with the subsequent deployment steps.

[0109] s404. Build a high-availability cluster;

[0110] After the cluster is built, it does not yet have high availability capabilities. It is necessary to configure high availability based on the network information of the host.

[0111] S405. Cluster bottom performance test;

[0112] At this point, the high-availability cluster has been successfully built. The underlying testers can use this to test the performance of the underlying storage cluster without being limited by the influence of the control end, thus achieving the separation of control and backend deployment.

[0113] S406. Collect and deploy network and disk information of underlying storage;

[0114] S407. Deploy the management and control end (management and control cluster) and the database used.

[0115] In the above deployment process, the management and storage clusters are deployed separately, and the component identities are identified based on the disk array. This process only requires changing the configuration file during capacity expansion, which can realize management and control reuse, mixed deployment of multiple types of storage clusters, and cluster expansion and networking in the production environment.

[0116] The previous section introduced the initial deployment process of the Lustre cluster. As the number of users increases or users demand new types of storage devices and network types, the original cluster capacity and carrying capacity become more challenging. At this time, technicians are required to expand the capacity of the underlying storage cluster and connect it to the network so that it can operate effectively in large-scale HPC and other application scenarios that require high throughput and low latency. Figure 6 Three types of cluster expansion and networking scenarios are demonstrated. The first type is the linear expansion of Lustre storage components OST and MDT, which can add storage hardware media to server nodes and add the newly added node servers to the cluster to achieve linear expansion of cluster capacity; the second type is to add Lustre disk MDS (Metadata Server) and disk OSS (Object Storage Server) to achieve distributed expansion. The node server has a newly mapped disk array, and the new node and storage media are expanded in the original cluster to achieve distributed expansion, increase capacity and improve IO read and write performance; the third type is to expand new types of clusters based on the original control end, such as building a cluster with high-performance storage media or a cluster with high-performance network bandwidth.

[0117] The flow chart of cluster expansion and grid connection is as follows: Figure 7 As shown in the figure, the steps for cluster expansion and grid connection are as follows:

[0118] Step S701, in response to a cluster expansion instruction for a deployed service cluster, determining a configuration file corresponding to the cluster expansion instruction;

[0119] Step S702, matching the cluster expansion information contained in the configuration file with each cluster configuration information; Step S703, if the match is successful, determining the expansion type of the expansion content;

[0120] Step S704: if the expansion content is a disk, determine a target node server that matches the disk, and add the disk to the target node server;

[0121] Step S705: If the expanded content is a node server, the node server is added to the target service cluster corresponding to the target cluster configuration information that successfully matches the node server;

[0122] Step S706: If the matching fails, a new update service cluster corresponding to the cluster expansion information is created based on the cluster expansion information.

[0123] Specifically, first, the control cluster checks whether the cluster expansion information in the configuration file already exists. If it does, it is determined that the cluster already in the control cluster is expanded and connected to the network. If it does not exist, it is a new Lustre storage cluster that reuses the control end. If it is an existing cluster, it is determined whether the node server is already in the cluster. If it is, the newly added storage media on the node server are collected and these storage disks are merged into the storage cluster. If the node server is not yet in the cluster, the storage media, network configuration, system architecture and other integrated information on the new node server are collected, and these new storage servers are merged into the existing cluster after deployment. If a new type of cluster is built, the deployment process is consistent with the initial deployment, and finally it needs to be merged into the control end. The above cluster expansion and networking operations realize the reuse of the same control end, and can flexibly expand and configure the cluster capacity, newly added distributed nodes and newly added types of clusters. After expansion, better load balancing and performance optimization can be achieved.

[0124] In a specific embodiment, Figure 8 As shown, a cluster deployment method is also provided, including:

[0125] Step S801, obtaining cluster identification information corresponding to each of a plurality of cluster categories and disk information corresponding to each of a plurality of node servers;

[0126] Step S802: for each cluster category, collect information about the cluster category based on the cluster identification information to obtain cluster configuration information of the cluster category;

[0127] Step S803: Based on the cluster configuration information, build a cluster architecture for each cluster type, analyze the disk information of each node server, and determine the disk type corresponding to each node server;

[0128] Step S804, for each node server, determining the number of disk types included in the node server;

[0129] Step S805, when the number of types is one, based on the disk capacity of each disk included in the node server, determine a target architecture whose cluster type matches the node server from each cluster architecture;

[0130] Step S806, when there are multiple types, determine the number of disks corresponding to each disk type. If the number of disks corresponding to the first disk type among the disk types is one, determine the cluster architecture corresponding to the first disk type as the target architecture that matches the node server.

[0131] Step S807, deploying each node server in the corresponding target architecture to obtain a service cluster corresponding to each cluster category;

[0132] Step S808, in response to the cluster expansion instruction for the deployed service cluster, determining a configuration file corresponding to the cluster expansion instruction;

[0133] Step S809, matching the cluster expansion information contained in the configuration file with each cluster configuration information; Step S810, if the match is successful, determining the expansion type of the expansion content;

[0134] Step S811, if the expansion content is a disk, determine a target node server that matches the disk, and add the disk to the target node server;

[0135] Step S812: If the expanded content is a node server, the node server is added to the target service cluster corresponding to the target cluster configuration information that successfully matches the node server;

[0136] Step S813: If the matching fails, a new update service cluster corresponding to the cluster expansion information is created based on the cluster expansion information.

[0137] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0138] Based on the same inventive concept, the embodiment of the present application also provides a cluster deployment device for implementing the cluster deployment method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more cluster deployment device embodiments provided below can refer to the limitations of the cluster deployment method above, and will not be repeated here.

[0139] In an exemplary embodiment, Fig. 9 As shown, a cluster deployment device 900 is provided, including: an information acquisition module 902, a cluster architecture building module 904, a type analysis module 906, a target architecture determination module 908 and a cluster deployment module 910, wherein:

[0140] The information acquisition module 902 is used to acquire cluster configuration information corresponding to each of the plurality of cluster categories and disk information corresponding to each of the plurality of node servers;

[0141] The cluster architecture building module 904 is used to build the cluster architecture of each cluster category based on the cluster configuration information;

[0142] The type analysis module 906 is used to perform type analysis on the disk information of each node server to determine the disk type corresponding to each node server;

[0143] A target architecture determination module 908 is used to determine, for each node server, a target architecture whose cluster type matches the node server from each cluster architecture based on the disk type;

[0144] The cluster deployment module 910 is used to deploy each node server in the corresponding target architecture to obtain a service cluster corresponding to each cluster category.

[0145] In an exemplary embodiment, the information acquisition module 902 is specifically used to: acquire cluster identification information corresponding to each of the plurality of cluster categories;

[0146] For each cluster category, information about the cluster category is collected based on the cluster identification information to obtain cluster configuration information of the cluster category.

[0147] In an exemplary embodiment, the target architecture determination module 908 is specifically used to: for each node server, determine the number of types of disk types included in the node server;

[0148] When the number of types is one, based on the disk capacity of each disk included in the node server, a target architecture whose cluster type matches the node server is determined from each cluster architecture.

[0149] In an exemplary embodiment, the cluster deployment device 900 further includes a target architecture matching module, which is specifically used to: determine the number of disks corresponding to each disk type when there are multiple types;

[0150] If the number of disks corresponding to the first disk type among the disk types is one, the cluster architecture corresponding to the first disk type is determined as the target architecture that matches the node server.

[0151] In an exemplary embodiment, the cluster deployment device 900 further includes a cluster expansion module, including:

[0152] An instruction response unit, configured to respond to a cluster expansion instruction for a deployed service cluster and determine a configuration file corresponding to the cluster expansion instruction;

[0153] A content adding unit is used to match the cluster expansion information contained in the configuration file with each cluster configuration information. If the match is successful, the expansion content contained in the configuration file is added to the target service cluster corresponding to the target cluster configuration information that successfully matches the cluster expansion information;

[0154] The cluster creation unit is used to create a new update service cluster corresponding to the cluster expansion information based on the cluster expansion information if the matching fails.

[0155] In an exemplary embodiment, the content adding unit is specifically used to: determine the expansion type of the expanded content;

[0156] If the expansion content is a disk, determine the target node server that matches the disk, and add the disk to the target node server;

[0157] If the expansion content is a node server, the node server is added to the target service cluster corresponding to the target cluster configuration information that successfully matches the node server.

[0158] Each module in the above cluster deployment device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0159] In an exemplary embodiment, a computer device is provided. The computer device may be a management terminal, and its internal structure diagram may be as shown in FIG. Fig.10As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with the external management end in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (Near Field Communication, NFC) or other technologies. When the computer program is executed by the processor, a cluster deployment method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0160] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0161] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above method when executing the computer program.

[0162] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0163] In one embodiment, a computer program product is provided, comprising a computer program, which implements the steps of the above method when executed by a processor.

[0164] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0165] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0166] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0167] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A cluster deployment method, characterized in that: The method comprises: Obtain cluster configuration information corresponding to each of multiple cluster categories and disk information corresponding to each of multiple node servers; Based on the cluster configuration information, build a cluster architecture for each cluster category; Performing type analysis on the disk information of each of the node servers respectively to determine the disk type corresponding to each of the node servers; For each of the node servers, based on the disk type, determining a target architecture whose cluster type matches the node server from each of the cluster architectures; Each of the node servers is deployed in a corresponding target architecture to obtain a service cluster corresponding to each of the cluster categories.

2. The method according to claim 1, characterized in that The obtaining of cluster configuration information corresponding to each of the plurality of cluster categories includes: Obtain cluster identification information corresponding to each of multiple cluster categories; For each of the cluster categories, information is collected on the cluster category based on the cluster identification information to obtain cluster configuration information of the cluster category.

3. The method according to claim 1, characterized in that For each of the node servers, based on the disk type, determining a target architecture whose cluster type matches the node server from each of the cluster architectures includes: For each of the node servers, determining the number of disk types included in the node server; In the case that the number of the types is one, based on the disk capacity of each disk included in the node server, a target architecture whose cluster type matches the node server is determined from the cluster architectures.

4. The method according to claim 2, characterized in that: The method further comprises: In the case where there are multiple types, determining the number of disks corresponding to each of the disk types; If the number of disks corresponding to the first disk type among the disk types is one, the cluster architecture corresponding to the first disk type is determined as the target architecture that matches the node server.

5. The method according to claim 1, characterized in that The method further comprises: In response to a cluster expansion instruction for a deployed service cluster, determining a configuration file corresponding to the cluster expansion instruction; Matching the cluster expansion information contained in the configuration file with each of the cluster configuration information, and if the match is successful, adding the expansion content contained in the configuration file to the target service cluster corresponding to the target cluster configuration information that successfully matches the cluster expansion information; If the matching fails, a new update service cluster corresponding to the cluster expansion information is created based on the cluster expansion information.

6. The method according to claim 5, characterized in that The adding the expansion content included in the configuration file to the target service cluster corresponding to the target cluster configuration information that successfully matches the cluster expansion information includes: Determining an expansion type of the expansion content; If the expansion content is a disk, determine a target node server that matches the disk, and add the disk to the target node server; If the expanded content is a node server, the node server is added to the target service cluster corresponding to the target cluster configuration information that successfully matches the node server.

7. A cluster deployment device, characterized in that: The device comprises: An information acquisition module, used to acquire cluster configuration information corresponding to each of the multiple cluster categories, and disk information corresponding to each of the multiple node servers; A cluster architecture building module, used to build a cluster architecture of each cluster category based on the cluster configuration information; A type analysis module, used to perform type analysis on the disk information of each of the node servers to determine the disk type corresponding to each of the node servers; A target architecture determination module, configured to determine, for each of the node servers, a target architecture whose cluster type matches the node server from among the cluster architectures based on the disk type; The cluster deployment module is used to deploy each of the node servers in the corresponding target architecture to obtain a service cluster corresponding to each of the cluster categories.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images