A distributed cluster configuration method based on a SHELL script

Abstract

The application provides a distributed cluster configuration method based on a SHELL script, comprising the following steps: architecture information and service distribution information of a cluster are intercepted by a script tool, and the intercepted information is stored in an associative array of the script tool; whether the cluster architecture information is a storage-computing integrated version or a storage-computing separated version is judged; a target configuration file is backed up, and the backup file is stored in a specified path; an instruction containing an operation type is received, the operation type comprises set, restart and rollback, and a parameter modification target is determined according to the instruction; before and after the parameter is modified, the parameter state is recorded and output. The application has the beneficial effects that the script tool is used to realize convenient modification of 8a cluster parameters, efficient restart of cluster services and quick recovery of configuration files, and through the script tool, configuration files of all cluster nodes corresponding services can be modified by one instruction.

Description

Technical Field

[0001] This invention belongs to the field of databases, and in particular relates to a distributed cluster configuration method based on SHELL scripts. Background Technology

[0002] Nanjing University General Large-Scale Distributed Parallel Database Cluster System, abbreviated as GBase 8a MPP Cluster, is a distributed parallel database cluster with a shared nothing architecture developed on the basis of GBase 8a column-store database. It has the characteristics of high performance, high availability, and high scalability, and can provide a cost-effective general computing platform for data management of various scales. It is widely used to support various data warehouse systems, BI systems and decision support systems.

[0003] Currently, the following issues exist when modifying read-only parameters on an 8a cluster:

[0004] Modifying 8a read-only parameters requires manually modifying the configuration file of the process to be modified on each node. When the cluster is large and the number of nodes is large, this operation is labor-intensive and prone to confusion. Once a configuration problem is found, the parameters need to be checked, confirmed, and restored again, which is a complex and time-consuming process. In addition, the modified 8a read-only parameters must be restarted by restarting the services of all nodes in the 8a cluster to take effect. Without a distributed cluster management tool, it is necessary to manually perform the restart on each node, which is inefficient. Furthermore, if the modification of the 8a configuration file is incorrect, it may cause the cluster service to fail to restart normally, and the recovery operation also needs to be performed on each node, resulting in a long exception handling time. Summary of the Invention

[0005] In view of this, the present invention aims to propose a distributed cluster configuration method based on SHELL scripts, so as to at least solve one of the problems in the background art.

[0006] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0007] A distributed cluster configuration method based on SHELL scripts includes:

[0008] The architecture and service distribution information of the cluster are captured by a script tool and stored in the associative array of the script tool. The architecture and service distribution information includes the node scale of the gcluster service, the node scale of the gnode service, and the IP address of the node.

[0009] According to the cluster architecture information, it is determined whether it is a storage-computing integrated version or a storage-computing separated version: if it is the storage-computing integrated version, the operation for the computing cluster is issued to all computing nodes, and if it is the storage-computing separated version, the operation for the computing cluster is only issued to the first computing node;

[0010] Backup the target configuration file, and store the backup file in a specified path;

[0011] Receive an instruction containing an operation type, the operation type including set, restart and rollback, and determine the parameter modification target according to the instruction;

[0012] Before and after modifying the parameters, record and output the parameter state.

[0013] Further, when the operation type is set, the node type of the target cluster is determined according to the input parameters, the specified parameter value is modified or appended in the target configuration file, and the script tool uses sed command and awk command to locate the process label range.

[0014] Further, when the operation type is restart, the script tool obtains the gcluster and gnode service node IP information, stores the host name and IP address to an associative array, and sends a restart command to the target cluster node.

[0015] Further, when the operation type is rollback, the script tool restores the configuration file content from the backup file, uses the diff command to compare the file differences and outputs the rollback result.

[0016] Further, the determination of the parameter modification target according to the instruction includes: when three parameters are received and the first parameter is set, the target cluster is determined according to the second parameter:

[0017] When the parameter is c, the configuration file of the gcluster cluster is modified;

[0018] When the parameter is n, the configuration file of the gnode cluster is modified;

[0019] When the parameter is cn, the configuration files of the gcluster and gnode clusters are modified;

[0020] When the parameter is pr, the environment variable configuration file is modified;

[0021] The third parameter format is "parameter name=parameter value", and if multiple parameters are included, the multiple parameters are separated by ";";

[0022] Iterate through the content matching the parameter name in the target configuration file, and execute according to the following rules:

[0023] If the number of matching lines is greater than 1, delete the duplicate lines one by one until only one line is left, and update the parameter value;

[0024] If the number of matching lines is 0, append the parameter under the corresponding process tag of the target configuration file;

[0025] After the modification is completed, the modified state of the parameter is retrieved and recorded again.

[0026] Further, when the received first parameter is restart, the script tool performs the restart of the cluster service through the following steps:

[0027] Get the IP information of all gcluster and gnode service nodes;

[0028] Store the hostname of each server and its IP address in an associative array to avoid repeatedly issuing commands in the gnode multi-IP deployment scenario;

[0029] Determine the restart target according to the second parameter:

[0030] When the parameter is c, only restart the gcluster cluster;

[0031] When the parameter is n, only restart the gnode cluster;

[0032] When the parameter is cn, restart the gcluster and gnode clusters simultaneously;

[0033] Iterate through the IP addresses in the associative array and send a restart command to the corresponding server one by one.

[0034] Further, the script tool adopts the following steps when performing the rollback operation:

[0035] Receive the first parameter as rollback and the second parameter as the target cluster type;

[0036] According to the target cluster type, restore the corresponding configuration file content from the backup file;

[0037] Use the diff command to compare the differences between the current configuration file and the backup file, and output the rollback content to the screen.

[0038] Further, the present scheme discloses a server, comprising at least one processor, and a memory in communication connection with the processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the processor to make the at least one processor execute a distributed cluster configuration method based on a SHELL script.

[0039] Further, the scheme discloses a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize a SHELL script-based distributed cluster configuration method.

[0040] Compared with the prior art, the SHELL script-based distributed cluster configuration method has the following beneficial effects:

[0041] The SHELL script-based distributed cluster configuration method can conveniently modify 8a cluster parameters, efficiently restart cluster services and quickly recover configuration files through a script tool, can modify configuration files of all cluster nodes through one instruction, can real-time feedback parameter states, can restart all cluster processes except gcware-related services through one instruction, can automatically return execution information, can provide configuration file backup and rollback functions, and can quickly recover to a state before modification when the modification result is not satisfied. BRIEF DESCRIPTION OF DRAWINGS

[0042] The accompanying drawings, which form a part of the present application, are intended to provide further understanding of the present application and are incorporated herein for explanation by way of the non-limiting embodiments of the present application. In the drawings:

[0043] Figure 1 A SHELL script-based distributed cluster configuration method according to an embodiment of the present application is shown in the accompanying drawings. DETAILED DESCRIPTION

[0044] It should be noted that the embodiments and features in the present application can be combined with each other without conflict.

[0045] The present application will be described in detail below with reference to the accompanying drawings and embodiments.

[0046] The present application discloses a SHELL script-based distributed cluster configuration method, which first explains the involved terms:

[0047] Read-only parameter: a parameter that cannot be modified online, and can only be modified by modifying a cluster configuration file, and then reading configuration file content by a database server process to take effect when the cluster is started;

[0048] Cluster configuration file: usually refers to

coordination cluster node configuration file: gbase_8a_gcluster.cnf, calculation cluster node configuration file: gbase_8a_gbase.cnf, cluster node environment variable configuration file: gbase_profile

[0049] 8a: GBase 8a MPP Cluster large-scale distributed parallel processing data warehouse in the following;

[0050] aset: the script tool based on shell language in the present application, used in Linux command line;

[0051] gcware: the consistency management process of GBase 8a MPP Cluste cluster;

[0052] gcluster: the coordination scheduling process of GBase 8a MPP Cluster cluster;

[0053] gnode: the data computing and storage process of GBase 8a MPP Cluster cluster;

[0054] gcadmin: GBase 8a MPP Cluster cluster operation and management command.

[0055] The specific scheme is as follows:

[0056] The aset acquires the current cluster architecture information and corresponding service distribution by intercepting the information returned by the gcadmin command, records the number of gcluster, gnode services and the IP address information, and stores them in different associative arrays of aset. When performing distribution action, the values in the corresponding service array are traversed to realize effective distribution operation of different nodes and different services. At the same time, the architecture version of the cluster is parsed. If it is a storage-computing integrated version, the operation on the computing cluster is issued to all computing nodes each time; if it is a storage-computing separated version, the operation on the computing cluster is only issued to the computing node in the first computing cluster each time. The value of the wh_num variable in aset can be modified to specify the modification plan for different computing clusters.

[0057] At each call, the aset will make a round of check judgment. First, check whether there is a gcware process in the current environment, if there is a gcware process, check whether the OS user of the current calling script and the OS user of starting the gcware process are consistent, if the OS user of starting the gcware is gbase8a and the user of the script calling is root, the aset will output the prompt information "please log in as gbase DBA user" after checking the inconsistency. Then, the aset will check whether the parameters are brought in the command and the number of parameters, if there is no parameter brought in, the prompt information and the help will be output, if the number of parameters is different from the specified one, the subsequent check will be directly skipped and the error prompt will be returned. The specified number of parameters in the check rule is 2 and 3 respectively, if 2 is met, the first parameter is brought into the judgment rule, if it is rollback or restart, the subsequent parameter will be checked, otherwise the check will be skipped and the error prompt will be returned; if 3 is met, the first parameter is brought into the judgment rule, if it is set or close, the subsequent parameter will be checked, otherwise the check will be skipped and the error result will be returned.

[0058] In each call of the script tool, the aset will obtain the current cluster size information in real time, without manual configuration. The aset will obtain the cluster architecture returned by the gcadmin command of 8a, format and split the gcluster cluster node information and gnode cluster node information by awk and sed commands, record all the node information to different arrays, and directly traverse the corresponding array to obtain the required node information when operating different cluster configuration files in the subsequent operation. The automatic acquisition not only provides the convenience of operation, but also guarantees the real-time and accuracy of each modification action.

[0059] When the aset receives three incoming parameters, and the first parameter is set, it continues to check the next two parameters, the second parameter is the corresponding target cluster, and the third parameter corresponds to the 8a cluster parameter to be modified in the configuration file. If the second parameter is c, modify the content input by the third parameter in the configuration file of all nodes of the gcluster cluster; if the second parameter is n, modify the content input by the third parameter in the configuration file of all nodes of the gnode cluster; if the second parameter is cn, modify the content input by the third parameter in the configuration file of the gcluster cluster and the gnode cluster; if the second parameter is pr, modify the environment variable in the environment variable file of the gcluster cluster and the gnode cluster. The content in the third incoming parameter is the 8a cluster parameter to be modified in the 8a cluster configuration file, and the incoming format of each 8a cluster is in the form of “parameter name = parameter value”, and multiple 8a cluster parameters can be separated by “;”. The aset will cut off the parameter name of each 8a cluster parameter, and do a full word matching search in the configuration file, output the content of the line that meets the matching condition as the pre-state record and display, and then count the number of output lines into the variable i, if i>1, it indicates that there is a situation of repeated setting of 8a cluster parameters in the configuration file, and the line numbers of these lines are traversed by the sed command, and then they are deleted, and i is reduced by 1 each time a line is deleted, when i=1, no longer delete and replace the parameter value of the line parameter with the target parameter value, if i=0 at the beginning, that is, there is no 8a cluster parameter in the configuration file, then the parameter is directly appended in the specified position in the redirection mode. After completing the modification, the parameter name is brought in again to do a full word matching on the configuration file, and the matching line content is output as the current state record and displayed. There are numerous parameters and different process tags in the configuration file of the 8a, and each process tag name is wrapped in square brackets, and the parameters under the process tag only take effect for the process corresponding to the tag, so the modification and addition of the parameters are effective only under the target process tag. Before each round of modification of the aset on the configuration file, the start_line variable is counted into the start_line variable, and the end_line variable is counted into the end_line variable, and the operation interval of the configuration file is controlled by the-n and p of sed to bring in the variable values of the above two line numbers, which effectively guarantees the correct effective bit of the parameter and the misoperation on the other process synchronization parameters.

[0060] Each server has its own unique hostname. When the first parameter received by the aset is restart, the IP information of all gcluster and gnode processes is obtained, the hostname of these IPs is obtained in turn, and the content of the hostname is stored in the associative array as the key value, and the corresponding IP address is stored as the value. In this way, even if a server has multiple IP addresses, only one IP will correspond to its hostname in the associative array. When issuing the restart operation, the value in the associative array is traversed, and the restart command is sent to the server with the corresponding IP. Even in the mixed deployment of gcluster and gnode or the deployment of multiple IP instances of gnode, the server will not receive the issued command repeatedly. At the same time, the aset will determine the object of the restart operation according to the content of the second parameter. If the parameter is c, only the gcluster cluster is restarted. If the parameter is n, only the gnode cluster is restarted. If the parameter is cn, the gcluster and gnode clusters are restarted at the same time.

[0061] Before performing a modification action on the configuration file each time, a backup of the current configuration file is made on each node, and then the configuration file is modified. In this way, each configuration file has a backup of its previous configuration state. If the first parameter received by the aset is rollback, the rollback action is triggered, the previously backed up configuration information is copied back to the current configuration file, and the parameter file is quickly restored. At the same time, the aset will determine the object of the rollback operation according to the content of the second parameter. If the parameter is c, only the configuration file of the gcluster cluster is rolled back. If the parameter is n, only the configuration file of the gnode cluster is rolled back. If the parameter is pr, only the configuration file of the environment variable is rolled back. During the rollback operation, the diff command is used to compare the differences between the current configuration file and the backup configuration file, and sed and awk are used to standardize the output format. The specific rollback parameters and parameter values are output to the screen for manual confirmation.

[0062] Through the implementation of the above scheme, the functional effects achieved are:

[0063] The tool execution user can be verified. If the system user currently calling the script tool is not the database administrator user in the system, the script tool will intercept the execution action and prompt the use of the database administrator in the system to execute;

[0064] The 8a cluster size (the IP address used by the cluster) is automatically located, the service process distribution (the IP address of each node where the service process is located) is automatically located, and the configuration file absolute path corresponding to the service process is automatically located;

[0065] Automatically determine whether it is a storage-computing integrated version of 8a cluster or a storage-computing separated version of 8a cluster, and automatically match the corresponding execution strategy according to different cluster versions;

[0066] The modification operation can be separately issued to the configuration file of all coordination cluster nodes, the modification operation can be separately issued to the configuration file of all computing cluster nodes, the modification operation can be simultaneously issued to the configuration files of all coordination cluster nodes and computing cluster nodes, and the modification operation can be separately issued to the environment variable configuration file of all cluster nodes;

[0067] Multiple read-only parameters to be modified can be issued by one instruction, and separated by “;” and modified by aset together;

[0068] The parameter in the configuration file of a certain service of a certain node can be output in real time during the modification process, and the parameter value state before modification and the parameter value state after modification;

[0069] Through one instruction, the cluster processes of non-gcware related services on all cluster nodes can be restarted, and the execution information on each node is returned;

[0070] The modification action is supported to be rolled back, when the result of the last parameter modification is not satisfied, the last modification operation can be restored to adjust the configuration file of a certain cluster, and restored to the state before the last modification through the rollback function.

[0071] Those skilled in the art can appreciate that the units and method steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in the above description in general. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Professionals can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0072] In several embodiments provided in the present application, it should be understood that the disclosed method and system can be implemented in other ways. For example, the division of the above-mentioned units is only a logical functional division, and actual implementation can have another division method, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The above-mentioned units can be or can not be physically separated, and the components displayed as units can be or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. According to the actual needs, part or all of the units can be selected to achieve the purpose of the embodiment of the present application.

[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not limited thereto; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application, and they should be covered in the scope of the claims and the description of the present application.

[0074] The above only describes the preferred embodiments of the present application, and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A distributed cluster configuration method based on SHELL scripts, characterized in that, include: The architecture and service distribution information of the cluster are captured by a script tool and stored in the associative array of the script tool. The architecture and service distribution information includes the node scale of the gcluster service, the node scale of the gnode service, and the IP address of the node. The architecture version of the cluster is analyzed to determine whether it is an integrated storage and compute version or a separate storage and compute version: if it is an integrated storage and compute version, operations on the compute cluster are distributed to all compute nodes; if it is a separate storage and compute version, operations on the compute cluster are distributed to only the first compute node. Backup the target configuration file and store the backup file in the specified path; Receive an instruction containing an operation type, including set, restart, and rollback, and determine the target parameter modification based on the instruction; When the operation type is set, the node type of the target cluster to be modified is determined according to the input parameters, and the specified parameter value is modified or appended in the target configuration file. The script tool uses the sed command and awk command to locate the process tag range. When the operation type is restart, the script tool obtains the IP information of gcluster and gnode service nodes, stores the hostname and IP address in an associative array, determines the target cluster node to restart based on the second parameter, and sends a restart command to the target cluster node; When the operation type is rollback, the first parameter is rollback and the second parameter is the target cluster type. Based on the target cluster type, the corresponding configuration file content is restored from the backup file. Use the diff command to compare the differences between the current configuration file and the backup file, and output the rollback content to the screen; Before and after modifying the parameters, the parameter status is recorded and output.

2. The distributed cluster configuration method based on SHELL scripts according to claim 1, characterized in that, The step of determining the parameter modification target according to the instruction includes: when three parameters are received and the first parameter is "set", the target cluster is determined according to the second parameter. When the parameter is 'c', modify the configuration file of the gcluster cluster. When the parameter is n, modify the configuration file of the gnode cluster; When the parameter is cn, modify the configuration files for the gcluster and gnode clusters; When the parameter is pr, the environment variable configuration file is modified; The third parameter is in the format "parameter name=parameter value". If there are multiple parameters, they are separated by ";". Iterate through the contents of the target configuration file that match the parameter name and execute according to the following rules: If the number of matched lines is greater than 1, remove duplicates line by line until only one line remains, and then update the parameter value. If the number of matched lines is 0, then append the parameter to the corresponding process tag in the target configuration file; After modification, perform a full word match in the target configuration file again with the parameter name, and record and display the matched line content as the current status.

3. The distributed cluster configuration method based on SHELL scripts according to claim 1, characterized in that, The target for modifying parameters according to instructions includes: When two parameters are received, and the first parameter is "restart", the script tool performs the following steps to restart the cluster service: Get the IP information of all gcluster and gnode service nodes; Store the hostname and IP address of each server in an associative array to avoid repeatedly issuing commands in a multi-IP deployment scenario of gnode; The restart target is determined based on the second parameter: When the parameter is set to 'c', only the gcluster cluster will be restarted. When the parameter is n, only the gnode cluster is restarted; When the parameter is cn, both the gcluster and gnode clusters are restarted simultaneously. Iterate through the IP addresses in the associative array and send a restart command to the corresponding server one by one.

4. A server, characterized in that: The system includes at least one processor and a memory communicatively connected to the processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the processor to cause the at least one processor to perform a distributed cluster configuration method based on a shell script as described in any one of claims 1-3.

5. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, it implements the distributed cluster configuration method based on SHELL scripts as described in any one of claims 1-3.

Images