Distributed cluster configuration method based on SHELL script

Abstract

The invention provides a distributed cluster configuration method based on an SHELL script, which comprises the following steps: intercepting architecture information and service distribution information of a cluster through a script tool, and respectively storing the intercepted information in an association array of the script tool; judging whether the cluster architecture is a storage and calculation integrated version or a storage and calculation separated version according to the cluster architecture information; backing up the target configuration file, wherein the backup file is stored in a specified path; an instruction containing operation types is received, the operation types comprise set, reset and roll back, and a parameter modification target is determined according to the instruction; and before and after the parameters are modified, recording and outputting parameter states. The method has the advantages that convenient modification of 8a cluster parameters, efficient restart of cluster services and quick recovery of configuration files are achieved through the script tool, and the configuration files of services corresponding to all cluster nodes can be modified through one instruction through the script tool.

Description

Technical Field

[0001] The invention belongs to the field of databases, and in particular relates to a distributed cluster configuration method based on SHELL scripts. Background Art

[0002] NCU's general-purpose large-scale distributed parallel database cluster system, referred to as GBase 8a MPP Cluster, is a distributed parallel database cluster with Shared Nothing architecture developed on the basis of GBase 8a column storage database. It has the characteristics of high performance, high availability, and high scalability. It can provide a cost-effective general-purpose computing platform for data management of various scales, and is widely used to support various data warehouse systems, BI systems, and decision support systems. Currently, there are the following problems when modifying read-only parameters of an 8a cluster: When modifying the 8a read-only parameters, you need to find the configuration file of the process that needs to be modified node by node and modify it manually. When the cluster is large and the number of nodes is large, this operation is labor-intensive and easy to confuse. Once a configuration problem is found, you need to check, confirm and restore the parameters again. The process is complicated and time-consuming. At the same time, the modified 8a read-only parameters must be restarted by restarting all node services of the 8a cluster to take effect. If a distributed cluster management tool is not built, you need to manually restart each node, which is inefficient. If the modification of the 8a configuration file is incorrect, the cluster service may not be able to restart normally. The recovery operation also needs to be performed on each node, and the exception handling time is long. Summary of the invention

[0003] In view of this, the present invention aims to propose a distributed cluster configuration method based on SHELL script to at least solve one problem in the background technology.

[0004] To achieve the above object, the technical solution of the present invention is achieved as follows: A distributed cluster configuration method based on SHELL script, comprising: The architecture information and service distribution information of the cluster are intercepted by the script tool, and the intercepted information is stored in the associative array of the script tool respectively, wherein the architecture information and service distribution information include the node scale of the gcluster service, the node scale of the gnode service and the IP address of the node; Determine whether it is a storage-computing integrated version or a storage-computing separated version based on the cluster architecture information: If it is a storage-computing integrated version, operations on the computing cluster are sent to all computing nodes; if it is a storage-computing separated version, operations on the computing cluster are sent only to the first computing node; Back up the target configuration file and store the backup file in the specified path; receiving an instruction including an operation type, wherein the operation type includes set, restart, and rollback, and determining a parameter modification target according to the instruction; Before and after modifying the parameters, the parameter status is recorded and output.

[0005] Furthermore, when the operation type is set, the node type of the target cluster is modified according to the input parameters, and the specified parameter value is modified or appended in the target configuration file. The script tool uses sed command and awk command to locate the process label range.

[0006] Furthermore, when the operation type is restart, the script tool obtains the IP information of the gcluster and gnode service nodes, stores the host name and the IP address into an associative array, and sends a restart command to the target cluster node.

[0007] Furthermore, when the operation type is rollback, the script tool restores the configuration file content from the backup file, uses the diff command to compare the file differences and outputs the rollback result.

[0008] Further, determining the parameter modification target according to the instruction includes: when three parameters are received and the first parameter is set, determining the target cluster according to the second parameter: When the parameter is c, modify the configuration file of the gcluster cluster; When the parameter is n, modify the configuration file of the gnode cluster; When the parameter is cn, modify the configuration files of the gcluster and gnode clusters; When the parameter is pr, modify the environment variable configuration file; The format of the third parameter is "parameter name = parameter value". If it contains multiple parameters, the multiple parameters are separated by ";"; Traverse the contents of the target configuration file that match the parameter name and execute according to the following rules: If the number of matching rows is greater than 1, delete duplicates row by row until only one row is left, and update the parameter value; If the number of matching lines is 0, then append the parameter under the corresponding process tag in the target configuration file; After the modification is completed, retrieve and record the modified status of the parameters again.

[0009] Furthermore, when the first parameter received is restart, the script tool executes the restart of the cluster service through the following steps: Get the IP information of all gcluster and gnode service nodes; Store each server's host name and its IP address in an associative array to avoid repeated commands in gnode multi-IP deployment scenarios; The restart target is determined by the second parameter: When the parameter is c, only the gcluster cluster is restarted; When the parameter is n, only the gnode cluster is restarted; When the parameter is cn, restart the gcluster and gnode clusters at the same time; Traverse the IP addresses in the associative array and send restart commands to the corresponding servers one by one.

[0010] Furthermore, when the script tool performs a rollback operation, the following steps are taken: The first parameter received is rollback, and the second parameter is the target cluster type; Restore the corresponding configuration file content from the backup file according to the target cluster type; Use the diff command to compare the current configuration file with the backup file and output the rollback content to the screen.

[0011] Furthermore, the present solution discloses a server, comprising at least one processor and a memory communicatively connected to the processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor so that the at least one processor executes a distributed cluster configuration method based on a SHELL script.

[0012] Furthermore, the present solution discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, a distributed cluster configuration method based on a SHELL script is implemented.

[0013] Compared with the prior art, the distributed cluster configuration method based on SHELL script described in the present invention has the following beneficial effects: The distributed cluster configuration method based on SHELL script described in the present invention realizes convenient modification of 8a cluster parameters, efficient restart of cluster services and rapid recovery of configuration files through script tools. Through the script tool, configuration files of services corresponding to all cluster nodes can be modified through one instruction, parameter status can be fed back in real time, and all cluster processes except gcware-related services in the cluster can be restarted through one instruction, execution information can be automatically returned, and configuration file backup and rollback functions can be provided. When the modification result is not satisfactory, it can be quickly restored to the state before the modification. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings: Figure 1 A schematic diagram of a distributed cluster configuration method based on SHELL script according to an embodiment of the present invention. DETAILED DESCRIPTION

[0015] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0016] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0017] This solution discloses a distributed cluster configuration method based on SHELL script. First, the terms involved are explained: Read-only parameters: Parameters that cannot be modified online. They can only be modified by modifying the cluster configuration file, and then the database server process reads the configuration file content to take effect when the cluster is started. Cluster configuration file: usually refers to [coordination cluster node configuration file: gbase_8a_gcluster.cnf, computing cluster node configuration file: gbase_8a_gbase.cnf, cluster node environment variable configuration file: gbase_profile]; 8a: GBase 8a MPP Cluster large-scale distributed parallel processing data warehouse in the following text; aset: This invention is a script tool written in shell language and used in Linux command line; gcware: GBase 8a MPP Cluster cluster consistency management process; gcluster: GBase 8a MPP Cluster coordination and scheduling process; gnode: data computing and storage process of GBase 8a MPP Cluster; gcadmin: GBase 8a MPP Cluster operation and maintenance management command.

[0018] The specific plan is as follows: aset intercepts the information returned by the gcadmin command to obtain the current cluster architecture information and the corresponding service distribution, records the number and IP address information of gcluster and gnode services, and stores them in different associative arrays of aset. When executing the distribution action, it traverses the values ​​in the corresponding service array to achieve effective distribution operations on different nodes and services. At the same time, the cluster architecture version is parsed. If it is a storage-computing integrated version, each operation on the computing cluster is sent to all computing nodes; if it is a storage-computing separated version, each operation on the computing cluster is only sent to the computing nodes in the first computing cluster. The modification plan for different computing clusters can be specified by modifying the value of the wh_num variable in aset.

[0019] Aset will perform a round of verification and judgment each time it is called. First, check whether there is a gcware process in the current environment. If there is a gcware process, check whether the OS user who currently calls the script is consistent with the OS user who starts the gcware process. If the OS user who starts gcware is gbase8a, and the user who calls the script is root, after aset checks for inconsistencies, it will output a prompt message "Please log in as gbase DBA user". After that, aset will check whether the command contains parameters and the number of parameters. If no parameters are included, it will output prompt information and usage help. If the number of parameters substituted is different from the specified number, it will directly skip subsequent verification and return an error prompt. The number of parameters in the verification rules is 2 and 3 respectively. If 2 are met, the first parameter will be brought into the judgment rule. If it is rollback or restart, the subsequent parameters will be verified. Otherwise, the verification will be jumped out and an error prompt will be returned. If 3 are met, the first parameter will be brought into the judgment rule. If it is set or close, the subsequent parameters will be verified. Otherwise, the verification will be jumped out and an error result will be returned.

[0020] Each time the script tool is called, aset will obtain the current cluster scale information in real time without manual configuration. aset will format and split the cluster architecture returned by the gcadmin command of 8a through awk and sed commands, and record the gcluster cluster node information and gnode cluster node information in different arrays. When performing operations on different cluster configuration files in the future, you can directly traverse the corresponding array to obtain the required node information. Automatic acquisition not only provides convenience of operation, but also ensures the real-time and accuracy of each modification action.

[0021] When aset receives three incoming parameters and the first parameter is set, it will continue to check the next two parameters. The second parameter is the corresponding target cluster, and the third parameter corresponds to the 8a cluster parameter to be modified in the configuration file. If the second parameter is c, modify the content of the third parameter in the configuration files of all nodes of the gcluster cluster; if the second parameter is n, modify the content of the third parameter in the configuration files of all nodes of the gnode cluster; if the second parameter is cn, modify the content of the third parameter in the configuration files of both the gcluster cluster and the gnode cluster; if the second parameter is pr, modify the environment variables in the environment variable files of both the gcluster cluster and the gnode cluster. The content in the third incoming parameter is the 8a cluster parameter to be modified in the 8a cluster configuration file. The incoming format of each 8a cluster is "parameter name = parameter value". Multiple 8a cluster parameters can be separated by ";". aset will extract the parameter name of each 8a cluster parameter, perform a full-word match search in the configuration file, output the content of the line that meets the matching conditions, record and display it as the pre-state, and then count the number of output lines into the variable i. If i>1, it means that there are repeated settings of 8a cluster parameters in the configuration file. The line numbers of these lines are traversed through the sed command, and then deleted. Every time a line is deleted, i is reduced by 1. When i=1, no deletion is performed and the parameter value of the line parameter is replaced with the target parameter value. If i=0 at the beginning, that is, there is no 8a cluster parameter in the configuration file, the parameter will be directly appended at the specified location in a redirected appending manner. After the modification is completed, the parameter name will be brought in again to perform a full-word match on the configuration file, and the matching line content will be output as the current state record and displayed. There are many parameters and different process tags in the 8a configuration file. Each process tag name is wrapped in brackets. The parameters under the process tag are only effective for the process corresponding to the tag, so the modification and appending of parameters must be operated under the target process tag to be effective. Before each round of modification of the configuration file, aset will count the line number where the target process label starts into the start_line variable and the line number where the next process label starts into the end_line variable. The variable values ​​of the two line numbers are introduced through sed -n and p to control the operation range of the configuration file each time. This effectively ensures the correct effectiveness of the parameters and avoids misoperation of synchronization parameters of other processes.

[0022] Each server has its own unique hostname. When the first parameter received by aset is restart, it will obtain the IP information of all gcluster and gnode processes, obtain the hostnames of these IPs in turn, and store the hostname content in the associative array as a key value. The value value stores the corresponding IP address. In this way, even if a server has multiple IP addresses, there is only one IP in the associative array that corresponds to its hostname. When issuing a restart operation, it will traverse the value in the associative array and send the restart command to the server with the corresponding IP. Even in the scenario of mixed deployment of gcluster and gnode or deployment of gnode with multiple IP instances, a single server will not receive repeated commands. At the same time, aset will determine the object of the restart operation based on the content of the second parameter. When the parameter is c, only the gcluster cluster will be restarted. If it is n, only the gnode cluster will be restarted. If it is cn, both the gcluster and gnode clusters will be restarted.

[0023] Before modifying the configuration file each time, a backup of the current configuration file will be made on each node, and then the configuration file will be modified. In this way, each configuration file has a backup of its previous configuration state. If the first parameter received by aset is rollback, a rollback operation will be triggered to copy the previously backed-up configuration information back to the current configuration file, so as to achieve rapid recovery of the parameter file. At the same time, aset will determine the object of the rollback operation based on the content of the second parameter. When the parameter is c, only the configuration file of the gcluster cluster will be rolled back. When the parameter is n, only the configuration file of the gnode cluster will be rolled back. When the parameter is pr, only the configuration file of the environment variable will be rolled back. During the rollback operation, the diff command will be used to compare the differences between the current configuration file and the backup configuration file, and the output format will be standardized through sed and awk, and the specific rolled-back parameters and parameter values ​​will be output to the screen for manual confirmation.

[0024] By implementing the above solution, the functional effects achieved are: You can verify the tool execution user. If the system user currently calling the script tool is not the database administrator user in the system, the script tool will intercept the execution action and prompt the database administrator in the system to execute it. Automatically locate the size of the 8a cluster (the IP address used by the cluster), automatically locate the distribution of service processes (the IP addresses of each node where a service process is located), and automatically locate the absolute path of the configuration file of the corresponding service process; Automatically determine whether it is an 8a cluster with integrated storage and computing or an 8a cluster with separated storage and computing, and automatically match the corresponding execution strategy according to different cluster versions; You can send modification operations to the configuration files of all coordination cluster nodes separately, you can send modification operations to the configuration files of all computing cluster nodes separately, you can send modification operations to the configuration files of all coordination cluster nodes and computing cluster nodes at the same time, and you can send modification operations to the environment variable configuration files of all cluster nodes separately; You can send multiple read-only parameters to be modified in one command, separate them with ";" and send them together with aset for modification; During the modification process, a parameter in a service configuration file of a node can be output in real time, including the parameter value status before modification and the parameter value status after modification. You can restart the cluster processes of non-gcware related services on all cluster nodes with one command and return the execution information on each node; Supports rollback of modification actions. If you are not satisfied with the result of the last parameter modification, you can use the rollback function to restore the last modification operation on a certain type of cluster configuration file to the state before the last modification.

[0025] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0026] In the several embodiments provided in the present application, it should be understood that the disclosed methods and systems can be implemented in other ways. For example, the division of the units described above is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The above-mentioned units may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0027] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

[0028] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A distributed cluster configuration method based on SHELL script, characterized in that: include: The architecture information and service distribution information of the cluster are intercepted by the script tool, and the intercepted information is stored in the associative array of the script tool respectively, wherein the architecture information and service distribution information include the node scale of the gcluster service, the node scale of the gnode service and the IP address of the node; Determine whether it is a storage-computing integrated version or a storage-computing separated version based on the cluster architecture information: If it is a storage-computing integrated version, operations on the computing cluster are sent to all computing nodes; if it is a storage-computing separated version, operations on the computing cluster are sent only to the first computing node; Back up the target configuration file and store the backup file in the specified path; receiving an instruction including an operation type, wherein the operation type includes set, restart, and rollback, and determining a parameter modification target according to the instruction; Before and after modifying the parameters, the parameter status is recorded and output.

2. A distributed cluster configuration method based on SHELL script according to claim 1, characterized in that: When the operation type is set, the node type of the target cluster is modified according to the input parameters, and the specified parameter value is modified or appended in the target configuration file. The script tool uses the sed command and the awk command to locate the process label range.

3. The distributed cluster configuration method based on Shell script according to claim 1, characterized in that: When the operation type is restart, the script tool obtains the IP information of the gcluster and gnode service nodes, stores the host name and IP address into an associative array, and sends a restart command to the target cluster node.

4. The distributed cluster configuration method based on Shell script according to claim 1, characterized in that: When the operation type is rollback, the script tool restores the configuration file content from the backup file, uses the diff command to compare file differences, and outputs the rollback result.

5. The distributed cluster configuration method based on SHELL script according to claim 2, characterized in that: Determining the parameter modification target according to the instruction includes: when three parameters are received and the first parameter is set, determining the target cluster according to the second parameter: When the parameter is c, modify the configuration file of the gcluster cluster; When the parameter is n, modify the configuration file of the gnode cluster; When the parameter is cn, modify the configuration files of the gcluster and gnode clusters; When the parameter is pr, modify the environment variable configuration file; The format of the third parameter is "parameter name = parameter value". If it contains multiple parameters, the multiple parameters are separated by ";"; Traverse the contents of the target configuration file that match the parameter name and execute according to the following rules: If the number of matching rows is greater than 1, delete duplicates row by row until only one row is left, and update the parameter value; If the number of matching lines is 0, then append the parameter under the corresponding process tag in the target configuration file; After the modification is completed, retrieve and record the modified status of the parameters again.

6. The distributed cluster configuration method based on SHELL script according to claim 3, characterized in that: When the first parameter received is restart, the script tool restarts the cluster service through the following steps: Get the IP information of all gcluster and gnode service nodes; Store each server's host name and its IP address in an associative array to avoid repeated commands in gnode multi-IP deployment scenarios; The restart target is determined by the second parameter: When the parameter is c, only the gcluster cluster is restarted; When the parameter is n, only the gnode cluster is restarted; When the parameter is cn, restart the gcluster and gnode clusters at the same time; Traverse the IP addresses in the associative array and send restart commands to the corresponding servers one by one.

7. The distributed cluster configuration method based on SHELL script according to claim 4, characterized in that: The script tool uses the following steps when performing a rollback operation: The first parameter received is rollback, and the second parameter is the target cluster type; Restore the corresponding configuration file content from the backup file according to the target cluster type; Use the diff command to compare the current configuration file with the backup file and output the rollback content to the screen.

8. A server, characterized in that: It includes at least one processor and a memory communicatively connected to the processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor so that the at least one processor executes a distributed cluster configuration method based on a SHELL script as described in any one of claims 1-7.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the distributed cluster configuration method based on SHELL script described in any one of claims 1 to 7 is implemented.

Images