Node fault repairing method, device and equipment and readable storage medium

Abstract

The invention discloses a node fault repair method, device and equipment and a readable storage medium, which are applied to the technical field of distributed storage, and comprise the following steps: determining a service repair sequence based on fault node characteristics, and the hierarchy and dependency relationship of services in fault nodes; determining a repair method of each service in the fault node and a judgment method of successful repair, and creating a repair list according to the repair method, the judgment method of successful repair and a service repair sequence; determining a repair method of the fault node based on the repair list, and repairing the fault node based on the repair method; the repairing method comprises the step of repairing the service with the stored data according to the repairing sequence and the repairing method in the repairing list. Compared with the current method that professional personnel need to reconstruct the data, the method has the advantages that automatic repair can be directly carried out on the basis of the repair list generated by the fault node, the professional personnel do not need to reconstruct the data, and the repair efficiency of the fault node is improved.

Description

Technical Field

[0001] The present invention relates to the field of distributed storage technology, and in particular to a node fault repair method, device, equipment and readable storage medium. Background Art

[0002] Distributed storage systems are based on servers, which often use two system disks in RAID mode to ensure reliability. Even if there are two system disks, it is inevitable that they will be damaged and not replaced in time, resulting in damage to the entire system and node unavailability. The common solution in the prior art is to first shrink the node out of the cluster, reinstall the system, and then expand the node into the cluster. Professionals then reconstruct the data. This method has a long repair cycle, and it cannot be used to complete reconstruction in scenarios with a small number of nodes and low redundancy.

[0003] It can be seen that how to improve the efficiency of node fault repair is a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0004] In view of this, an object of the present invention is to provide a node fault repair method, device, equipment and readable storage medium, which solve the technical problem of low efficiency of node fault repair in the prior art.

[0005] In order to solve the above technical problems, the present invention provides a node fault repair method, comprising:

[0006] Analyze the cluster status, alarm information and configuration files of the node to determine the characteristics of the faulty node; wherein the cluster status is the running status information of the cluster, the alarm information is the information to remind the cluster of abnormality, and the configuration file includes the files of various services of the node;

[0007] Determine the service repair order based on the fault node characteristics and the dependency relationship of services in the fault node;

[0008] Determine the repair method and successful repair judgment method of each service in the faulty node, and create a repair list list according to the repair method, the successful repair judgment method and the service repair sequence;

[0009] A repair method for the faulty node is determined based on the repair list, and the faulty node is repaired based on the repair method; wherein the repair method includes repairing the service having storage data according to the repair order and repair method in the repair list.

[0010] On the one hand, the node fault repair method further includes:

[0011] Determine a configuration list according to the characteristics of the faulty node, the configuration list including repair items and non-repair items; wherein the repair items include services that need to be repaired during the system reinstallation process and the capacity expansion process; the non-repair items are services that need to be configured without repair;

[0012] Accordingly, the method of determining the repair method of each service in the faulty node and the method of judging successful repair, and creating a repair list according to the repair method, the method of judging successful repair and the order of service repair, includes:

[0013] The repair list is created based on the configuration list, the repair method, the method for determining successful repair, and the service repair sequence.

[0014] On the one hand, the method of determining the repair method of each service in the faulty node and the method of judging successful repair, and creating a repair list according to the repair method, the method of judging successful repair and the order of service repair, includes:

[0015] Determine whether the repair list includes an expansion item, the expansion item being a hard disk data service;

[0016] When the expansion item is included in the repair list, determining whether the hard disk drive data of the expansion item is included in the storage pool;

[0017] When the hard disk drive data of the expansion item is included in the storage pool, repair is performed according to the repair order and repair method in the repair list;

[0018] When the hard disk drive data of the expansion item is not included in the storage pool, the expansion item is reduced in size;

[0019] When the repair list does not include the expansion item, determining whether the faulty node is connected to the cluster;

[0020] When the faulty node is connected to the cluster, the repair service is obtained according to the repair order in the repair list, and it is determined whether the current repair service is the expansion item;

[0021] If the current repair service is the expansion item, determine to perform expansion processing on all the expansion items that have been shrunk and reinstalled;

[0022] If the current repair service is not the expansion item, determine to perform repair according to the repair order and repair method in the repair list.

[0023] On the one hand, if the current repair service is not the expansion item, after determining to repair according to the repair order and repair method in the repair list, it also includes:

[0024] Determine whether the currently repaired service is a critical service based on the key items configured in the repair list; wherein the critical service is a service set according to the impact of the service on the node;

[0025] When the currently repaired service is a critical service, the repaired critical service is tested, and if the test passes, the next service is repaired;

[0026] When the currently repaired service is a non-critical service, the repair of the next service is directly performed.

[0027] On the one hand, when the faulty node is not connected to the cluster, it includes:

[0028] Determine to wait according to a waiting principle; wherein the waiting principle is a principle of waiting according to a set overall waiting time and a waiting frequency;

[0029] When waiting based on the waiting frequency in the waiting principle, if connectivity is detected within the set overall waiting time, repair is performed based on the repair list; when the set overall waiting time is exceeded, a failure prompt is displayed and the reason for the failure is displayed.

[0030] On the one hand, the information in the repair list includes serial number, configuration item, module, level, key item, whether repair is required, dependency, repair method and repair success detection method.

[0031] On the one hand, after determining a repair method for the faulty node based on the repair list and repairing the faulty node based on the repair method, the method further includes:

[0032] Perform an overall test on the repair results; wherein, the overall test includes testing the repair list to determine whether there are any abnormal items, and ending directly when no abnormal items exist, and determining the overall test results based on the number of services to be repaired and the types of services that failed to be repaired, wherein the overall test results include successful overall service repair, service repair failure, and successful partial service repair.

[0033] The embodiment of the present invention further provides a node fault repair device, comprising:

[0034] A fault node feature determination module is used to analyze the cluster state, alarm information and configuration files of the node to determine the fault node features; wherein the cluster state is the running state information of the cluster, the alarm information is the information to remind the cluster of abnormality, and the configuration files include files of various services of the node;

[0035] A service repair sequence determination module, used to determine the service repair sequence based on the fault node characteristics and the dependency relationship of services in the fault node;

[0036] A repair list determination module is used to determine the repair method of each service in the faulty node and the method for judging successful repair, and to create a repair list according to the repair method, the method for judging successful repair and the order of service repair;

[0037] A repair module is used to determine a repair method for a faulty node based on the repair list, and repair the faulty node based on the repair method; wherein the repair method includes repairing a service having stored data according to the repair order and repair method in the repair list.

[0038] The embodiment of the present invention further provides a node fault repair device, including:

[0039] Memory for storing computer programs;

[0040] A processor is used to execute the computer program to implement the steps of the above-mentioned node failure repair method.

[0041] An embodiment of the present invention further provides a readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned node failure repair method are implemented.

[0042] The present invention also provides a computer program product, including a computer program / instruction, which implements the steps of the above-mentioned node fault repair method when executed by a processor.

[0043] The purpose of the embodiments of the present invention is to provide a node fault repair method, device, equipment and readable storage medium, which can solve the technical problem of low efficiency of node fault repair.

[0044] To solve the above technical problems, an embodiment of the present invention provides a node fault repair method, which may include: analyzing the cluster status, alarm information and configuration file of the node to determine the characteristics of the faulty node; wherein the cluster status is the running status information of the cluster, the alarm information is the information to remind the cluster of abnormalities, and the configuration file includes files of various services of the node; determining the service repair order based on the characteristics of the faulty node and the dependencies of the services in the faulty node; determining the repair method and the judgment method of successful repair for each service in the faulty node, and creating a repair list list according to the repair method, the judgment method of successful repair and the service repair order; determining the repair method of the faulty node based on the repair list list, and repairing the faulty node based on the repair method; wherein the repair method includes repairing the service with stored data according to the repair order and repair method in the repair list list.

[0045] It can be seen from the above technical scheme that the beneficial effect of the present invention is that compared with the current method of repairing node failures by shrinking, refilling, and expanding nodes, and requiring professionals to reconstruct the data, in the present invention, for nodes with data that need to be repaired, automatic repair can be performed directly based on the repair list list generated by the faulty node, without the need for professionals to reconstruct the data, thereby improving the efficiency of repairing faulty nodes. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 A flowchart of a node fault repair method provided by an embodiment of the present invention;

[0048] Figure 2 A schematic diagram of a distributed storage system framework provided by an embodiment of the present invention;

[0049] Figure 3 A flowchart of a node fault repair method provided by an embodiment of the present invention;

[0050] Figure 4 A schematic diagram of the structure of a node fault repair device provided by an embodiment of the present invention;

[0051] Figure 5 A schematic diagram of the structure of a node fault repair device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] The terms "including" and "having" in the specification of the present invention and the above-mentioned drawings, as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may include steps or units that are not listed.

[0054] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0055] Next, a flowchart of a node fault repair method provided by an embodiment of the present invention is described in detail. Figure 1 A node fault repair method provided by an embodiment of the present invention may include:

[0056] S101, analyzing the cluster state, alarm information and configuration files of the node to determine the characteristics of the faulty node; wherein the cluster state is the running state information of the cluster, the alarm information is the information to remind the cluster of abnormality, and the configuration file includes the files of various services of the node.

[0057] The executor of this embodiment is an electronic device. The electronic device may be a computer, a mobile phone, etc. This embodiment does not limit the specific cluster state. For example, the cluster state in this embodiment may be a degraded state; or the cluster state in this embodiment may be a node failure state; or the cluster state in this embodiment may be an abnormal monitoring service state. This embodiment does not limit the specific alarm information. For example, the alarm information in this embodiment may be the alarm information of the hard disk interface, or the alarm information in this embodiment may also be the alarm information of the network port. This embodiment does not limit the specific configuration file. The configuration file in this embodiment may be a configuration file of a faulty node, so as to obtain more detailed information of the node from the configuration file.

[0058] S102, determining a service repair sequence based on the fault node characteristics and the dependency relationship of services in the fault node.

[0059] The service repair order in this embodiment means that when there are multiple nodes with faults, if node B depends on node A, node A needs to be repaired first, or the level of node C is higher than that of node B, node C needs to be repaired first. It can be understood that the services in this embodiment can be services of different levels. For example, the services in this embodiment can be services of the operation layer, storage layer and protocol layer. The hierarchical relationship of the operation layer, storage layer and protocol layer can be understood from the perspective of computer network or software architecture. These layers are usually used in design to abstract and manage various aspects of complex systems to achieve the design goals of high cohesion and low coupling. Protocol layer: This layer is mainly responsible for defining the format and rules of data exchange. In software architecture, the protocol layer refers to the application program interface (API) or the inter-service communication protocol, which specifies how different modules or services interact with each other. Operation layer: The operation layer usually involves the processing and management of data. This may include operations such as creating, updating and deleting data. In a database management system, the operation layer may be the part of SQL (database) statement execution, responsible for parsing SQL commands and operating the database. Storage layer: The storage layer focuses on data persistence, that is, how to save data to a disk or other storage medium and retrieve it when needed. This embodiment does not limit specific services. For example, the service in this embodiment can be a basic service; or the service in this embodiment can also be a storage service; or the service in this embodiment can also be a network service; or the service in this embodiment can also be a computing service.

[0060] S103, determining the repair method of each service in the faulty node and the method for judging successful repair, and creating a repair list according to the repair method, the method for judging successful repair and the order of service repair.

[0061] The repair method for each service in this embodiment can be based on existing tools or newly designed tools. The judgment method for successful repair corresponding to each service is not limited in this embodiment. For example, the judgment method for network repair can be to perform a network query. If the status is up and it can ping with other nodes, the repair is confirmed to be successful. The status of up usually means that a device or service is running and ready to accept requests. And being able to ping with other nodes means that the device or service can communicate with other devices through the network without blocking. The judgment method for successful repair of the cluster version in this embodiment can be command query, return version number, and the version number is the required version number. This embodiment does not limit the specific repair list. For example, the repair list in this embodiment may include configuration items, modules to which it belongs, whether it needs to be repaired, dependencies, etc. Alternatively, the information in the repair list list of this embodiment includes sequence number, configuration items, modules to which it belongs, levels, key items, whether it needs to be repaired, dependencies, repair methods, and repair success detection methods.

[0062] It should be further explained that in order to improve the accuracy of the repair checklist, the above-mentioned node fault repair method may also include: determining a configuration checklist based on the characteristics of the faulty node, the configuration checklist including repair items and non-repair items; wherein the repair items include services that need to be repaired again during the system refilling process and the capacity expansion process; non-repair items are services that do not need to be repaired and need to be configured; accordingly, determining the repair method and the method for judging successful repair for each service in the faulty node, and creating a repair checklist based on the repair method, the method for judging successful repair and the order of service repair, including: creating a repair checklist based on the configuration checklist, the repair method, the method for judging successful repair and the order of service repair. This embodiment confirms the repair items and non-repair items so that when repairing according to the repair checklist, only the repair items can be repaired and the non-repair items can be configured. For example, the repair items in this embodiment can be basic recovery modules, data service modules, non-data service modules (services configured during the expansion process), and advanced settings modules (advanced settings performed after the original expansion is completed). Non-repair items are some basic configuration conditions, such as whether the node HDD (abbreviation of Hard Disk Drive, called hard disk drive in Chinese) is in the storage pool and other information. The services in the cluster are classified into four-level modules according to their levels. If the node is a mon (monitoring) node, the mon service is added to the list. If it is not a mon node, the (mon service) is not added. This embodiment requires that the repair items be repaired again after a failure occurs to ensure the normal operation of the system. Non-repair items can be directly configured, which improves the efficiency of node failure repair.

[0063] It should be further explained that the above-mentioned determination of the repair method of each service in the faulty node and the method for judging successful repair, and the creation of a repair list according to the repair method, the method for judging successful repair and the order of service repair may include:

[0064] S1031, determining whether the repair list includes an expansion item, where the expansion item is a hard disk data service;

[0065] S1032: When the repair list includes an expansion item, it is determined whether the hard disk drive data of the expansion item is included in the storage pool.

[0066] S1033: When the hard disk drive data of the expansion item is included in the storage pool, repair is performed according to the repair order and repair method in the repair list.

[0067] S1034: When the hard disk drive data of the expansion item is not included in the storage pool, the expansion item is reduced in size.

[0068] S1035: When the repair list does not include the capacity expansion item, it is determined whether the faulty node is connected to the cluster.

[0069] S1036: When the faulty node is connected to the cluster, a repair service is obtained according to the repair order in the repair list, and it is determined whether the current repair service is a capacity expansion item.

[0070] S1037: If the current repair service is an expansion item, determine to perform expansion processing on all the expansion items that have been shrunk and reinstalled.

[0071] S1038: If the current repair service is not a capacity expansion item, determine to perform repair according to the repair order and repair method in the repair list.

[0072] In this embodiment, judging whether the expansion item is included can be understood as whether the OSD (Object Storage Daemon) service needs to be restored. When it is not included, the hard disk drive data can be judged, for example, whether the HDD (Hard Disk Drive) is in the storage pool. OSD is the upper layer service of HDD. The expansion item in this embodiment is the hard disk data service. The hard disk data service is used to determine whether the service with data needs to be restored. When it is not needed, the data can be repaired according to the repair list. When the service with data needs to be restored, it can be determined whether the data is in the storage pool. If it is in the storage pool, it means that a new repair method needs to be used for repair. Otherwise, it means that the service that does not include data can be directly downsized. In this embodiment, when the hard disk drive data is not included in the storage pool, it can be repaired based on the existing process, that is, the first step: downsizing. First, the node is downsized from the cluster for the faulty node, that is, the service is downsized, such as the osd (disk management service) service, mon service, protocol service, etc. The second step: filling and basic recovery. After the downsizing is completed, the node is restored and the new system is filled. The third step: expansion. This step requires first analyzing the roles and services that the node originally assumed in the cluster, and then expanding services such as osd, mon, and protocols by expanding services. This embodiment provides methods for repairing different services, so that services that do not include data can be directly scaled out of the cluster, reinstalled, and expanded into the cluster. Services that include data can be directly repaired according to the repair list without data reconstruction, thereby improving the efficiency of node fault repair.

[0073] It should be further explained that in order to improve the accuracy of node repair, if the current repair service is not an expansion item, after determining to repair according to the repair order and repair method in the repair list, it can also include: determining whether the currently repaired service is a critical service based on the key items configured in the repair list; wherein, the critical service is a service set according to the impact of the service on the node; if the currently repaired service is a critical service, the repaired critical service is detected, and if the detection passes, the repair of the next service is executed; if the currently repaired service is a non-critical service, the repair of the next service is directly executed. In this embodiment, it is determined whether it is a critical item. If it is a critical item (such as version number detection, the version is incorrect, and it ends directly), a module detection is performed, and if it does not pass, it ends. If the key item passes, it continues to determine whether it is the last item. If it is not, it continues to the next item. If it is the last item, the repair ends. In this embodiment, when it is determined that the currently repaired service is a critical service configured for the key item, it will detect whether the critical service is successfully repaired. Since the critical service has a relatively large impact on the entire node, it is necessary to judge the critical service.

[0074] It should be further explained that when the faulty node is not connected to the cluster, the above-mentioned process may include: determining to wait according to the waiting principle; wherein the waiting principle is the principle of waiting according to the set overall waiting time and waiting frequency, and when it is determined based on the waiting principle that the overall waiting time is exceeded, the failure logic is detected, the failure is prompted, and the reason for the failure is displayed. This embodiment does not limit the specific overall waiting time. For example, the waiting time in this embodiment may be 2 minutes; or the waiting time in this embodiment may also be 5 minutes. The waiting frequency in this embodiment may be detected once every 5 seconds, or once every minute. Since critical services will affect the repair of the node and make the node unusable, it is necessary to prompt the failure so that it can be resolved in time. And through the device waiting principle, the waiting time is prevented from being too long, which affects the repair efficiency.

[0075] S104, determining a repair method for the faulty node based on the repair list, and repairing the faulty node based on the repair method; wherein the repair method includes repairing the service having stored data according to the repair order and repair method in the repair list.

[0076] This embodiment does not limit the specific service for storing data. For example, the service for storing data in this embodiment may be a service including HDD (Hard Disk Drive) data; or the service for storing data in this embodiment may be an SSD (Solid State Drive). The repair method in this embodiment may repair the service with stored data according to the repair order and repair method in the repair list, and for the service without data, it may be repaired directly by shrinking, reinstalling and expanding.

[0077] It should be further explained that after determining the repair method of the faulty node based on the repair list, and repairing the faulty node based on the repair method, it can also include: performing an overall detection on the repair result; wherein the overall detection includes detecting the repair list to determine whether there are abnormal items, and ending directly when there are no abnormal items, and determining the overall detection result according to the number of services to be repaired and the type of services that failed to be repaired, wherein the overall detection result includes successful overall service repair, failed service repair, and successful partial service repair. After the overall detection in this embodiment is completed, the overall detection is first called once to find out the abnormal items. If there are no abnormal items, it ends successfully directly. The specific process is to first determine whether the node IP is connected. If not, it is considered that the system disk has failed, all items are marked as failed, and the repair module is called for repair; if it is connected, an overall detection is performed, and the failed items are marked for repair. After the detection is completed, check whether the number of items to be repaired is greater than 0. If it is equal to 0, it is directly successfully passed. If it is greater than 0, it is further determined whether the calling module is an analysis module or a repair module. If it is an analysis module, the repair module is continued to be called for repair, and then it is determined whether the key items are included. If included, the overall result is considered to be failed, and if not included, the overall result is successful. The module detection is mainly to detect the current service after the repair is completed, and then continue if it passes, and directly perform detection and analysis to obtain a detection and analysis report if it fails. This embodiment provides a method for overall detection, which can perform dual detection on the service and improve the accuracy of detection.

[0078] It should be further explained that, in order to improve the accuracy of repairing services that do not have data included in the storage pool, the above-mentioned method of repairing a faulty node based on the repair list list, and repairing the faulty node based on the repair method, may include: when all the services in the current repair list list are services for data included in the storage pool, determining the repair method as a method of repairing a faulty node based on the repair list list; when all the services in the repair list list are services for data not included in the storage pool, determining the repair method as a method of first shrinking the node out of the cluster, then refilling the node, and finally expanding the node to the cluster; when the services in the repair list list are services for data partially included in the storage pool, determining the method of repairing the faulty node based on the repair list list for the services for data included in the storage pool, and first shrinking the node out of the cluster, then refilling the node, and finally expanding the node to the cluster for the services for data not included in the storage pool. This embodiment provides specific node repair methods corresponding to different repair list lists, improves the accuracy of node fault repair, and improves the efficiency of node repair.

[0079] A node fault repair method provided by an embodiment of the present invention may include: S101, analyzing the cluster state, alarm information and configuration file of the node to determine the characteristics of the faulty node; wherein the cluster state is the running state information of the cluster, the alarm information is the information to remind the cluster of abnormality, and the configuration file includes the files of various services of the node; S102, determining the service repair order based on the characteristics of the faulty node, the hierarchy and dependency of the services in the faulty node; S103, determining the repair method and the judgment method of successful repair of each service in the faulty node, and creating a repair list list according to the repair method, the judgment method of successful repair and the service repair order; S104, determining the repair method of the faulty node based on the repair list list, and repairing the faulty node based on the repair method; wherein the repair method includes repairing the service with storage data according to the repair order and repair method in the repair list list. The present invention can automatically repair the node with data that needs to be repaired directly based on the repair list list generated by the faulty node, without the need for professionals to reconstruct the data, thereby improving the efficiency of repairing the faulty node.

[0080] Distributed storage clusters are generally composed of multiple servers. Different servers may have different physical hardware and often perform different storage service roles. For example, monitoring nodes are usually an odd number and are distributed in different cabinets. In this way, power failure or failure of a single cabinet will not affect the use of the entire cluster. Figure 2 , Figure 2A schematic diagram of a distributed storage system framework provided by an embodiment of the present invention. The distributed storage service system business generally includes hardware / parts, OS (operation) layer, storage layer, function layer and protocol layer. In the fusion scenario, not only the roles are different, but also the upper layer protocol functions may be different. Some nodes provide file services (SMB (Server Message Block, client / server file sharing protocol), NFS (Network File System, distributed file system protocol) and other services), and some nodes prompt object services (S3 (Amazon Simple Storage Service, object storage service) / Swift (OpenStack ObjectStorage, object storage system) and other services). Therefore, once a node fails, it needs to be restored to the same as before, and it is necessary to analyze layer by layer. For example, there are 1 20-node cluster, 2 storage pools, 7 mon (monitoring) nodes, which are 1, 3, 5, 7, 11, 13, and 15 nodes respectively. The first 10 nodes are nfs services (Network File System, a distributed file system protocol), and the last 10 nodes are rgw (Rados Gateway, the gateway of the distributed object storage system) services. In this way, the services that need to be restored at each layer of the failed node 1 and the failed node 12 are different.

[0081] In order to make the present invention easier to understand, please refer to Figure 3 , Figure 3 A flowchart of a node fault repair method provided by an embodiment of the present invention may specifically include:

[0082] S301, determine the characteristics of the faulty node by analyzing the cluster state, alarm information and configuration file of the node; wherein the cluster state is the running state of the cluster, the alarm information is the information to remind the cluster abnormality and cluster failure, and the configuration file includes the configuration files of various services of the node.

[0083] In this embodiment, the cluster status, alarm information and configuration files of the node are analyzed, including the bottom layer analysis, including the node name, node version, IP, clock source, alarm timing task, security compliance, etc.; the storage layer analysis, including mon (monitoring node), osd (object storage device), whether osd is in the storage pool, etc.; the protocol layer analysis, including load balancing, high availability, file, block, object service). The analysis process in this embodiment can be used as an analysis module.

[0084] S302, determining a list of node services and configurations to be repaired based on the characteristics of the faulty node; wherein the configuration nodes include repair items and non-repair items, and the non-repair items are services that do not require data recovery but require configuration.

[0085] S303, determining the dependency relationship between services at different levels according to the dependency relationship of the services, and determining the service repair sequence and key items of the faulty node.

[0086] S304, determining the repair method of each service in the faulty node and the method for judging successful repair, and creating a repair list according to the service repair sequence and key items of the faulty node.

[0087] For ease of understanding, this embodiment provides a specific repair list, as shown in Table 1. Table 1 is a repair list provided by an embodiment of the present invention.

[0088] Table 1 A list of repair checklists

[0089]

[0090] S305, determining whether the repair list includes a capacity expansion item.

[0091] Whether the embodiment includes scalability refers to whether the OSD service needs to be restored.

[0092] S306, if the expansion item is included, determine whether the HDD is included in the storage pool. If it is not in the storage pool, directly repair it according to the first repair method; the first repair method is a process of shrinking, filling and expanding. If it is in the storage pool, execute S307.

[0093] S307: Determine whether the faulty node is connected to the cluster.

[0094] This embodiment can obtain connectivity based on the waiting principle.

[0095] S308: If connected, repair each service of the node according to the repair order in the repair list to obtain a repaired node.

[0096] S309, if not connected, the waiting interface prompts the user to configure the network IP for filling and connection.

[0097] S310: After the repair is completed, all expansion items are marked as repair completed.

[0098] It should be noted that when repairing, the repair results of the set key service items can be tested to determine whether the repair is successful. If successful, the next item is tested. If the repair is unsuccessful, the report results are determined and the failed key items are listed.

[0099] The embodiment of the present invention provides a node fault repair method, which can quickly and efficiently perform automatic repairs according to a repair list, and any operation and maintenance personnel can quickly perform repairs, thereby reducing the difficulty of operation and maintenance, improving the success rate of repairs, shortening the business reconstruction cycle, reducing the time period of impact on the business, and ensuring business continuity and data reliability.

[0100] The node fault repairing device provided in an embodiment of the present invention is introduced below. The node fault repairing device described below and the node fault repairing method described above can be referred to each other.

[0101] Figure 4 A schematic diagram of the structure of a node fault repair device provided by an embodiment of the present invention may include:

[0102] The fault node feature determination module 100 is used to analyze the cluster state, alarm information and configuration file of the node to determine the fault node feature; wherein the cluster state is the operation state information of the cluster, the alarm information is the information to remind the cluster abnormality, and the configuration file includes the files of various services of the node;

[0103] A service repair sequence determination module 200, configured to determine a service repair sequence based on the fault node characteristics and the dependency relationship of services in the fault node;

[0104] A repair list determination module 300 is used to determine the repair method of each service in the faulty node and the method for determining successful repair, and to create a repair list according to the repair method, the method for determining successful repair and the order of service repair;

[0105] The repair module 400 is used to determine the repair method of the faulty node based on the repair list, and repair the faulty node based on the repair method; wherein the repair method includes repairing the service with stored data according to the repair order and repair method in the repair list.

[0106] Further, based on the above embodiment, the node fault repair device may further include:

[0107] A configuration list determination module is used to determine a configuration list according to the characteristics of the faulty node, wherein the configuration list includes repair items and non-repair items; wherein the repair items include services that need to be repaired during the system reinstallation process and the capacity expansion process; and the non-repair items are services that need to be configured without repair;

[0108] Accordingly, the repair list determination module 300 includes:

[0109] A repair list determination unit is used to create the repair list based on the configuration list, the repair method, the method for determining successful repair, and the service repair sequence.

[0110] Further, based on the above embodiment, the repair list determination module 300 includes:

[0111] A first expansion item judgment unit, used to determine whether the repair list includes an expansion item, the expansion item being a hard disk data service;

[0112] a unit for determining whether the hard disk drive data is included in the storage pool, for determining whether the hard disk drive data of the expansion item is included in the storage pool when the expansion item is included in the repair list;

[0113] A first repair unit, configured to repair the hard disk drive data of the expansion item according to the repair sequence and repair method in the repair list when the hard disk drive data of the expansion item is included in the storage pool;

[0114] A capacity reduction unit, configured to reduce the capacity of the expansion item when the hard disk drive data of the expansion item is not included in the storage pool;

[0115] A connectivity determination unit, configured to determine whether the faulty node is connected to the cluster when the capacity expansion item is not included in the repair list;

[0116] A second expansion item judgment unit is used to obtain a repair service according to the repair order in the repair list when the faulty node is connected to the cluster, and to judge whether the current repair service is the expansion item;

[0117] An expansion processing unit, configured to determine to perform expansion processing on all the shrunk and refilled expansion items if the current repair service is the expansion item;

[0118] The second repair unit is used to determine to perform repair according to the repair order and repair method in the repair list if the current repair service is not the expansion item.

[0119] Further, based on the above embodiment, the node fault repair device may further include:

[0120] A critical service determination unit, configured to determine whether the currently repaired service is a critical service based on the key items configured in the repair list; wherein the critical service is a service set according to the impact of the service on the node;

[0121] A next service determination unit, configured to detect the repaired key service when the currently repaired service is a key service, and if the detection passes, perform repair of the next service;

[0122] The next service repairing unit is used to directly perform repairing of the next service when the currently repaired service is a non-critical service.

[0123] Further, based on the above embodiment, the above node fault repair method may also include:

[0124] A waiting connection module is used to determine whether to wait according to a waiting principle; wherein the waiting principle is a principle of waiting according to a set overall waiting time and a waiting frequency;

[0125] The failure prompt module is used to, when waiting based on the waiting frequency in the waiting principle, detect connectivity within the set overall waiting time, and then perform repairs based on the repair list; when the set overall waiting time is exceeded, prompt failure and display the reason for failure.

[0126] Further, based on the above embodiment, the information in the repair list includes serial number, configuration item, module, level, key item, whether repair is required, dependency, repair method and repair success detection method.

[0127] Further, based on the above embodiment, the node fault repair device may further include:

[0128] A detection module is used to perform an overall detection on the repair results; wherein, the overall detection includes detecting the repair list to determine whether there are abnormal items, and directly ending the process when no abnormal items exist, and determining the overall detection results based on the number of services to be repaired and the types of services that failed to be repaired, wherein the overall detection results include successful overall service repair, service repair failure, and successful partial service repair.

[0129] It should be noted that the order of the modules and units in the above-mentioned node fault repair device can be changed without affecting the logic.

[0130] Figure 4 The description of the features in the corresponding embodiments can be found in Figure 4 The relevant descriptions of the corresponding embodiments will not be repeated here one by one.

[0131] The node fault repair device provided by the embodiment of the present invention may include: a fault node feature determination module 100, which is used to analyze the cluster state, alarm information and configuration file of the node to determine the fault node feature; wherein the cluster state is the running state information of the cluster, the alarm information is the information to remind the cluster abnormality, and the configuration file includes the files of various services of the node; a service repair order determination module 200, which is used to determine the service repair order based on the fault node feature and the dependency relationship of the services in the fault node; a repair list determination module 300, which is used to determine the repair method and the judgment method of successful repair of each service in the fault node, and create a repair list list according to the repair method, the judgment method of successful repair and the service repair order; a repair module 400, which is used to determine the repair method of the fault node based on the repair list list, and repair the fault node based on the repair method; wherein the repair method includes repairing the service with storage data according to the repair order and repair method in the repair list list. The present invention can automatically repair the node with data that needs to be repaired directly based on the repair list list generated by the fault node, without the need for professionals to reconstruct the data, thereby improving the efficiency of fault node repair.

[0132] A node fault repair device provided by an embodiment of the present invention is introduced below. The node fault repair device described below and the node fault repair method described above can be referenced to each other.

[0133] Figure 5 A schematic diagram of the structure of a node fault repair device provided by an embodiment of the present invention, such as Figure 5 As shown, the node failure repair device may include: a memory 60 for storing a computer program;

[0134] The processor 61 is used to implement the steps of the node failure repair method in the above embodiment when executing the computer program.

[0135] The node fault repair device provided in this embodiment may include but is not limited to a smart phone, a tablet computer, a laptop computer, or a desktop computer.

[0136] Among them, the processor 61 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 61 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 61 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 61 may be integrated with a graphics processing unit (GPU), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 61 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0137] The memory 60 may include one or more computer-readable storage media, which may be non-transitory. The memory 60 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 60 is at least used to store the following computer program 601, wherein, after the computer program is loaded and executed by the processor 61, the relevant steps of the node fault repair method disclosed in any of the aforementioned embodiments can be implemented. In addition, the resources stored in the memory 60 may also include an operating system 602 and data 603, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 602 may include Windows, Unix, Linux, etc. Data 603 may include but is not limited to data required for node fault repair, etc.

[0138] In some embodiments, the node fault repair device may also include a display screen 62 , an input and output interface 63 , a communication interface 64 , a power supply 65 , and a communication bus 66 .

[0139] Those skilled in the art will understand that Figure 5 The structure shown in the figure does not constitute a limitation on the node failure recovery device, and may include more or fewer components than those shown in the figure.

[0140] It is understandable that if the node fault repair method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the current technology, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, magnetic disk or optical disk, etc. Various media that can store program codes.

[0141] Based on this, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned node fault repair method are implemented.

[0142] Based on this, an embodiment of the present invention further provides a computer program product, including a computer program / instruction, which implements the steps of the above-mentioned node fault repair method when executed by a processor.

[0143] The above is a detailed introduction to a node fault repair method, device, equipment and readable storage medium provided by an embodiment of the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0144] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0145] The above is a detailed introduction to a node fault repair method, device, equipment and readable storage medium provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A node fault repair method, characterized in that: include: Analyze the cluster status, alarm information and configuration files of the node to determine the characteristics of the faulty node; wherein the cluster status is the running status information of the cluster, the alarm information is the information to remind the cluster of abnormality, and the configuration file includes the files of various services of the node; Determine the service repair order based on the fault node characteristics and the dependency relationship of services in the fault node; Determine the repair method and successful repair judgment method of each service in the faulty node, and create a repair list list according to the repair method, the successful repair judgment method and the service repair sequence; A repair method for the faulty node is determined based on the repair list, and the faulty node is repaired based on the repair method; wherein the repair method includes repairing the service having storage data according to the repair order and repair method in the repair list.

2. The node failure repair method according to claim 1, characterized in that: The node fault repair method further includes: Determine a configuration list according to the characteristics of the faulty node, the configuration list including repair items and non-repair items; wherein the repair items include services that need to be repaired during the system reinstallation process and the capacity expansion process; the non-repair items are services that need to be configured without repair; Accordingly, the method of determining the repair method of each service in the faulty node and the method of judging successful repair, and creating a repair list according to the repair method, the method of judging successful repair and the order of service repair, includes: The repair list is created based on the configuration list, the repair method, the method for determining successful repair, and the service repair sequence.

3. The node failure repair method according to claim 1, characterized in that: The determining of the repair method of each service in the faulty node and the method for judging successful repair, and creating a repair list according to the repair method, the method for judging successful repair and the order of service repair, includes: Determine whether the repair list includes an expansion item, the expansion item being a hard disk data service; When the expansion item is included in the repair list, determining whether the hard disk drive data of the expansion item is included in the storage pool; When the hard disk drive data of the expansion item is included in the storage pool, repair is performed according to the repair order and repair method in the repair list; When the hard disk drive data of the expansion item is not included in the storage pool, the expansion item is reduced in size; When the repair list does not include the expansion item, determining whether the faulty node is connected to the cluster; When the faulty node is connected to the cluster, the repair service is obtained according to the repair order in the repair list, and it is determined whether the current repair service is the expansion item; If the current repair service is the expansion item, determine to perform expansion processing on all the expansion items that have been shrunk and reinstalled; If the current repair service is not the expansion item, determine to perform repair according to the repair order and repair method in the repair list.

4. The node failure repair method according to claim 3, characterized in that: If the current repair service is not the expansion item, after determining to repair according to the repair order and repair method in the repair list, the method further includes: Determine whether the currently repaired service is a critical service based on the key items configured in the repair list; wherein the critical service is a service set according to the impact of the service on the node; When the currently repaired service is a critical service, the repaired critical service is tested, and if the test passes, the next service is repaired; When the currently repaired service is a non-critical service, the repair of the next service is directly performed.

5. The node failure repair method according to claim 3, characterized in that: When the faulty node is disconnected from the cluster, the following events may occur: Determine to wait according to a waiting principle; wherein the waiting principle is a principle of waiting according to a set overall waiting time and a waiting frequency; When waiting based on the waiting frequency in the waiting principle, if connectivity is detected within the set overall waiting time, repair is performed based on the repair list; when the set overall waiting time is exceeded, a failure prompt is displayed and the reason for the failure is displayed.

6. The node failure repair method according to claim 1, characterized in that: The information in the repair list includes serial number, configuration item, module, level, key item, whether repair is needed, dependency, repair method and repair success detection method.

7. The node failure repair method according to claim 1, characterized in that: After determining a repair method for the faulty node based on the repair list and repairing the faulty node based on the repair method, the method further includes: Perform an overall test on the repair results; wherein, the overall test includes testing the repair list to determine whether there are any abnormal items, and ending directly when no abnormal items exist, and determining the overall test results based on the number of services to be repaired and the types of services that failed to be repaired, wherein the overall test results include successful overall service repair, service repair failure, and successful partial service repair.

8. A node fault repair device, characterized in that: include: A fault node feature determination module is used to analyze the cluster state, alarm information and configuration files of the node to determine the fault node features; wherein the cluster state is the running state information of the cluster, the alarm information is the information to remind the cluster of abnormality, and the configuration files include files of various services of the node; A service repair sequence determination module, used to determine the service repair sequence based on the fault node characteristics and the dependency relationship of services in the fault node; A repair list determination module is used to determine the repair method of each service in the faulty node and the method for judging successful repair, and to create a repair list according to the repair method, the method for judging successful repair and the order of service repair; A repair module is used to determine a repair method for a faulty node based on the repair list, and repair the faulty node based on the repair method; wherein the repair method includes repairing a service having stored data according to the repair order and repair method in the repair list.

9. A node fault repair device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program to implement the steps of the node failure repair method as claimed in any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the node failure repair method according to any one of claims 1 to 7 are implemented.

Images