A method, apparatus, device, and readable storage medium for node fault repair.

Abstract

This invention discloses a node fault repair method, apparatus, device, and readable storage medium, applied in the field of distributed storage technology. The method includes: determining the service repair order based on the characteristics of the faulty node, the hierarchy of services within the faulty node, and their dependencies; determining the repair methods for each service within the faulty node and the method for determining successful repair; creating a repair list based on the repair methods, the method for determining successful repair, and the service repair order; determining the repair method for the faulty node based on the repair list, and repairing the faulty node based on the repair method; the repair method includes repairing services containing stored data according to the repair order and repair method in the repair list. Compared to current methods that require professional personnel to reconstruct the data, this invention can directly perform automatic repair based on the repair list generated from the faulty node, eliminating the need for professional personnel to reconstruct the data and improving the efficiency of faulty node repair.

Description

Technical Field

[0001] This invention relates to the field of distributed storage technology, and in particular to a node fault repair method, apparatus, device, and readable storage medium. Background Technology

[0002] Distributed storage systems are server-based, and servers often use a RAID configuration with two system disks to ensure reliability. However, even with two system disks, failure and delayed replacement can still lead to system corruption and node unavailability. A common solution is to first scale the node down from the cluster, reinstall the system, and then scale the node back into the cluster. Professionals then reconstruct the data. This method has a long recovery period and is unsuitable for scenarios with few nodes and low redundancy.

[0003] It is evident that improving the efficiency of node fault repair is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a node fault repair method, apparatus, device and readable storage medium, which solves the technical problem of low node fault repair efficiency in the prior art.

[0005] To address the aforementioned technical problems, this invention provides a node fault repair method, comprising:

[0006] The cluster status, alarm information, and configuration files of the nodes are analyzed to determine the characteristics of the faulty nodes; wherein, the cluster status refers to the running status information of the cluster, the alarm information refers to the information that alerts the cluster to abnormalities, and the configuration files include the files of various services of the node.

[0007] The service repair order is determined based on the characteristics of the faulty node and the dependencies of services within the faulty node.

[0008] Determine the repair methods and successful repair criteria for each service in the faulty node, and create a repair list based on the repair methods, successful repair criteria, and service repair order.

[0009] Based on the repair list, a repair method for the faulty node is determined, and the faulty node is repaired based on the repair method; wherein, the repair method includes repairing services with stored data according to the repair order and repair method in the repair list.

[0010] On the one hand, the node fault repair method also includes:

[0011] A configuration list is determined based on the characteristics of the faulty node. The configuration list includes repair items and non-repair items. The repair items include services that need to be repaired during system reinstallation and capacity expansion. The non-repair items are services that do not need to be repaired but need to be configured.

[0012] Accordingly, the method for determining the repair of each service in the faulty node, the method for judging successful repair, and the creation of a repair list based on the repair methods, the method for judging successful repair, and the service repair order include:

[0013] The repair list is created based on the configuration list, the repair method, the successful repair determination method, and the service repair order.

[0014] On the one hand, the method for determining the repair of each service in the faulty node, the method for judging successful repair, and the creation of a repair list based on the repair methods, the method for judging successful repair, and the service repair order include:

[0015] Determine whether the repair list includes an expansion item, where the expansion item is a hard disk data service;

[0016] When the expansion item is included in the repair list, it is determined whether the hard disk drive data of the expansion item is included in the storage pool.

[0017] When the hard drive data of the expansion item is included in the storage pool, the repair is performed in the order and method of the repair list.

[0018] If the hard drive data of the expansion item is not included in the storage pool, then the expansion item is reduced in size;

[0019] If the expansion item is not included in the repair list, then it is determined whether the faulty node is connected to the cluster.

[0020] When the faulty node connects to the cluster, it obtains repair services according to the repair order in the repair list and determines whether the current repair service is the expansion item.

[0021] If the current repair service is the expansion item, determine to perform expansion processing on all expansion items that have been reduced in size and refilled;

[0022] If the current repair service is not one of the expansion items, then the repair will be performed according to the repair order and repair method in the repair list.

[0023] On the one hand, if the current repair service is not the expansion item, after determining to repair according to the repair order and repair method in the repair list, it also includes:

[0024] Based on the key items configured in the repair list, it is determined whether the service currently being repaired is a critical service; wherein, the critical service is a service defined according to its impact on the node;

[0025] When the service currently being repaired is a critical service, the repaired critical service is tested. If the test passes, the repair of the next service is executed.

[0026] If the service currently being repaired is a non-critical service, then the repair of the next service will be performed directly.

[0027] On the one hand, when a faulty node is not connected to the cluster, including:

[0028] It is determined that waiting will be conducted according to the waiting principle; wherein, the waiting principle is the principle of waiting based on the set overall waiting time and waiting frequency;

[0029] When waiting based on the waiting frequency in the aforementioned waiting principle, if connectivity is detected within the set overall waiting time, repair is performed based on the repair list; if the set overall waiting time is exceeded, a failure message is displayed, along with the reason for the failure.

[0030] On the one hand, the information in the repair list includes serial number, configuration item, module, level, key item, whether repair is required, dependency item, repair method, and repair success detection method.

[0031] On the one hand, after determining the repair method for the faulty node based on the repair list and repairing the faulty node based on the repair method, the method further includes:

[0032] A comprehensive inspection of the repair results is performed. This comprehensive inspection includes checking the repair list to determine if there are any abnormal items. If no abnormal items are found, the process ends directly. The comprehensive inspection result is determined based on the number of services to be repaired and the types of services that failed to be repaired. The comprehensive inspection result includes overall service repair success, service repair failure, and partial service repair success.

[0033] This invention also provides a node fault repair device, comprising:

[0034] The fault node feature determination module is used to analyze the cluster status, alarm information and configuration files of nodes to determine the characteristics of fault nodes; wherein, the cluster status is the running status information of the cluster, the alarm information is the information that reminds the cluster of abnormalities, and the configuration files include the files of various services of the node.

[0035] The service repair order determination module is used to determine the service repair order based on the characteristics of the fault node and the dependencies of services in the fault node.

[0036] The repair list determination module is used to determine the repair methods and successful repair judgment methods for each service in the fault node, and to create a repair list based on the repair methods, the successful repair judgment methods and the service repair order.

[0037] The repair module is used to determine the repair method for the faulty node based on the repair list, and to repair the faulty node based on the repair method; wherein, the repair method includes repairing services with stored data according to the repair order and repair method in the repair list.

[0038] This invention also provides a node fault repair device, comprising:

[0039] Memory, used to store computer programs;

[0040] A processor is used to execute the computer program to implement the steps of the node fault repair method described above.

[0041] This invention also provides a readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the node fault repair method described above.

[0042] The present invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-described node fault repair method.

[0043] The purpose of this invention is to provide a node fault repair method, apparatus, device, and readable storage medium, which can solve the technical problem of low node fault repair efficiency.

[0044] To address the aforementioned technical problems, embodiments of the present invention provide a node fault repair method, which may include: analyzing the cluster status, alarm information, and configuration files of a node to determine the characteristics of the faulty node; wherein, the cluster status refers to the running status information of the cluster, the alarm information refers to information reminding the cluster of abnormalities, and the configuration files include files for various services of the node; determining the service repair order based on the faulty node characteristics and the dependencies between services in the faulty node; determining the repair methods and successful repair judgment methods for each service in the faulty node, and creating a repair list based on the repair methods, successful repair judgment methods, and service repair order; determining the repair methods for the faulty node based on the repair list, and repairing the faulty node based on the repair methods; wherein, the repair methods include repairing services with stored data according to the repair order and repair methods in the repair list.

[0045] As can be seen from the above technical solution, the beneficial effects of the present invention are as follows: Compared with the current method of repairing node failures by using shrinking, refilling, and expanding nodes, and requiring professional personnel to reconstruct the data, the present invention can automatically repair nodes with data that need to be repaired directly based on the repair list generated by the faulty node, without requiring professional personnel to reconstruct the data, thus improving the efficiency of faulty node repair. Attached Figure Description

[0046] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 A flowchart of a node fault repair method provided in an embodiment of the present invention;

[0048] Figure 2 This is a schematic diagram of a distributed storage system framework provided in an embodiment of the present invention;

[0049] Figure 3 A flowchart illustrating a node fault repair method provided in an embodiment of the present invention;

[0050] Figure 4 This is a schematic diagram of a node fault repair device provided in an embodiment of the present invention;

[0051] Figure 5 This is a schematic diagram of a node fault repair device provided in an embodiment of the present invention. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0053] The terms "comprising" and "having," and any variations thereof, in the specification and accompanying drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may include steps or units not listed.

[0054] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] Next, a flowchart of a node fault repair method provided by an embodiment of the present invention will be described in detail. Figure 1 A node fault repair method provided in this embodiment of the invention may include:

[0056] S101, analyze the cluster status, alarm information and configuration files of the nodes to determine the characteristics of the faulty nodes; where cluster status is the running status information of the cluster, alarm information is the information that reminds the cluster of abnormalities, and configuration files include the files of various services of the node.

[0057] The execution subject of this embodiment is an electronic device. This electronic device can be a computer, mobile phone, etc. This embodiment does not limit the specific cluster state; for example, the cluster state in this embodiment can be a degraded state; or the cluster state in this embodiment can be a node failure state; or the cluster state in this embodiment can be a monitoring service malfunction state. This embodiment does not limit the specific alarm information. For example, the alarm information in this embodiment can be a hard drive interface alarm, or the alarm information in this embodiment can also be a network port alarm. This embodiment does not limit the specific configuration file. The configuration file in this embodiment can be the configuration file of the faulty node, so as to obtain more detailed information about the node from the configuration file.

[0058] S102, determine the service repair order based on the characteristics of the faulty node and the service dependencies in the faulty node.

[0059] The service repair order in this embodiment refers to the following: when multiple nodes fail, if node B depends on node A, node A needs to be repaired first; or if node C is at a higher level than node B, node C needs to be repaired first. It is understood that the services in this embodiment can be services at different levels. For example, the services in this embodiment can be operations, storage, and protocols. The hierarchical relationship between the operations, storage, and protocols layers can be understood from the perspective of computer networks or software architecture. These layers are typically used in design to abstract and manage various aspects of complex systems to achieve the design goals of high cohesion and low coupling. Protocol Layer: This layer is mainly responsible for defining the format and rules of data exchange. In software architecture, the protocol layer refers to the application programming interface (API) or inter-service communication protocol, which specifies how different modules or services interact. Operations Layer: The operations layer typically involves data processing and management. This may include operations such as data creation, updating, and deletion. In a database management system, the operations layer may be the part that executes SQL (database) statements, responsible for parsing SQL commands and manipulating the database. Storage Layer: The storage layer focuses on data persistence, i.e., how to save data to disk or other storage media and retrieve it when needed. This embodiment does not limit the specific service; for example, the service in this embodiment can be a basic service; or the service in this embodiment can be a storage service; or the service in this embodiment can be a network service; or the service in this embodiment can be a computing service.

[0060] S103, determine the repair methods and successful repair criteria for each service in the faulty node, and create a repair list based on the repair methods, successful repair criteria, and service repair order.

[0061] The repair methods for each service in this embodiment can be based on existing tools or newly designed tools. This embodiment does not limit the method for determining successful repair for each service. For example, the method for determining network repair can be to perform a network query; if the status is "up" and other nodes can be pinged, then the repair is considered successful. A "up" status typically indicates that a device or service is running and ready to accept requests. Being able to ping other nodes means that the device or service can communicate with other devices through the network without obstruction. The method for determining successful repair of the cluster version in this embodiment can be a command query that returns a version number, which is the required version number. This embodiment does not limit the specific repair list. For example, the repair list in this embodiment may include configuration items, the module to which it belongs, whether repair is needed, dependencies, etc. Alternatively, the information in the repair list in this embodiment may include a sequence number, configuration item, module to which it belongs, level, key item, whether repair is needed, dependencies, repair method, and repair success detection method.

[0062] It should be further explained that, to improve the accuracy of the repair list, the above-mentioned node fault repair method may further include: determining a configuration list based on the characteristics of the faulty node, wherein the configuration list includes repair items and non-repair items; wherein, repair items include services that need to be repaired again during system reinstallation and capacity expansion; non-repair items are services that do not need repair but need to be configured; accordingly, determining the repair methods and successful repair judgment methods for each service in the faulty node, and creating a repair list based on the repair methods, successful repair judgment methods, and service repair order, including: creating a repair list based on the configuration list, repair methods, successful repair judgment methods, and service repair order. This embodiment, by identifying repair items and non-repair items, allows for repair of only the repair items and configuration of the non-repair items when repairing according to the repair list. For example, in this embodiment, the repair items can be the basic recovery module, data service module, and non-data service module (services configured during the expansion process), and the advanced settings module (advanced settings performed after the original expansion). Non-repair items include basic configuration information such as whether the node's HDD (Hard Disk Drive) is in the storage pool. Services in the cluster are categorized into four levels of modules based on their hierarchy. For example, if a node is a mon (mon) node, the mon service is added to the list; if it is not a mon node, it is not added (mon service). This embodiment addresses the issue that repair items need to be re-repaired after a failure to ensure normal system operation, while non-repair items can be directly configured, improving the efficiency of node failure repair.

[0063] It should be further explained that the above-mentioned methods for determining the repair of each service in the faulty node, the methods for judging successful repair, and the creation of a repair list based on the repair methods, the methods for judging successful repair, and the service repair order may include:

[0064] S1031, Determine whether the repair list includes an expansion item, where the expansion item is a hard disk data service;

[0065] S1032, when the repair list includes expansion items, determine whether the hard disk drive data of the expansion items is included in the storage pool.

[0066] S1033, when the hard disk drive data of the expansion item is included in the storage pool, repair is performed according to the repair order and repair method in the repair list.

[0067] S1034, If the hard disk drive data of the expansion item is not included in the storage pool, then the expansion item is reduced in size.

[0068] S1035, if the expansion item is not included in the repair list, then determine whether the faulty node is connected to the cluster.

[0069] S1036, when the faulty node is connected to the cluster, the repair service is obtained in the order of repair in the repair list, and it is determined whether the current repair service is an expansion item.

[0070] S1037, If the current repair service is an expansion item, determine to perform expansion processing on all expansion items that have been reduced in size and reinstalled.

[0071] S1038, If the current repair service is not an expansion item, determine to repair according to the repair order and repair method in the repair list.

[0072] In this embodiment, determining whether an expansion item is included can be understood as whether the OSD (Object Storage Daemon) service needs to be restored. If it's not included, it can be determined whether the hard disk drive data, such as the HDD (Hard Disk Drive), is in the storage pool. OSD is a service above HDD. In this embodiment, the expansion item is the hard disk data service. The hard disk data service is used to determine whether the service containing data needs to be restored. When it's not needed, the data can be repaired according to the repair checklist. When the service containing data needs to be restored, it can be determined whether the data is in the storage pool. If it is, it means a new repair method needs to be used; otherwise, the service without data can be directly scaled down. In this embodiment, when the hard disk drive data is not included in the storage pool, repair can be performed based on the existing process: Step 1: Scale-down. First, the faulty node is scaled down from the cluster, which means service scaling down, such as OSD (Disk Management Service), MON service, protocol service, etc. Step 2: Loading and basic recovery. After scaling down, the node is restored, and the new system is loaded. Step 3: Expansion. This step requires first analyzing the node's original role and services within the cluster, and then expanding services such as OSD, MON, and protocols by extending those services. This embodiment provides methods for repairing different services, allowing services that do not contain data to be directly scaled out of the cluster, reinstalled, or expanded back into the cluster. For services that include data, repairs can be performed directly according to the repair checklist without data reconstruction, thus improving the efficiency of node fault repair.

[0073] It should be further explained that, to improve the accuracy of node repair, after determining that the repair should be performed according to the repair order and method in the repair list, if the current repair service is not an expansion item, it may also include: determining whether the service being repaired is a critical service based on the key items configured in the repair list; where critical services are services defined according to their impact on the node; if the current service being repaired is a critical service, the repaired critical service is tested; if the test passes, the repair of the next service is executed; if the current service being repaired is a non-critical service, the repair of the next service is executed directly. In this embodiment, if it is a critical item (e.g., version number detection, incorrect version, direct termination), a module test is performed; if it fails, the process ends; if the critical item passes, the process continues to determine whether it is the last item; if not, the next item is executed; if it is the last item, the repair ends. In this embodiment, when it is determined that the service being repaired is a critical service configured as a key item, the success of the repair of the critical service is checked. Since critical services have a significant impact on the entire node, it is necessary to determine whether the service is a critical service.

[0074] It should be further explained that when a faulty node is not connected to the cluster, the above-mentioned waiting process may include: determining to wait according to a waiting principle; wherein, the waiting principle is based on a set overall waiting time and waiting frequency. When it is determined based on the waiting principle that the overall waiting time has been exceeded, failure logic is detected, a failure message is displayed, and the reason for the failure is shown. This embodiment does not limit the specific overall waiting time. For example, the waiting time in this embodiment can be 2 minutes; or the waiting time in this embodiment can also be 5 minutes. The waiting frequency in this embodiment can be once every 5 seconds or once per minute. Since critical services will affect the repair of nodes, making nodes unusable, a failure message needs to be displayed so that the problem can be resolved in a timely manner. Furthermore, the device waiting principle prevents the waiting time from being too long, which would affect the repair efficiency.

[0075] S104, determine the repair method for the faulty node based on the repair list, and repair the faulty node based on the repair method; wherein, the repair method includes repairing services with stored data according to the repair order and repair method in the repair list.

[0076] This embodiment does not limit the specific data storage service. For example, the data storage service in this embodiment can be a service that includes HDD (Hard Disk Drive) data; or the data storage service in this embodiment can be an SSD (Solid State Drive). The repair method in this embodiment can repair services with stored data according to the repair order and repair method in the repair list. For services without data, repair methods such as shrinking, re-installing, and expanding can be used directly.

[0077] It should be further explained that after determining the repair method for the faulty node based on the repair checklist and repairing the faulty node based on the repair method, the process can also include: performing an overall check on the repair results. This overall check includes checking the repair checklist to determine if there are any anomalies; if none are found, the process ends directly. It also involves determining the overall check result based on the number of services to be repaired and the types of services that failed to be repaired. The overall check result includes overall service repair success, service repair failure, and partial service repair success. In this embodiment, after repair is complete, an overall check is first called to identify anomalies. If no anomalies are found, the process ends directly. Specifically, the process first checks if the node IP is reachable. If not, the system disk is considered faulty, all items are marked as failed, and the repair module is called to repair them. If reachable, an overall check is performed, and failed items are marked as needing repair. After the check, the number of items to be repaired is checked to see if it is greater than 0. If it is equal to 0, the process passes directly. If it is greater than 0, it further determines whether the called module is the analysis module or the repair module. If it is the analysis module, the repair module is called to repair it. Then, it checks if key items are included. If they are included, the overall result is considered a failure; otherwise, the overall result is a success. Module detection primarily involves performing a check after the current service has been repaired. If the check passes, the process continues; otherwise, a detection and analysis report is generated directly. This embodiment provides a method for overall detection, enabling dual detection of the service and improving detection accuracy.

[0078] It should be further explained that, in order to improve the accuracy of repairing services for which data is not included in the storage pool, the above-mentioned repair method for determining faulty nodes based on the repair list, and repairing the faulty nodes based on the repair method, can include: when all services in the current repair list include data in the storage pool, the repair method is determined to be the method for determining faulty nodes based on the repair list; when all services in the repair list do not include data in the storage pool, the repair method is determined to be the method of first scaling the node down from the cluster, then reloading the node, and finally scaling the node back into the cluster; when some services in the repair list include data in the storage pool, the method for determining faulty nodes based on the repair list is used for services including data in the storage pool, and the method for first scaling the node down from the cluster, then reloading the node, and finally scaling the node back into the cluster is used for services not including data in the storage pool. This embodiment provides specific node repair methods corresponding to different repair lists, improving the accuracy and efficiency of node fault repair.

[0079] This invention provides a node fault repair method, which may include: S101, analyzing the cluster status, alarm information, and configuration files of the node to determine the characteristics of the faulty node; wherein, the cluster status refers to the running status information of the cluster, the alarm information refers to information reminding the cluster of abnormalities, and the configuration files include the files of various services of the node; S102, determining the service repair order based on the faulty node characteristics, the service hierarchy and dependencies in the faulty node; S103, determining the repair methods and successful repair judgment methods for each service in the faulty node, and creating a repair list based on the repair methods, successful repair judgment methods, and service repair order; S104, determining the repair methods for the faulty node based on the repair list, and repairing the faulty node based on the repair methods; wherein, the repair methods include repairing services with stored data according to the repair order and repair methods in the repair list. This invention can automatically repair nodes with data requiring repair directly based on the repair list generated from the faulty node, without requiring professional personnel to reconstruct the data, thus improving the efficiency of faulty node repair.

[0080] A distributed storage cluster typically consists of multiple servers. These servers may have different physical hardware and often perform different storage service roles. For example, there are usually an odd number of monitoring nodes, distributed across different racks. This way, a power outage or failure of a single rack does not affect the operation of the entire cluster. Please refer to [reference needed]. Figure 2 , Figure 2This is a schematic diagram of a distributed storage system framework provided by an embodiment of the present invention. Distributed storage service systems generally include hardware / parts, an OS (operation) layer, a storage layer, a functional layer, and a protocol layer. In converged scenarios, not only are the roles different, but the upper-layer protocol functions may also differ. Some nodes provide file services (SMB (Server Message Block, client / server file sharing protocol), NFS (Network File System, distributed file system protocol), etc.), while others provide object services (S3 (Amazon Simple Storage Service, object storage service) / Swift (OpenStack Object Storage System, object storage system), etc.). Therefore, if a node fails and needs to be restored to its original state, it is necessary to analyze each layer one by one. For example, in a 20-node cluster with two storage pools and seven monitoring nodes (nodes 1, 3, 5, 7, 11, 13, and 15), the first 10 nodes provide NFS (Network File System, a distributed file system protocol), and the last 10 nodes provide RGW (Rados Gateway, a distributed object storage system gateway) services. Thus, the services that need to be restored at each layer differ between a failed node 1 and a failed node 12.

[0081] For a clearer understanding of this invention, please refer to the following details. Figure 3 , Figure 3 A flowchart illustrating a node fault repair method provided in this embodiment of the invention may specifically include:

[0082] S301 analyzes the cluster status, alarm information, and configuration files of the nodes to determine the characteristics of the faulty nodes; where cluster status refers to the running status of the cluster, alarm information refers to information reminding the cluster of abnormalities and faults, and configuration files include the configuration files of various services of the node.

[0083] This embodiment analyzes the cluster status, alarm information, and configuration files of nodes, including: analyzing the underlying layer (node ​​name, node version, IP, clock source, alarm scheduler, security compliance, etc.); analyzing the storage layer (monitoring node, OSD (object storage device), whether the OSD is in the storage pool, etc.); and analyzing the protocol layer (load balancing, high availability, file, block, and object services). The analysis process in this embodiment can be considered an analysis module.

[0084] S302, determine the list of node services and configurations to be repaired based on the characteristics of the faulty node; wherein, the configuration node includes repair items and non-repair items, and the non-repair items are services that do not require data recovery but need to be configured.

[0085] S303, determine the dependencies between services at different levels based on service dependencies, determine the service repair order for faulty nodes, and identify key items.

[0086] S304, determine the repair methods for each service in the faulty node, the method for judging successful repair, and create a repair list based on the repair order and key items of the services in the faulty node.

[0087] For ease of understanding, this embodiment provides a specific repair list, as shown in Table 1. Table 1 is a repair list provided by an embodiment of the present invention.

[0088] Table 1. List of Repair Measures

[0089]

[0090] S305, Determine whether the repair list includes an expansion item.

[0091] Whether this embodiment includes scalability refers to whether the OSD service needs to be restored.

[0092] S306, if expansion items are included, determine whether the HDD is included in the storage pool. If it is not in the storage pool, proceed directly to the first repair method. The first repair method is the process of shrinking, filling and expanding the storage pool. If it is in the storage pool, execute S307.

[0093] S307 determines whether the faulty node is connected to the cluster.

[0094] This embodiment can achieve connectivity based on the waiting principle.

[0095] S308, if connected, then repair the various services of the node according to the repair order in the repair list to obtain the repaired node.

[0096] S309, if the connection is not established, wait for the interface to prompt the user to configure the network IP for filling and connection.

[0097] S310, after the repair is complete, mark all expansion items as repair complete.

[0098] It should be noted that during the repair process, the repair results of key service items can be checked to determine whether the repair was successful. If successful, proceed to the next item. If the repair fails, generate a report listing the key failure items.

[0099] This invention provides a node fault repair method that can quickly and efficiently automate repair based on a repair checklist. Any maintenance personnel can quickly perform repairs, reducing the difficulty of maintenance, increasing the success rate of repairs, shortening the business reconstruction cycle, reducing the time period of impact on business, and ensuring business continuity and data reliability.

[0100] The node fault repair device provided in the embodiments of the present invention will be described below. The node fault repair device described below can be referred to in correspondence with the node fault repair method described above.

[0101] Figure 4 A schematic diagram of a node fault repair device provided in an embodiment of the present invention may include:

[0102] The fault node feature determination module 100 is used to analyze the cluster status, alarm information and configuration files of the node to determine the fault node features; wherein, the cluster status is the running status information of the cluster, the alarm information is the information that reminds the cluster of abnormalities, and the configuration files include the files of various services of the node.

[0103] The service repair order determination module 200 is used to determine the service repair order based on the characteristics of the fault node and the dependencies of services in the fault node.

[0104] The repair list determination module 300 is used to determine the repair methods and successful repair judgment methods for each service in the fault node, and to create a repair list based on the repair methods, the successful repair judgment methods and the service repair order.

[0105] The repair module 400 is used to determine the repair method for the faulty node based on the repair list, and to repair the faulty node based on the repair method; wherein, the repair method includes repairing services with stored data according to the repair order and repair method in the repair list.

[0106] Furthermore, based on the above embodiments, the node fault repair device may further include:

[0107] The configuration list determination module is used to determine a configuration list based on the characteristics of the faulty node. The configuration list includes repair items and non-repair items. The repair items include services that need to be repaired during system reinstallation and capacity expansion. The non-repair items are services that do not need to be repaired but need to be configured.

[0108] Accordingly, the repair list determination module 300 includes:

[0109] The repair list determination unit is used to create the repair list based on the configuration list, the repair method, the successful repair judgment method, and the service repair order.

[0110] Furthermore, based on the above embodiments, the repair list determination module 300 includes:

[0111] The first expansion item determination unit is used to determine whether the repair list includes an expansion item, wherein the expansion item is a hard disk data service;

[0112] The hard disk drive data inclusion determination unit is used to determine whether the hard disk drive data of the expansion item is included in the storage pool when the expansion item is included in the repair list list.

[0113] The first repair unit is used to repair the hard disk drive data of the expansion item according to the repair order and repair method in the repair list when the hard disk drive data is included in the storage pool.

[0114] A shrinking unit is used to shrink the expansion item when the hard disk drive data of the expansion item is not included in the storage pool.

[0115] A connectivity determination unit is used to determine whether the faulty node is connected to the cluster when the expansion item is not included in the repair list.

[0116] The second expansion item judgment unit is used to obtain repair services according to the repair order in the repair list when the faulty node is connected to the cluster, and to determine whether the current repair service is the expansion item.

[0117] The capacity expansion processing unit is configured to determine, if the current repair service is the capacity expansion item, to perform capacity expansion processing on all capacity-shrinking and re-installing capacity expansion items.

[0118] The second repair unit is used to determine, if the current repair service is not the expansion item, to perform repair according to the repair order and repair method in the repair list.

[0119] Furthermore, based on the above embodiments, the node fault repair device may further include:

[0120] A critical service determination unit is used to determine whether the service currently being repaired is a critical service based on the critical items configured in the repair list; wherein, the critical service is a service defined according to the service's impact on the node;

[0121] The next service determination unit is used to detect the repaired critical service when the currently repaired service is a critical service. If the detection passes, the repair of the next service is executed.

[0122] The next service repair unit is used to directly perform the repair of the next service when the currently repaired service is a non-critical service.

[0123] Furthermore, based on the above embodiments, the above node fault repair method may further include:

[0124] A connection waiting module is used to determine whether to wait according to a waiting principle; wherein, the waiting principle is the principle of waiting based on a set overall waiting time and waiting frequency;

[0125] The failure notification module is used to perform repairs based on the repair list if connectivity is detected within the set overall waiting time when waiting is based on the waiting frequency in the waiting principle; if the set overall waiting time is exceeded, a failure notification is displayed, along with the reason for the failure.

[0126] Furthermore, based on the above embodiments, the information in the repair list includes serial number, configuration item, module, level, key item, whether repair is required, dependency item, repair method, and repair success detection method.

[0127] Furthermore, based on the above embodiments, the node fault repair device may further include:

[0128] The detection module is used to perform an overall detection of the repair results. The overall detection includes detecting the repair list to determine whether there are any abnormal items. If no abnormal items are found, the process ends directly. The overall detection result is determined based on the number of services to be repaired and the types of services that failed to be repaired. The overall detection result includes overall service repair success, service repair failure, and partial service repair success.

[0129] It should be noted that the order of the modules and units in the above-mentioned node fault repair device can be changed without affecting the logic.

[0130] Figure 4 The description of the features in the corresponding embodiments can be found in [reference needed]. Figure 4 The relevant descriptions of the corresponding embodiments will not be repeated here.

[0131] The node fault repair device provided in this embodiment of the invention may include: a fault node feature determination module 100, used to analyze the cluster status, alarm information, and configuration files of the node to determine the fault node features; wherein, the cluster status refers to the running status information of the cluster, the alarm information refers to information reminding the cluster of abnormalities, and the configuration files include files for various services of the node; a service repair order determination module 200, used to determine the service repair order based on the fault node features and the dependencies of services in the fault node; a repair list determination module 300, used to determine the repair methods and successful repair judgment methods for various services in the fault node, and create a repair list according to the repair methods, the successful repair judgment methods, and the service repair order; and a repair module 400, used to determine the repair methods for the fault node based on the repair list, and repair the fault node based on the repair methods; wherein, the repair methods include repairing services with stored data according to the repair order and repair methods in the repair list. This invention can automatically repair nodes with data requiring repair directly based on the repair list generated from the fault node, without requiring professional personnel to reconstruct the data, thus improving the efficiency of fault node repair.

[0132] The following describes a node fault repair device provided by an embodiment of the present invention. The node fault repair device described below can be referred to in correspondence with the node fault repair method described above.

[0133] Figure 5 This is a schematic diagram of the structure of a node fault repair device provided in an embodiment of the present invention, as shown below. Figure 5 As shown, the node fault repair device may include: a memory 60 for storing computer programs;

[0134] The processor 61 is used to implement the steps of the node fault repair method as described in the above embodiment when executing a computer program.

[0135] The node fault repair device provided in this embodiment may include, but is not limited to, smartphones, tablets, laptops, or desktop computers.

[0136] The processor 61 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 61 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 61 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 61 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 61 may also include an Artificial Intelligence (AI) processor, which handles computational operations related to machine learning.

[0137] The memory 60 may include one or more computer-readable storage media, which may be non-transitory. The memory 60 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 60 is used to store at least the following computer program 601, which, after being loaded and executed by the processor 61, is capable of implementing the relevant steps of the node fault repair method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 60 may also include an operating system 602 and data 603, and the storage method may be temporary or permanent storage. The operating system 602 may include Windows, Unix, Linux, etc. The data 603 may include, but is not limited to, data required for node fault repair.

[0138] In some embodiments, the node fault repair device may further include a display screen 62, an input / output interface 63, a communication interface 64, a power supply 65, and a communication bus 66.

[0139] Those skilled in the art will understand that Figure 5 The structure shown does not constitute a limitation on the node fault repair device and may include more or fewer components than shown.

[0140] It is understood that if the node fault repair method in the above embodiments is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the current technology, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods in the various embodiments of the present invention. The aforementioned storage medium includes: USB flash drive, mobile hard drive, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, magnetic disk or optical disk, and other media capable of storing program code.

[0141] Based on this, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the node fault repair method described above.

[0142] Based on this, an embodiment of the present invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-described node fault repair method.

[0143] The foregoing has provided a detailed description of a node fault repair method, apparatus, device, and readable storage medium provided by embodiments of the present invention. The various embodiments are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0144] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0145] The present invention has provided a detailed description of a node fault repair method, apparatus, device, and readable storage medium. Specific examples have been used to illustrate the principles and implementation methods of the invention. The descriptions of these embodiments are merely illustrative of the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to the invention without departing from its principles, and these improvements and modifications also fall within the scope of protection of the claims.

Claims

1. A node fault repair method, characterized in that, include: The cluster status, alarm information, and configuration files of the nodes are analyzed to determine the characteristics of the faulty nodes; wherein, the cluster status refers to the running status information of the cluster, the alarm information refers to the information that alerts the cluster to abnormalities, and the configuration files include the files of various services of the node. The service repair order is determined based on the characteristics of the faulty node and the dependencies of services within the faulty node. Determine the repair methods and successful repair criteria for each service in the faulty node, and create a repair list based on the repair methods, successful repair criteria, and service repair order. Based on the repair list, a repair method for the faulty node is determined, and the faulty node is repaired based on the repair method; wherein, the repair method includes repairing services with stored data according to the repair order and repair method in the repair list; The process of determining the repair methods for each service in the faulty node, the method for judging successful repair, and creating a repair list based on the repair methods, the method for judging successful repair, and the service repair order includes: Determine whether the repair list includes an expansion item, where the expansion item is a hard disk data service; When the expansion item is included in the repair list, it is determined whether the hard disk drive data of the expansion item is included in the storage pool. When the hard drive data of the expansion item is included in the storage pool, the repair is performed in the order and method of the repair list. If the hard drive data of the expansion item is not included in the storage pool, then the expansion item is reduced in size; If the expansion item is not included in the repair list, then it is determined whether the faulty node is connected to the cluster. When the faulty node connects to the cluster, it obtains repair services according to the repair order in the repair list and determines whether the current repair service is the expansion item. If the current repair service is the expansion item, determine to perform expansion processing on all expansion items that have been reduced in size and refilled; If the current repair service is not one of the expansion items, then the repair will be performed according to the repair order and repair method in the repair list.

2. The node fault repair method according to claim 1, characterized in that, The node fault repair method further includes: A configuration list is determined based on the characteristics of the faulty node. The configuration list includes repair items and non-repair items. The repair items include services that need to be repaired during system reinstallation and capacity expansion. The non-repair items are services that do not need to be repaired but need to be configured. Accordingly, the method for determining the repair of each service in the faulty node, the method for judging successful repair, and the creation of a repair list based on the repair methods, the method for judging successful repair, and the service repair order include: The repair list is created based on the configuration list, the repair method, the successful repair determination method, and the service repair order.

3. The node fault repair method according to claim 1, characterized in that, If the current repair service is not the expansion item, after determining to repair according to the repair order and repair method in the repair list, the following is also included: Based on the key items configured in the repair list, it is determined whether the service currently being repaired is a critical service; wherein, the critical service is a service defined according to its impact on the node; When the service currently being repaired is a critical service, the repaired critical service is tested. If the test passes, the repair of the next service is executed. If the service currently being repaired is a non-critical service, then the repair of the next service will be performed directly.

4. The node fault repair method according to claim 1, characterized in that, When a faulty node is not connected to the cluster, including: It is determined that waiting will be conducted according to the waiting principle; wherein, the waiting principle is the principle of waiting based on the set overall waiting time and waiting frequency; When waiting based on the waiting frequency in the aforementioned waiting principle, if connectivity is detected within the set overall waiting time, repair is performed based on the repair list; if the set overall waiting time is exceeded, a failure message is displayed, along with the reason for the failure.

5. The node fault repair method according to claim 1, characterized in that, The information in the repair list includes the serial number, configuration item, module, level, key item, whether repair is required, dependencies, repair method, and repair success detection method.

6. The node fault repair method according to claim 1, characterized in that, After determining the repair method for the faulty node based on the repair list and repairing the faulty node based on the repair method, the method further includes: A comprehensive inspection of the repair results is performed. This comprehensive inspection includes checking the repair list to determine if there are any abnormal items. If no abnormal items are found, the process ends directly. The comprehensive inspection result is determined based on the number of services to be repaired and the types of services that failed to be repaired. The comprehensive inspection result includes overall service repair success, service repair failure, and partial service repair success.

7. A node fault repair device, characterized in that, include: The fault node feature determination module is used to analyze the cluster status, alarm information and configuration files of nodes to determine the characteristics of fault nodes; wherein, the cluster status is the running status information of the cluster, the alarm information is the information that reminds the cluster of abnormalities, and the configuration files include the files of various services of the node. The service repair order determination module is used to determine the service repair order based on the characteristics of the fault node and the dependencies of services in the fault node. The repair list determination module is used to determine the repair methods and successful repair judgment methods for each service in the fault node, and to create a repair list based on the repair methods, the successful repair judgment methods and the service repair order. The repair module is used to determine the repair method for the faulty node based on the repair list, and to repair the faulty node based on the repair method; wherein, the repair method includes repairing services with stored data according to the repair order and repair method in the repair list; The repair list determination module includes: The first expansion item determination unit is used to determine whether the repair list includes an expansion item, wherein the expansion item is a hard disk data service; The hard disk drive data inclusion determination unit is used to determine whether the hard disk drive data of the expansion item is included in the storage pool when the expansion item is included in the repair list list. The first repair unit is used to repair the hard disk drive data of the expansion item according to the repair order and repair method in the repair list when the hard disk drive data is included in the storage pool. A shrinking unit is used to shrink the expansion item when the hard disk drive data of the expansion item is not included in the storage pool. A connectivity determination unit is used to determine whether the faulty node is connected to the cluster when the expansion item is not included in the repair list. The second expansion item judgment unit is used to obtain repair services according to the repair order in the repair list when the faulty node is connected to the cluster, and to determine whether the current repair service is the expansion item. The capacity expansion processing unit is configured to determine, if the current repair service is the capacity expansion item, to perform capacity expansion processing on all capacity-shrinking and re-installing capacity expansion items. The second repair unit is used to determine, if the current repair service is not the expansion item, to perform repair according to the repair order and repair method in the repair list.

8. A node fault repair device, characterized in that, include: memory for storing computer programs; A processor for executing the computer program to implement the steps of the node fault repair method as described in any one of claims 1 to 6.

9. A readable storage medium, characterized in that, The readable storage medium stores a computer program that, when executed by a processor, implements the steps of the node fault repair method as described in any one of claims 1 to 6.

Images