Abnormal node processing method and apparatus, metadata server, and storage medium

Abstract

The application discloses an abnormal node processing method and device, a metadata server and a storage medium, and relates to the distributed storage field.The method comprises the following steps: firstly, obtaining a node information table from a database server, wherein the node information table records the identifiers of a plurality of processable abnormal nodes and the processing states of each processable abnormal node, and the processing state comprises an uncompleted state and a completed state; then, for each processable abnormal node in the uncompleted state, performing deletion processing on a system configuration file and a data recovery information table in the database server based on the identifier of the processable abnormal node; finally, calling a preset script to uninstall the processable abnormal node, and modifying the processing state of the processable abnormal node in the node information table to the completed state, so that the abnormal node in the distributed storage system is processed, and the overall function of the distributed storage system is not affected.

Description

Technical Field

[0001] The present invention relates to the field of distributed storage, and in particular to an abnormal node processing method, device, metadata server and storage medium. Background Art

[0002] Compared with traditional storage systems, distributed storage systems are widely used because they store data in a dispersed manner on multiple independent devices, have higher reliability, availability, and access efficiency, and are easy to expand.

[0003] A distributed storage system consists of multiple nodes. When a distributed storage system encounters a device failure that prevents power on, an operating system that is damaged beyond repair, or a hardware failure that requires replacement, abnormal nodes may appear among the multiple nodes, affecting the overall functionality of the distributed storage system.

[0004] Therefore, how to handle abnormal nodes in a distributed storage system so that they do not affect the overall function of the distributed storage system is a problem that needs to be solved. Summary of the Invention

[0005] In order to overcome the deficiencies of the prior art, embodiments of the present invention provide a method and apparatus for processing abnormal nodes, a metadata server, and a computer-readable storage medium.

[0006] The technical solution of the embodiment of the present invention can be implemented as follows:

[0007] In a first aspect, the present invention provides a method for processing abnormal nodes, which is applied to a metadata server in a distributed storage system, wherein the distributed storage system further includes a database server, the metadata server is in communication with the database server, and an abnormal node exists in the distributed storage system. The method comprises:

[0008] Obtaining a node information table from the database server, wherein the node information table records identifiers of a plurality of processable abnormal nodes and a processing status of each processable abnormal node, wherein the processing status includes an unfinished state and a completed state;

[0009] For each of the processable abnormal nodes in an unfinished state, based on the identifier of the processable abnormal node, deleting the system configuration file and the data recovery information table in the database server;

[0010] A preset script is called to uninstall the processable abnormal node, and the process status of the processable abnormal node in the node information table is modified to a completed status.

[0011] Optionally, before the step of obtaining the node information table from the database server, the method further includes a step of determining whether the abnormal node is a processable abnormal node, which includes:

[0012] If the abnormal node is in an offline state, obtaining the node role information and node file information of the abnormal node from the database server;

[0013] Determining whether the abnormal node meets a preset processing condition based on the node role information and the node file information;

[0014] If the abnormal node meets the preset processing conditions, the abnormal node is added to the node information table as a processable abnormal node, and the deletion status of the abnormal node is set to an unfinished state;

[0015] If the abnormal node does not meet the preset deletion condition, then after a preset time interval, the process returns to the step of obtaining the node attribute information and node file information of the abnormal node from the database server until the abnormal node meets the preset processing condition.

[0016] Optionally, the step of determining whether the abnormal node meets a preset processing condition based on the node role information and the node file information includes:

[0017] Obtaining the number of non-abnormal nodes in the distributed storage system from the database server;

[0018] If the number of non-abnormal nodes meets the preset erasure ratio and the abnormal node is determined to be a non-master node based on the node role information, then determine whether the abnormal node stores unrecovered files based on the node file information;

[0019] If so, it is determined that the abnormal node does not meet the preset processing conditions;

[0020] If not, it is determined that the abnormal node meets the preset processing conditions.

[0021] Optionally, the step of deleting the system configuration file and the data recovery information table in the database server based on the identifier of the processable abnormal node includes:

[0022] If it is determined, based on the identifier of the processable abnormal node, that the system configuration file contains configuration information of the processable abnormal node, deleting the configuration information of the processable abnormal node from the system configuration file;

[0023] If it is determined, based on the identifier of the processable abnormal node, that the data recovery information table records information of recovered files corresponding to the processable abnormal node, the information of recovered files corresponding to the processable abnormal node is deleted from the data recovery information table.

[0024] Optionally, the method further includes:

[0025] For each of the processable abnormal nodes in a completed state, based on the identifier of the processable abnormal node, searching the system configuration file and the data recovery information table in the database server to obtain a search result;

[0026] If the search result is empty, the identifier and processing status of the processable abnormal node are deleted from the node information table, and the status value of the system capacity balancing table in the database server is set to a first preset value, where the first preset value indicates whether the distributed storage system needs to be confirmed to perform capacity balancing;

[0027] If the search result is not empty, the processing status of the processable abnormal node in the node information table is modified to an unfinished state.

[0028] Optionally, the method further includes:

[0029] Obtaining a status value of a system capacity balance table in the database server;

[0030] If the state value is a first preset value, obtaining the used capacity of each non-abnormal node in the distributed storage system;

[0031] Determining whether the distributed storage system meets a preset capacity balancing condition based on the used capacity of each of the non-abnormal nodes;

[0032] If the distributed storage system meets the preset capacity balancing condition, setting the state value of the system capacity balancing table to a second preset value;

[0033] If the distributed storage system does not meet the preset capacity balancing condition, the state value of the system capacity balancing table is set to a third preset value.

[0034] Optionally, the step of determining whether the distributed storage system meets a preset capacity balancing condition based on the used capacity of each of the non-abnormal nodes includes:

[0035] Calculating an average value of the used capacities of all the non-abnormal nodes;

[0036] If the difference between the used capacity of each of the non-abnormal nodes and the average value is not greater than a preset threshold, it is determined that the distributed storage system does not meet the preset capacity balance condition;

[0037] If there is at least one non-abnormal node whose difference between the used capacity and the average value is greater than a preset threshold, it is determined that the distributed storage system meets a preset capacity balance condition.

[0038] In a second aspect, an embodiment of the present invention provides an abnormal node processing device, which is applied to a metadata server in a distributed storage system, wherein the distributed storage system further includes a database server, the metadata server is in communication with the database server, and an abnormal node exists in the distributed storage system. The device includes:

[0039] an acquisition module, configured to acquire a node information table from the database server, wherein the node information table records identifiers of a plurality of processable abnormal nodes and a processing status of each processable abnormal node, wherein the processing status includes an unfinished state and a completed state;

[0040] Processing module for:

[0041] For each of the processable abnormal nodes in an unfinished state, based on the identifier of the processable abnormal node, deleting the system configuration file and the data recovery information table in the database server;

[0042] A preset script is called to uninstall the processable abnormal node, and the process status of the processable abnormal node in the node information table is modified to a completed status.

[0043] In a third aspect, an embodiment of the present invention provides a metadata server comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the abnormal node processing method as described in the first aspect when executing the computer program.

[0044] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the abnormal node processing method as described in the first aspect is implemented.

[0045] Compared to the prior art, an embodiment of the present invention provides a method for handling abnormal nodes in a metadata server in a distributed storage system. First, a node information table is obtained from a database server. The node information table records the identifiers of multiple processable abnormal nodes and the processing status of each processable abnormal node, including an unfinished state and a completed state. Then, for each processable abnormal node in an unfinished state, based on the identifier of the processable abnormal node, a system configuration file and a data recovery information table in the database server are deleted. Finally, a preset script is called to uninstall the processable abnormal node and modify the processing status of the processable abnormal node in the node information table to a completed state. Because the embodiment of the present invention sets a node information table for recording processable abnormal nodes, after deleting the processable abnormal node in the node information table in an unfinished state according to its identifier, the system configuration file and the data recovery information table are deleted, and then the preset script is called to uninstall it, thereby achieving the processing of abnormal nodes in the distributed storage system so that it does not affect the overall function of the distributed storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 A schematic diagram of the structure of a distributed storage system provided by an embodiment of the present invention;

[0048] Figure 2 A schematic diagram of a process for handling abnormal nodes provided by an embodiment of the present invention Figure 1 ;

[0049] Figure 3 A flowchart of a method for determining abnormal nodes that can be handled according to an embodiment of the present invention;

[0050] Figure 4 A schematic diagram of a process for handling abnormal nodes provided by an embodiment of the present invention Figure 2 ;

[0051] Figure 5 A schematic diagram of a load balancing process according to an embodiment of the present invention;

[0052] Figure 6 An example diagram of a method for handling abnormal nodes provided by an embodiment of the present invention;

[0053] Figure 7 A functional unit block diagram of an abnormal node processing device provided by an embodiment of the present invention;

[0054] Figure 8 A schematic block diagram of the structure of a metadata server provided by an embodiment of the present invention.

[0055] Icons: 100 - abnormal node processing device; 101 - acquisition module; 102 - processing module; 103 - judgment module; 104 - capacity balancing module; 200 - metadata server; 210 - memory; 220 - processor. DETAILED DESCRIPTION

[0056] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0057] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention.

[0058] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0059] In addition, the terms "first", "second", etc., if used, are merely used to distinguish and describe, and should not be understood as indicating or implying relative importance.

[0060] For the erasure ratio N+M, N and M are concepts in erasure codes, N is the number of original data copies, and M is the number of check data copies. The original data can be restored through any N copies of the N+M copies. That is, if less than or equal to M copies of data fail, it can still be restored through the remaining data.

[0061] like Figure 1As shown in the figure, the distributed storage system based on erasure code mainly consists of metadata server (MetaServer, MS), data server (Data node, DN), TGT server, client (Software Development Kit, SDK), Web front-end, Web back-end (such as Flask), and database server (including PGSQL database and Redis database).

[0062] Among them, the metadata server is responsible for managing the file metadata, load balancing, data migration, data recovery, etc. of all data servers. The data servers usually have one master and multiple backups.

[0063] According to the erasure ratio N+M of the distributed storage system, the minimum number of nodes num that constitute the distributed storage system can be determined min =(N+M) / M, assuming the actual number of nodes in the distributed storage system is num cur , usually, when num cur <5, the number of metadata servers is 3. cur When >=5, the number of metadata servers is 5.

[0064] When a distributed storage system experiences power failure, operating system corruption beyond repair, or hardware failure requiring replacement, abnormal nodes may appear in the system. Existing technologies recover data after an abnormal node appears, but then fail to address the abnormal node. This approach fails to consider the potential for unforeseen issues that may arise if a node fails and cannot be recovered, leading to partial unavailability of distributed storage system functions and significantly reducing the practicality of the distributed storage system.

[0065] In order to overcome the deficiencies of the prior art, the present invention provides an embodiment of the present invention. Figure 1 The following is a detailed description of how to handle abnormal nodes in the metadata server.

[0066] Please refer to Figure 2 The abnormal node processing method includes steps S101 to S103.

[0067] S101, obtaining a node information table from a database server.

[0068] The node information table records the identifiers of multiple processable abnormal nodes and the processing status of each processable abnormal node, where the processing status includes an unfinished state and a completed state.

[0069] As a possible implementation, the node information table can be set in Figure 1In the PGSQL database, the identifier of the processable abnormal node can include the node IP and node ID, and different processing states can be represented by setting different status values.

[0070] The table structure of the node information table may be Dnid-Ip-Timestamp-Status, where Dnid refers to the ID of the processable abnormal node, Ip refers to the IP address of the processable abnormal node, and Timestamp refers to the time when the processable abnormal node is added to the node information table.

[0071] The Status in the table structure refers to the processing status. When the Status value is 1, it means the processing status is unfinished. When the Status value is 2, it means the processing status is completed.

[0072] It is understandable that before executing step S101, it is necessary to judge the abnormal nodes in the distributed storage system, determine the abnormal nodes that can be processed, and add them into the system. Figure 1 The node information table in the PGSQL database.

[0073] Therefore, please refer to Figure 3 The implementation process of “determining whether an abnormal node is a processable abnormal node” may include steps S201 to S203.

[0074] S201: If the abnormal node is in an offline state, obtain the node role information and node file information of the abnormal node from the database server.

[0075] Among them, the metadata server can Figure 1 Get the online status of abnormal nodes from the PGSQL database.

[0076] In an embodiment of the present invention, an abnormal node is generally a node that has failed, which can usually be selected by a Web front-end and then its identification is transmitted to the metadata server through the corresponding interface. In order to avoid mistakenly clicking a normal node as an abnormal node, the metadata server needs to determine whether the abnormal node transmitted by the Web front-end is in an offline state.

[0077] When the abnormal node is online, it means that the abnormal node is not a node that actually fails, but a normal node that is mistakenly clicked. Therefore, the metadata server needs to report the processing failure to the Web front end.

[0078] When the abnormal node is in the offline state, it means that the abnormal node is the node that has actually failed. Figure 1 Get the node role information of the node from the PGSQL database in Figure 1The Redis database in the node obtains the node file information for the next step of judgment.

[0079] S202: Determine whether the abnormal node meets the preset processing conditions based on the node role information and the node file information.

[0080] The implementation process of step S202 may be as follows:

[0081] S202-1, obtain the number of non-abnormal nodes in the distributed storage system from the database server.

[0082] As a possible implementation, we can Figure 1 Get the total number of nodes in the distributed storage system from the PGSQL database total , by the total number of nodes num total Subtract the number of abnormal nodes num that are offline and passed in by the Web front end unormal The value obtained is the number of non-abnormal nodes num normal .

[0083] S202-2: If the number of non-abnormal nodes meets the preset erasure ratio and the abnormal node is determined to be a non-master node based on the node role information, determine whether the abnormal node stores unrecovered files based on the node file information.

[0084] Among them, the roles of nodes in the distributed storage system include MS node, redis node, PGSQL node, DN node, etc. For any node, if MS node, redis node or PGSQL node appears in the node role information, then the node is the master role node. If MS node, redis node or PGSQL node does not appear in the node role information, then the node is a non-master role node.

[0085] Assuming that the preset erasure ratio of the distributed storage system is N+M, "the number of non-abnormal nodes that meets the preset erasure ratio" refers to the number of non-abnormal nodes num normal >=(N+M) / M.

[0086] Since the minimum number of nodes in the distributed storage system is num min =(N+M) / M, it is understandable that when num normal >=num min When , the number of non-abnormal nodes meets the preset erasure ratio.

[0087] As a possible implementation, the node file information is from Figure 1All file information and corresponding file and object information on the abnormal node are obtained from the DN and file mapping information table TBL_DN_BLK-DnId-DiskId in the Redis database.

[0088] When there are still files of the abnormal node in the mapping information table TBL_DN_BLK-DnId-DiskId, it means that the abnormal node still stores unrecovered files, and then step S202-3 is executed.

[0089] When the file of the abnormal node does not exist in the mapping information table TBL_DN_BLK-DnId-DiskId, it means that the abnormal node does not store any unrecovered files, and step S202-4 is executed.

[0090] S202-3: Determine whether the abnormal node meets the preset processing conditions.

[0091] To ensure data reliability in the distributed storage system, abnormal nodes that do not meet the preset processing conditions cannot be processed, and the metadata server needs to report the processing failure to the Web front end.

[0092] S202-4: Determine whether the abnormal node meets the preset processing conditions.

[0093] In the embodiment of the present invention, when the abnormal node meets the preset processing conditions, step S203 is executed.

[0094] S203: Add the abnormal node as a processable abnormal node into the node information table, and set the deletion status of the abnormal node to an unfinished state.

[0095] As a possible implementation, Figure 1 A new node information table, bs_node_del_info, is added to the PGSQL database to record the identification and processing status of processable abnormal nodes. The table structure is Dnid-Ip-Timestamp-Status. In the table structure, Dnid refers to the ID of the processable abnormal node, Ip refers to the IP address of the processable abnormal node, and Timestamp refers to the time when the processable abnormal node is added to the node information table.

[0096] In an embodiment of the present invention, when an abnormal node does not meet the preset processing conditions, the process returns to the step of obtaining node attribute information and node file information of the abnormal node from the database server after a preset time interval until the abnormal node meets the preset processing conditions.

[0097] Among them, the preset time can be set according to actual needs.

[0098] The above process of "determining whether an abnormal node is a processable abnormal node" can be simplified as follows:

[0099] Step 1: Determine whether the abnormal node is offline;

[0100] Step 2: If the abnormal node is offline, determine whether the number of non-abnormal nodes in the distributed storage system meets the preset erasure ratio;

[0101] Step 3: If the abnormal node is offline and the number of non-abnormal nodes meets the preset erasure ratio, determine whether the abnormal node is a non-master node;

[0102] Step 4: If the abnormal node is offline, the number of non-abnormal nodes meets the preset erasure ratio, and the abnormal node is a non-master node, determine whether the abnormal node still stores unrecovered files;

[0103] In step 5, if the abnormal node is offline, the number of non-abnormal nodes meets the preset erasure ratio, the abnormal node is a non-master node, and the abnormal node does not store any unrecovered files, then the abnormal node can be determined to be a processable abnormal node.

[0104] It is understandable that an abnormal node cannot be treated as a processable abnormal node in the following four situations.

[0105] Case 1: The abnormal node is online;

[0106] Case 2: The abnormal node is offline, but the number of non-abnormal nodes in the distributed storage system does not meet the preset erasure ratio;

[0107] Case 3: The abnormal node is offline, the number of non-abnormal nodes in the distributed storage system meets the preset erasure ratio, but the abnormal node is not a non-master node;

[0108] Case 4: The abnormal node is offline, the number of non-abnormal nodes in the distributed storage system meets the preset erasure ratio, the abnormal node is a non-master node, but the abnormal node stores unrecovered files.

[0109] Since new nodes can be added to the distributed storage system and the roles of nodes can be reselected, the file recovery process takes a certain amount of time. Therefore, the above method process can be re-executed after a preset time interval to determine whether the abnormal node can be used as a processable abnormal node.

[0110] As a possible implementation, a specific background thread can be set up in the metadata server to execute Figure 3 The method flow of "determining whether an abnormal node is a processable abnormal node" shown in Figure 1Update the node information table in the PGSQL database.

[0111] S102 : For each processable abnormal node in an unfinished state, based on the identifier of the processable abnormal node, a system configuration file and a data recovery information table in the database server are deleted.

[0112] The system configuration file records the configuration information of each node in the storage system, which can be set in Figure 1 The PGSQL database in .

[0113] The data recovery information table records the information of each recovered file in the distributed storage system and can be set in the redis database.

[0114] Optionally, the implementation process of step S102 may be as follows:

[0115] S102-1: If it is determined that the system configuration file contains configuration information of the processable abnormal node according to the identifier of the processable abnormal node, the configuration information of the processable abnormal node is deleted from the system configuration file.

[0116] Since the configuration information of the node includes the node identifier, it is possible to determine whether the system configuration file records the configuration information of the processable abnormal node by comparing the identifier of the processable abnormal node with the node identifier included in the configuration information of each node in the system configuration file.

[0117] S102-2: If it is determined based on the identifier of the processable abnormal node that the data recovery information table records information of recovered files corresponding to the processable abnormal node, the information of recovered files corresponding to the processable abnormal node is deleted from the data recovery information table.

[0118] Since the information of each recovered file includes the identifier of the node where the file was located before recovery, it is possible to determine whether the data recovery information table records the information of the recovered file corresponding to the processable abnormal node by comparing the identifier of the processable abnormal node with the identifier of the node where the file was located before recovery included in the information of each recovered file in the data recovery information table.

[0119] S103, calling a preset script to uninstall the abnormal nodes that can be processed, and modifying the processing status of the abnormal nodes that can be processed in the node information table to a completed status.

[0120] Among them, the preset script is pre-written and can be used to uninstall any node in the distributed storage system.

[0121] In an embodiment of the present invention, after deleting the information related to the processable abnormal node in the system configuration file and the data recovery information table, the corresponding script is called to uninstall the processable abnormal node, thereby avoiding the node from being put back online or re-migrating data to the node during the uninstallation process, thereby ensuring the normal operation of the distributed storage system.

[0122] For each processable exception node in the completed state in the node information table, refer to Figure 4 The abnormal node processing method further includes steps S104 to S106 in parallel with steps S102 to S103.

[0123] S104, for each processable abnormal node in the completed state, based on the identifier of the processable abnormal node, searching the system configuration file and the data recovery information table in the database server to obtain a search result;

[0124] In the node information table, the processing status of each newly added processable abnormal node is in an unfinished state. Only after the process of steps S102 to S103 is executed on it, its processing status will be changed to a completed state.

[0125] In order to determine whether each processable abnormal node in the completed state is actually uninstalled, it is necessary to determine whether the configuration information of the node in the system configuration file is deleted, and whether the information of the recovered file corresponding to the node in the data recovery information table is deleted.

[0126] S105: If the search result is empty, the identifier and processing status of the processable abnormal node are deleted from the node information table, and the status value of the system capacity balancing table in the database server is set to a first preset value.

[0127] The first preset value indicates whether the distributed storage system to be confirmed needs to perform capacity balancing.

[0128] When the search result is empty, it means that the configuration information of the node in the system configuration file is deleted, and the information of the restored file corresponding to the node in the data recovery information table is deleted, and the node has been truly uninstalled.

[0129] S106: If the search result is not empty, the processing status of the abnormal node that can be processed in the node information table is modified to an unfinished state.

[0130] When the search result is not empty, it means that the configuration information of the node in the system configuration file has not been deleted, and the information of the recovered file corresponding to the node in the data recovery information table has not been deleted. The node has not been actually uninstalled. At this time, the processing status of the node needs to be reset to an unfinished state so that the process of steps S102 to S103 can be re-executed to achieve uninstallation.

[0131] As a possible implementation, a specific background thread may be set in the metadata server to poll and execute the above-mentioned processes of steps S102 to S103 and steps S104 to S106.

[0132] After the abnormal node is processed, there may be a capacity imbalance between the nodes in the system. Therefore, please refer to Figure 5 The abnormal node processing method further includes steps S301 to S305.

[0133] S301: Obtain a status value of a system capacity balancing table in a database server.

[0134] Among them, the system capacity balance table can be set in Figure 1 The redis database in .

[0135] In an embodiment of the present invention, the status value of the capacity balancing table can be a first preset value, a second preset value, or a third preset value. The first preset value indicates whether the distributed storage system to be confirmed needs to perform capacity balancing; the second preset value indicates that the distributed storage system needs to perform capacity balancing; and the third preset value indicates that the distributed storage system does not need to perform capacity balancing.

[0136] When the status value of the system capacity balancing table is the first preset value, it is necessary to execute the process of steps S302 to S303 to determine whether the distributed storage system needs capacity balancing, and reset the status value of the system capacity balancing table to the second preset value or the third preset value based on the judgment result.

[0137] S302: If the status value is a first preset value, obtain the used capacity of each non-abnormal node in the distributed storage system.

[0138] Since the metadata server stores metadata of files on each node, the used capacity of each node can be determined based on the metadata of files on each node.

[0139] S303: Determine whether the distributed storage system meets a preset capacity balancing condition based on the used capacity of each non-abnormal node.

[0140] The implementation process of step S303 may be as follows:

[0141] S303-1, calculating the average value of the used capacity of all non-abnormal nodes.

[0142] S303-2: If the difference between the used capacity of each non-abnormal node and the average value is not greater than the preset threshold, it is determined that the distributed storage system does not meet the preset capacity balance condition.

[0143] S303-3: If the difference between the used capacity and the average value of at least one non-abnormal node is greater than a preset threshold, it is determined that the distributed storage system meets the preset capacity balance condition.

[0144] In the embodiment of the present invention, if it is determined that the distributed storage system meets the preset capacity balance condition, step S304 is executed; if the distributed storage system does not meet the preset capacity balance condition, step S305 is executed.

[0145] S304: Set the status value of the system capacity balancing table to a second preset value.

[0146] S305: Set the status value of the system capacity balancing table to a third preset value.

[0147] It should be noted that if the status value of the system capacity balancing table obtained by executing step S301 is the second preset value, the following implementation method can be used to load balance the distributed storage system.

[0148] First, obtain the used capacity of each non-abnormal node in the distributed storage system and calculate the average used capacity of all non-abnormal nodes;

[0149] Then, the difference between the used capacity of each non-abnormal node and the average value is calculated, and the non-abnormal node with the largest difference is used as the data migration node, and the non-abnormal node with the smallest difference is used as the data migration node;

[0150] Next, migrate part of the data in the data migration node to the data migration node;

[0151] Finally, determine whether the difference between the used capacity of each non-abnormal node and the average value is not greater than a preset threshold. If so, set the state value of the system capacity balancing table to a third preset value.

[0152] If not, the capacity of the distributed storage system is re-balanced until the state value of the system capacity balancing table is set to a third preset value.

[0153] The following examples will be used to further illustrate the above method provided by the embodiment of the present invention.

[0154] Assume that a distributed storage system with an erasure ratio of 4+2 has 6 nodes, namely node 1, node 2, node 3, node 4, node 5 and node 6. Among them, nodes 1 to 5 are all MS nodes, Redis nodes or PGSQL nodes. That is, nodes 1 to 5 are master nodes, and node 6 is a non-master node.

[0155] Since node 2 fails and cannot rest, it is offline. The user clicks on this node as an abnormal node through the web front end and transmits its identification to the metadata server.

[0156] The metadata server first determines whether node 2 is offline.

[0157] Since node 2 is offline, the metadata server continues to determine whether the number of non-abnormal nodes in the distributed storage system meets the preset erasure ratio.

[0158] Since the number of non-abnormal nodes in the distributed storage system is 5, and the minimum number of nodes calculated according to the erasure ratio of 4+2 is 3, that is, the number of non-abnormal nodes in the distributed storage system meets the preset erasure ratio, the metadata server continues to determine whether node 2 is a non-master role node.

[0159] Since node 2 is an MS node, that is, it is not a non-master role node, the metadata server ends the judgment process at this time and returns to the Web front-end processing failure.

[0160] After waiting for 30 minutes, the metadata server receives another node deletion request from the Web front end. At this time, it re-determines whether node 2 can be used as a processable abnormal node.

[0161] During the waiting process, node 6 is reselected as the MS / redis / PGSQL node. At this time, node 2 is a non-protagonist node. After the metadata server re-completes the above judgment on node 2, it continues to determine whether the file of node 2 still exists in the mapping information table TBL_DN_BLK-DnId-DiskId in the redis database.

[0162] If the file of node 2 exists in TBL_DN_BLK-DnId-DiskId, the metadata server ends the judgment process and returns a message indicating that the processing failed on the Web front end. It then waits for another 30 minutes. After receiving the node deletion request from the Web front end, it re-judges whether node 2 can be treated as a processable abnormal node.

[0163] If the file for node 2 does not exist in TBL_DN_BLK-DnId-DiskId, node 2 is determined to be a processable abnormal node. According to the table structure Dnid-Ip-Timestamp-Status of the node information table bs_node_del_info in the PGSQL database, the ID and IP information of node 2 are added, and its Status value is set to 1.

[0164] The background thread ms_delnode of the metadata server polls the node information table bs_node_del_info in the PGSQL database every 5 seconds.

[0165] For node 2 whose Status value is 1, the metadata server's background thread ms_delnode clears the configuration information of node 2 in the system configuration file dn_hosts based on the ID and IP address of node 2.

[0166] The background thread ms_delnode of the metadata server scans the data recovery table TBL_RECOVERD in the redis database based on the ID and IP of node 2, and deletes the information of the recovered files corresponding to node 2.

[0167] The metadata server's background thread ms_delnode then calls the pre-written node uninstallation script to uninstall node 2;

[0168] The metadata server's background thread ms_delnode sets the Status value of node 2 in the node information table bs_node_del_info to 2.

[0169] For node 2 whose Status value is 2, the background thread ms_delnode of the metadata server retrieves the system configuration file dn_hosts and the data recovery table TBL_RECOVERD based on the ID and IP address of node 2.

[0170] Because the search result is empty, the background thread ms_delnode of the metadata server deletes the relevant information of node 2 in the node information table bs_node_del_info and sets the status value of the capacity balancing table TBL_EXPAND_MIGRATE in the Redis database to 1.

[0171] The background thread ms_mgr of the metadata server polls the status value of the capacity balancing table TBL_EXPAND_MIGRATE in the Redis database every 60 seconds.

[0172] When the background thread ms_mgr of the metadata server detects that the status value of the capacity balancing table TBL_EXPAND_MIGRATE is 1, it obtains the used capacity of node 1, node 3, node 4, node 5, and node 6.

[0173] like Figure 6 As shown, the used capacity CAP of node 1, node 3, node 4, node 5 and node 6 节点1 CAP 节点3 CAP 节点4 CAP 节点5 and CAP 节点6 They are 7500GB, 6000GB, 5500GB, 5500GB and 7500GB respectively.

[0174] Calculate the average CAP of the used capacity of each node ave :

[0175] CAP ave =(7500+6000+5500+5500+7500) / 6=6000G

[0176] Calculate the used capacity and CAP of each node ave The difference is compared with the preset threshold CAP labs =500G for comparison:

[0177] CAP end节点1 =7500-6000=1500G>CAP labs

[0178] CAP end节点3 =6000-6000=0G <CAP labs

[0179] CAP end节点4 =5500-6000=-500G <CAP labs

[0180] CAP end节点5 =5500-6000=-500G <CAP labs

[0181] CAP end节点6 =7500-6000=1500G>CAP labs

[0182] Since the used capacity of nodes 1 and 6 is equal to CAP ave The difference is greater than CAP labs , so it is necessary to balance the capacity of the distributed storage system.

[0183] The background thread ms_mgr of the metadata server performs capacity balancing operations as follows:

[0184] Used capacity and CAP ave The nodes with the largest difference, namely node 1 and node 6, are used as data migration nodes.

[0185] For node 1, obtain 100 files of node 1 (each file is 64MB in size) from the mapping information table TBL_DN_BLK-DnId-DiskId in the Redis database and add them to the capacity balancing table TBL_MIGRATION in the Redis database.

[0186] Used capacity and CAP ave The nodes with the smallest difference, sufficient capacity to create files, and nodes that meet the erasure ratio, namely nodes 4 and 5, are used as data migration nodes and migrate the 100 files from node 1.

[0187] Assume that all 100 files on node 1 are migrated to node 4. The migration capacity is: 100*64M=6400M=6.25G. At this time, the used capacity of each node changes as follows:

[0188] CAP 节点1 =7500G-6.25G=7493.75G;

[0189] CAP 节点3 =6000G;

[0190] CAP 节点4 =5500G+6.25G=5506.25G;

[0191] CAP 节点5 =5500G;

[0192] CAP 节点6 =7500G;

[0193] Calculate the used capacity and CAP of each node again ave The difference is compared with the preset threshold CAP labs =500G, if the used capacity of each node is equal to CAP ave The difference is less than CAP labs When the capacity balancing table TBL_MIGRATION in the redis database is set to 0, it means that the capacity balancing has been completed.

[0194] If the used capacity of each node and CAP are still not satisfied ave The difference is less than CAP labs, then continue with a new round of capacity balancing.

[0195] In order to execute the corresponding steps in the above method embodiment and various possible implementations, an implementation of the abnormal node processing device 100 is respectively given below.

[0196] Please refer to Figure 7 The abnormal node processing device 100 includes an acquisition module 101 , a processing module 102 , a judgment module 103 and a capacity balancing module 104 .

[0197] The acquisition module 101 is configured to acquire a node information table from a database server. The node information table records identifiers of multiple processable abnormal nodes and a processing status of each processable abnormal node. The processing status includes an unfinished state and a completed state.

[0198] Processing module 102 is configured to delete the system configuration file and data recovery information table in the database server for each processable abnormal node in an uncompleted state based on the identifier of the processable abnormal node; call a preset script to uninstall the processable abnormal node, and change the processing status of the processable abnormal node in the node information table to a completed state.

[0199] Optionally, the judgment module 103 is used to obtain the node role information and node file information of the abnormal node from the database server if the abnormal node is in an offline state; based on the node role information and node file information, determine whether the abnormal node meets the preset processing conditions; if the abnormal node meets the preset processing conditions, add the abnormal node as a processable abnormal node to the node information table, and set the deletion status of the abnormal node to an unfinished state; if the abnormal node does not meet the preset deletion conditions, after a preset time interval, return to execute the step of obtaining the node attribute information and node file information of the abnormal node from the database server until the abnormal node meets the preset processing conditions.

[0200] Optionally, when the judgment module 103 is used to judge whether the abnormal node meets the preset processing conditions based on the node role information and the node file information, it is specifically used to obtain the number of non-abnormal nodes in the distributed storage system from the database server; if the number of non-abnormal nodes meets the preset correction and erasure ratio and the abnormal node is determined to be a non-master role node based on the node role information, then based on the node file information, it is judged whether the abnormal node stores unrecovered files; if so, it is determined that the abnormal node does not meet the preset processing conditions; if not, it is determined that the abnormal node meets the preset processing conditions.

[0201] Optionally, when the processing module 102 is used to perform deletion processing on the system configuration file and the data recovery information table in the database server based on the identifier of the processable abnormal node, it is specifically used to, if it is determined based on the identifier of the processable abnormal node that the system configuration file records the configuration information of the processable abnormal node, delete the configuration information of the processable abnormal node from the system configuration file; if it is determined based on the identifier of the processable abnormal node that the data recovery information table records the information of the recovered file corresponding to the processable abnormal node, delete the information of the recovered file corresponding to the processable abnormal node from the data recovery information table.

[0202] Optionally, the processing module 102 is further configured to retrieve and process the system configuration file and the data recovery information table in the database server for each processable abnormal node in a completed state based on the identifier of the processable abnormal node to obtain a retrieval result; if the retrieval result is empty, the identifier and the processing status of the processable abnormal node are deleted from the node information table, and the status value of the system capacity balancing table in the database server is set to a first preset value, which indicates whether the distributed storage system to be confirmed needs capacity balancing; if the retrieval result is not empty, the processing status of the processable abnormal node in the node information table is modified to an unfinished state.

[0203] Optionally, the capacity balancing module 104 is used to obtain the status value of the system capacity balancing table in the database server; if the status value is a first preset value, the used capacity of each non-abnormal node in the distributed storage system is obtained; based on the used capacity of each non-abnormal node, it is determined whether the distributed storage system meets the preset capacity balancing condition; if the distributed storage system meets the preset capacity balancing condition, the status value of the system capacity balancing table is set to a second preset value; if the distributed storage system does not meet the preset capacity balancing condition, the status value of the system capacity balancing table is set to a third preset value.

[0204] Optionally, when the capacity balancing module 104 is used to determine whether the distributed storage system meets the preset capacity balancing condition based on the used capacity of each non-abnormal node, it is specifically used to calculate the average value of the used capacity of all non-abnormal nodes; if the difference between the used capacity of each non-abnormal node and the average value is not greater than a preset threshold, it is determined that the distributed storage system does not meet the preset capacity balancing condition; if there is at least one non-abnormal node whose difference between the used capacity and the average value is greater than the preset threshold, it is determined that the distributed storage system meets the preset capacity balancing condition.

[0205] Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working process of the abnormal node processing device 100 described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0206] Furthermore, the embodiment of the present invention also provides a metadata server 200, please refer to Figure 8 , the metadata server 200 may include a memory 210 and a processor 220 .

[0207] The processor 220 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the abnormal node processing method provided in the above-mentioned method embodiment.

[0208] The memory 210 can be a ROM or other type of static storage device that can store static information and instructions, a RAM or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to this. The memory 210 can exist independently and be connected to the processor 220 via a communication bus. The memory 210 can also be integrated with the processor 220. Among them, the memory 210 is used to store machine executable instructions for executing the scheme of the present application. The processor 220 is used to execute the machine executable instructions stored in the memory 210 to implement the above-mentioned method embodiment.

[0209] An embodiment of the present invention further provides a computer-readable storage medium containing a computer program. When the computer program is executed, it can be used to perform relevant operations in the abnormal node processing method provided by the above method embodiment.

[0210] In summary, an embodiment of the present invention provides an abnormal node processing method, device, metadata server, and storage medium. First, a node information table is obtained from a database server. The node information table records the identifiers of multiple processable abnormal nodes and the processing status of each processable abnormal node, and the processing status includes an unfinished state and a completed state. Then, for each processable abnormal node in an unfinished state, based on the identifier of the processable abnormal node, a system configuration file and a data recovery information table in the database server are deleted. Finally, a preset script is called to uninstall the processable abnormal node and modify the processing status of the processable abnormal node in the node information table to a completed state. Because the embodiment of the present invention sets a node information table for recording processable abnormal nodes, after deleting the processable abnormal node in the node information table in an unfinished state according to its identifier, the system configuration file and the data recovery information table are deleted, and then the preset script is called to uninstall it, thereby achieving the processing of abnormal nodes in the distributed storage system so that it does not affect the overall function of the distributed storage system.

[0211] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for handling abnormal nodes, characterized in that: A metadata server is applied to a distributed storage system, the distributed storage system further comprising a database server, the metadata server being in communication with the database server, and an abnormal node exists in the distributed storage system. The method comprises: Obtaining a node information table from the database server, wherein the node information table records identifiers of a plurality of processable abnormal nodes and a processing status of each processable abnormal node, wherein the processing status includes an unfinished state and a completed state; For each of the processable abnormal nodes in an unfinished state, based on the identifier of the processable abnormal node, deleting the system configuration file and the data recovery information table in the database server; A preset script is called to uninstall the processable abnormal node, and the process status of the processable abnormal node in the node information table is modified to a completed status.

2. The method according to claim 1, wherein Before the step of obtaining the node information table from the database server, the method further includes the step of determining whether the abnormal node is a processable abnormal node, which includes: If the abnormal node is in an offline state, obtaining the node role information and node file information of the abnormal node from the database server; Determining whether the abnormal node meets a preset processing condition based on the node role information and the node file information; If the abnormal node meets the preset processing conditions, the abnormal node is added to the node information table as a processable abnormal node, and the deletion status of the abnormal node is set to an unfinished state; If the abnormal node does not meet the preset deletion condition, then after a preset time interval, the process returns to the step of obtaining the node attribute information and node file information of the abnormal node from the database server until the abnormal node meets the preset processing condition.

3. The method according to claim 2, wherein The step of determining whether the abnormal node meets a preset processing condition based on the node role information and the node file information includes: Obtaining the number of non-abnormal nodes in the distributed storage system from the database server; If the number of non-abnormal nodes meets the preset erasure ratio and the abnormal node is determined to be a non-master node based on the node role information, then determine whether the abnormal node stores unrecovered files based on the node file information; If so, it is determined that the abnormal node does not meet the preset processing conditions; If not, it is determined that the abnormal node meets the preset processing conditions.

4. The method according to claim 1, wherein The step of deleting the system configuration file and the data recovery information table in the database server based on the identifier of the processable abnormal node includes: If it is determined, based on the identifier of the processable abnormal node, that the system configuration file contains configuration information of the processable abnormal node, deleting the configuration information of the processable abnormal node from the system configuration file; If it is determined, based on the identifier of the processable abnormal node, that the data recovery information table records information of recovered files corresponding to the processable abnormal node, the information of recovered files corresponding to the processable abnormal node is deleted from the data recovery information table.

5. The method according to claim 1, wherein The method further comprises: For each of the processable abnormal nodes in a completed state, based on the identifier of the processable abnormal node, searching the system configuration file and the data recovery information table in the database server to obtain a search result; If the search result is empty, the identifier and processing status of the processable abnormal node are deleted from the node information table, and the status value of the system capacity balancing table in the database server is set to a first preset value, where the first preset value indicates whether the distributed storage system needs to be confirmed to perform capacity balancing; If the search result is not empty, the processing status of the processable abnormal node in the node information table is modified to an unfinished state.

6. The method according to claim 5, wherein The method further comprises: Obtaining a status value of a system capacity balance table in the database server; If the state value is a first preset value, obtaining the used capacity of each non-abnormal node in the distributed storage system; Determining whether the distributed storage system meets a preset capacity balancing condition based on the used capacity of each of the non-abnormal nodes; If the distributed storage system meets the preset capacity balancing condition, setting the state value of the system capacity balancing table to a second preset value; If the distributed storage system does not meet the preset capacity balancing condition, the state value of the system capacity balancing table is set to a third preset value.

7. The method according to claim 6, wherein The step of determining whether the distributed storage system meets a preset capacity balancing condition based on the used capacity of each non-abnormal node includes: Calculating an average value of the used capacities of all the non-abnormal nodes; If the difference between the used capacity of each of the non-abnormal nodes and the average value is not greater than a preset threshold, it is determined that the distributed storage system does not meet the preset capacity balance condition; If there is at least one non-abnormal node whose difference between the used capacity and the average value is greater than a preset threshold, it is determined that the distributed storage system meets a preset capacity balance condition.

8. An abnormal node processing device, characterized in that: A metadata server is applied to a distributed storage system, the distributed storage system further comprising a database server, the metadata server being in communication with the database server, and an abnormal node existing in the distributed storage system. The apparatus comprises: an acquisition module, configured to acquire a node information table from the database server, wherein the node information table records identifiers of a plurality of processable abnormal nodes and a processing status of each processable abnormal node, wherein the processing status includes an unfinished state and a completed state; Processing module for: For each of the processable abnormal nodes in an unfinished state, based on the identifier of the processable abnormal node, deleting the system configuration file and the data recovery information table in the database server; A preset script is called to uninstall the processable abnormal node, and the process status of the processable abnormal node in the node information table is modified to a completed status.

9. A metadata server, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the abnormal node processing method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that It stores a computer program, which, when executed by a processor, implements the abnormal node processing method according to any one of claims 1 to 7.

Images